VShell® Server FAQ

VShell Server FAQ

Can VShell be exported outside the United States?

VShell is under export control as a strong encryption product by the U.S. Department of Commerce. VShell can be exported under the ENC export license exemption to customers in member states of the European Union (EU) and Australia, Canada, Czech Republic, Hungary, Japan, New Zealand, Norway, Poland, and Switzerland. Outside the EU and eight other countries, VShell may be exported to non-government users only.

For more information on exporting VShell, see the Encryption Export page.

How is the VShell Server Licensed?

There are three elements that are all essential when considering what type of VShell license purchase will work for your organization:

  1. VShell "Edition"
  2. Server Machine Count
  3. Functionality

All of these elements must be considered together when determining the licensing requirements that apply to your planned deployment of VShell.

VShell "Edition"

VShell licenses are categorized into three separate "Editions" which determine the maximum number of allowed concurrent client connections to VShell. VShell license editions allowing fewer concurrent connections are priced lower than editions that allow a larger number of concurrent connections.

License Edition Concurrent (authenticated) client connections allowed
Administrator 2 concurrent connections
Workgroup 10 concurrent connections
Enterprise Unlimited concurrent connections

VShell operates as a 'Workgroup' edition during the 30-day evaluation period, allowing up to 10 connections at a time to the server.

Note: "Client connections" is not the same as "users". The number of concurrent client connections is the number of connections where the client has successfully authenticated to VShell, regardless of the user account associated with the authentication. Two separate connections to the VShell service will count as two concurrent connections even if the same user account is used for authentication in each case.

Server Machine Count

VShell is licensed on a per-computer installation basis, whether it be physical or virtual. A VShell license (regardless of edition – Administrator, Workgroup, or Enterprise) can only be installed on a single machine. If you plan to deploy VShell on 7 different machines, for example, 7 VShell licenses must be purchased. There is one exception to this requirement:

In the event that a failover, disaster recovery or other secondary server is maintained offline ("Failover Server") and such Failover Server is solely used when the primary server is not functioning, then, for each license for the primary server, one backup copy of the Software may be installed on the Failover Server corresponding to the primary server at no additional charge. High availability or load balancing server deployments require VShell licenses be purchased for each node in the cluster.

Functionality

All versions of VShell provide SSH2/SFTP/SCP functionality.

VShell with FTPS

A version of VShell with FTPS (FTP over SSL) is available, allowing you to provide support for a wider variety of connectivity protocols depending on your clients' needs.

I can connect to VShell Server with some SCP and SFTP clients but not others. Which ones does VShell support?

VShell 3.0 and later:

VShell versions 3.0 and later now have built-in SCP file transfer support. File transfer clients that only support the SCP protocol will now work with VShell without any further server side setup. Currently, server to server SCP transfers will not work due to VShell handling the transfer internally and not passing the request off to an external SCP executable.

VShell 2.6 and earlier:

VShell supports SCP and SFTP clients that use the SFTP protocol as defined in the SSH2 IETF draft. Many SCP clients, including the one that comes with OpenSSH 2.3.0, are based on the UNIX-style rcp remote copy implementation from SSH1 rather than on the SSH2 SFTP protocol.

It is possible to use OpenSSH's SFTP client with the batch mode -f batchfile to automate some transfers, although this is limited in that SFTP does not seem to perform recursive copies. It is also possible to put an open source SCP utility in your path allowing VShell to work with OpenSSH's SCP utility.

The software known as "SFTP 0.9.6", which is distributed through the http://www.xbill.org/sftp site, is not based on the IETF draft and is not supported by VShell.

What's the difference between SFTP and FTP over SSH2?

In an SFTP transfer, the SSH2 server both encrypts the data and handles the file transfer itself. This is true for VShell as well as OpenSSH and other UNIX or Linux servers. FTP over SSH2 uses the port forwarding capabilities of SSH2 to forward standard FTP over an encrypted tunnel, while the actual FTP file transfer is handled by a separate FTP server.

SFTP is generally more secure than FTP over SSH2. FTP over SSH2 is considered secure only if the FTP server resides on the same machine as the SSH2 server. If you have a choice between using SFTP or FTP over SSH2, SFTP is the better choice, especially when you don't know whether the FTP and SSH2 servers are on the same machine or on a trusted network.

Is there a way I can see my command prompt when I'm in scrollback (TTY) mode using a Kornshell (KSH) or C Shell (CSH) command shell?

If you have trouble getting the appropriate $ or % command prompt when you use KSH or CSH with VShell, try using the interactive argument (-i) when invoking your shell. You should be able to see your prompt then.

How do I get commands to echo correctly in scrollback (TTY) mode when using 4NT as the alternate command shell?

Because of the way in which 4NT implemented its shell, scrollback (TTY) mode will not work properly with VShell.

How do I get VNC to run securely over SSH2 using SecureCRT and VShell?

On the VShell/VNC server:

  1. In the VNC interface, set the following parameters:
    1. Check the "Accept Socket Connections" check box.
    2. Set "Display number" to "02" (this will configure the server to listen on port 5902).
    3. Set the password.
    Note: Setting "Display number" directly to "5902" may cause an error with the client.
  2. Create the following Registry entry

    AllowLoopBack REG_DWORD "1"

    (hex) under the following key:

      HKEY_LOCAL_MACHINE\SOFTWARE\ORL\WinVNC3

  3. Restart the VNC service.

In your SecureCRT client:

  1. Select "File / Connect..." to open the Session Manager.
  2. Click on the "New Session" button and configure a SSH2 session that connects to the VShell/VNC server.
  3. Select the "Connections / Port Forwarding" category and press the "Add" button.
  4. Create a port forward entry with the following settings:
    1. Name: VNC (or whatever name is meaningful to you)
    2. Local Port: 5902
    3. Remote Port: 5902
    4. Clear the "Destination host is different for the SSH server" check box.
  5. Click on the "OK" button until you are back at the Session Manager and then click on the "Connect" button to initiate your new session.

In your VNCViewer:

  1. Enter "localhost:02" in the Connection details dialog.

Why am I getting key exchange failures and what can I do about them?

Several things cause key exchange failures. If you are having problems, check the following conditions.

  1. When you generated your host key, did you use DSA or RSA? Make sure that your client supports whichever algorithm you selected.
  2. Make sure that at least one of the ciphers that your client is using matches at least one of the ciphers selected in the VShell Control Panel.
  3. Make sure that at least one of the MACs that your client is using matches at least one of the Macs selected in the VShell Control Panel.
  4. Make sure that you have Compression enabled in the VShell Control Panel if your client is requesting compression.

How do I get RAdmin to run over VShell?

The following steps will help you get RAdmin running over VShell. These steps assume that you are using SecureCRT as your port-forwarding client.

Note: This procedure uses the default listening port for the product. It should be possible, on the server end, to set the port to whichever port you want to use.

Configure RAdmin (created using RAdmin 2.1)

  1. Install the RAdmin server on the same machine that is running VShell. Configure the RAdmin password/security options as needed (this is a part of installation process).
  2. Install the RAdmin client on the remote workstation to which you want to connect.

Configure SecureCRT

  1. In SecureCRT, select "File / Connect..." to open the Session Manager.
  2. Click on the "New Session" button and configure a SSH2 session that connects to the VShell/ RAdmin server.
  3. Select the "Connections / Port Forwarding" category and press the "Add" button.
  4. Create a port forward entry with the following settings:
      a. Name: RAdmin (or whatever name is meaningful to you)
      b. Local Port: 4899
  5. Click on the "OK" button until you are back at the Session Manager and then click on the "Connect" button to initiate your new session.

Establish Connection

  1. From the RAdmin client, start Remote Administrator Viewer.
  2. From the "Connection" menu, select "Connect To..."
  3. Enter localhost as the IP address or DNS name.
  4. Click on the "Connect" button.
  5. Enter the password that you specified in Step 1 of the "Configure RAdmin" section above.

How do I get pcAnywhere to run over VShell?

The following steps will help you get Symantec pcAnywhere running over VShell. These steps assume that you are using SecureCRT® as your port-forwarding client.

Note: This procedure uses the default listening port for the product. It should be possible, on the server end, to set the port to whichever port you want to use.

Configure pcAnywhere (created using pcAnywhere 10.0)

  1. Install the pcAnywhere server on the same machine that is running VShell.
  2. Install the pcAnywhere client on the remote workstation from which you want to connect.
  3. On the VShell/pcAnywhere server, start pcAnywhere Manager.
  4. Click on the "Host" button then right-click on "Direct" and select "Properties".
  5. On the "Connection Info" tab, check "TCP/IP" and uncheck "COM1".
  6. On the "Callers" tab, add users and privileges as needed.
  7. Double-click on "Direct" and follow the prompts to start the host service.
  8. On the pcAnywhere client, run pcAnywhere Manager.
  9. Click on the "Remotes" button then right-click on "Direct" and select "Properties".
  10. On the "Connection Info" tab, check "TCP/IP" and uncheck "COM1".
  11. On the "Settings" tab, enter localhost as the "Network host PC to control or IP address".
  12. Click the "OK" button to save your settings.

Configure SecureCRT

  1. In SecureCRT, select "File / Connect..." to open the Session Manager.
  2. Click on the "New Session" button and configure a SSH2 session that connects to the VShell/pcAnywhere server.
  3. Select the "Connections / Port Forwarding" category and press the "Add" button.
  4. Create a port forward entry with the following settings:
      a. Name: pcAnywhere Data Port (or whatever name is meaningful to you)
      b. Local Port: 5631
  5. Create a second port forward entry with the following settings:
      a. Name: pcAnywhere Status Port (or whatever name is meaningful to you)
      b. Local Port: 5632
  6. Click on the "OK" button until you are back at the Session Manager and then click on the "Connect" button to initiate your new session.

Establish Connection

  1. From the pcAnywhere Manager client, click on the "Remotes" button and then double-click "Direct" to connect to the VShell/pcAnywhere server.

How do I configure SecureCRT to work with VShell?

The following steps describe how to configure SecureCRT to work with VShell. The procedure below assumes that you have installed and are using SecureCRT 3.1.2 to SecureCRT 3.3.4. In SecureCRT 3.4, all of the steps below are performed automatically by selecting "VShell" from the protocol list.

  1. Obtain the keymap file VShell.key from the VShell installation folder (users without administrator privileges may need to get the file from their administrator). Save the file on your system (the default location is C:\Program Files\SecureCRT 3.0\Keymaps).
  2. Start SecureCRT.
  3. In the "File" menu, click on "Connect..." to open the Session Manager.
  4. Click on the "New Session" toolbar button to create a new session.
  5. In the "Session Options" dialog, click on the "Connection" category.
  6. Choose SSH2 as the "Protocol".
  7. Click on the "SSH2" subcategory and:
      a. if you are using version 3.1.x, choose Standard as the "SSH Server".
      b. if you are using 3.3 or newer, choose Auto Detect as the "SSH Server".
  8. Click on the "Emulation" category, choose VT220 as the "Terminal" and check the "ANSI Color" check box.
  9. In the "Keyboard" entry box, choose Custom (listed above Default) and browse for the keymap file that you saved in Step 1.
  10. Enter 25 in "Logical Rows", and enter 80 in "Logical Columns".
  11. Click on the "Appearance" category and click on the "Font" button.
  12. In the "Font" dialog, select Terminal in the "Font" window.
  13. Click on the "OK" button to save your settings.

How do I get VIM to work with VShell?

To get VIM working with VShell, you must first install VIM for Microsoft Windows. The installer can be downloaded from the following location:

http://www.vim.org/download.php#pc

Once you have the required version of VIM installed, log on to VShell using your SSH client configured in VT220 mode.

Note: If you are connecting to VShell with terminal type TTY (scrollback mode), VIM will not work.

Why doesn't the scrollback buffer seem to work (even making the scroll buffer bigger doesn't help)?

Your client may not be configured correctly to use the VShell® scrollback feature. You need to set up your client to send a terminal type TTY message when it connects.

Note: When using the scrollback buffer mode, depending on your command shell, it may not be possible to use cursor-based applications like Edit, VI or VIM. Also, you may not be able to use the back arrow to get to the first line of a multi-line command (e.g., if the command is very long and wraps to the next line, you may not be able to cursor back up to the first line).

You may need to experiment with command-line switches to optimize the function of your particular command shell.

To configure SecureCRT to use the VShell scrollback buffer, follow these instructions:

  1. Open the SecureCRT "Session Options" dialog and select the "Emulation / Advanced" category.
  2. Check the "Terminal type" check box, enter TTY in the corresponding entry box and click the "OK" button to save your changes.

What is Scraper.exe? What purpose does it serve?

Scraper.exe is VShell's screen-scraping application used to translate data formatted for Windows systems so that it can be accessed by users running terminal emulators. The screen scraper also reformats user input from terminal emulators so that it can be understood by applications running on Windows systems.

Note: Scraper.exe does not run when the client connected to VShell has requested the use of the scrollback buffer. (For more information about the scrollback buffer, see FAQ, "Why doesn't the scrollback buffer seem to work".)

I'm having trouble connecting to VShell® and am getting a "Could not load environment" error message. What's the problem?

This problem is specific to Windows NT and is a known behavior. The cause is an empty string in the following registry key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SessionManager\Environment

When VShell makes a CreateEnvironmentBlock system call (to get a user's environment when the user logs on), NT is not able to handle empty strings in this key and the function fails. Uninstalling Oracle is known to leave empty strings within the environment block.

The solution to this problem is to open the registry and delete any empty string values that you find in the above registry key.

Warning: If you use the registry editor incorrectly, you may cause serious problems that may require you reinstall your operation system. Before making chnages to the registry, you should back up any valued data on your computer.

What file access and user rights does a VShell user need to access VShell?

Your access rights need to be reconfigured if you have problems accessing VShell or if you receive the following error message:

00002: Could not start program (Scraper.exe /X 80 /Y 24 C:\WINDOWS\system32\CMD.EXE): The user does not have the right to create processes; check for "logon locally" user right

To correct this, ensure that the account that you are trying to connect to VShell with has its settings configured as follows:

  • In the VShell Control Panel, select the "Access Control" category. Ensure that the "Logon" access right is checked.
  • In Windows, ensure that the "Log on locally" option is selected.

Instructions for Windows XP:

  1. Open the "Start" menu and click on "Control Panel".
  2. Open "Performance and Maintenance".
  3. Open "Administrative Tools" and select "Local Security Policy".
  4. In the folder tree, select "Local Policies / User Rights Assignments".
  5. Make sure that the account in question is not listed or a member of a group listed in the "Deny logon locally" policy.

Instructions for Windows 7 and Later:

  1. Open the "Start" menu and click on "Control Panel".
  2. Open "System and Security".
  3. Open "Administrative Tools" and select "Local Security Policy".
  4. In the folder tree, select "Local Policies / User Rights Assignments".
  5. Make sure that the account in question is listed in the "Allow log on locally" policy and is not listed in the "Deny log on locally" policy.

How do I run VShell as a user other than "System"?

To run VShell as a user other than "System", follow these steps:

Note: The user account must be a member of the Administrators groups (according to Microsoft's documentation on loading profiles).

  1. Ensure that the user account that you want to run VShell (the service) as has the following policy settings:
    • Act as part of the operating system
    • Logon as a service
    • Replace a process level token
  2. On the machine running VShell, go to the Windows Control Panel and locate and open "Services".
  3. For Windows 7, 8, 10, and Server 2008/2012/2016:
    1. Right-click on the "VanDyke Software VShell" entry and select "Properties" from the menu.
    2. Select the "Log On" page and click on the "This account" radio button.
  4. For Windows NT Server:
    1. Select the "VanDyke Software VShell" entry and click on the "Startup..." button.
    2. Click on the "This account" radio button.
  5. Enter the user account that you want to use with the appropriate password.
  6. VShell should now be running as the user that you entered.

Why is the message-of-the-day displayed twice on Solaris systems?

In the Solaris operating environment, the login shell by default displays the message of the day. Therefore, if you also include the MOTDPath option in your vshelld_config file, VShell® misinterprets this as a second instruction to display the message of the day. To suppress the second message, comment out or remove the "MOTDPath/etc/motd" setting in the vshelld_config file.

Why does my OpenSSH agent crash when I try to add a VShell-generated key pair?

You cannot use VShell-generated key pairs with the OpenSSH SSH-agent because the agent is unable to read private keys generated by vkeygen (unlike public keys, private keys do not have a standardized format). However, VanDyke servers and clients do understand private and public keys generated by OpenSSH applications.

How can I configure syslogd to log vshelld messages to a separate file?

The following steps describe how to configure vshelld to log its messages to a separate file using the syslog (8) facility.

  1. Change the vshelld_config (5) option "SyslogFacility" to a unique syslog (8) facility that is not currently used by any other applications on the system. For example, the following line will cause vshelld to log to syslog using the "local3" facility: SyslogFacility LOG_LOCAL3
  2. Modify syslog.conf (5) so that messages of the facility you chose in the previous step will be logged to a specific file. For example, the following line will steer any messages to the "local3" facility to the file /var/log/vshelld.
    local3.* /var/log/vshelld
    Log messages from vshelld will now be appended to /var/log/vshelld.
  3. Depending on your syslog.conf settings, some or all of your vshelld messages may continue to be appended to other files (such as /var/log/messages). To prevent this, use the "none" syslog level to exclude vshelld messages from being sent to other destinations. For example, including "local3.none" in the following line prevents any vshelld messages (if configured as above) from being appended to /var/log/messages:
    # Log anything of level info or higher (except local3).
    local3.none;*.info /var/log/messages

For more information on configuring vshelld and syslog, see the vshelld_config (5) and syslog.conf (5) manual pages.

My public-key authentication failed on Red Hat and VShell logged an "Unable to authenticate" error. Why?

If VShell could not authenticate and logged the following error message:

"Disconnecting from server: Unable to authenticate using any of the configured authentication methods."

there is most likely a UMASK problem.

On Red Hat, the default UMASK is 0002, which means that files are created by default with the group-writable bit set. This causes public-key authentication with vshelld to fail. To keep this from happening on newly created files, change the setting on your UMASK to make them not group-writable (set them to 0022, for example). To correct this problem with existing files, change the permissions on your .vshell and publickey folders and any files in your publickey folder so that they are writable only by the owner.

I can't get public key-only authentication in VShell for Windows to work. What's wrong?

For public key-only authentication to work with VShell for Windows, you must have the VShell Public Key LSA Module installed. This module comes with VShell and should have been installed by default with the VShell installation package. You may just need to reboot your machine for the module to take affect.

If rebooting your machine does not correct the problem, try reinstalling VShell and be sure that the Public Key LSA Module installation option is enabled. After installing VShell with the module, you must reboot your machine.

When I try to establish a shell, my connection gets dropped. Why?

If you are connecting to a Windows 2003 (or newer) server using GSSAPI authentication and cannot establish an interactive shell, the problem may be that you do not have access to the cmd.exe program.

To run GSSAPI, VShell® uses the MS SSPI interface, which does not provide a way to do interactive logons and, therefore, does not provide the INTERACTIVE security identifier (SID). Under Windows 2003 server, many of the programs in "windows\system32" grant access via the INTERACTIVE SID; if you do not have the INTERACTIVE SID, you will not be able to run cmd.exe.

To work around this issue, you need to be specifically added to the ACL for cmd.exe or have the NETWORK SID added to the ACL for that file.

Note: This issue also applies to cscript.exe, atrib.exe, cacls.exe and other programs.

How do I remotely execute built-in Windows commands like "dir" and "del"?

Many Windows DOS commands such as "DEL" and "DIR" are built-in commands that are only available within the command shell cmd.exe. Remote execution within VShell for Windows does not include running the client-specified command within a command shell. Therefore, the command shell must be run as part of the remote execution as specified from the client's side. Remote execution of cmd /c <command> should work successfully.

For example:

vsh user@host cmd /c dir - remotely list the directory
vsh user@host cmd /c del <filename> - remotely delete a file

How can I stop getting GSSAPI library warnings from VShell for UNIX?

If you do not have the GSSAPI library installed on your server or you have a different version than the one required by your version of VShell, you may be seeing GSSAPI library warnings stating that the library could not be loaded. If you want to stop seeing these warnings, remove GSSAPI from the AuthenticationsAllowed option in the vshelld_config file. Once the parameter is removed, the vshelld server will no longer attempt to load the library and, therefore, won't report the error.

For example, change the configuration option to the following:

AuthenticationsAllowed { publickey, password, keyboard-interactive }

This will stop GSSAPI library warnings from appearing in VShell log files.

VShell Service Fails to Start (Error 1053), or attempting to start the VShell control panel results in "application configuration is incorrect" error.

Note: VShell 3.9 and later has addressed this issue by no longer using the Windows Side-by-Side assembly (WinSxS) for dependent libraries. If you are running a version of VShell earlier than 3.9, please upgrade to the latest official release. If you are unable to upgrade, please follow the below steps to resolve the problem.

The following information applies to VShell versions 3.0 through 3.8.

Background Information

VShell for Windows requires specific DLL files (Microsoft redistributable libraries). If required DLL files are not already available on a machine when VShell is installed, the VShell for Windows installer stores them in the Windows "Side-by-Side" (SxS) directory in order to ensure that they are available for use when the VShell service starts, or when the VShell control panel is launched.

In rare situations, a Microsoft Windows update or non-VanDyke software maintenance action may unexpectedly remove required DLLs or corrupt related manifest files located in the "WinSxS" system folder.

If these unexpected changes occur while the VShell service is running, the error will not be seen until either the VShell service is restarted, or the machine is rebooted.

It is unknown what application, actions, or circumstances cause the removal of these required WinSxS DLL files and/or manifest files.

When this situation occurs, the VShell service fails to start with the following error message:

Could not start the VanDyke Software VShell SSH2/SFTP service on Local Computer.
Error 1053: The service did not respond to the start or control request in a timely fashion.

Additionally, the VShell control panel fails to launch, with the system reporting the following error message:

This application has failed to start because the application configuration is incorrect. Reinstalling the application may fix this problem.

Resolution

To correct the WinSxS problem, a system administrator can "repair" the VShell installation. This "repair" operation will not modify your VShell configuration, and can quickly be done using the following steps as a guide:

  1. Launch the Windows Control Panel as the user that originally installed VShell on the machine.
  2. Launch "Add/Remove Programs".
  3. Find "VanDyke Software VShell" and press the "Change" button. If "VanDyke Software VShell" does not appear within the list, then you must re-launch the Add/Remove Programs applet within the user context of the account that was originally used to install the VShell application.
  4. When the "Welcome" screen appears, press the "Next" button.
  5. On the "Program Maintenance" screen, select "Repair" and press the "Next" button.
  6. On the "Ready To Repair" screen, press the "Install" button to repair your VShell installation. Note that a repair will not modify your current VShell configuration, unless you have installed a custom VShell deployment package which includes VShell configuration information.
  7. Once the installer has completed the "repair" operation, you should be able to launch the VShell control panel and start the VShell SSH2/SFTP service (as well as the VShell FTPS service, if installed).

    Note: The repair operation does not affect core Windows operating system files. It only replaces the DLL files that have been unexpectedly removed -- files that the VShell service requires for functional operation.