VShell Server FAQ

Can VShell be exported outside the United States?

VShell is under export control as a strong encryption product by the U.S. Department of Commerce. VShell can be exported under the ENC export license exemption to customers in member states of the European Union (EU) and Australia, Canada, Czech Republic, Hungary, Japan, New Zealand, Norway, Poland, and Switzerland. Outside the EU and eight other countries, VShell may be exported to non-government users only.

For more information on exporting VShell, see the Encryption Export page.

How is the VShell Server Licensed?

There are three elements that are all essential when considering what type of VShell license purchase will work for your organization:

  1. VShell "Edition"
  2. Server Machine Count
  3. Functionality

All of these elements must be considered together when determining the licensing requirements that apply to your planned deployment of VShell.

VShell "Edition"

VShell licenses are categorized into three separate "Editions" which determine the maximum number of allowed concurrent client connections to VShell. VShell license editions allowing fewer concurrent connections are priced lower than editions that allow a larger number of concurrent connections.

License Edition Concurrent (authenticated) client connections allowed
Administrator 2 concurrent connections
Workgroup 10 concurrent connections
Enterprise Unlimited concurrent connections

VShell operates as a 'Workgroup' edition during the 30-day evaluation period, allowing up to 10 connections at a time to the server.

Note: "Client connections" is not the same as "users". The number of concurrent client connections is the number of connections where the client has successfully authenticated to VShell, regardless of the user account associated with the authentication. Two separate connections to the VShell service will count as two concurrent connections even if the same user account is used for authentication in each case.

Server Machine Count

VShell is licensed on a per-computer installation basis, whether it be physical or virtual. A VShell license (regardless of edition – Administrator, Workgroup, or Enterprise) can only be installed on a single machine. If you plan to deploy VShell on 7 different machines, for example, 7 VShell licenses must be purchased. There is one exception to this requirement:

In the event that a failover, disaster recovery or other secondary server is maintained offline ("Failover Server") and such Failover Server is solely used when the primary server is not functioning, then, for each license for the primary server, one backup copy of the Software may be installed on the Failover Server corresponding to the primary server at no additional charge. High availability or load balancing server deployments require VShell licenses be purchased for each node in the cluster.


All versions of VShell provide SSH2/SFTP/SCP functionality.

VShell with FTPS

A version of VShell with FTPS (FTP over TLS) is available, allowing you to provide support for a wider variety of connectivity protocols depending on your clients' needs.

Which clients does VShell support?

It is expected that any client that conforms to the SSH2 protocol is supported by VShell. Similarly, any client that conforms to the FTP/FTPS protocol is supported by VShell with FTPS.

If you experience an issue connecting to VShell with a particular client application, please contact support@vandyke.com.

Why am I getting key exchange failures and what can I do about them?

Several things cause key exchange failures. If you are having problems, check the following conditions.

  1. What type of host key are you using (DSA, ECDSA, Ed25519, RSA)? Make sure that your client supports the algorithm you selected.
  2. Make sure that at least one of the ciphers that your client is using matches at least one of the ciphers enabled in the VShell Control Panel.
  3. Make sure that at least one of the MACs that your client is using matches at least one of the Macs enabled in the VShell Control Panel.
  4. Make sure that you have Compression enabled in the VShell Control Panel if your client is requesting compression.

I can't get public key-only authentication in VShell for Windows to work. What's wrong?

For public key-only authentication to work with VShell for Windows, you must have the VShell Public Key LSA Module installed. This module comes with VShell and should have been installed by default with the VShell installation package. You may just need to reboot your machine for the module to take affect.

If rebooting your machine does not correct the problem, try reinstalling VShell and be sure that the Public Key LSA Module installation option is enabled. After installing VShell with the module, you must reboot your machine.

How do I run VShell as a user other than "System"?

To run VShell as a user other than "System", follow these steps:

Note: The user account must be a member of the Administrators groups (according to Microsoft's documentation on loading profiles).

  1. Ensure that the user account that you want to run VShell (the service) as has the following policy settings:
    • Act as part of the operating system
    • Logon as a service
    • Replace a process level token
  2. Stop any VShell service(s) that are currently running.
  3. On the machine running VShell, go to the Windows Control Panel and locate and open "Services".
    1. Right-click on the "VanDyke Software VShell" entry and select "Properties" from the menu.
    2. Select the "Log On" page and click on the "This account" radio button.
  4. Enter the user account that you want to use with the appropriate password.
  5. Re-start the VShell service(s).
  6. VShell should now be running as the user that you entered.

My public-key authentication failed on Red Hat and VShell logged an "Unable to authenticate" error. Why?

If VShell could not authenticate and logged the following error message:

"Disconnecting from server: Unable to authenticate using any of the configured authentication methods."

there is most likely a UMASK problem.

On Red Hat, the default UMASK is 0002, which means that files are created by default with the group-writable bit set. This causes public-key authentication with vshelld to fail. To keep this from happening on newly created files, change the setting on your UMASK to make them not group-writable (set them to 0022, for example). To correct this problem with existing files, change the permissions on your .vshell and publickey folders and any files in your publickey folder so that they are writable only by the owner.

Why am I receiving the error message, 'The user does not have the right to create processes; check for "logon locally" user right'?

If you receive the following error message:

00002: Could not start program (Scraper.exe /X 80 /Y 24 C:\WINDOWS\system32\CMD.EXE): The user does not have the right to create processes; check for "logon locally" user right

First, ensure that the account that you are trying to connect to VShell with has its settings configured as follows:

  • In the VShell Control Panel, select the "Access Control" category. Ensure that the "Logon" access right is checked.
  • In Windows, ensure that the "Log on locally" option is selected.
  1. Open the "Start" menu and click on "Control Panel".
  2. Open "System and Security".
  3. Open "Administrative Tools" and select "Local Security Policy".
  4. In the folder tree, select "Local Policies / User Rights Assignments".
  5. Make sure that the account in question is listed in the "Allow log on locally" policy and is not listed in the "Deny log on locally" policy.

Why does my OpenSSH agent crash when I try to add a VShell-generated key pair?

You cannot use VShell-generated key pairs with the OpenSSH SSH-agent because the agent is unable to read private keys generated by vkeygen (unlike public keys, private keys do not have a standardized format). However, VanDyke servers and clients do understand private and public keys generated by OpenSSH applications.

When I try to establish a shell, my connection gets dropped. Why?

If you are connecting to a Windows 2003 (or newer) server using GSSAPI authentication and cannot establish an interactive shell, the problem may be that you do not have access to the cmd.exe program.

To run GSSAPI, VShell® uses the MS SSPI interface, which does not provide a way to do interactive logons and, therefore, does not provide the INTERACTIVE security identifier (SID). Under Windows 2003 server, many of the programs in "windows\system32" grant access via the INTERACTIVE SID; if you do not have the INTERACTIVE SID, you will not be able to run cmd.exe.

To work around this issue, you need to be specifically added to the ACL for cmd.exe or have the NETWORK SID added to the ACL for that file.

Note: This issue also applies to cscript.exe, atrib.exe, cacls.exe and other programs.

How do I remotely execute built-in Windows commands like "dir" and "del"?

Many Windows DOS commands such as "DEL" and "DIR" are built-in commands that are only available within the command shell cmd.exe. Remote execution within VShell for Windows does not include running the client-specified command within a command shell. Therefore, the command shell must be run as part of the remote execution as specified from the client's side. Remote execution of cmd /c <command> should work successfully.

For example:

vsh user@host cmd /c dir - remotely list the directory
vsh user@host cmd /c del <filename> - remotely delete a file

How do I get VIM to work with VShell?

To get VIM working with VShell, you must first install VIM for Microsoft Windows. The installer can be downloaded from the following location:


Once you have the required version of VIM installed, log on to VShell using your SSH client configured in VT220 mode.

Note: If you are connecting to VShell with terminal type TTY (scrollback mode), VIM will not work.

What is Scraper.exe? What purpose does it serve?

Scraper.exe is VShell's screen-scraping application used to translate data formatted for Windows systems so that it can be accessed by users running terminal emulators. The screen scraper also reformats user input from terminal emulators so that it can be understood by applications running on Windows systems.

Note: Scraper.exe does not run when the client connected to VShell has requested the use of the scrollback buffer. (For more information about the scrollback buffer, see FAQ, "Why doesn't the scrollback buffer seem to work".)

Why doesn't the scrollback buffer seem to work (even making the scroll buffer bigger doesn't help)?

Your client may not be configured correctly to use the VShell® scrollback feature. You need to set up your client to send a terminal type TTY message when it connects.

Note: When using the scrollback buffer mode, depending on your command shell, it may not be possible to use cursor-based applications like Edit, VI or VIM. Also, you may not be able to use the back arrow to get to the first line of a multi-line command (e.g., if the command is very long and wraps to the next line, you may not be able to cursor back up to the first line).

You may need to experiment with command-line switches to optimize the function of your particular command shell.

To configure SecureCRT to use the VShell scrollback buffer, follow these instructions:

  1. Open the SecureCRT Session Options dialog and select the Emulation / Advanced category.
  2. Check the Terminal type check box, enter TTY in the corresponding entry box and click the OK button to save your changes.

How can I configure syslogd to log vshelld messages to a separate file?

The following steps describe how to configure vshelld to log its messages to a separate file using the syslog (8) facility.

  1. Change the vshelld_config (5) option "SyslogFacility" to a unique syslog (8) facility that is not currently used by any other applications on the system. For example, the following line will cause vshelld to log to syslog using the "local3" facility: SyslogFacility LOG_LOCAL3
  2. Modify syslog.conf (5) so that messages of the facility you chose in the previous step will be logged to a specific file. For example, the following line will steer any messages to the "local3" facility to the file /var/log/vshelld.
    local3.* /var/log/vshelld
    Log messages from vshelld will now be appended to /var/log/vshelld.
  3. Depending on your syslog.conf settings, some or all of your vshelld messages may continue to be appended to other files (such as /var/log/messages). To prevent this, use the "none" syslog level to exclude vshelld messages from being sent to other destinations. For example, including "local3.none" in the following line prevents any vshelld messages (if configured as above) from being appended to /var/log/messages:
    # Log anything of level info or higher (except local3).
    local3.none;*.info /var/log/messages

For more information on configuring vshelld and syslog, see the vshelld_config (5) and syslog.conf (5) manual pages.

How can I stop getting GSSAPI library warnings from VShell for UNIX?

If you do not have the GSSAPI library installed on your server or you have a different version than the one required by your version of VShell, you may be seeing GSSAPI library warnings stating that the library could not be loaded. If you want to stop seeing these warnings, remove GSSAPI from the AuthenticationsAllowed option in the vshelld_config file. Once the parameter is removed, the vshelld server will no longer attempt to load the library and, therefore, won't report the error.

For example, change the configuration option to the following:

AuthenticationsAllowed { publickey, password, keyboard-interactive }

This will stop GSSAPI library warnings from appearing in VShell log files.

How do I get VNC to run securely over SSH2 using SecureCRT and VShell?

The following steps will help you get VNC running over VShell. These steps assume that you are using SecureCRT as your port-forwarding client.

On the VShell/VNC server:

  1. In the VNC interface, set the following parameters:
    1. Check the Accept Socket Connections check box.
    2. Set "Display number" to "02" (this will configure the server to listen on port 5902).
    3. Set the password.
    Note: Setting "Display number" directly to "5902" may cause an error with the client.
  2. Create the following Registry entry

    AllowLoopBack REG_DWORD "1"

    (hex) under the following key:


  3. Restart the VNC service.

In your SecureCRT client:

  1. Select File / Connect... to open the Session Manager.
  2. Click on the New Session button and configure a SSH2 session that connects to the VShell/VNC server.
  3. Select the Connections / Port Forwarding category and press the Add button.
  4. Create a port forward entry with the following settings:
    1. Name: VNC (or whatever name is meaningful to you)
    2. Local Port: 5902
    3. Remote Port: 5902
    4. Clear the Destination host is different for the SSH server check box.
  5. Click on the OK button until you are back at the Session Manager and then click on the Connect button to initiate your new session.

In your VNCViewer:

  1. Enter "localhost:02" in the Connection details dialog.

Why is the message-of-the-day displayed twice on Solaris systems?

In the Solaris operating environment, the login shell by default displays the message of the day. Therefore, if you also include the MOTDPath option in your vshelld_config file, VShell® misinterprets this as a second instruction to display the message of the day. To suppress the second message, comment out or remove the "MOTDPath/etc/motd" setting in the vshelld_config file.

VShell Service Fails to Start (Error 1053), or attempting to start the VShell control panel results in "application configuration is incorrect" error.

Note: VShell 3.9 and later has addressed this issue by no longer using the Windows Side-by-Side assembly (WinSxS) for dependent libraries. If you are running a version of VShell earlier than 3.9, please upgrade to the latest official release. If you are unable to upgrade, please follow the below steps to resolve the problem.

The following information applies to VShell versions 3.0 through 3.8.

Background Information

VShell for Windows requires specific DLL files (Microsoft redistributable libraries). If required DLL files are not already available on a machine when VShell is installed, the VShell for Windows installer stores them in the Windows "Side-by-Side" (SxS) directory in order to ensure that they are available for use when the VShell service starts, or when the VShell control panel is launched.

In rare situations, a Microsoft Windows update or non-VanDyke software maintenance action may unexpectedly remove required DLLs or corrupt related manifest files located in the "WinSxS" system folder.

If these unexpected changes occur while the VShell service is running, the error will not be seen until either the VShell service is restarted, or the machine is rebooted.

It is unknown what application, actions, or circumstances cause the removal of these required WinSxS DLL files and/or manifest files.

When this situation occurs, the VShell service fails to start with the following error message:

Could not start the VanDyke Software VShell SSH2/SFTP service on Local Computer.
Error 1053: The service did not respond to the start or control request in a timely fashion.

Additionally, the VShell control panel fails to launch, with the system reporting the following error message:

This application has failed to start because the application configuration is incorrect. Reinstalling the application may fix this problem.


To correct the WinSxS problem, a system administrator can "repair" the VShell installation. This "repair" operation will not modify your VShell configuration, and can quickly be done using the following steps as a guide:

  1. Launch the Windows Control Panel as the user that originally installed VShell on the machine.
  2. Launch "Add/Remove Programs".
  3. Find "VanDyke Software VShell" and press the "Change" button. If "VanDyke Software VShell" does not appear within the list, then you must re-launch the Add/Remove Programs applet within the user context of the account that was originally used to install the VShell application.
  4. When the "Welcome" screen appears, press the "Next" button.
  5. On the "Program Maintenance" screen, select "Repair" and press the "Next" button.
  6. On the "Ready To Repair" screen, press the "Install" button to repair your VShell installation. Note that a repair will not modify your current VShell configuration, unless you have installed a custom VShell deployment package which includes VShell configuration information.
  7. Once the installer has completed the "repair" operation, you should be able to launch the VShell control panel and start the VShell SSH2/SFTP service (as well as the VShell FTPS service, if installed).

    Note: The repair operation does not affect core Windows operating system files. It only replaces the DLL files that have been unexpectedly removed -- files that the VShell service requires for functional operation.