Using Sawmill To Analyze VShell® Server W3C Log Files

Introduction

Beginning with version 3.5, VShell for Windows has the ability to log using the W3C extended log file format. This new format allows the use of third-party log analysis tools to analyze VShell's log files. Log analysis tools like Sawmill combined with VShell's W3C logging support offer an easy way to answer many questions you may have concerning the activity on your VShell server. These tools parse the VShell log files and report information regarding the server traffic. This tip discusses how to set up the Sawmill log analysis tool with VShell's log files and how to apply its analysis and filtering capabilities.

Log analysis tools

There are many commercial and free log analysis tools available. Some tools provide basic server traffic statistics, such as total visits or total bandwidth during specific time periods. Other tools provide very detailed reports allowing you to determine exactly who has logged into the server, when they logged in, what files they uploaded or downloaded, and how many bytes were uploaded and downloaded.

The following links to the Open Directory Project provide a list of both commercial and free log analysis tools:

Open Directory Project commercial log analysis tools

Open Directory Project freeware and open source log analysis tools

Sawmill - Getting started

The Sawmill log analysis tool provides an intuitive web-based interface. The reports generated by Sawmill make it easy to answer the following questions and many more about VShell server usage:

• Who is connecting to the server?
• When is the server getting the most traffic?
• Where are connections originating?
• What files are being accessed?

At this point we assume you have successfully installed Sawmill on the same machine that the VShell server is running on and have gone through the initial setup for Sawmill licensing and administrative user setup.

It is important to verify that the VShell server is indeed logging using the W3C log file format. To do this, open the VShell Control Panel, go to the Logging options category and verify that the Use W3C log file format option is enabled. This option is on by default. Any logs on the machine that were created with a version of VShell prior to 3.5 will not be in the analyzable W3C format.

After starting Sawmill, the first step is to create a profile. A profile is just the set of log files that will be analyzed. You can specify a specific log file, or use wildcards or regular expressions to select all or a specific set of log files. Selecting the Start here link will launch the New Profile wizard. From here, browse to the VShell log folder (typically C:\Program Files\VanDyke Software\VShell\Log) and select either a specific log file, or the folder itself so that all logs within the folder are part of the profile.

Sawmill wizard - specifying log files

After specifying the log files, click the Next button.

Sawmill automatically determines the log format; it should detect your W3C log format. Once it does, choose the option Continue with the above detected log format and click on the Next button.

Now choose the log fields that you want included in the analysis. Checking all options will result in the most information available in the report, but will also increase the size of the Sawmill statistical database and the time to analyze. For this exercise, select all options other than Hits and Page views (Hits and Page views are geared to web servers and track specific web page hits and views). The Visitors option provides information on which users have connected to the server, while the Bytes, Server-to-Client bytes, and Client-to-Server bytes options track the total number of bytes transferred, number of bytes downloaded, and the number of bytes uploaded, respectively.

Sawmill wizard - choosing fields

After the options have been selected, click the Next button. Finally, enter a descriptive name for the profile and select Finish, then close the wizard window.

After the profile has been created, Sawmill automatically reads in the logs, analyzes them, and creates the database of statistics. Once this is done, the default page displays a general overview of the logs analyzed, including total number of visitors and the number of bytes transferred. If logs from multiple days were used, then daily average figures will be shown as well.

Sawmill report overview

Analysis, filtering, and drilling down

Once the profile has been analyzed, the statistics can be filtered and viewed in any number of ways. On the left side of the interface are the major categories for filtering the data. These categories include the following:

• Date and time: year, month, day, and hour filtering
• Visitor demographics: client IP and authenticated username filtering
• Server: server IP and port filtering
• X_sources: service filter (i.e., filter on either SFTP or FTPS service)
• X_topics: corresponds to log topics - sftp, ftps, conn, auth, info, err
• X_sessions: connection ID's
• Pages: files and folders touched
• X_messages: server message associated with an operation
• Single-page Summary: displays a summary of all of the above categories
• Log detail - displays the actual log entries for an operation

A very powerful feature of Sawmill is its ability to "drill down". By repeatedly filtering on the categories described earlier, you are able to select only those parts of data that you are interested in seeing. For example, an administrator may be interested in finding out who has been accessing a particular file on the system. To find this information you first select the Pages category. This lists all the files that have been uploaded to or downloaded from the system.

Sawmill Pages lists all files transferred

Select the file you are interested in to bring up the file's overview page. Then click on the Zoom to report >> drop-down menu and select the next category you want to filter on. For example, select the Authenticated usernames category to view a list of all users who accessed the file.

Sawmill Pages Zoom to report

Once you select a username, you can drill down even further, for instance, to select one of the date categories, which lets you see exactly when the file was last accessed by this user, or the Client IPs category, to see where the user was connecting from.

Sawmill zoom view on username analysis overview page

The Single-page Summary is another useful page that quickly provides information on all of the categories reported. This page displays bar graphs for all date and time categories, allowing you to easily see when the server traffic is at its highest and lowest.

Sawmill summary - date and time data

All authenticated usernames are listed, as well as all files that have been transferred.

Sawmill summary - display usernames