Skip to main content

Check out Interactive Visual Stories to gain hands-on experience with the SSE product features. Click here.

Skyhigh Security

Best practice: Working with Progress Indication Methods

  1. Last updated
  2. Save as PDF

To provide progress indication for a user who requested a file upload or download, you can work with suitable progress indication methods according to your working environment.

The following methods are available:

  • Progress pages — Progress pages keep the user informed about downloading and scanning times and provide a link for obtaining the completely processed file.
    We recommend using progress pages as the default method. Apply other methods only if progress pages are not eligible, for example, when the web browser used for downloading is not Mozilla-compatible.
    NOTE: When the default Progress Indication rule set is implemented, use of progress indication methods follows this recommendation.
  • Data trickling — Data trickling informs the user about the estimated overall processing time without indicating the portion that is required for anti-malware scanning.
    This method can, however, be used for any kind of download regardless of the web browser type.
  • FTP upload timeout prevention — This method can only be used for uploading files when Web Gateway is configured to run as an FTP proxy.
    The upload is not performed using a web browser, but requires a standalone FTP client, which can be implemented, for example, using Filezilla.

Working with progress pages

You can use progress pages with web browsers that are Mozilla-compatible. Taking packet captures allows you to track the progress indication workflow and detect issues.

Mozilla-compatible browsers

Progress pages only work for web browsers that announce their compatibility with Mozilla in the User-Agent headers of HTTP requests. This includes, for example, Mozilla Firefox, Microsoft Internet Explorer, Google Chrome, and Safari, but not Opera.

A packet capture of an HTTP request created, for example, using Wireshark, shows whether your browser is Mozilla-compatible. The capture contains a line about the User-Agent, such as the following:

User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; .NET CLR
1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR
3.5.30729)

This line shows that your browser is Microsoft Internet Explorer (MSIE) and that it is indeed Mozilla-compatible. For other browsers, the User-Agent lines might look as follows.

Firefox:

User-Agent: Mozilla/5.0 (Windows NT 5.2; rv:23.0) Gecko/20100101 Firefox/23.0

Chrome:

User-Agent: Mozilla/5.0 (Windows NT 5.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/
29.0.1547.57 Safari/537.36

Safari:

User-Agent: Mozilla/5.0 (Windows NT 5.2) AppleWebKit/534.57.2 (KHTML, like Gecko) Version/
5.1.7 Safari/534.57.2

Opera:

User-Agent: Opera/9.80 (Windows NT 5.2) Presto/2.12.388 Version/12.15

The line shows that this browser is not Mozilla-compatible.

Workflow for progress pages

When progress pages are enabled on Web Gateway, a page showing the download and scanning progress is provided, as well as a page that announces download completion and offers a link for obtaining the fully downloaded file.

By creating packet captures with, for example, Wireshark, you can track the workflow as needed to detect issues. In the following example, three devices with the following IP addresses appear in the workflow:

  • A client of Web Gateway: 10.10.80.1
  • Web Gateway: 10.10.80.57
  • A web server: 10.10.80.200

The workflow includes the following main steps.

  1. A client sends a request for downloading a large file from a web server.
<message number and timestamp> 10.10.80.1 3365 10.10.80.57 9090 HTTP 596 GET 
http://10.10.80.200/big_archive.zip
  1. Web Gateway redirects the client to the progress page, sending the following HTTP status: 307 Moved Temporarily.

    The redirection specifies a location with the same IP address as in the request that was originally sent by the client, but with the subdirectory information changed to enable redirection to the progress page.
Location: http://10.10.80.200/mwg-internal/de5fs23hu73ds/progress?id=kw0Rd85RXX
  1. The client opens a new TCP connection to Web Gateway, sends a request for the location that it was redirected to, and starts downloading the progress page.

    The download begins with a message that contains the following request.
GET http://10.10.80.200/mwg-internal/de5fs23hu73ds/progress?id=29qNbsp9oR HTTP/1.1

Due to the mwg-internal subdirectory information in the request, Web Gateway knows that the requested page is locally available. It provides this page to the client, rather than querying the web server that is identified by the IP address in the request.

  1. After downloading the progress page, the client requests an update from Web Gateway every five seconds, for example, as follows.
GET http://10.10.80.200/mwg-internal/de5fs23hu73ds/progress?id=29qNbsp9oR&a=1&13771061
54955 HTTP/1.1
  1. Web Gateway responds according to the progress made, sending the following HTTP status: 200 OK. With this status, line-based text data is sent to indicate the progress.

    The text data includes five values with particular meanings.

    For example, in the following response, the text data indicates that Web Gateway has started downloading the requested file.
16.3 MB; 204.5 MB; 7; 0; 0

The meaning of each value in this response is as follows.

  • 1 – Amount of data downloaded so far from web server
  • 2 – Total amount of data to be downloaded
  • 3 – Percentage of download completion
  • 4 – Anti-malware scanning complete? (0 = No, 1 = Yes)
  • 5 – Time (in seconds) consumed so far by anti-malware scanning

So, here the text data indicates that Web Gateway has already downloaded 16.3 MB from a file that is 204.5 MB large, which amounts to 7 percent, and that anti-malware scanning has not started yet.

  • In this response, the text data indicates that Web Gateway has downloaded the file and started anti-malware scanning, which is not yet complete.
204.5 MB; 204.5 MB; 100; 0; 153
  • In this response, the text data finally indicates that Web Gateway has completed anti-malware scanning.
204.5 MB; 204.5 MB; 100; 1; 4512

The web browser now presents the user with a link for downloading the requested file to the client.

Workflow for data trickling

When data trickling is enabled, Web Gateway downloads a requested file and sends tiny pieces of it to the requesting client.

This keeps the connection alive while Web Gateway downloads the whole file and scans it for infections. This means that while the scanning process is going on, the data moves at a very slow rate.

If Web Gateway detected an infection after the file had been completely downloaded, only a small amount of the file would have been passed on. So the client would not have received enough malicious data to let any harm be caused.

In more detail, the workflow for data trickling is as follows.

  1. Web Gateway sends an initial chunk, which is 4,096 bytes long, followed by 1 byte for every 1,000 that are downloaded, as long as the file is scanned for infections.

    As estimations of the download time are provided by the web browser, which is not aware of any Web Gateway activities, estimated download times look extremely long to the user at the beginning.

    For example, about 58 hours are estimated for downloading a 204 MB file, as shown in the following lines within the web browser.
big_archive.zip from 10.10.80.200
Estimated time left 58 hr 10 min (4.80 KB of 204 MB copied)
Download to: C:\Documents and Settings\big_archive.zip
  1. When Web Gateway has finished anti-malware scanning and found no infections, it sends the remaining portion of the file to the client at full network speed.

Workflow for FTP upload timeout prevention

When FTP upload timeout prevention is enabled, Web Gateway sends progress indication messages to the FTP client that sent a file for uploading to an FTP server.

The messages are sent every five seconds. This keeps the TCP connection alive and gives Web Gateway time to scan the file for infections.

The messages can be viewed on the FTP server using, for example, the Filezilla tool that was also used to install the FTP client.

In more detail, the workflow for FTP upload timeout prevention is as follows.

  1. Web Gateway uploads a file from the client and starts anti-malware scanning. While the scanning is in progress, Web Gateway sends a progress indication message to the client every five seconds.
Response: 150 File status OK; about to open data connection.
Response: 226-data processing in progress
Response: 226-data processing in progress
...

When the file has been uploaded to Web Gateway, this is indicated by a completion message.

C:\Documents and Settings ...     --> /home admin small_archive.zip     876,241
Normal Transferring
    00:00:00 elapsed             -:-:-left             100%     876,241 bytes

The file is not yet visible, however, on the FTP server.

  1. After Web Gateway has finished anti-malware scanning and found no infection, it uploads the file to the FTP server and sends a status message to the FTP client.
...
Response: 226-data processing in progress
Response: 226-data processing in progress
Response: 226 File receive OK
Status: File transfer successful, transferred 876,241 bytes in 26 seconds

The file is now visible on the FTP server.

  1. Back to top
  • Was this article helpful?

Recommended articles

  1. Article type
    Topic
  2. Tags
    1. Web Gateway