Progress Indication Methods
While Skyhigh Secure Web Gateway performs anti-virus/anti-malware scanning on downloaded fi les,Progress Indication sends data back to the client computer to keep the connection alive as well as to provide visual progress information to the user. Progress Indication assures that the user and web browser know that Secure Web Gateway received the request and is processing the download.
Why should I use it?
Progress Indication should be used for two major reasons:
- To prevent TCP connections from timing-out between the Secure Web Gateway and the client computer while the Secure Web Gateway scans large files for viruses and malware
- To provide users with visual feedback in their web browser, indicating that their download has started and scanning is in progress (this function is only provided by Progress Pages)
Types of Progress Indication
Progress Page
Progress Pages are particularly important when the Secure Web Gateway is configured to do anti-virus/anti-malware scanning. The scanning of large or "dense" files can take significant periods of time. A "dense" file might be relatively small in size, but contain many sub-files that are archives or executables such as: .dll, .exe, .zip, .rar, .jar, .iso, .xap. All of which must be individually unarchived and scanned.
A fi le that is large and dense, such as a 200MB ZIP file containing software developer tools,can take 30+ minutes to be fully scanned by the anti-virus/anti-malware engine. If progress indication is not provided to the user with Progress Pages, most users become impatient,or worse, believe there is a network or server problem. The impatient user will relaunch their download multiple times, causing the anti-virus/anti-malware engine to scan redundant copies concurrently, adding additional load to the Secure Web Gateway.
Progress Pages give the user visual feedback, indicating that their fi le is being downloaded and anti-virus/anti-malware scanned, reducing helpdesk calls.
Data Trickling
Data Trickling can work for any client device or software that downloads data. It is an alternative to Progress Pages, but does not provide any particular visual progress indication beyond that provided by the web browser or client software itself. Data Trickling,from the point of view of an end-user, will make downloads appear very slow. Therefore,we recommend Progress Pages always be enabled and Data Trickling enabled as a fallback for clients that cannot use Progress Pages (this is exactly how the default rule sets are configured).
Data Trickling effectively throttles the download connection speed between the Secure Web Gateway and the client, sending only a tiny amount of data that prevents the TCP connection from timing out. When the Gateway finishes scanning the fi le for viruses and malware (finding none), the remaining portion of the fi le is sent to the client at full network speed.
Because estimations of download time are provided by the web browser, and it is not aware of what the Secure Web Gateway is doing, estimated download times will initially look extremely long to a user. The image below shows download progress indication provided by Internet Explorer v8 while Data Trickling is occurring
FTP Upload Timeout Prevention
When the Secure Web Gateway is used as an FTP proxy, FTP Upload Timeout Prevention enables the Secure Web Gateway to send FTP status messages to FTP clients that prevent the TCP connection from timing-out. This gives Secure Web Gateway time to scan uploaded fi les for viruses and malware. The image below shows progress indication that was sent by the Secure Web Gateway to Filezilla FTP client software.
.
How to Get it
Recommendations
We recommend that you import copies of Progress Indication rules from the Rule Set Library, keeping them all enabled and using their default configurations unless you have been advised to do otherwise by Skyhigh Technical Support or you are certain your environment requires changes. Default settings work well for most customers.
Importing from the Rule Set Library
If you do not already have these rules as part of your Policy, you can import them from the Rule Set Library. Under "Policies" on the "Rule Sets" tab, click "Add" and then choose "Rule Set from Library...". Find the Rule Set named "Progress Indication". In some version of Skyhigh Secure Web Gateway, it will be nested inside Rule Set "Common Rules".
Detailed Information
Progress Page
When can it be used?
Progress Pages only work for web browsers that announce their compatibility with Mozillain their HTTP User-Agent header. This includes: Internet Explorer, FireFox, Chrome and Safari (but not Opera).
What is Mozilla?
In the early days of graphic web browsers, Mozilla was the project code name of Netscape Navigator during its development. Netscape was an early innovator, introducing new and advanced features for the day. Websites checked the HTTP User-Agent header. If it was a Mozilla web browser, advanced content would be served; if not, basic content. Eventually Netscape's competitors implemented similar features, enabling their browsers to display the same content. However, not all websites updated their User-Agent string check. As a result,competitors such as Microsoft added "Mozilla compatible" to the User-Agent sent by their browsers. This continues today.
How do I know if my browser is Mozilla compatible?
It is easy to figure out if your web browser is Mozilla compatible by doing a quick packet capture. Start Wire shark, browser to a website, then in the capture, fi nd an HTTP request. In the example below, you can see that the web browser is Microsoft Internet Explorer v8 and is Mozilla v4 compatible.
HTTP User-Agent Header Samples
Mozilla Compatible:
Internet Explorer v8
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0;.NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR3.0.4506.2152; .NET CLR 3.5.30729)
FireFox v23
User-Agent: Mozilla/5.0 (Windows NT 5.2; rv:23.0) Gecko/20100101Firefox/23.0
Chrome v29
User-Agent: Mozilla/5.0 (Windows NT 5.2) AppleWebKit/537.36 (KHTML, likeGecko) Chrome/29.0.1547.57 Safari/537.36
Safari v5
User-Agent: Mozilla/5.0 (Windows NT 5.2) AppleWebKit/534.57.2 (KHTML, likeGecko) Version/5.1.7 Safari/534.57.2
Not Mozilla Compatible:
Opera v9
User-Agent: Opera/9.80 (Windows NT 5.2) Presto/2.12.388 Version/12.15
How does it work?
- The client opens a TCP connection to the Secure Web Gateway (if not already open)and requests a file to download.
- The Secure Web Gateway begins to download the file. If it is not text or HTML and takes the Gateway more than 5 seconds to download and process the file, the Secure Web Gateway will redirect the client to a Progress Page with an HTTP status of"307 Temporarily Moved". The redirection will be to the same URL Host as the original request, but the subdirectory will be changed to a unique value similar in format to: "/mwg-internal/de5fs23hu73ds/progress?id=29qNbsp9oR".
- The client then opens a new TCP connection to the Secure Web Gateway and requests the URL to which it was redirected.
- Because the requested URL contains subdirectory "mwg-internal", the Secure Web Gateway knows the requested file(s) are local and stored on the Secure Web Gateway.The Secure Web Gateway itself serves a Progress Page to the client, rather than querying the webserver represented by the URL Host of the request.
- Every 5 seconds the client will request a progress update from the Secure Web Gateway.
- The Web Gateway's progress update response contains fi ve values, in a format similar to: 204.5 MB;204.5 MB;100;1;4512
- 1st value = amount of data downloaded so far, from web server to Secure Web Gateway.
- 2nd value = total size of file being downloaded
- 3rd value = percentage of download complete (0-100)
- 4th value = is AV scanning complete? (0=no; 1=yes)
- 5th value = cumulative elapsed time during anti-virus/anti-malwarescanning
What does the user experience?
When a user downloads a non-text/html fi le that takes more than five seconds to be processed by the Secure Web Gateway, the Secure Web Gateway will redirect the client to a Progress Page, which will display dynamic progress indication while the download is occurring.The Progress Page shows the amount of data downloaded, total size of the fi le being downloaded and a progress bar.
Once the Gateway has finished downloading the requested file, it will begin anti-virus/anti-malware scanning. During this phase, the Progress Page counts the elapsed seconds during scanning. There is also an animated progress bar.
Once scanning is complete, the Progress Page presents the user with a link. When clicked, the fi le is downloaded from the Secure Web Gateway to the client.
When the download link is clicked, the web browser presents the user with a File Download dialog box.
What settings can be modified?
We recommend that you use default settings as found in the Rule Set Library, unless you have been advised to do otherwise by Skyhigh Technical Support or you are certain your environment requires changes. The defaults work well for most users.
However, there are a few settings that can be configured if you drill-down into the Enable Progress Page rule's Event settings container.
Items in the "Templates" section allow you choose and edit the appearance of the Progress Page's HTML. Boxes allow you to choose the collection which contains the templates you want to use, as well as the specific templates for each of the three stages of a Progress Page: the Progress Bar Page, Download Finished Page and Download Canceled Page.
The lower section contains three time values:
"Delay for redirects to Progress Page" in seconds: after a client requests a file, if the Secure Web Gateway needs more than this many seconds to perform its filtering and scanning duties, the Secure Web Gateway will redirect the client to a Progress Page in place of the file.
"File availability time before download" in minutes: after the Secure Web Gateway has finished downloading and scanning a file, it will be available on the Secure Web Gateway for this amount of time until a user downloads a copy.
"File availability time after download" in minutes: after a user downloads a copy of the file, it will remain available on the Secure Web Gateway for this amount of time for additional downloads.
What does a packet capture look like?
In the packet capture images below, you will see the following devices communicating:
- client computer = 10.10.80.1
- Skyhigh Secure Web Gateway = 10.10.80.57
- web server = 10.10.80.200
The client computer requests a web page from the Secure Web Gateway. The Gateway will redirect the client to the Progress Page with a "307 Moved Temporarily" HTTP status.The client will then open a new TCP connection to the Secure Web Gateway and request the location to which it was redirected.
The client downloads the Progress Page (HTML, CSS, JPEGs, PNGs, etc.). Every 5 seconds the client will then request a progress update to which the Secure Web Gateway will respond. The progress update in the image below indicates that the Secure Web Gateway is downloading the requested file.
Below, the Secure Web Gateway has downloaded the fi le and begun anti-virus/anti-malware scanning.
The Secure Web Gateway has completed anti-virus/anti-malware scanning. The web browser now presents the user with a link from which the requested fi le can be downloaded .
Data Trickling
When can it be used?
Data Trickling can be used for any client that downloads data.
How does it work?
When Data Trickling is enabled, the Secure Web Gateway sends tiny pieces of the requested download file back to the client. This keeps the connection alive while the Secure Web Gateway downloads the whole fi le and scans it for viruses and malware. By default, the Gateway sends an initial chunk of data that is 4,096 bytes and there after sends 1 byte for every 1,000 it downloads. The data moves at such a slow rate that if the Secure Web Gateway does detect a virus or malware after the file is downloaded, such a small amount of the file has been passed onto the client that no harm should be possible.This is because the client will not have received enough of the malicious data for it to be executed.
What does the user experience?
When Data Trickling is enabled, downloading a fi le seems like a typical file download that is not going through a proxy: the web browser's download manager appears,manages and displays information. The drawback of this, however, is that the estimated time displayed will seem outrageously long to a user, because the Secure Web Gateway only sends tiny amounts of data while anti-virus/anti-malware scanning is occurring.
In the image below, you can see that Internet Explorer is estimating 58 hours to download a 204MB fi le across an Ethernet LAN.
What settings can be modified?
We recommend that you use default settings as found in the Rule Set Library, unless you have been advised to do otherwise by Skyhigh Technical Support or you are certain your environment requires changes. The defaults work well for most users.
However, there are two settings that can be configured if you drill-down into the Enable Data Trickling rule's Event settings container:
- "Size of first chunk" in bytes: when data trickling begins, the first chunk of downloaded and scanned data sent by the Secure Web Gateway to the client will be this size.
- "Forwarding rate" per thousand, in bytes: after the first chunk, the Secure Web Gateway will send this many bytes to the client for every 1,000 bytes downloaded by the Secure Web Gateway, keeping the client/Secure Web GatewayTCP connection alive.
What does a packet capture look like?
In the packet capture image below, you will see the following devices communicating:
client computer = 10.10.80.1
Skyhigh Secure Web Gateway = 10.10.80.56
web server = 10.10.80.200
A packet capture will look like any other fi le downloaded without a proxy. The difference will be that packets contain only a small amount of data while the Secure Web Gateway is doing anti-virus/anti-malware scanning.
In the image below of a packet capture, after the initial bytes are sent, the download settles into a consistant rate of 8 bytes per packet from the Secure Web Gateway to the client:
FTP Upload Timeout Prevention
When can it be used?
FTP Upload Timeout Prevention is only used for FTP uploads via Skyhigh Secure Web Gateway's FTP Proxy Port (default listening port number is 2121). Settings for the FTP proxy, including enabling it, are configured on the same page as the HTTP Proxy Port.
However, their settings are independent of one another and the FTP Proxy Port is disabled by default. See the image below for help finding the FTP Proxy Port settings.
To use the Secure Web Gateway's FTP Proxy Port, you will also need to use a stand-alone FTP client. You can use a program like Filezilla. (FYI: a web browser will not work foruploading data via FTP.)
The image below shows a basic configuration inside Filezilla that allows traffic to FTP server 10.10.80.200 to be sent through Skyhigh Secure Web Gateway's FTP Proxy Port at10.10.80.57 (NOTE: in this example, the Secure Web Gateway does not require proxy authentication for FTP traffic).
How does it work?
When FTP Upload Timeout Prevention is enabled, the Secure Web Gateway will send a progress indication every 5 seconds while performing anti-virus/anti-malware scanning. The progress indication message is sent as an FTP 226 status code, which generically means: "requested file action successful". Specifically, the Skyhigh Secure Web Gateway sends a message that reads: "226-data processing in progress".
What does the user experience?
Every 5 seconds, while the Secure Web Gateway is performing anti-virus/anti-malware scanning, the user will see the message: "226-data processing in progress" inside their FTP client program. After uploading a fi le, the user will see that 100% of the file has uploaded, but the fi le will not yet be visible on the FTP server. This is because the Skyhigh Secure Web Gateway is acting as an FTP proxy. The fi le indeed was 100% uploaded,but to the Secure Web Gateway, not the FTP server. Thus the file is not visible on the server.The Secure Web Gateway will send "226-data processing in progress" messages to the client every 5 seconds until anti-virus/anti-malware scanning is complete. If no virus or malware was found, then the Secure Web Gateway will upload the file to the FTP server. Onceuploaded, it will become visible on the FTP server and a message will be sent to theuser's FTP client program indicating success, such as: "File transfer successful".
The image below shows that the fi le "small_archive.zip" has been uploaded to the Secure Web Gateway and anti-virus/anti-malware scanning as begun:
The following image shows that the Secure Web Gateway has finished anti-virus/anti-malware scanning and the fi le has been uploaded onto the FTP server:
What settings can be modified?
FTP Upload Timeout Prevention does not have user-modifiable settings.
What does a packet capture look like?
In the packet capture images below, you will see the following devices communicating:
- client computer = 10.10.80.1
- Skyhigh Secure Web Gateway = 10.10.80.57
- FTP server = 10.10.80.200
If you create a packet capture on the Secure Web Gateway or the client PC, you can clearly seethe progress indication sent by the Secure Web Gateway while it does anti-virus/anti-malware scanning: "226-data processing in progress":
Common Problems
OMG! - Data Trickling Will Take Forever!
No, actually it will not. From the point of view of a user sitting at a web browser, at first it will look like the download is estimated to take hours. In reality it will fi nish much quicker.See information above to better understand what occurs when Data Trickling is in use.
How do I diagnose it?
If a user complains of slow downloads, ask them what exactly they are experiencing. If their web browser is estimating a huge time value for a download, check your Secure Web Gateway's Progress Indication settings. It is possible that Data Trickling is enabled butnot Progress Pages.
How do I fix it?
If you want better information passed to users than is provided by Data Trickling,consider enabling Progress Pages if they are not.
Progress Page not Found Due to External Load Balancing Issues
If you are using load balancing devices in front of a group of Secure Web Gateway, sometimes a user will receive a Progress Page, but before it completes downloading and scanning, the user will receive a "Page Not Found" error. This is because your load balancer's "stickiness"value is not long enough: it chose to send a client's later request for a Progress Page update to a Secure Web Gateway that did not create the Progress Page. A Progress Page created for a user exists only on the Web Gateway that created it. Therefore, a client's request must arrive at that same Secure Web Gateway until downloading and scanning is complete.
How do I diagnose it?
The easiest way to diagnose this issue is running simultaneous packet captures on all Secure Web Gateway through which the client might send traffic. When the client receives the "Page Not Found" error from a Secure Web Gateway, stop the captures and inspect. You will see requests arrive at one Secure Web Gateway every five seconds, then all of a sudden,arrive on another Secure Web Gateway which will immediately send an HTTP 404 Page Not Found error.
How do I fix it?
To avoid this problem, you will need to increase the "stickiness" value on your load balancer for client source IP addresses.
Progress Page Not Displayed, Even WhenEnabled
Even when properly enabled, sometimes Progress Pages do not start when expected. This most often happens if your web browser is not announcing "Mozilla
Compatibility " in the HTTPUser-Agent header (such as when using Opera). Or, this could happen if you have changed the rule from defaults settings.
How do I diagnose it?
Do a packet capture and inspect the HTTP User-Agent header sent by the client's web browser. If it contains "Mozilla", it should receive a Progress Page. Also inspect your Progress Indication rules. Are Progress Pages enabled? Have the criteria been changed?
How do I fix it?
If your web browser does not announce Mozilla compatibility, this cannot be changed.However, some customers have had luck adding criteria to the Progress Page rule so it matches "Opera" or other criteria. Also, if your rules have already been modified and you want to return to defaults, you can import a fresh copy from the Rule Set Library.Compare the fresh copy to yours. Cut and paste rules as needed. Delete the un used newly-imported rules before saving changes.
Web Page Parts Are Missing, Embedded Content Not Loaded
Sometimes Progress Pages will be triggered for content embedded on an HTML web page,such as Silverlight content. Under normal operation, after a Progress Page is activated,once downloading and scanning is complete a user must click the download link in order for the client computer to receive the fi le. If a user does not see the link or cannot click it,the content is not received.
Dynamic Silverlight content on an HTML page is often contained in a XAP file, which is an archive in ZIP format. Within a XAP file there are many sub-files including executables. If downloading and scanning takes more than fi ve seconds, which it often does for XAPs, a Progress Page will be triggered. However, the user will never see this Progress Page:generally speaking, the web browser will only display an HTML Progress Page in place of an HTML page. It will not display an HTML Progress Page in place of an embedded fi le such as XAP, JPEG or CSS that is part of an HTML page.
More specifically, a Progress Page will be displayed to a user in place of a fi le directly requested by the user. This typically means the user clicked a link to view a JPEG, an HTML page or to download a file. These are "primary" fi les requested directly by the user. A Progress Page, on the other hand, will not be displayed in place of a "secondary" file requested by the web browser. Secondary files are typically used to create primary files.For example, if a user types the address of an HTML page and hits enter, the web browser will load the HTML file which is a "primary" fi le. Then, the web browser will automatically request and load all files referenced by that HTML file, such as pictures or CSS which are used to render the HTML page as described by the HTML fi le. These are "secondary" files.
How do I diagnose it?
A packet capture on the client computer or Web Gateway will allow you to see this issue. The web browser will appear to be empty or missing content (no Secure Web Gateway Progress or Block Page will be seen by the user). However, in the packet capture, you will be able to see the Secure Web Gateway redirect the client computer to a Progress Page(for more information about packet captures containing Progress Pages, see above).
How do I fix it?
A white list entry will fix this problem, bypassing the fi le from Progress Pages and/oranti-virus/anti-malware scanning. You could add it to the Global Whitelist, which will avoid both. Or, you could create a new whitelist within the Progress Indication,Progress Page or Gateway Anti-Malware rule sets. Be sure you make the correct typeof whitelist entry, which depends on the criteria used for matching.