Understand User Agents
You might be asking yourself, "Okay, why do I care about the user agent?". Well, the user agent can be important and can help you out in lots of situations. Let's say iTunes keeps prompting users to enter in their credentials, or a windows media player stream simply will not play. You might need to bypass these programs from certain rules to get them functioning the way they should be. You can easily do that with 1 header, and that is the user agent header.
What is a User Agent?
A User Agent is a header in an HTTP request that identifies the client software originating the request.
Why is this header important?
This is important because it identifies what piece of software may be making a request so you can properly identify where it is coming from. This could be anything from the browser, to windows media player. They all have a user agent to identify what software it is.
Why would I need to create a rule to match on the user agent?
There are times when you need to match the user agent criteria for a rule. These times usually include bypassing a specific rule set within Secure Web Gateway that may be causing issues with the user agent. The beauty about Secure Web Gateway 7 is that you don’t have to bypass all of the rules. Say you are having an issue where a specific user agent is not working with authentication (which is one of the most common times) so you only want to bypass authentication. You can easily do that. There will be some examples of this and other common scenarios at the end.
How do I find out what the user agent is for a request?
There are a few ways to find out this sort of information. Below I will list and explain some common ways to find this out:
It is possible to log the user agent in your access logs. By default, the Secure Web Gateway logs the user agent to the access logs.
There are lots of online resources available to find out the user agent. A lot of them are made publicly available. It is possible to do a Google search, and there is also a very useful website that has a list of a lot of the common user agents. This URL is http://www.useragentstring.com. Also, if you would like to find out what your current browser user agent is you can go to this website: http://whatsmyuseragent.com.
It is possible to find the user agent header for a GET request in a TCP Dump by following the TCP Stream of a request. You can do this using a packet tracing software, such as ‘Wireshark’. You would do this by finding the specific request you want, right-clicking on it, and selecting “Follow TCP Stream”. This would then bring up a window that shows the request and its headers. You can also use a filter within Wireshark to search for a specific user agent. For example, you could use a filter such as:
http.user_agent matches "Mozilla"
The Secure Web Gateway has capabilities to gather a TCP Dump so that you can open it with Wireshark to analyze. For help with gathering a TCP Dump see Performing Packet Tracing in Secure Web Gateway (SWG).
Here are some common user agents that we come across quite frequently:
Windows Media Player 10
**Windows Media Player will always use both of these user agent strings**
NOTE: Most browser user agents have “Mozilla/” at the beginning, but do not confuse this with Mozilla Firefox. This does not mean the user agent is for Firefox, it is just what all browser user agents start with.
Create a rule based on the User Agent header in Secure Web Gateway
There are several reasons for why you might want to create a rule to match a user agent. You might want to either allowlist or block a specific user agent. For example, if you didn’t want users to use the Internet Explorer 6 browser, you could block the user agent. Another reason is that some user agents do not know how to respond to proxy authentication. Some common ones we have encountered in the past that have trouble with authentication are: Silverlight,iTunes, Windows Media Player, Real Player, Adobe Updater, Adobe Flash. These user agents typically do not know how to respond to a 407 proxy authentication request and simply do not answer when the Secure Web Gateway asks for it, so they need to be bypassed.
The Secure Web Gateway does not have a specific property for the user agent so it needs to be entered manually.
- First you would select the header.request.get property from the list
- Then, you would press the “parameters” button
- Now, here is where you have to manually type in: User Agent
- Next, you have a choice to either have it match a single User Agent, or have it match in a list of user agents. The choice is yours of which you want to do, but lists are always recommended because then if you have new entries you want to add later, all you have to do is add it to the already existing list. Here is an example of a list of user agents it can match.
- The action you want to choose is completely up to you. You can either block it, stop rule set, or stop cycle. It all depends on what you are trying to accomplish with the rule.
Hopefully, this has given you a better understanding of what the user agent header is exactly, and what purpose it has. Also, hopefully, this gets you on the right path to creating a rule based on the user agent property to use within your Secure Web Gateway appliance.
Rule Set Examples
Bypass a user agent from authentication
Bypass a user agent from the SSL Scanner
Create a Global Block list of User Agents
TCP Dump Troubleshooting Examples
Here is an example of what you might see in a TCP Dump that shows when a user agent doesn't respond to proxy NTLM authentication requests, and also how to find the user agent in a GET request.
Program is not responding to NTLM authentication
Here you can see that iTunes keeps trying to go to the iTunes store, and we keep asking for authentication. However, each time, iTunes keeps making the same request and never responds with any credentials. In a normal situation, it would complete the NTLM handshake by first sending back the same request with an NTLM negotiate, which the Secure Web Gateway would then respond back with an NTLM challenge, and then iTunes would respond back with an NTLM authenticate and provide the NTLM credentials to the Secure Web Gateway.
How to find the user agent in a GET request
Here you can see I have selected a GET request in a TCP Dump, and under the "Hypertext Transfer Protocol" section below you can see the User Agent header. I did this using a program called "Wireshark".