Best Practices - Use URL Properties for Configuring Bypass Lists

Last updated
Save as PDF

URL properties, such as URL, URL.Host, URL.Host.BelongsToDomains, and others, can be used in the criteria of rules to configure bypass lists for web objects. These lists are also referred to as allowlists in the following.

When a web object is allowlisted, users are allowed to access it, for example, to view a web page or download a file. Allowlisting rules are inserted into appropriate rule sets within the rule set system of Web Gateway. They usually stop further rule processing with regard to the current request for accessing a web object to prevent other rules from blocking this access.

Different URL properties can be used for different kinds of allowlisting. To allow access to an individual web object, for example, to ensure users can download a particular file, the URL property is best used together with a list that contains the full URL for this file.

The following examples explain which URL properties are best used for different kinds of allowlisting and how to do it.

In addition to this, some tips and examples are given regarding the:

Values that different URLs are set to when a sample URL is processed that has been sent to Web Gateway in a request for web access
Use of the two operators is in list and matches in list in the criteria of a rule
Good and bad entries in the lists that are used with different URL properties

Allowlisting individual web objects – URL

Goal - Allow users to access individual web objects. For example, download the file Stinger.exe, which can be accessed using the URL
https://www.mcafee.com/en-us/consume...s/stinger.html
How to do it - Use the URL string property with a list of full URLs in the criteria of a rule.

The rule could, for example, be configured as follows:

URL is in list URLAllowList –> Stop Rule Set

If you add the URL https://www.mcafee.com/en-us/consumer-corporate/mcafee-labs/free-tools/stinger.html to the list URLAllowList, the file Stinger.exe is allowlisted when the rule is processed.

NOTE: In a similar way, you can block access to the file using the following rule from the default URL Filtering rule set:

URL matches in list URLBlockList –> Block

If you add the URL in question to the list URLBlockList, the file is blocked when the rule is processed.

If the matches in list operator is used instead of is in list, expressions containing wildcards can be entered into the list that is used by the property. The property can then also be used to allowlist multiple web objects.

However, if all web objects provided by a particular host should be allowlisted, this can be achieved more easily using the URL.Host property.

Allowlisting hosts – URL.Host

Goal - Allow users to access the web objects that are provided on particular hosts. For example, download the file Stinger.exe or any other file that is provided on the host https://www.mcafee.com/en-us/consume...s/stinger.html.
How to do it - Use the URL.Host string property with a list for host names in the criteria of a rule.

A rule that the URL.Host property is used in could, for example, be configured as follows:

URL.Host is in list HostAllowList –> Stop Rule Set

If you add the host download.mcafee.com to the list HostAllowList, all web objects that are provided by this host are allowlisted when the rule is processed.

If the matches in list operator is used instead of is in list, expressions containing wildcards can be entered into

the list that is used by the property. The property can then also be used to allowlist multiple hosts.

However, if all hosts within a particular domain should be allowlisted, this can be achieved more easily using the URL.Host.BelongsToDomains property.

Allowlisting domains – URL.Host.BelongsToDomains

Goal - Allow users to access the web objects that are provided within particular domains. For example, download the file Stinger.exe and any other file that is provided by the host download.mcafee.com, as well as any other downloadable file provided by any other host within the domain mcafee.com.
How to do it - Use the URL.Host:BelongsToDomains Boolean property with a list of domain names in the criteria of a rule.

The rule could, for example, be configured as follows:

URL.Host.BelongsToDomains("Domain List") equals true –> Stop Rule Set

If you add the domain mcafee.com to the list Domain List, all web objects within this domain are allowlisted when the rule is processed.

The list Domain List is configured as a parameter of the URL.Host:BelongsToDomains property, which is of the Boolean type.

When, for example, the URL https://www.mcafee.com/en-us/consumer-corporate/mcafee-labs/free-tools/stinger.html is processed, the value of the property (true or false) depends on whether the mcafee.com domain has been entered into the list Domain List or not.

The following example shows which entries in the list Domain List lead to a match when the property is used for allowlisting:

mcafee.com

dell.com

k12.ga.us

twitter.com

xxx

Then the criteria:

URL.Host.BelongsToDomains("Domain List") equals true

matches for the following URLs:

https://contentsecurity.skyhigh.cloud/

https://my.mcafee.com

http://my.support.dell.com

http://www.dekalb.k12.ga.us

http://twitter.com

http://www.twitter.com

any.site.xxx

but not for:

https://www.mymcafee.com

http://www.treasury.ga.us

http://malicioustwitter.com

Using the URL.Host.BelongsToDomains property also avoids the effort of creating more complicated solutions to achieve the same, for example:

Using two entries in a list of wildcard expressions, such as:
- twitter.com
- *twitter.com
Using a single, complex entry in a list of wildcard expressions, such as:
- regex((.*\.|.?)twitter\.com)

Property values for a sample URL

When the sample URL https://www.skyhighsecurity.com/en-us/products/secure-web-gateway.html is processed, the URL properties below are set to different values as follows.

Property	Value for sample URL
URL	https://www.skyhighsecurity.com/en-u...b-gateway.html
URL.Host	https://www.skyhighsecurity.com/
URL.Host.BelongsToDomain	true or false In the list that is configured as a parameter of this property, the following would have to be entered for the domain: https://www.skyhighsecurity.com/.
URL.FileName	web-gateway.aspx
URL.Path	/us/products/web-gateway.aspx
URL.Protocol	http

Use of operators for different types of matches

It makes an important difference whether the is in list or matches in list operator is used in the criteria of a rule.

Property

Description

is in list

Requires an exact string match.

If there are wildcard characters in a list entry, they are interpreted as literal strings.

matches in list

Allows and evaluates wildcards in list entries.

Good and bad entries in lists for URL properties

Entries in the lists that are used by the different URL properties can be good are bad, according to how they fit in with the intended use of a property. The following are examples of good and bad list entries.

URL property	Good and bad list entries
*URL with is in list* operator**	Good https://www.skyhighsecurity.com/en-us/products/secure-web-gateway.html The full URL is entered, as it is required for this property. No wildcards are specified, as these are not evaluated when the is in list operator is used. Bad https://www.skyhighsecurity.com/en-u...b-gateway.html The entry does not specify the full URL, as the protocol information, http://, is not included.
*URL with matches in list* operator**	Good https://www.skyhighsecurity.com/* This entry contains a wildcard for allowing access to any web object provided by the host www.mcafee.com, which is appropriate when the matches in list operator is used. NOTE: The entry will not match for https://www.skyhighsecurity.com/ regex(htt(p\|ps)://(.\.\|\.?)skyhighsecurity.com(\/.\|\/?)) This entry is more complex, as it uses regular expressions. When matched, it allows access, under the HTTP or HTTPS protocol, to any web object within the domain mcafee.com and its subdomains. regex(htt(p\|ps)://(.\.\|\.?)skyhighsecurity.(com\|co.us)(\/.\|\/?)) This entry is the same as the previous, but shows how other top-level domains, such as .com or .co.us, can be allowisted. Bad .skyhighsecurity.com The entry does not exclude unwanted matches, for example, a match for the URL http://malicious-download-site.cc/ma...-file.exe?url= www.skyhighsecurity.com.
*URL.Host with is in list* operator**	Good www.skyhighsecurity.com A host name is entered, which fits in with the intended use for this property. No wildcards are specified, which is appropriate when the is in list operator is used. Bad skyhighsecurity.com The entry specifies a domain name (mcafee.com), whereas the value of the property is a host name (www.mcafee.com if, for example, the URL https://www.skyhighsecurity.com/en-us/products/secure-web gateway.html is processed). No match will be produced this way. .skyhighsecurity.com The entry contains a wildcard, which is not evaluated when the is in list operator is used. .skyhighsecurity.com/us* The entry includes path information (/us), which does not fit in with the intended use of the property. In addition to this, a wildcard is specified, which is not evaluated when the is in list operator is used.
*URL.Host with matches in list* operator**	Good .skyhighsecurity.com The entry matches for on any host within the domain skyhighsecurity.com* but not for skyhighsecurity.com itself. regex((.\.\|\.?)skyhighsecurity.com)* The entry uses regular expressions to allowlist the domain skyhighsecurity.com and any of the hosts within it. Bad .skyhighsecurity.com The entry does not exclude unwanted matches, for example, http://www.skyhighsecurity.com .malicious-download-site.cc/. .www.skyhighsecurity.com/us The entry includes path information (/us), which does not fit in with the intended use of the property.
URL.HostBelongsToDomains	Good mcafee.com entered in the list Domain List, which is configured as a parameter of the property. The entry matches for the mcafee.com domain and all hosts within it, for example,www.skyhighsecurity.com. www.skyhighsecurity.com The entry does not specify a domain, but is valid. It only allowlists the host www.skyhighsecurity.com. *NOTE: This can also be achieved by adding the entry to a list for the URL.Host* property used together with the is in list operator. Bad** .skyhighsecurity.com The entry contains a wildcard, which does not fit in with the intended use of the property. The property was rather developed to avoid the effort of using wildcards in list entries. Instead it requires an exact domain match, for example, a match for skyhighsecurity.com*.