Skip to main content
Skyhigh Security

Customer Maintained Subscribed Lists and External Lists

Secure Web Gateway has the capability to use lists that are maintained off-box from the Secure Web Gateway for use in your policy. A customer can utilize a subscribed list or an external list. One of the benefits of using a subscribed or external list is that the contents of the list can be managed on a web server or an external source by different groups within a company. Secure Web Gateway can then collect and use this list for use in the policy/rule engine. 

Common problems that we see in the field relate to the format of the files used for these lists. Failure to use the appropriate format can cause various problems such as rule engine errors and list update issues. The purpose of this article is to show how to setup these lists in the correct format and how to troubleshoot potential problems. 

Download the example files here

Subscribed Lists versus External List 

  Subscribed List External List
Key Characteristic
  • List is declared in SWG
  • List content is stored in SWG Storage
  • Behaves like any internal list, only content is not edited but periodically reloaded from external sources
  • List is not declared and only exists externally
  • Dynamic reference to the lists from the policy engine
  • Cache of used lists with a defined TTL or size limit and LRU 
Advantages
  • List content is maintained externally
  • Admin can see the list content in the UI
  • List is immediately available after restart/crash
  • No delay when accessing the list, update happens asynchronously. 
  • Best scalability to any number of lists in a cluster.

Example: In an ISP/SaaS environment tens of thousands of lists can exist externally and a node will only load those lists that are required for the current transactions on this node.

Disadvantages

Does not scale well to thousands of lists

  • Without a self-designed pre-fetch strategy, additional latency when accessing lists that are not in cache.
  • Cache is lost if SWG restarts/crashes
  • Lists cannot be viewed in the UI (could be shown via error message template in the browser thru special internal request though) 

Subscribed Lists

Subscribed lists are more static in nature and are updated on a configurable schedule. Changes to the list are maintained on the web server or location where the list is stored, not on the Secure Web Gateway.

NOTE: Only the list configuration is saved in a Secure Web Gateway backup file and not the actual contents of the list.

A common use for subscribed lists is to aid in helpdesk tasks. For example, help desk engineers can maintain allowlists or blocklists off-box instead of locally on the Secure Web Gateway. This could be useful if you do not want to give any Secure Web Gateway UI rights to a specific user. 

Setup Instructions 

STEP 1: Format the list properly for the Secure Web Gateway

Subscribed lists can be simple (TXT and XML) or complex (XML). We'll show you some examples of each below.

Simple Lists 

a) Text 

Here is the format of a Simple Text File: 

type=list type 

"data item 1" "data item 1 comment" 

"data item 2" "data item 2 comment" 

When creating your simple list, you'll need to substitute in the appropriate list type. Thebold text below indicates the list type you'll need to specify: 

  • Application Name = applcontrol 
  • Dimension = dimension 
  • String = string 
  • Category = category 
  • IP = ip 
  • IPRange = iprange 
  • MediaType = mediatype 
  • Number = number 
  • Wildcard Expression = regex 

EXAMPLES:

type=category 

com.scur.category.115 

com.scur.category.166 

NOTE: Trusted Source category codes can be downloaded here: McAfee Trusted Source Web Database Reference Guide. 

type=applcontrol 

"Google News" 

"Ebay.com" "Ray asked me to add this application" 

NOTE: For a listing of Applications, you can view the System List in the Secure Web Gateway UI:(Policy > Lists > Systems Lists > Application Name) 

type=regex 

"*.espn.com/nfl*" "HD-0002 - VP wants to check NFL scores" 

"*.ebay.com/auctions*" 

type=string 

"www.google.com" "HD-0003 - Jon needs access" 

"www.startribune.com" "HD-0004 - Tim needs to view" 

type=string 

"Hello\"Michael\" \"Michael!\"" "" 

"*empty comment\"\"\" "" 

NOTE: The escape character \ is used to allow a double quote as part of the data or comments as seen above. 

b) XML 

Here is the format of a Simple XML File:

<content type="list type"> 

<listentry> 

<entry> data item 1 </entry> 

<description> data item 1 comment</description> 

</listentry> 

<listentry> 

<entry> data item 2 </entry> 

<description> data item 2 comment</description> 

</listentry> 

</content> 

<content type="regex"> 

<listEntry> 

<entry>*.windowsupdate.com</entry> 

<description>HD Ticket: 1234</description> 

</listEntry> 

<listEntry> 

<entry>*.microsoft.com</entry> 

<description>HD Ticket: 1235</description> 

</listEntry> 

</content> 

 

Complex Lists 

Complex lists can only use the .xml file format. The first line of any complex list will begin with a line that looks like this: 

<content type="complex.nexthopproxy">

The content type, in this example nexthopproxy, relates to the type of complex list you're creating. Below is a list of content types you can use. 

NOTE: The bold text below indicates the content type syntax you will need to use. 

  • Certificate Authority = complex.ca 
  • Extended List = complex.extendedlist 
  • Element = complex.element 
  • HostAndCertificate = complex.hostandcertificate 
  • ICAP Server = complex.icapserver 
  • NextHopProxy = complex.nexthopproxy 
  • MapType = complex.maptype 

Due to the complexity of the format, it is recommended that you use the following steps to create your complex list: 

  1. Create a sample complex List: Use the Secure Web Gateway to create a complex list of the 'type' you want. In the Secure Web Gateway UI, go to: Policy> Lists > Add > give the list a name and choose a content type. Press the OK button. Lastly, add content to the list and Save changes. 
  2. Export the complex List: In the Secure Web Gateway UI, go to: Policy > Lists- Select the list and then click the Export button. Save the list to your local machine. 
  3. Modify the list: The exported list adds extra lines that we will need to remove. Edit the exported list in a text editor. 
  4. Delete all lines in the file that precede the opening <content>tag and follow the closing </content>tag. 
  5. Modify the opening <content>tag to read <content type="complex.content type">, for example <content type="complex.nexthopproxy"> 

Here is what an exported and modified next hop proxy complex list example would look like: 

<content type="complex.nexthopproxy"> 

<listEntry> 

<complexEntry defaultRights="2"> 

<configurationProperties>

<configurationProperty key="name" type="com.scur.type.string" value="Default NextHop Proxy" /> 

<configurationProperty key="host" type="com.scur.type.string" value="10.10.79.50" /> 

<configurationProperty key="port" type="com.scur.type.string" value="9090" /> 

<configurationProperty key="user" type="com.scur.type.string" value="" /> 

<configurationProperty key="password" type="com.scur.type.string" value="" /> 

<configurationProperty key="retries" type="com.scur.type.string" value="1" /> 

<configurationProperty key="waittime" type="com.scur.type.string" value="10" /> 

<configurationProperty key="persistent" type="com.scur.type.boolean" value="true" /> 

</configurationProperties> 

</complexEntry> 

<description /> 

</listEntry> 

</content> 

 

Here is a map type complex subscribed list example: 

<content type="complex.maptype"> 

<listEntry> 

<complexEntry defaultRights="2"> 

<configurationProperties> 

<configurationProperty key="key" type="com.scur.type.string" value="bob" /> 

<configurationProperty key="value" type="com.scur.type.string" value="pas$w0rD"/> 

</configurationProperties> 

</complexEntry> 

<description /> 

</listEntry>

</content> 

STEP 2: Add list configuration to the Secure Web Gateway

Two options for adding a list: 

Option 1

clipboard_e749506a66229cb9fefd51622e3a74d91.png

  1. Navigate to Policy > Lists> Add List (green circle icon) 
  2. Enter a name for the list and select list type. Note: the type selected here must match the actual list you created. Otherwise you will get an error when trying to save changes. 
  3. Select "List Content is managed remotely" 
  4. Select "Customer Maintained List" 
  5. Click "Setup" 
  6. Setup window opens: 

clipboard_e38927a1a8d418c52a54f7771e7734d85.png

Setup options: 

  • URL to download list from: HTTP, HTTPS, or FTP supported 
  • Use this "Certificate Authority Chain": select if HTTPS used and import certificate in.PEM format from local workstation 
  • Ignore certificate errors: select for HTTP or FTP, or if you wish to ignore certificate errors on HTTPS 
  • URL Authentication: can leave blank if none required to access URL for download 
  • Proxy: Only configure if there is an upstream proxy 
  • List content update: Select how often you want the Secure Web Gateway to update the list 
  • When done configuring your list, click "OK".
  • If successful, you will see your list appended with the label "Customer maintained" and list contents will be displayed. 

clipboard_e96e9ef0fc944e5380af82f18af54131c.png

 

Option 2
  1. Navigate to Policy > Rule Sets 
  2. Edit an existing rule or create a new one 
  3. For "Rule Criteria", edit or add criteria that matches your list type, for example,URL.Categories. 
  4. Click "Add List..." 
  5. Then fill out customer-maintained list according to the setup procedures shown in option 1. 

Example of using a customer-maintained subscribed list in a rule. 

clipboard_e9a6711b2af1e75611632efddf61b2244.png

Additional settings

  • Navigate to Configuration > Appliances > Appliances> your Appliance > Central Management > Advanced Subscribed Lists Settings.
  • Ensure that “Allow to download customer subscribed lists” is selected.
  • Set the following values:
    • Maximum number of allowed threads
    • Maximum number of allowed entries in list
    • Maximum list size

Troubleshooting Subscribed Lists 

The following logs and methods can be used to discover problems with your customer-maintained subscribed list.

  1. Logs showing errors, successful updates, and changes to list configuration: 
    Troubleshooting> Log files> update> customer_maintained_lists.log
  2. Logs showing if elements in the list are rejected: 
    Dashboard > Alerts. You may see an error such as "Could not parse 1 list element(s) for list YOUR-LIST-NAME of type YOUR-LIST-TYPE. Invalid elements are ignored." 
    Troubleshooting> Log files> mwg-errors> mwg-core.errors.log
  3. Logs showing which user created or deleted a customer-maintained subscribed list (search for the list name to find quickly): 
    Troubleshooting> Log files> audit> audit.log
  4. Setup Incident Alert(s) for the following IDs. 
    250 (An entry in a list is invalid and will be ignored). 
    323 (Update of customer subscribed list x succeeded on node y.) 
    324 (Update of customer subscribed list x failed on nodes y, z, ...) 
    324 (Status of customer subscribed list x on node y is up to date.) 
    326 (Download of customer subscribed list x failed on nodes y, z, ...) 

External Lists

An external list retrieves data from a data source like a web server, file on disk, LDAP, or database and then converts them into the appropriate data type by the use of properties in the Secure Web Gateway. An important benefit of external lists is that they are processed and fetched dynamically on the appliance. All retrieving and conversion of external list data is triggered when the data is first used in a rule in your policy.

Setup Instructions:

  1. Navigate to Policy > Settings > Engines > External Lists > right-click > Add.
  2. Enter name for list.
  3. Select data source type. Each type will have its own required fields. Follow instructions on the screen. 

clipboard_e7cbd4297f7091a95c0d9fcff35b967e3.png

 

Data Source Types 

Web Service 

A web service will be accessed under HTTP, HTTPS, or FTP protocol. The file format will need to be in XML, JSON, or Text format. 

a) XML 

The format of this file simply needs to follow typical XML standards. Secure Web Gateway can use a standard called 'XPath' that will allow it to navigate through elements of the XML file it is retrieving and filter for specific data you want populated in the external list. Additional information about creating an Xpath can be found at http://www.w3schools.com/xpath/

XML file example: 

<user-info> 

<user> 

<name lang="eng">tom</name> 

<level>1</level> 

</user> 

<user> 

<name lang="eng">bob</name> 

<level>5</level>

</user> 

</user-info> 

Using the xml example above, we will show you how to use xPath to retrieve certain elements of the XML file. 

- Settings under Policy> Settings> Engines> External Lists: 

XPath Expression = /user-info/user/name/text() 

This XPath expression will give the name of the user in the XML file. Essentially, this expression will search down the XML file looking for each tag you've specified in the path above and then use the 'text()' function to only pull out the text value of the <name> tag. 

XPath Expression = /user-info/user/level/text() 

This XPath expression will filter out the text value of the <level> tag for use in the external list. 

 

b) JSON 

The data source will return data as JSON object. 

- JSON examples can be found starting on page 152 in the product guide. 

- Use of parameters (placeholders) in the rules is explained starting on page 139 in theproduct guide. 

- Example using external JSON file list in rules: 

clipboard_ed17249707664df74299459bcdda64461.png

clipboard_e48c9bf603f8ffa362616e81447bb0402.png

 

c) TXT 

The format of this file need to simply be a text file. For text files a regular expression is used to locate the data for our list. 

Text file example: 

Member Group name 

------ ---------- 

group1 admin 

group1 sudoadmin 

group2 report 

group3 users 

group4 guest 

Added sudo admin group. Updated on 9/28/2017 

- The regular expression below uses a grouping operator so only the "group name"column will be retrieved for our list. 

- Settings under Policy> Settings> Engines> External Lists: 

Data Type = Plain Text 

Regular Expression = regex(^group. (.*)) 

- Example using external text file list in rules:

clipboard_e22d133e2ae3bfbf55476e1675a49ed75.png

File on disk 

File on disk refers to a file within your local file system (on the Secure Web Gateway appliance itself). This option is good for testing your external list when another external source is not yet available. Accepted list formats are the same as for web service data sources (XML, JSON, and TEXT). 

LDAP or LDAPS 

Another data source for external lists is LDAP. Once connected to an LDAP server, you can specify an LDAP Search Filter and Attribute to fetch specific data for the list. The search filter is an LDAP standard and the attribute will depend on your LDAP configuration. Below is an example that searches for the user bob and returns the 'Title' attribute for that user. 

- Settings under Policy> Settings> Engines> External Lists: 

Data Type = LDAP 

Search Filter = (sAMAccountName=bob) 

Attribute = Title 

Database 

Another data source for external lists is Database. You can choose between retrieving data from a PostgreSQL or SQLite3. Once connected to the database, you can specify a SQLquery to fetch specific data for the list. Below is an example SQL query used with External Lists. 

- Settings under Policy> Settings> Engines> External Lists: 

Data Type = Database 

SQL Query = SELECT user from userTable; 

 

Troubleshooting External Lists 

The following logs and methods can be used to discover problems with your external list.

  1. Dashboard errors.   
    - Dashboard > Alerts. 
    - Look for errors that include “External Lists Filter”, for example: 
    "No resource for data source is available (Origin: External Lists Filter, ID: 1806)"
  2. Error logs. 
    - Troubleshooting> Log files> mwg-errors> mwg-core.errors.log 
    Search for errors that include “ExternalListsFilter”, for example: 
    [ExternalListsFilterPlugin] [ExtListsError] Error in ExternalListsFilter: 'No such resource (file, URL, LDAP server, etc.)'. 
    [ExternalListsFilterPlugin] [ExtListsError] Error in ExternalListsFilter:'CFileDataSource: file don't exists! '/opt/mwg''. 
    - If extra error logs was created, for example, such as the one created with the YouTubeExternal list, it would be located here: Troubleshooting> Log files> user-defined-logs.
  3. Logs showing which user created or deleted an external list (search for ‘External Lists” to find quickly): 
    - Troubleshooting> Log files> audit> audit.log
  4. Check to make sure the Secure Web Gateway can access web service data source. For example, when checking if the Secure Web Gateway can retrieve the YouTube external list, run the following command on the CLI. If successful, you will see the output “Connected to…”. 
    curl -v https://www.googleapis.com
  5. Setup Incident Alert(s) for the following IDs. 
    - 1800 (The number of entries that can be retrieved from an external list has exceeded the configured limit.).
    - 1801 (The amount of data of entries that can be retrieved from an external list has exceeded the configured limit.)
    - 1802 (An error occurred when data was retrieved from an external list.)
    - 1803 (An error occurred when data that had been retrieved from an external list was converted.)
    - 1804 (A time-out occurred when data was retrieved from an external list.)
    - 1805 (Permission to retrieve data from an external list was denied.)
    - 1806 (A resource for retrieving external list data could not be found.)
  • Was this article helpful?