PAN Log Dual-Mode Processing
Dual-mode processing for Palo Alto Networks (PAN) logs in Skyhigh Cloud Connector provides an aging policy for the cache and prevents out of disk space issues. It also prevents Traffic events from being ignored if there is no matching entry in the URL log.
PAN logs can come as two separate log files, URL/THREAT and TRAFFIC, or as a single log file that has both URL/THREAT and TRAFFIC log lines. No events are generated from URL logs or log lines. URLs are used by traffic logs or log lines to set allow/denied field values.
To process PAN logs, use an Cloud Connector (CC) configuration that uses a custom type with Rule-Based Preprocessing:
- You can process separate URL/THREAT and TRAFFIC logs from the same CC log directory. Filter logs using a file filter such as url*.* and traffic*.*.
- To process URL/THREAT and TRAFFIC logs from different logs, your configuration must have two subconfigurations. (The dualmode_type field determines the log type.)
- For URL/THREAT logs, the dualmode_type field value is THREAT.
- For TRAFFIC logs, the dualmode_type field value is TRAFFIC.
- To process logs that have both URL/THREAT and TRAFFIC type lines in the same log, add the property dualmode.singlefile=true to the beginning of the file SystemWatcher.additionalConfig property.
When URL logs are processed, Cloud Connector keeps the cache session ID, port number, and IP address under the levedb/dualmode_urlcache directory. This directory is purged every two days, and renamed as the timestamp (for example, 1493249037012), so the history is stored for four days. The oldest two directories are removed every two days.
When Traffic logs are processed, CC tries to match the session ID, port number, and IP address from the Traffic logline to the cached data from the URL logs. It then replaces the action value.
- If the URL log has a block_url action, and the respective Traffic log has an allow action, the event is given a block_url action.
- If block_url is added to the fileSystemWatcher.denyString=block-url in the logprocessor.properties file, the event is given a denied action.
Unless the firewall policy (Traffic log) allows it, it never triggers URL filtering, so you can have allow and block actions for the same session. This would result in one blocked or denied session when parsing PAN logs.
Insufficient, incomplete, and not-applicable actions never trigger URL filtering, because the application identification has not identified the app, so it never sends the session to the URL filtering function. This results in no URL logs being generated for those sessions.
NOTE: The block-continue action is actually a web page from which you can continue to the URL. So really, it is an allow action.
If a URL/Threat log was never processed for a Traffic log, an error is logged to the CC debug log, like the following:
[ERROR] [main] c.s.l.c.FileProcessor | Detected Dual mode traffic file with out processing URL/AD information, it will be skipped from current run until URL processing is completed. File : /shn/shnlogsoriginal/pan/syslogOut_TRAFFIC-2016-12-16T10-26-08.033+01-00.log Error:Dual mode traffic log processing with timestamp: Wed Jan 11 03:26:07 UTC 2017 is attempted but URL log processed till null. This file processing deferred until URL processing is completed[Tenant: <Any>, Operation: CONTEXT, ErrorType: DUALMODE_NOT_READY, SubType: 0]:
A Traffic log is only processed once a URL/THREAT log is processed.
If Traffic log lines do not have a corresponding URL cached in leveldb (session ID, IP address, and port number), events are created with Traffic log line information without a change for action value.