Skip to main content

Check out Interactive Visual Stories to gain hands-on experience with the SSE product features. Click here

Skyhigh Security

About Expression Based Event Filtering

In the Skyhigh Cloud Connector logprocessor.local.properties file, you can drop events based on certain criteria, if processing them with current options and preprocessor rules would end up with false positives. 

To do this, use the property for custom event filter, called customEventFilter. This property holds a comma-separated list of Boolean MVEL expressions based on which the events can be dropped. An event is dropped if at least one expression executes to true.

For example, to drop all events where the client IP address is 192.168.0.1 and the HTTP method is connected, the value of the property is:

customEventFilter=((event.clienipaddress==192.168.0.1)&&(event.httpmethod=='CONNECT'))

You can also use a comma-separated list of expressions.

How to Filter 

  1. Get a sample log line from the log. For example:
1509807030.874 63 10.100.176.166 TCP_DENIED/403 185 CONNECT tcp://ib.adnxs.com:443/ - DOMAIN\4kSPJATRbwDam -/- - - Mozilla/5.0%20(Windows%20NT%206.1;%20WOW64;%20Trident/7.0;%20MSBrowserIE;%20rv:11.0)%20like%20Gecko - --- OIP:107.154.102.111 AIP:170.74.248.22:8888 CAT:"A_PAusers_Whitelist;A_Review_Reminder_Whitelist;Technology/Internet" NSL:- OCT:- ORT:- XFF: <xff-ip redacted> PSP:0 4968 SSL-C:none SSL-V:- XVID:-
1509807030.874 63 10.100.176.167 TCP_DENIED/403 185 CONNECT tcp://ib.adnxs.com:443/ - DOMAIN\4kSPJATRbwMam -/- - - Mozilla/5.0%20(Windows%20NT%206.1;%20WOW64;%20Trident/7.0;%20MSBrowserIE;%20rv:11.0)%20like%20Gecko - --- OIP:107.154.102.112 AIP:170.74.248.21:8888 CAT:"A_PAusers_Whitelist;A_Review_Reminder_Whitelist;Technology/Internet" NSL:- OCT:- ORT:- XFF: <xff-ip redacted> PSP:0 4968 SSL-C:none SSL-V:- XVID:-
  1. In this example, the corresponding preprocessor configuration is:
{
    "dateFormat": "AUTO",
    "topRule": {
        "type": "csv",
        "on": " ",
        "trim": "true"
    },
    "fields": {
        "timestamp": {
            "type": "select",
            "index": "1"
        },
        "time_taken": {
            "type": "select",
            "index": "2"
        },
        "source_ip": {
            "type": "priority_based",
            "invalidValue": [
                "",
                "XFF"
            ],
            "parsingRules": [
                {
                    "type": "select",
                    "index": 22,
                    "replacePattern": "XFF:([^-]+)",
                    "replaceWith": "$1"
                },
                {
                    "type": "select",
                    "index": 3
                }
            ]
        },
        "action": {
            "type": "chain",
            "rules": [
                {
                    "type": "select",
                    "index": "4"
                },
                {
                    "type": "csv",
                    "on": "/"
                },
                {
                    "type": "select",
                    "index": "1"
                }
            ]
        },
        "http_status": {
            "type": "select",
            "index": "4",
            "between": {
                "start": "/",
                "end": " "
            }
        },
        "destination_bytes": {
            "type": "select",
            "index": "5"
        },
        "method": {
            "type": "select",
            "index": "6"
        },
        "url": {
            "type": "select",
            "index": "7"
        },
        "source_user": {
            "type": "select",
            "index": "9"
        },
        "mime_type": {
            "type": "select",
            "index": "11"
        },
        "user_agent": {
            "type": "select",
            "index": "13"
        },
        "destination_ip": {
            "type": "select",
            "index": "16",
            "replacePattern": "OIP:",
            "replaceWith": ""
        },
        "source_bytes": {
            "type": "select",
            "index": "24"
        },
        "protocol_addOn": {
            "type": "chain",
            "rules": [
                {
                    "type": "select",
                    "index": "17"
                },
                {
                    "type": "csv",
                    "on": ":"
                },
                {
                    "type": "select",
                    "index": "3"
                }
            ]
        }
    }
}
  1. Filter the events that have a source IP address of 192.168.0.1 and the HTTP method is Connect. 
  2. According to the preprocessor configuration above, index 3 is the source IP mapped to the source_ip parameter. Similarly, index 6 is the HTTP method and it is mapped to method.
  3. Now, look up source_ip and method in the following table. The corresponding values are event.clienipaddress and event.httpmethod

Preprocessor fields

Event attribute

action event.proxyaction
custom0

Example:

(event.customAttributes!=null && event.customAttributes.get(0)=='HTTPS')

This filters any event for which the first custom attribute value is 'HTTPS'. The index starts at 0.

destination_bytes event.serivetoclientbytes
destination_host event.destinationhostip
destination_ip event.serverIP
destination_port event.port
http_status event.httpstatus
method event.httpmethod
mime_type event.contenttype
protocol event.protocol
referral event.referralUrl
source_bytes event.clienttoservicebytes
source_ip event.clienipaddress
source_user event.username
time_taken event.timetaken
timestamp event.timestamp
url_path event.uripath
url_query event.uriquery
user_agent event.useragent

 

  1. Use the values from the table in your filter. For example:

customEventFilter=((event.clienipaddress=='192.168.0.1')&&(event.httpmethod=='CONNECT'))
customEventFilter=((event.Proxyaction=='Blocked')&& ((event.Httpmethod=='POST') || (event.Httpmethod=='GET')))
customEventFilter=((event.Username=='william.vogt@louisvilleky.gov'))
customEventFilter=((event.Httpstatus=='200'))
customEventFilter=((event.Clienipaddress=='10.120.1.165') && (event.Httpmethod=='GET') && (event.Httpstatus=='200'))   /pre>

Convert UInt64 Hexadecimal Values

Skyhigh Cloud Connector can recognize and process hexadecimal, ocatdecimal, and binary values from log files, but you must configure the Additional Configuration field in Cloud Connector Configuration to make it work.

For example, if you have upload/download bytes in a UInt64/HEX (16 digit) value, Cloud Connector can parse these values. 

In the following logline, all values for flowStatistics.bytesSent and flowStatistics.bytesReceived are in hexadecimal format. For example, 0000000000000300

"Jan 4, 2019, 11:33:59 AM",Cisco FireSight @ 10.4.215.30,"DeviceType=Estreamer DeviceAddress=10.4.215.30 
CurrentTime=1546619639904 recordType=RNA_FLOW_STATISTICS recordLength=700 timestamp=04 Jan 2019 11:33:59 
netmapDomainRef=0 detectionEngineRef=9 ipAddress=0.0.0.0 MACAddress=00:00:00:00:00:00 hasIPv6=true 
eventSecond=0 eventMicroSecond=0 eventType=FLOW_STATISTICS fileNumber=B9862F5C filePosition=2A291101 
ipV6Address=0:0:0:0:0:0:0:0 flowStatistics.initiatorIPAddress=10.6.101.13 flowStatistics.responderIPAddress=10.8.26.161 
flowStatistics.originalClientIPAddress=0:0:0:0:0:0:0:0 flowStatistics.policyRevision=0000000000000000000000005C2ED025 
flowStatistics.ruleId=268437521 flowStatistics.tunnelRuleId=0 flowStatistics.ruleAction=2 
flowStatistics.ruleReason=0 flowStatistics.initiatorPort=8 flowStatistics.responderPort=0 
flowStatistics.tcpFlags=0 flowStatistics.protocol=1 flowStatistics.netFlowIPAddress=0:0:0:0:0:0:0:0 
flowStatistics.instanceId=18 flowStatistics.connectionCounter=25559 flowStatistics.firstPacketTimestamp=1546619238 
flowStatistics.lastPacketTimestamp=1546619478 flowStatistics.packetsSent=000000000000000C 
flowStatistics.packetsReceived=000000000000000C 
flowStatistics.bytesSent=0000000000000300 flowStatistics.bytesReceived=0000000000000300 
flowStatistics.initiatorPacketsDropped=0000000000000000 flowStatistics.responderPacketsDropped=0000000000000000 
flowStatistics.initiatorBytesDropped=0000000000000000 flowStatistics.responderBytesDropped=0000000000000000 
flowStatistics.qosAppliedInterface=00000000000000000000000000000000

To allow Cloud Connector to process hex values, update the Additional Configuration field in Cloud Connector Configuration with the conversions format. This is located on the Log Processing tab, in the Sub-configuration. See Cloud Connector Config Log Processing

Example:

preprocessor.rules={"dateFormat":"dd MMM yyyy HH:mm:ss","topRule":{"type":"chain","rules":[
{"type":"csv","on":"\\t","trim":"true"}
]},"fields":{"date":{"type":"select","index":"0","between":
{"start":"timestamp","end":"\\t"}
,"trim":"true"},"source_ip":{"type":"select","index":"0","between":
{"start":"flowStatistics.initiatorIPAddress=","end":"\\t"}
,"trim":"true"},"destination_ip":{"type":"select","index":"0","between":
{"start":"flowStatistics.responderIPAddress=","end":"\\t"}
,"trim":"true"},"source_port":{"type":"select","index":"0","between":
{"start":"flowStatistics.initiatorPort=","end":"\\t"}
,"trim":"true"},"destination_port":{"type":"select","index":"0","between":
{"start":"flowStatistics.responderPort=","end":"\\t"}
,"trim":"true"},"url":{"type":"select","index":"0","between":
{"start":"flowStatistics.clientAppURL=","end":"\\t"}
,"trim":"true"},"user_agent":{"type":"select","index":"0","between":
{"start":"flowStatistics.userAgent=","end":"\\t"}
,"trim":"true"},"action":{"type":"select","index":"0","between":
{"start":"flowStatistics.ruleAction=","end":"\\t"}
,"trim":"true"},"http_status":{"type":"select","index":"0","between":
{"start":"flowStatistics.httpResponse=","end":"\\t"}
,"trim":"true"},"source_bytes":{"type":"select","index":"0","between":
{"start":"flowStatistics.bytesSent=","end":"\\t"}
,"trim":"true","base":16},"destination_bytes":{"type":"select","index":"0","between":
{"start":"flowStatistics.bytesReceived=","end":"\\t"}
,"trim":"true","base":16}}}

For hexadecimal values, use "base": 16 in the required field. For octa values, use "base": 8. For binary values, use "base": 2

This is useful with selectors such as select, between, and more. 

  • Was this article helpful?