Skip to main content
McAfee MVISION Cloud

Cloud Connector Config Log Processing - Sub Configuration

Previously, when you installed MVISION Cloud Connector, you had to send a sample log file to MVISION Cloud Support in order to create your Log Processor configuration, also called the Log Parser. 

In the Log Processing Sub Configuration, the MVISION Cloud Connector Log Parser Wizard allows you to avoid that step. Now when you install Cloud Connector, a default Log Processor configuration is included. For Shadow IT, the Cloud Connector Log Processor analyzes the logs from your proxy or firewall device, and extracts the relevant information required to populate the MVISION Cloud Services and Users pages. 

The Log Parser Wizard provides four options:

  1. Upload Sample Log FileThe Log Parser Wizard allows you to upload your own sample log files from your proxy or firewall device to create your Log Processor configuration, without contacting MVISION Cloud Support. During this upload, your data never leaves your network. MVISION Cloud only puts the data in a temporary S3 bucket for the duration of the browser session. Once you log out or that browser session ends, the data is automatically deleted. 
  2. Ingest Log from Syslog. You can use a log file from Syslog to create the Log Parser configuration. Select any file from the 10 most recent files. The file is truncated to the first 1000 lines to create the parser configuration.
  3. Import Existing Configuration. If you have an existing Log Parser configuration, you can use it as a template to create a new sub-configuration. Just import it to a text file, then upload that text file here.
  4. Manual Configuration. This is the existing manual configuration method. Use this method if MVISION Cloud Support has given you a Log Parser configuration. 

Upload Sample Log File

To configure the Log Parser using a sample log file, perform the following steps:

  1. Go to Infrastructure > Cloud Connector.
  2. Select the Cloud Connector instance you want to configure. 
  3. Click the Log Processing tab.
  4. Click Add new Sub-Configuration.
  5. Click Upload Sample Log File.
  6. Browse and select a sample log file from your local machine. Click Upload.
  7. Once uploaded, a preview of the default Log Parser settings displays. You will customize the Log Parser settings on the basis of your log format. For details, see Settings to Parse the Log file
    clipboard_ea41ed17e918972650660b8607ce0f06b.png
  8. Click Next.
  9. On the Evaluate Parsed Result page, verify the parsed result. To modify any column, select the Edit pencil icon. For details, see Evaluate Parsed Result.
    clipboard_e7235c718072f46ea99e2d114a7a1f6df.png
  10. On the Map Required Log fields page, review the list of attributes and make sure that all fields are mapped to corresponding fields from the log file. To add or change the mapping of attributes, see Mapping Log Fields.
    clipboard_e90d9334bf7144557dca8b612c012119a.png

NOTE: Map at least 10 required log fields to enhance mapping accuracy.

  1. On the Validate Format page, review mapped formats for accuracy. Items in red must be mapped manually. For details, see Validate Mapped Formats.
    clipboard_e6e050605fae18ee54b7d3567535c2667.png
  2. In the Sub Configuration page, review your settings and enable the configuration.

Settings to Parse the Log File

A combination of options is used to parse the uploaded sample log file. View the output in the Evaluate Parsed Result page. Select the appropriate options in this section to parse the sample log file accurately and extract the required fields to create your Log Parser configuration.

Column Character

  • No Separator. Select No Separator if you know there are no separators used in the log files.
  • Tab. Select Tab if your log files are separated by a tab.
  • Space. Select Space if your log files are separated by a space.
  • Comma. Select Comma if your log files are separated by a comma.
  • Custom Character. Select this radio button option and enter any custom character indicating a separator for the data in the log file. 
  • First Line contains Column Header. Select this checkbox if the first line of your log files contains a column header and you want to exclude this data.
  • Key-Value Pair. Select this checkbox and enter any key-value pair to be ignored from the log data files.

Text Qualifier

  • Double Quotation. Select this option to ignore characters between double quotation marks.
  • Custom Character. Select this option to ignore the texts in the custom character that you enter in the field.
  • Ignore lines starting with. Select this checkbox and enter the starting text to ignore for data processing.
  • Use '\' for escaping special characters. Select this checkbox to replace escaping special characters with '\'.
  • Trim leading and trailing whitespace. Select this checkbox to trim leading and trailing whitespace.

File Format. Select any file format from the list.

Evaluate Parsed Result

Review the Evaluate Parsed Result table to make sure all required columns are mapped to the appropriate fields. If the column values are not in the expected format, you can parse them using Regular Expressions. If you have nested fields within a column, you can parse them again using the same or different set of rules. Click the Edit pencil icon to parse the column values.

You can parse the values using the following two options:

Parse Using Regex

You can use a Regex match to eliminate and replace an unwanted element from the column entry.

For example, to eliminate the port number :443 from Column 10, perform the following steps:
clipboard_e9b2ef1f12608dbeb807f8ec96c706e8e.png

  1. Click the Edit pencil icon  .
  2. Select Parse Using RegEx.
  3. Enter the value for Regex Match. In this case, :443.
  4. Enter the Regex Replace element. In this case, '.' period.
  5. Click Test.

clipboard_e1025bdbe6a34ee0d3228dddcfd7b6f21.png

The Preview pane displays the Regex Replace results. The URL static.criteo.net:443 is now changed to static.criteo.net.

Parse Nested Entries in this Column

You can use this option to break a complex column entry into a simpler or singular entity in order to feed the required information to the Log Parser. For example, to break up the information in Column 14, when you need only the browser information and OS details, you can perform the following steps:
clipboard_ebcf029db1ebcc83d276f0a23044225f2.png

  1. Click the Edit pencil icon  .
  2. Select Parse Nested Entries in this Column.
  3. Select a delimiter for the column. In this case, Space.
    clipboard_e88e74fd9c5dda6895b6ddfa30b77c36b.png
  4. Click Test.
    clipboard_e7216213f98a4018810264492d1f34f75.png

The Preview pane displays the column entry separated into a simpler column entity. Column 1 displays browser information and Column 2 displays OS details. A column can further be broken down into simpler entities using the same steps.

Mapping Log Fields

Review the list of attributes below and make sure that all fields are mapped to corresponding fields from the log file. To add or change the attribute mapping, click the menu and select the fields as appropriate.

Map Required Log Fields

  • Date
  • Timestamp
  • Time
  • URL 
  • Destination Host
  • Destination IP
  • Destination Bytes
  • Source IP
  • Source Bytes
  • Source User

Map Optional Log Fields

  • URL Query
  • Total Bytes
  • Referral
  • MIME Type
  • Port
  • Action
  • URL Path
  • HTTP Status
  • Raw 
  • Method

Validate Mapped Formats

Review mapped formats for accuracy. Items in red must be mapped manually. 

  • Timestamp
  • Date and Time
  • Time Taken
  • Client to Service Bytes
  • Service to Client Bytes

Ingest Log From Syslog

Files read from Syslog are written to a folder or directory. Select any file from the 10 most recent files. The file is truncated to the first 1000 lines to create the parser configuration.

To configure the Log Parser using  a log from Syslog, perform the following steps:

  1. Go to Infrastructure > Cloud Connector.
  2. Select the Cloud Connector instance you want to configure. 
  3. Click the Log Processing tab.
  4. Click Add new Sub-Configuration.
  5. Click Ingest Log From Syslog.

NOTE: If Syslog is NOT enabled for the Cloud Connector instance, you will see an error message.
clipboard_edd0194f60467ba088cd4942a0773f350.png

  1. Click Configure Syslog. (For instructions, see Cloud Connector Config Syslog.)
    clipboard_eac4d4a290b5c1107af0017da7a9fd03c.png
  2. Once uploaded, a preview of the default Log Parser settings displays. You will customize the Log Parser settings on the basis of your log format. For details, see Settings to Parse the Log file.
  3. Click Next.
  4. On the Evaluate Parsed Result page, verify the parsed result. To modify any column, select the edit icon. For details, see Evaluate Parsed Result.
    clipboard_ef8d5e56de4681800822c5d465b3bbcd5.png
  5. On the Map Required Log Fields page, review the list of attributes and make sure that all fields are mapped to corresponding fields from the log file.
    To add or change the attribute mapping, see Mapping Log Fields.
    clipboard_e26995d498cf9b0d8c1e7737616f65ee7.png

NOTE: Map at least 10 required log fields to enhance mapping accuracy.

  1. On the Validate Format page, review mapped formats for accuracy. Items in red must be mapped manually. For details, see Validate Mapped Formats.
    clipboard_e56b684797c66b4e42bbb7b6ef89d3355.png
  2. In the Sub Configuration page, review your settings and enable the configuration.

Import Existing Configuration

If you have an existing Log Parser configuration, you can use it as a template to create a new sub-configuration. Just import it to a text file, then upload that text file here.

To export an existing configuration from MVISION Cloud Connector, see Export an Existing Configuration

To configure Log Parser by importing an existing configuration, perform the following steps:

  1. Go to Infrastructure > Cloud Connector.
  2. Select the Cloud Connector instance you want to configure. 
  3. Click the Log Processing tab.
  4. Click Add new Sub-Configuration.
  5. Click Import Existing Configuration.
    clipboard_e76f804fb39d33ba72c8da33fb858d757.png
  6. Click Upload.
  7. Browse and select an existing configuration file from your local machine. Click Upload.
  8. Click Next.
  9. Verify the configuration details and click Save.
  10. In the Sub Configuration page, review your settings and enable the configuration.
    clipboard_eecc618f506ca8a78dab2614ebff6eba3.png

Export an Existing Configuration

To export an existing sub-configuration in order to import it, perform the following steps:

  1. Go to Infrastructure > Cloud Connector.
  2. Select the Cloud Connector instance you want to configure. 
  3. Click the Log Processing tab.
  4. Select the required Sub-Configuration.
  5. Click Action clipboard_ea0ddc1df3a97a4d6b412b4ff0e4481e9.png > Export
  6. The Sub-configuration is downloaded to your local machine.
    clipboard_e309b5eae5751a5d847896c6ed5ee781b.png

Manual Configuration

Create a new Log Parser Configuration manually. This option is available for advanced users with complex log formats, or for users who have a configuration provided by MVISION Cloud Support

To manually configure the Log Parser, perform the following steps:

  1. Go to Infrastructure > Cloud Connector.
  2. Select the Cloud Connector instance you want to configure. 
  3. Click the Log Processing tab.
  4. Click Add new Sub-Configuration.
  5. Click Manual Configuration.
  6. Click Next.
  7. Modify the field values, and click Save.

Refer to the following table for field values and their descriptions:

Basic Details

Field  Description
Sub Configuration Tag Tag for the sub configuration
Folder to look into This is the folder location to put firewall logs for MVISION Cloud Connector to check in order to process and generate events.
Post Processing Action Select to enable the post processing option after logs are processed. 1 NOTHING: Do not perform any action after log processing. 2 MOVE : MVISION Cloud Connector Moves the processed log files to configured location. 3 DELETE : MVISION Cloud Connector Deletes the processed log files
Move Log File Location Enter the location where processed log files will be moved, when the MOVE option is selected.
File Filter Enter a file filter Regex to match and process files.
Folder Filter Enter a folder filter Regex to match and search for raw log files.
Process ZIP Files Enter True to process compressed log ZIP files. Enter False to disable.
Additional Configuration Enter any additional configuration to add a custom pre-processor rule.
Preprocessor Class Name Select the name of the preprocessor.
Deny Strings Enter a list of deny strings to search for in raw firewall logs.
Recursive File Process Choose Yes to process files recursively.

 

Advanced Settings

Field  Description
File Format Select the format for your type of firewall and proxy log files. For example, Bluecoat or McAfee.
Customized Description for Sub-config Enter the customized description for the sub-configuration to process raw log files.
Custom Sample Enter the custom sample Log to generate preprocessor rule.
Require Custom Mapping Select Yes to enable custom mapping for log files. No to disable it.
Maximum Untouched Time Enter a number of milliseconds for the time to wait before MVISION Cloud Connector starts processing the raw log file that is stored in the log location.
Prefetch File Cache Size Enter the number of the cache size (in MB) to process files before they are fetched.
Subnet Aggregation Enter the subnet mask for aggregation of IP addresses registered in the raw logs.
Automation Frequency Enter the automation frequency in seconds for log processing.
Automation Setting Enter the automation settings for log processing.
  • Was this article helpful?