Support Center > Search Results > SecureKnowledge Details
Log Exporter - Check Point Log Export Technical Level
Solution

Check Point "Log Exporter" is an easy and secure method for exporting Check Point logs over syslog.

Exporting can be done in few standard protocols and formats.

Log Exporter supports:

  • SIEM applications: Splunk, LogRhythm, Arcsight, RSA, QRadar, McAfee, rsyslog, ng-syslog and any other SIEM application that can run a Syslog agent.
  • Protocols: Syslog over TCP or UDP.
  • Formats: Syslog, Splunk, CEF, LEEF, LogRhythm, RSA, Json, Generic.
  • Security: Mutual authentication TLS 1.2.
  • Log Types: The ability to export security logs, audit logs, or both.
    (Note: Audit logs exist on both the Management Server and the Log Server).
  • Filter out Firewall connection logs.
  • Filtering: choose what to export based on field values.
  • Links to Log Attachments: export links to the relevant log card in SmartView and the log attachments.

The table below contains the release information for the features:

Feature / Capability Description R80.20 R80.30 R80.40 R81 R81.10
Filtering Choose what to export based on field values Jumbo Hotfix Accumulator Take 103 and higher Jumbo Hotfix Accumulator Take 107 and higher GA GA GA
Links to Log Attachments Export links to the relevant log card in SmartView and the log attachments (Forensics/Threat Emulation reports) Jumbo Hotfix Accumulator Take 127 and higher Jumbo Hotfix Accumulator Take 107 and higher GA GA GA
Attachments IDs API Export identifiers of attachments for fetching them via Log API Jumbo Hotfix Accumulator Take 183 and higher Jumbo Hotfix Accumulator Take 217 and higher Jumbo Hotfix Accumulator Take 78 and higher GA GA
DNS Name Usage Configure DNS name (FQDN) as the target-server in addition to IP address Jumbo Hotfix Accumulator Take 190 and higher Jumbo Hotfix Accumulator Take 228 and higher Jumbo Hotfix Accumulator Take 92 and higher Jumbo Hotfix Accumulator Take 13 and higher GA
Reconnection to Load Balancer Support Initiate reconnection to load balancer every X minutes (configurable) Jumbo Hotfix Accumulator Take 190 and higher Jumbo Hotfix Accumulator Take 228 and higher Jumbo Hotfix Accumulator Take 92 and higher Jumbo Hotfix Accumulator Take 13 and higher GA

Table of Contents

  • How does it Work
  • Installation
  • Uninstall
  • Basic Deployment
  • Advanced Deployment - Additional Commands
  • Advanced Configuration Post Deployment
  • Format Configuration
  • Fields Configuration
  • TLS Configuration
  • Filter Configuration
  • Log Fields Mapping for Advanced Fields Configuration
  • SIEM Specific instructions
  • Transition from LEA to Log exporter
  • Transition from CPLogToSyslog to Log exporter
  • Troubleshooting
  • Known Limitations
  • Appendix
  • Change Log
  • Related Solutions
  • Revision History

Click Here to Show Entire Article

How Does It Work

Show / Hide the section

Log Exporter is a multi-threaded daemon service, running on a log server. Each log that is written on the log server is read by the log exporter daemon, transformed into the desired format and mapping and then sent to the end target. Therefore, it is recommended to deploy Log Exporter on every server contains logs that should be exported.

On MDS/MLM, if log exporter is deployed on several domains, each domain server will have its log exporter daemon service. If exporting the logs to several targets, each target will have its log exporter daemon.

The Log Exporter is implemented as ETL procedure:

  • Extract - Reads incoming logs from the Security Gateway, stored in local files.

  • Transform - Changes the logs according to configuration files (both exported format and field name/values, irrelevant fields removal).

  • Load - Sends the logs to the configured target server in TCP / UDP (takes into consideration filter configuration if exists).

  • Data integrity - Log Exporter stops exporting when disconnected from the 3rd party and remember the last position exported.

    Once the connection is re-established, log exporter will automatically start exporting logs from the last known position.

The Log Exporter is exporting both online and offline (if any) logs in parallel. In case the 3rd party server is slow, log exporter will reduce the offline exporting rate, to prioritize the online logs over them.

Installation

Show / Hide the section

Log Exporter is integrated in R80.10 with Jumbo Hotfix Accumulator Take_270 and higher versions.

  • R77.30

    Install this release on a R77.30 Multi-Domain Server, Multi-Domain Log Server, Security Management Server, Log Server or SmartEvent Server.

    Note: Log Exporter can be installed on top of R77.30 Jumbo Hotfix Accumulator Take 292 and above.

    *This hotfix must be installed after the Jumbo, and will need to be uninstalled to upgrade to a higher Jumbo Take, and then reinstalled after the newer Jumbo is in place.

    Version Date CPUSE Online Identifier CPUSE Offline Package
    R77.30 14 April 2019 Check_Point_R77.30_Log_Exporter_T36_sk122323_FULL.tgz (TGZ)

    Install the hotfix using CPUSE. See sk92449.

Uninstall

Show / Hide the section
  • Security Management Server - Uninstall the feature using CPUSE. See sk92449.

  • Multi-Domain Security Management Server - Uninstalling the package does not remove the configuration files.

    To uninstall and completely remove all configurations, do the following:

    1. Run:

      cp_log_export delete name all [domain-server all] --apply-now

      Note: Do not forget to add "<domain-server all>" on a MDS / MLM.

    2. On a Multi-Domain Security Management Server, go to the MDS context:

      # mdsenv

    3. Make sure the environment variable $EXPORTERDIR exists, and its value is $RTDIR/log_exporter (in R77.30 - $FWDIR/log_exporter). Run:

      # rm -rf $EXPORTERDIR

    4. Uninstall the hotfix using CPUSE. See sk92449.

    5. Reboot the Server.

Basic Deployment

Common method for creating / modifying log exporters / targets.

Show / Hide the section

There are two ways to configure Log Exporter: SmartConsole (Starting from R81) and CLI.

Syntax to configure the Log Exporter using CLI commands:

cp_log_export add name <Name> [domain-server <Name or IP address of Domain Server>] target-server <HostName or IP address of Target Server> target-port <Port on Target Server> protocol {udp | tcp} format {syslog | cef | splunk | logrhythm | leef | rsa | json | generic} [Optional Arguments]

  • On MDS / MLM: The "domain-server" argument is mandatory.

    • Use 'mds' as the value for the "domain-server" argument to export audit logs from the MDS level.
    • Use 'all' as the value for the "domain-server" argument to configure the Log Exporter instance on every Domain.
  • The "target-server" argument can use either the target server IP address or its DNS name.

This will create a new target directory with the unique name specified in the "name" parameter in the $EXPORTERDIR/targets/<deployment_name> directory, and set the target configuration parameters with the connection details: IP Address, port, protocol, format, and read-mode.

The above deployment will export logs in clear text. To export logs using encryption, see the section "Advanced Deployment - Additional Commands".

The new Log Exporter does not start automatically.

To start it, run:

cp_log_export restart

To configure the Log Exporter in SmartConsole (Starting from R81), see the Logging and Monitoring Administration Guide for your version.

Advanced Deployment - Additional Commands

Advanced parameters for creating / modifying log exporters / targets

Show / Hide the section

Usage

cp_log_export <Command Name> [Command Arguments]

To see the built-in help, run:

cp_log_export <Command Name> help

Commands

Name Description
add Deploys a new Log Exporter instance.
set Updates an existing Log Exporter instance configuration.
delete Removes an existing Log Exporter instance.
show Prints the current configurations of the existing Log Exporter instances.
status Prints the overview statuses of the existing Log Exporter instances.
start Starts the Log Exporter instance.
stop Stops the Log Exporter instance.
restart Restarts the Log Exporter instance.
reexport Resets the current read position and re-exports all logs per the Log Exporter instance configuration.

Parameters

Parameter Name Description add set delete show/status/start/stop/restart reexport
name Unique name of the exporter configuration. Mandatory Mandatory Mandatory Optional - Default all Mandatory
domain-server The relevant domain-server name or IP address. Mandatory Mandatory Mandatory Optional - Default all Mandatory
target-server Exporting the logs to this IP address. Mandatory Optional N/A N/A N/A
target-port The port on which the target is listening to. Mandatory Optional N/A N/A N/A
protocol Transport protocol to use. Mandatory Optional N/A N/A N/A
format The format in which the logs will be exported. Optional Optional N/A N/A N/A
read-mode The mode in which the log files will be read and exported. Optional Optional N/A N/A N/A
enabled Allow the log_exporter to start when running the cpstart or mdsstart command. Optional Optional N/A N/A N/A
encrypted Using TLS (SSL) encryption for exporting the logs. Optional Optional N/A N/A N/A
ca-cert Full path to the CA PEM certificate file.
Relevant only when the value of 'encrypted' is 'true'.
Optional Optional N/A N/A N/A
client-cert Full path to the client P12 certificate file.
Relevant only when the value of 'encrypted' is 'true'.
Optional Optional N/A N/A N/A
client-secret The challenge phrase that was used to create the client P12 certificate.
Relevant only when the value of 'encrypted' is 'true'.
Optional Optional N/A N/A N/A
filter-action-in

Exporting all logs with a specific action.
The value must be surrounded by double quotes ("").
Multiple values are supported and must be separated by a comma.

Optional Optional N/A N/A N/A
filter-origin-in

Exporting all logs from a specific origin.
The value must be surrounded by double quotes ("").
Multiple values are supported and must be separated by a comma.

Optional Optional N/A N/A N/A
filter-blade-in Exporting all logs that belong to a specific blade.
The value must be surrounded by double quotes ("").
Multiple values are supported and must be separated by a comma.
Predefined blade families can be selected (TP, Access, Endpoint, Mobile).
Optional Optional N/A N/A N/A
--apply-now Applying any change that was done in the add immediately. Optional Optional Mandatory N/A Mandatory
export-link Add a field to the exported log that represents a link to SmartView that shows the log card. Optional Optional N/A N/A N/A
export-attachment-link Add a field to the exported log that represents a link to SmartView that shows the log card and automatically opens the attachment. Optional Optional N/A N/A N/A
export-link-ip Make the 'export-link' and the 'export-attachment-link' use a customized IP address (for example, for a Log Server behind NAT). Optional Optional N/A N/A N/A
export-attachment-ids Add a field to the exported log that represents the ID of log's attachment (if exists). Optional Optional N/A N/A N/A
reconnect-interval Schedule reconnection to the target server. Optional Optional N/A N/A N/A

Note: Using the 'filter-action-in' / 'filter-origin-in' / 'filter-blade-in' will replace any other filter configuration that was pre declared on these fields directly in the filtering XML. Other fields filters will not be overridden.

Advanced Configuration Post Deployment

Modifying Log Exporter instance configuration without using the deployment script

Show / Hide the section

After deploying a new instance of Log Exporter, all related files to that deployment can be found under $EXPORTERDIR/targets/<deployment name>

On an MDS / MLM server, the EXPORTERDIR environment variable is per domain, and its value is changed automatically when switching between domain server contexts with the "mdsenv" command.

Note: You must restart the Log Exporter instance for the new settings to take effect.

Target Configuration XML

Log Exporter target configuration is saved in a file located in each deployment folder: $EXPORTERDIR/targets/<deployment_name>/targetConfiguration.xml

Listed below are some configuration options:

Parameter Description Possible/Default Values
<version></version> Current Log Exporter version - used for upgrades.  
<is_enabled></is_enabled> Determines whether the process will be monitored by the watch dog.
  • true
  • false

Destination Parameters

Parameter Description Possible/Default Values
type Reserved for future use  
<ip></ip> The IP address or DNS name of the target server that will receive the logs Any IPv4 address or DNS name
<port></port> The port on which the target is listening to Any valid port number
<protocol></protocol> The protocol that will be used in the connection. UDP / TCP
<reconnect_interval></reconnect_interval> Determines whenever to reinitiate the connection to the target server Number of minutes

Security Parameters

Discussed in more detail in the "TLS Configuration" section.

Parameter Description Possible/Default Values
<security></security> Determines whether the connection data will be sent in clear text or encrypted.
  • clear (default)
  • tls
<pem_ca_file></pem_ca_file> The location of the root CA PEM file.  
<p12_certificate_file></p12_certificate_file> The location of the client key pair in P12 format.  
<client_certificate_challenge_phrase></client_certificate_challenge_phrase> The challenge phrase that was used to create the P12 certificate.
The value will be hashed after restarting the process.
 

Source Parameters

Parameter Description Possible/Default Values
<folder></folder> The path where the log files are located Default location is $FWDIR/log/
<log_files></log_files> Determines which log files will be exported or how far back to read logs from  the $FWDIR/log/fw.log file
  • read logs from (<number> - (default=1)) days back (recommended)
  • <specific file name>
  • on-line
  • no value will use 'on-line'
<log_types></log_types> Determines which log will be exported based on their type
  • all (default)
  • log
  • audit
<read_mode></read_mode> Determines whether to export complete logs or delta only.
  • semi-unified (default)
  • raw

Resolver Parameters

Parameter Description Possible/Default Values
<mappingConfiguration></mappingConfiguration> The XML file containing the log field mapping scheme.
If left empty will use the default settings.
Default values are based on the 'format'.
<exportAllFields>true</exportAllFields>

When this field is set to 'true' all log fields will be sent regardless of whether they appear in the mapping scheme, except for specifically black-listed fields in the relevant log format mapping file (<exported>false</exported>).

When set to 'false', only those fields which appear in the relevant log format mapping file will be sent (with exported flag set to 'true': <exported>true</exported>)

  • true
  • false

Format Parameters

Parameter Description Possible/Default Values
<formatHeaderFile></formatHeaderFile> The XML file contains the log header format scheme.
If left empty will use the default settings.
Default values are based on the 'format'.

General Filter Configuration Path

Parameter Description Possible/Default Values
<dynamicFilter></dynamicFilter> The XML file containing the filtering configuration.
If left empty, default configuration will be used.
The default path is: conf/FilterConfiguration.xml

SmartView links parameters

Parameter Description Possible/Default Values
export_log_link Add a field to the exported log that represents a link to SmartView that shows the log card. True/False [default]
export_attachment_link Add a field to the exported log that represents a link to SmartView that shows the log card and automatically opens the attachment. True/False [default]
export_link_ip Make the 'export_log_link' and the 'export_attachment_link' use a customized IP address (for example, for a Log Server behind NAT).
  • empty (default)
  • IPv4 Address

Filter out firewall connections Parameters

This configuration allows Log Exporter instance to filter out Firewall traffic logs for several blades ('Firewall-1 & VPN-1', 'HTTPS Inspection' and 'Security Gateway/Management').

Note: Firewall session logs will still be exported (generated by tracking a Firewall rule per session).

Parameter Description Possible/Default Values
<filter filter_out_by_connection="false">

Determines if the access logs should be filtered out.

When set to 'true' VPN-1 & Firewall-1 connection logs will be filtered out

Note: No other blade filters are currently supported. This will be expanded upon in future releases.

  • true
  • false

Limitation: HTTPS Inspection logs, Firewall logs generated not from rules, and a few Firewall NAT update logs will still be exported.

Format Configuration

Show / Hide the section

Every format has its own predefined format configuration file that defines the format of the exported logs, what will be the delimiters, what fields will be part of the header, and so on.

These files are located under each deployment folder: $EXPORTERDIR/targets/<deployment_name>/conf/*FormatDefinition.xml

Note: Do not edit the original *FormatDefinition.xml files - doing that will cause data loss after upgrade. Instead, copy the file to a different file name and modify the copied file while leaving the original intact. After done modifying the file, refer to it (full path) in <formatHeaderFile> element in the relevant targetConfiguration.xml file.

Body

Parameter Description Syslog Splunk RSA CEF LEEF LogRhythm Generic
<start_message_body></start_message_body> The character preceding the log data payload. [            
<end_message_body></end_message_body> The character following the log data payload. ]            
<message_separator></message_separator> The delimiter that separates logs. &#10; (&#10;=='\n') &#10; ('\n') &#10; ('\n') &#10; ('\n') &#10; ('\n') &#10; ('\n') ('\n')
<fields_separatator></fields_separatator> The delimiter that separates log fields. '; '
(semicolon + space)
|
(pipe)
' '
(space)
' '
(space)
&#09
(<TAB>)
|
(pipe)
' '
(space)
<field_value_separatator></field_value_separatator> The assignment operator. : = = = = = =
<value_encapsulation_start>&quot;</value_encapsulation_start> The value encapsulation operator (start) "       "    "
<value_encapsulation_end>&quot;</value_encapsulation_end> The value encapsulation operator (end) "       "   "
<escape_chars>
  <char>
    <orig></orig>
    <escaped></escaped>
  </char>
</escape_chars>

Escaping unwanted characters.

The escape functionality will replace the string that's encapsulated by the 'orig' tags with the one encapsulated by the 'escaped' tags

;\ --> \\
" --> \"
&#10; --> ' '
] --> \]
| --> ;
= --> \=
&#10; --> ' '
= --> \=

&#10; --> ' '
;\ --> \\
= --> \=
&#10; --> ' '
| --> \|
= --> \=
&#10; --> ' '
| --> ;
= --> \=
&#10; --> ' '
\ --> \\
" --> '
&#10; --> ' '

Header

Parameter Description Default values for Syslog Default values for Splunk Default values for RSA Default values for CEF Default values for LogRhythm
<header_format></header_format> The delimiter between the header values and the number of values. Every {} will be replaced with one value. ' ' (space) time={}|hostname={}| <134> | ; LOGV2 {}|

Notes:

  • If you want to add constant string to the header, you can do this by adding the string to the <header_format> tag value.
  • If you want to add a new field to the header, you need to add a new header format replacement string (for example: {}) to the <header_format> and also add the relevant information under the <headers> tag.

Fields Configuration

Show / Hide the section

Every format has its own predefined fields configuration file that allow to change the name / value of the exported field, filter out irrelevant fields, and so on.

These files are located under each deployment folder: $EXPORTERDIR/targets/<deployment_name>/conf/*FieldsMapping.xml

Note: Do not edit the original *FieldsMapping.xml files - doing that will cause data loss after upgrade. Copy the file to a different file name and modify the copied file while leaving the original intact. After done modifying the file, refer to it (full path) in the <mappingConfiguration> element in the relevant targetConfiguration.xml file.

Parameter Description Values
<table> Some fields will appear in tables depending on the log format. This information can be found in the ELG log - one entry for every new field. A field can appear in multiple tables, each distinct instance is considered as a new field.  
<exported></exported>

Optional. You can filter out specific fields by using the 'exported' tag (value: true, or false) in the mapping configuration file. Alternatively, if the 'exportAllFields' tag in the 'targetConfiguration.xml' file is set to 'false', only those fields which are listed in the mapping file will be exported.

Note - This parameter can also be configured on a table field level to allow / prevent its export when the field is part of a table.

  • true
  • false
<origName></origName>

The name of the field that will be mapped to <dstName>.

Refer to Log Fields Mapping for Advanced Fields Configuration.

 
<dstName></dstName> The new mapping scheme name for the desired field.  
<required></required> Optional. When set to 'true', only logs which contain this field will be exported.
  • true
  • false

TLS Configuration

Show / Hide the section

How to set a secured connection between the log exporter and the syslog server

Log Exporter can export logs over an encrypted connection using TLS protocol.

When using TLS, it is important to know that only mutual authentication is allowed.

For mutual authentication, the Log Exporter needs the following certificates:

  • CA certificate that signed both client (Log Exporter side) and server (syslog server side) certificates. Required format: PEM.
  • Client (Log Exporter side) certificate. Required format: P12.

Notes:

  • CA server needs to be routable from Log Server to establish the connection.
  • In addition to these two certificates, a third certificate should be installed on the syslog server side based on the server requirement. It is also possible to use self-signed certificates.

The following procedure uses openssl commands that have to be run on non-Check Point server.

Creating a CA Certificate

  1. Create CA key:

    openssl genrsa -out ca.key 2048

  2. Create CA Cert:

    openssl req -x509 -new -nodes -key ca.key -days 2048 -out ca.pem

    You will be prompted to provide information regarding the certificate. Apart from the Common Name (it is recommended to use the device IP address as the Common Name), all other fields are optional and can be skipped. If you are purchasing an SSL certificate from a certificate authority, it is often required that these additional fields.

Creating a Log Exporter Certificate

  1. Create Log Exporter key:

    openssl genrsa -out cp_client.key 2048

  2. Create Log Exporter CSR file:

    openssl req -new -key cp_client.key -out cp_client.csr

  3. Create Log Exporter CRT file:

    openssl x509 -req -in cp_client.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out cp_client.crt -days 2048 -sha256

  4. Create Log Exporter P12 file:

    openssl pkcs12 -inkey cp_client.key -in cp_client.crt -export -out cp_client.p12

Creating a Syslog Server Certificate

    1. Create Target Server key:

      openssl genrsa -out server.key 2048

    2. Create Target Server CSR file:

      openssl req -new -key server.key -out server.csr

    3. Create a Target Server CRT file:

      openssl x509 -req -in server.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out server.crt -days 2048 -sha256

Note: Several SIEM applications require the Server certification to be in a specific format. For more information, refer to the section "SIEM Specific instructions".

Filter Configuration

Choose what to export based on field values.

Show / Hide the section

The Log Exporter can filter logs based on field values.

Because fields mapping operation is done before filtering, make sure to use the dst name / value of the fields if configured specifically in the *FieldsMapping.xml file (based on the format). Otherwise, use the name / value as written in the raw log file.

It is possible to configure what to export or what not to export.

The filter configuration file is located under each deployment folder: $EXPORTERDIR/targets/<deployment-name>/conf/FilterConfiguration.xml

Parameter Description Possible/Default Values
<filterGroup operator=""></filterGroup>

A group of fields that will determine what to export.

The relation between the fields is determined by the operator value.

Refer to Log Fields Mapping for Advanced Fields Configuration.

operator[and | or]

<field name="" operator="">
  <value operation=""></value>
</field>

Declare a single field filter that will participate in the filter group.

  • name - The name of the field to filter on.
  • operator - Declares the operator (and / or) between the various declared operations.
  • operation - Declares the matching logic regarding the declared value.
  • value - The specific value to filter on. Multiple values for a single operation is supported and should be added as a separate row.

operator[and | or]

operation[eq - equal | neq - not equal | gt - greater than | lt - less than ]

There are two ways to configure the filtering feature:

  1. Using the cp_log_export command.

    This command allows you to configure filtering for action / blade / origin fields only.

    The syntax is:

    cp_log_export set name <name> filter-action-in "value1,value2"

    cp_log_export set name <name> filter-origin-in "value1,value2"

    cp_log_export set name <name> filter-blade-in "value2"

    In addition, it is possible to use predefined families for "filter-blade-in" value:

    • TP - For exporting Threat Prevention logs only (Anti-Bot, Anti-Malware, Threat Emulation, IPS, IPS-1, SmartDefense, Anti-Virus, New Anti Virus, Anti-Spam and Email Security, Threat Extraction, MTA).
    • Access - For exporting Access logs only (Security Gateway/Management, VPN-1 & FireWall-1, Firewall, Application Control, URL Filtering, Content Awareness, Connectra, Mobile Access, Compliance blade, Core, DDoS Protector, Identity Awareness, Identity Logging, UA WebAccess).
    • Mobile - For exporting Mobile logs only (WIFI Network, Mobile App, OS Exploits, Device, Network Security, Cellular Network, Network Access, iOS Profiles, Text Message, On-device Network Protection).
    • EndPoint - For exporting Endpoint logs only (Anti-Bot, Anti Malware, Threat Emulation,  IPS, IPS-1, SmartDefense, Anti-Virus, New Anti Virus, Anti-Spam and Email Security, Threat Extraction, MTA ).
  2. Modifying the FilterConfiguration.xml file manually.

    It is allowed to add new fields to this file - for example:

    <filters>
      <filterGroup operator="and">
        <field name="action" operator="and">
        </field>
        <field name="origin" operator="and">
        </field>
        <field name="product" operator="or">
        </field>
        <field name="severity" operator="or">
          <value operation="eq">3</value>
            <value operation="eq">4</value>
        </field>
      </filterGroup>
    </filters>

Limitations:

  • The relation between the values of the same operation is only "OR".

    Example:

    cp_log_export set name <target-name> filter-action-in "accept,drop"

    Only logs with action = "accept" OR action = "drop" are exported.

  • Filtering is not supported for any of the following fields:

    time, category, UUID,appi_name ,app_desc, app_category, matched_category, app_properties, app_rule_name, app_risk, HTTPS_inspection_rule_name, cvpn_resource, cvpn_category, name, desc, properties.

  • Filtering on a certain field with the condition: "not equal(value1) OR not equal(value2)" is not supported. When editing the filtering XML, make sure to have a maximum of one line of "neq" operation in each field.

Log Fields Mapping for Advanced Fields Configuration

Show / Hide the section

Check Point Infinity solution includes multiple log fields, representing the diversity of Check Point's products.
The log fields mapping will help you understand security threats, logs language, complex queries and SIEM.

For information on Check Point's Log Fields Mapping, refer to: sk144192.

SIEM Specific instructions

How to configure SIEM applications to receive logs optimally.

Show / Hide the section

Rsyslog

Rsyslog is not configured to use RFC5424 timestamp format by default. Therefore, you should manually change Rsyslog setting for it to be compliant with Log Exporter output format.

On the Syslog server:

  1. Edit the /etc/rsyslog.conf file.

  2. Comment out this line, if it is not commented out already:

    #"$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat"

  3. Add this line in the file:

    $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format

  4. Save the change in the file and close it.

  5. Restart the Rsyslog:

    service rsyslog restart

ArcSight

ArcSight recommends naming the server certificate file as 'syslog-ng'.

  1. Convert the key to the P12 format:

    openssl pkcs12 -inkey syslogServer.key -in syslogServer.crt -export -out syslog-ng.p12 -name "syslogng-alias" -password pass:changeit

  2. Make sure the value of the environment variable ARCSIGHT_HOME is path to the connector installation directory.

  3. Run the certificate manager on the Linux KDE console:

    $ARCSIGHT_HOME/current/bin/arcsight agent keytoolgui

  4. From the File menu, open the keystore:

    $ARCSIGHT_HOME/current/jre/lib/security/cacerts

    The password is: changeit
  5. From the menu, select Import Trusted Certificate

  6. From the file dialog, select ca.pem and save it.

  7. Save and close the certificate manager.

  8. Edit the agent.properties file to enable the mutual authentication:

    vi $ARCSIGHT_HOME//current/user/agent/agent.properties

  9. Configure the value 'true' for the 'syslogng.mutual.auth.enabled' parameter:

    syslogng.mutual.auth.enabled=true

  10. Add these lines at the bottom:

    syslogng.tls.keystore.file=user/agent/syslog-ng.p12

    syslogng.tls.keystore.alias=syslogng-alias

  11. Save the change in the file and close it.

  12. Restart the service:

    /etc/init.d/arc_connector_name restart

Splunk

We recommend to use Check Point App for Splunk when exporting logs to a Splunk server.

For more information about installation and deployment, see the Check Point App for Splunk User Guide.

In addition, to configure an encrypted connection, do the following:

  1. Generate the server PEM file according to Splunk TLS Documentation:

    cat syslogServer.crt syslogServer.key RootCA.pem > splunk.pem

  2. Edit the 'inputs.conf' file on the Splunk server:

    vi /opt/splunk/etc/apps/<Name of the app, where the configuration is saved>/local/inputs.conf

  3. Update the file to use TLS:

    [SSL]
    serverCert = <full path to server PEM file>
    sslPassword = <challenge password>
    requireClientCert = true
    [tcp-ssl://<port>]
    index = <index>
    
  4. Save the change in the file and close it.

  5. Edit the 'server.conf' file on the Splunk server:

    vi /opt/splunk/etc/system/local/server.conf

  6. Update the file to use the relevant CA PEM file:

    [sslConfig]
    sslRootCAPath = <full path to CA PEM file>
    
    [SSL]
    cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH
    
  7. Save the change in the file and close it.

  8. Restart the Splunk daemon:

    /opt/splunk/bin/splunk restart

QRadar

  1. In the Authentication Mode field, choose TLS And Client Authentication.

  2. Upload the Check Point certificate and private key to QRadar.

  3. Provide the absolute path to those under the Provide Certificate option.

Notes:

  • When using Client Authentication, you must provide the absolute path to the client certificate.
  • Make sure the "Common Name" is unique in every certificate.

Transition from LEA to Log exporter

Recommended method to move from the existing LEA connector to the new log exporter.

Show / Hide the section
  1. Delete the OPSEC application object from the GUI, if you only use it for the OPSEC application. Alternatively, remove the LEA client entity from it, if you do not use it:

  2. If this is the only OPSEC LEA client, configure the $FWDIR/conf/fwopsec.conf file to not allow LEA (if this is not the only OPSEC LEA client, skip this step):

    Change these lines:

    From:

    To:

    #
    lea_server auth_port 18184
    lea_server port 0
    #
    
    #
    #lea_server auth_port 18184
    #lea_server port 0
    #
    
  3. Install the Log Exporter according to the installation guide above.

Transition from CPLogToSyslog to Log exporter

Recommended method to move from the existing CPLogToSyslog to the new log exporter.

Show / Hide the section
  1. Uninstall the CPLogToSyslog package using CPUSE. See the instructions in sk92449 - section 4-C.
  2. Install the Log Exporter according to the installation guide above.

Troubleshooting

Show / Hide the section
Symptoms Cause Suggested solution
Logs are not exported after adding a filter to the FilterConfiguration.xml file, or by using the cp_log_export command. cp_log_export adds the default values to the FilterConfiguration.xml file, while the field names should be the same as the exported name. It causes the filter mechanism to not match any log.
  1. In the relevant XXXFieldsMapping.xml file, look for the relevant mapped field.

  2. Find the element named <dstName> and copy it.

  3. Edit the <exporter-dir>/conf/FilterConfiguration.xml file.

  4. Replace the field name to the previously copied one.

Assigning a field in the FieldsMapping file as export false, but keep exporting the field.

For example: When the user assign for the field "layer_uuid export=false", but he keeps seeing this field as part of the log in the Log Server.

The field is part of a table in the log, and the standard configuration to filter out a field is not effective on a table field.

To prevent these fields from been exported, you need to:

  1. Go to the $EXPORTERDIR/targets/<exporter_name>/conf/ directory.

  2. Edit the "Fields Mapping" file you use (that corresponds to the format you export).

  3. Look for 'match_table' tag:

    <tableName>match_table</tableName>
    
  4. Add the required lines:

    • If the 'match_table' tag does not exist, add these lines inside the 'fields' tag:

      
      <table>
        <tableName>match_table</tableName>
         <fields>
          <field><origName>field_name</origName><exported>false</exported></field>
         </fields>
      </table>
      
    • If the 'match_table' tag exists, add this line inside the 'fields' tag:

      <table>
        <tableName>match_table</tableName>
         <fields>
          <field><origName>field_name</origName><exported>false</exported></field>
         </fields>
      </table>
      

    The file should look like this:

    <fields>
    <!-- Filter out fields -->
      <field><origName>field_name1</origName><exported>false</exported></field>
      <field><origName>field_name2</origName><exported>false</exported></field>
      <field><origName>field_name3</origName><exported>false</exported></field>
        ... ...
        <table><tableName>match_table</tableName>
          <fields>
            <field> ... </field>
            ... ...
            <field><origName>field_name</origName><exported>false</exported></field>
          </fields>
        </table>
    
    <!-- End of filter out -->
    </fields>
    
  5. Save the change in the file and close it.

  6. Restart the Log Exporter to load the new settings:

    cp_log_export restart name <exporter_name>

Known Limitations

Show / Hide the section
  • If you want to change this environment, you must first consult your Check Point partner or vendor. This release is built for the specific environment. Upgrades or other changes can overwrite Hotfix functionality and environment customizations.
  • When exporting logs from a certain domain using UDP protocol, the exported log's header will contain the MDS IP and not the CMA/CLM's IP address.

Appendix

Show / Hide the section

Special log fields

Field Description
loguid

Some checkpoint logs are updated over time. Update logs will have the same loguid value. Check Point SmartLog client will correlate those updates into a single unified log.

When sending update logs to 3rd party (SIEM) servers, in the default raw read-mode, they will arrive as distinct logs. Best use the semi-unified read-mode, which will send few instances of the log, but each instance will contain the entire event chain until this update. Admins can alternatively use the loguid field to correlate update logs and get the full event chain themselves.

Note: All related log & log-updates will share the same initial time as the 1st log (in semi-unified mode).

Example of update logs includes the total number of bytes sent and received over time or the severity field which will be updated over time as more information becomes available.

hll_key

Stands for High-Level Log key. This concept was introduced in R80.10, where Multiple connection logs can comprise one session with one shared hll_key.

For example, when browsing a webpage, you might have multiple connection logs which are related to the same session. Connection logs which are part of the same session will share the same hll_key value.

Syslog-NG Listener configuration

When configuring a source on a Syslog NG server it is recommended to use the syslog-protocol flag.

For example:

source s_network { network(transport("tcp") port(514) flags(syslog-protocol) ); };

Splunk Listener configuration

It is recommended to add these time settings to your source type:

TIME_FORMAT = %s
TIME_PREFIX = time=
MAX_TIMESTAMP_LOOKAHEAD = 15

ArcSight Listener configuration

The Log Exporter solution does not work with the OPSEC LEA connector. Instead, you must install the ArcSight Syslog-NG connector.

ArcSight Common Event Format (CEF) Mapping

CEF is an extensible, text-based format designed to support multiple device types by offering the most relevant information. Message syntaxes are reduced to work with ESM normalization. Specifically, CEF defines a syntax for log records comprised of a standard header and a variable extension, formatted as key-value pairs. The CEF format can be used with on-premise devices by implementing the ArcSight Syslog SmartConnector. CEF can also be used by cloud-based service providers by implementing the SmartConnector for ArcSight Common Event Format REST.

CEF Header format

Version Device Vendor Device Product Device Version Device Event Class ID Name Severity
Default CEF:0 Check Point Log Update Check Point Log Log 0
Values - - Product Name (Blade) -

Attack Name

Protection Type

Verdict

Matched Category

DLP Data Type

Application Category

Application Properties

Protection Name

Application Name

Message Info

Service ID

Service

Application Risk

Risk

Severity

QRadar Log Event Extended Format (LEEF) Mapping

The Log Event Extended Format (LEEF) is a customized event format for IBM Security QRadar.

LEEF Header format

  LEEF:Version Vendor Product Version EventID
Default LEEF:2.0 Check Point Log Update 1.0 Check Point Log
Values - - Product Name (Blade) -

Protection Name

Application Name

Action

Note: The time format is not compliant with the official LEEF format.

Until such a time as IBM will add support for epoch time format Log Exporter with LEEF format is only partially supported.

Change Log

Show / Hide the section

Note: Takes listed in the table below are related to the Log Exporter, and not to Jumbo Hotfix Accumulators.

Issue ID Description
Take 51 (14 April 2019)
SL-1822

Installation of R80_10_JHF_LOGOUT fails because of the script updateExistingExporters.sh when there are no exporters in the server.

Take 50 (05 March 2019)
SL-2003

Added filtering support. Now you can decide which logs to export.

Take 43 (20 January 2019)
PMTR-13842

Added support for exporting logs to the new Check Point's Splunk application.

SL-1817

Log Exporter was getting stuck after 7 hours of uptime

SL-1932

Log Exporter could not be installed on top of R80.10 Jumbo Hotfix Accumulator Take_169 and above.

Revision History

Show / Hide the section
Date Description
Aug 15, 2021 Updated that configuration in SmartConsole is available starting from R81.
July 28, 2021 Added RSA, and Json formats.
July 13, 2021 Improved the article formatting
Mar 17, 2021 SK was updated with all existing functionalities and better descriptions
Oct 13, 2020 Updated Known Limitation section
Aug 19, 2020 Updated the Installation section
Jul 06, 2020 Edited R80.30 installation section
Jun 09, 2020 Changed note to: Audit logs exist on every Log Server
Apr 13, 2020 Updated Limitation for Filtering Configuration section
Feb 19, 2020 Updated to Check_Point_R80.20_JHF_T118_Log_Exporter_Enhancements_T5_sk122323_FULL.tgz which fixed an issue with ds.conf when installing on top of R80.20 Jumbo Hotfix Accumulator Take_118.
Jan 19, 2020 Added scenario to Troubleshooting section
Jan 1, 2020 Added LogRhythm format for R80.20 on top of JHF T118 and R80.30 on top of JHF T111.
Dec 12, 2019 Added Troubleshooting section.
Oct 2, 2019 Added SmartView links For R80.20 on top of Jumbo Hotfix Accumulator Take_103
July 8, 2019

Added "Log Fields Mapping for Advanced Fields Configuration" section

Give us Feedback
Please rate this document
[1=Worst,5=Best]
Comment