Log Exporter is a multi-threaded daemon service, running on a log server. Each log that is written on the log server is read by the log exporter daemon, transformed into the desired format and mapping and then sent to the end target. Therefore, it is recommended to deploy Log Exporter on every server contains logs that should be exported.

On MDS/MLM, if log exporter is deployed on several domains, each domain server will have its log exporter daemon service. If exporting the logs to several targets, each target will have its log exporter daemon.

The Log Exporter is implemented as ETL procedure:

• Extract - Reads incoming logs from the Security Gateway, stored in local files.

• Transform - Changes the logs according to configuration files (both exported format and field name/values, irrelevant fields removal).

• Load - Sends the logs to the configured target server in TCP / UDP (takes into consideration filter configuration if exists).

• Data integrity - Log Exporter stops exporting when disconnected from the 3rd party and remember the last position exported.

  Once the connection is re-established, log exporter will automatically start exporting logs from the last known position.

The Log Exporter is exporting both online and offline (if any) logs in parallel. In case the 3rd party server is slow, log exporter will reduce the offline exporting rate, to prioritize the online logs over them.

Log Exporter is integrated in R80.10 with Jumbo Hotfix Accumulator Take_270 and higher versions.

• R77.30

  Install this release on a R77.30 Multi-Domain Server, Multi-Domain Log Server, Security Management Server, Log Server or SmartEvent Server.

  Note: Log Exporter can be installed on top of R77.30 Jumbo Hotfix Accumulator Take 292 and above.

  *This hotfix must be installed after the Jumbo, and will need to be uninstalled to upgrade to a higher Jumbo Take, and then reinstalled after the newer Jumbo is in place.

  Version	Date	CPUSE Online Identifier	CPUSE Offline Package
  R77.30	14 April 2019	Check_Point_R77.30_Log_Exporter_T36_sk122323_FULL.tgz	(TGZ)

  Install the hotfix using CPUSE. See sk92449.

• Security Management Server - Uninstall the feature using CPUSE. See sk92449.

• Multi-Domain Security Management Server - Uninstalling the package does not remove the configuration files.

  To uninstall and completely remove all configurations, do the following:

  1. Run:

     cp_log_export delete name all [domain-server all] --apply-now

     Note: Do not forget to add "<domain-server all>" on a MDS / MLM.

  2. On a Multi-Domain Security Management Server, go to the MDS context:

     # mdsenv

  3. Make sure the environment variable $EXPORTERDIR exists, and its value is $RTDIR/log_exporter (in R77.30 - $FWDIR/log_exporter). Run:

     # rm -rf $EXPORTERDIR

  4. Uninstall the hotfix using CPUSE. See sk92449.

  5. Reboot the Server.

There are two ways to configure Log Exporter: SmartConsole and CLI.

Syntax to configure the Log Exporter using CLI commands:

cp_log_export add name <Name> [domain-server <Name or IP address of Domain Server>] target-server <HostName or IP address of Target Server> target-port <Port on Target Server> protocol {udp | tcp} format {syslog | cef | splunk | logrhythm | generic} [Optional Arguments]

• On MDS / MLM: The "domain-server" argument is mandatory.

  • Use 'mds' as the value for the "domain-server" argument to export audit logs from the MDS level.
  • Use 'all' as the value for the "domain-server" argument to configure the Log Exporter instance on every Domain.

• The "target-server" argument can use either the target server IP address or its DNS name.

This will create a new target directory with the unique name specified in the "name" parameter in the $EXPORTERDIR/targets/<deployment_name> directory, and set the target configuration parameters with the connection details: IP Address, port, protocol, format, and read-mode.

The above deployment will export logs in clear text. To export logs using encryption, see the section "Advanced Deployment - Additional Commands".

The new Log Exporter does not start automatically.

To start it, run:

cp_log_export restart

To configure the Log Exporter in SmartConsole, see the Logging and Monitoring Administration Guide for your version.

Usage

cp_log_export <Command Name> [Command Arguments]

To see the built-in help, run:

cp_log_export <Command Name> help

Commands

Parameters

Note: Using the 'filter-action-in' / 'filter-origin-in' / 'filter-blade-in' will replace any other filter configuration that was pre declared on these fields directly in the filtering XML. Other fields filters will not be overridden.

After deploying a new instance of Log Exporter, all related files to that deployment can be found under $EXPORTERDIR/targets/<deployment name>

On an MDS / MLM server, the EXPORTERDIR environment variable is per domain, and its value is changed automatically when switching between domain server contexts with the "mdsenv" command.

Note: You must restart the Log Exporter instance for the new settings to take effect.

Target Configuration XML

Log Exporter target configuration is saved in a file located in each deployment folder: $EXPORTERDIR/targets/<deployment_name>/targetConfiguration.xml

Listed below are some configuration options:

Destination Parameters

Security Parameters

Discussed in more detail in the "TLS Configuration" section.

Source Parameters

Resolver Parameters

Format Parameters

General Filter Configuration Path

SmartView links parameters

Filter out firewall connections Parameters

This configuration allows Log Exporter instance to filter out Firewall traffic logs for several blades ('Firewall-1 & VPN-1', 'HTTPS Inspection' and 'Security Gateway/Management').

Note: Firewall session logs will still be exported (generated by tracking a Firewall rule per session).

Limitation: HTTPS Inspection logs, Firewall logs generated not from rules, and a few Firewall NAT update logs will still be exported.

Every format has its own predefined format configuration file that defines the format of the exported logs, what will be the delimiters, what fields will be part of the header, and so on.

These files are located under each deployment folder: $EXPORTERDIR/targets/<deployment_name>/conf/*FormatDefinition.xml

Note: Do not edit the original *FormatDefinition.xml files - doing that will cause data loss after upgrade. Instead, copy the file to a different file name and modify the copied file while leaving the original intact. After done modifying the file, refer to it (full path) in <formatHeaderFile> element in the relevant targetConfiguration.xml file.

Body

<escape_chars>
  <char>
    <orig></orig>
    <escaped></escaped>
  </char>
</escape_chars>

;\ --> \\
" --> \"

 --> ' '
] --> \]

| --> ;
= --> \=

 --> ' '

= --> \=


 --> ' '

;\ --> \\
= --> \=

 --> ' '
| --> \|

= --> \=

 --> ' '

| --> ;
= --> \=

 --> ' '

\ --> \\
" --> '

 --> ' '

Header

Notes:

• If you want to add constant string to the header, you can do this by adding the string to the <header_format> tag value.
• If you want to add a new field to the header, you need to add a new header format replacement string (for example: {}) to the <header_format> and also add the relevant information under the <headers> tag.

Every format has its own predefined fields configuration file that allow to change the name / value of the exported field, filter out irrelevant fields, and so on.

These files are located under each deployment folder: $EXPORTERDIR/targets/<deployment_name>/conf/*FieldsMapping.xml

Note: Do not edit the original *FieldsMapping.xml files - doing that will cause data loss after upgrade. Copy the file to a different file name and modify the copied file while leaving the original intact. After done modifying the file, refer to it (full path) in the <mappingConfiguration> element in the relevant targetConfiguration.xml file.

How to set a secured connection between the log exporter and the syslog server

Log Exporter can export logs over an encrypted connection using TLS protocol.

When using TLS, it is important to know that only mutual authentication is allowed.

For mutual authentication, the Log Exporter needs the following certificates:

• CA certificate that signed both client (Log Exporter side) and server (syslog server side) certificates. Required format: PEM.
• Client (Log Exporter side) certificate. Required format: P12.

Notes:

• CA server needs to be routable from Log Server to establish the connection.
• In addition to these two certificates, a third certificate should be installed on the syslog server side based on the server requirement. It is also possible to use self-signed certificates.

The following procedure uses openssl commands that have to be run on non-Check Point server.

Creating a CA Certificate

1. Create CA key:

   openssl genrsa -out ca.key 2048

2. Create CA Cert:

   openssl req -x509 -new -nodes -key ca.key -days 2048 -out ca.pem

   You will be prompted to provide information regarding the certificate. Apart from the Common Name (it is recommended to use the device IP address as the Common Name), all other fields are optional and can be skipped. If you are purchasing an SSL certificate from a certificate authority, it is often required that these additional fields.

Creating a Log Exporter Certificate

1. Create Log Exporter key:

   openssl genrsa -out cp_client.key 2048

2. Create Log Exporter CSR file:

   openssl req -new -key cp_client.key -out cp_client.csr

3. Create Log Exporter CRT file:

   openssl x509 -req -in cp_client.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out cp_client.crt -days 2048 -sha256

4. Create Log Exporter P12 file:

   openssl pkcs12 -inkey cp_client.key -in cp_client.crt -export -out cp_client.p12

Creating a Syslog Server Certificate

1. Create Target Server key:

   openssl genrsa -out server.key 2048

2. Create Target Server CSR file:

   openssl req -new -key server.key -out server.csr

3. Create a Target Server CRT file:

   openssl x509 -req -in server.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out server.crt -days 2048 -sha256

Note: Several SIEM applications require the Server certification to be in a specific format. For more information, refer to the section "SIEM Specific instructions".

The Log Exporter can filter logs based on field values.

Because fields mapping operation is done before filtering, make sure to use the dst name / value of the fields if configured specifically in the *FieldsMapping.xml file (based on the format). Otherwise, use the name / value as written in the raw log file.

It is possible to configure what to export or what not to export.

The filter configuration file is located under each deployment folder: $EXPORTERDIR/targets/<deployment-name>/conf/FilterConfiguration.xml

<field name="" operator="">
  <value operation=""></value>
</field>

There are two ways to configure the filtering feature:

1. Using the cp_log_export command.

   This command allows you to configure filtering for action / blade / origin fields only.

   The syntax is:

   cp_log_export set name <name> filter-action-in "value1,value2"

   cp_log_export set name <name> filter-origin-in "value1,value2"

   cp_log_export set name <name> filter-blade-in "value2"

   In addition, it is possible to use predefined families for "filter-blade-in" value:

   • TP - For exporting Threat Prevention logs only (Anti-Bot, Anti-Malware, Threat Emulation, IPS, IPS-1, SmartDefense, Anti-Virus, New Anti Virus, Anti-Spam and Email Security, Threat Extraction, MTA).
   • Access - For exporting Access logs only (Security Gateway/Management, VPN-1 & FireWall-1, Firewall, Application Control, URL Filtering, Content Awareness, Connectra, Mobile Access, Compliance blade, Core, DDoS Protector, Identity Awareness, Identity Logging, UA WebAccess).
   • Mobile - For exporting Mobile logs only (WIFI Network, Mobile App, OS Exploits, Device, Network Security, Cellular Network, Network Access, iOS Profiles, Text Message, On-device Network Protection).
   • EndPoint - For exporting Endpoint logs only (Anti-Bot, Anti Malware, Threat Emulation,  IPS, IPS-1, SmartDefense, Anti-Virus, New Anti Virus, Anti-Spam and Email Security, Threat Extraction, MTA ).

2. Modifying the FilterConfiguration.xml file manually.

   It is allowed to add new fields to this file - for example:

   <filters>
     <filterGroup operator="and">
       <field name="action" operator="and">
       </field>
       <field name="origin" operator="and">
       </field>
       <field name="product" operator="or">
       </field>
       <field name="severity" operator="or">
         <value operation="eq">3</value>
           <value operation="eq">4</value>
       </field>
     </filterGroup>
   </filters>

Limitations:

• The relation between the values of the same operation is only "OR".

  Example:

  cp_log_export set name <target-name> filter-action-in "accept,drop"

  Only logs with action = "accept" OR action = "drop" are exported.

• Filtering is not supported for any of the following fields:

  time, category, UUID,appi_name ,app_desc, app_category, matched_category, app_properties, app_rule_name, app_risk, HTTPS_inspection_rule_name, cvpn_resource, cvpn_category, name, desc, properties.

• Filtering on a certain field with the condition: "not equal(value1) OR not equal(value2)" is not supported. When editing the filtering XML, make sure to have a maximum of one line of "neq" operation in each field.

Check Point Infinity solution includes multiple log fields, representing the diversity of Check Point's products.
The log fields mapping will help you understand security threats, logs language, complex queries and SIEM.

For information on Check Point's Log Fields Mapping, refer to: sk144192.

Rsyslog

Rsyslog is not configured to use RFC5424 timestamp format by default. Therefore, you should manually change Rsyslog setting for it to be compliant with Log Exporter output format.

On the Syslog server:

1. Edit the /etc/rsyslog.conf file.

2. Comment out this line, if it is not commented out already:

   #"$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat"

3. Add this line in the file:

   $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format

4. Save the change in the file and close it.

5. Restart the Rsyslog:

   service rsyslog restart

ArcSight

ArcSight recommends naming the server certificate file as 'syslog-ng'.

1. Convert the key to the P12 format:

   openssl pkcs12 -inkey syslogServer.key -in syslogServer.crt -export -out syslog-ng.p12 -name "syslogng-alias" -password pass:changeit

2. Make sure the value of the environment variable ARCSIGHT_HOME is path to the connector installation directory.

3. Run the certificate manager on the Linux KDE console:

   $ARCSIGHT_HOME/current/bin/arcsight agent keytoolgui

4. From the File menu, open the keystore:

   $ARCSIGHT_HOME/current/jre/lib/security/cacerts

   The password is: changeit

5. From the menu, select Import Trusted Certificate

6. From the file dialog, select ca.pem and save it.

7. Save and close the certificate manager.

8. Edit the agent.properties file to enable the mutual authentication:

   vi $ARCSIGHT_HOME//current/user/agent/agent.properties

9. Configure the value 'true' for the 'syslogng.mutual.auth.enabled' parameter:

   syslogng.mutual.auth.enabled=true

10. Add these lines at the bottom:

    syslogng.tls.keystore.file=user/agent/syslog-ng.p12

    syslogng.tls.keystore.alias=syslogng-alias

11. Save the change in the file and close it.

12. Restart the service:

    /etc/init.d/arc_connector_name restart

Splunk

We recommend to use Check Point App for Splunk when exporting logs to a Splunk server.

For more information about installation and deployment, see the Check Point App for Splunk User Guide.

In addition, to configure an encrypted connection, do the following:

1. Generate the server PEM file according to Splunk TLS Documentation:

   cat syslogServer.crt syslogServer.key RootCA.pem > splunk.pem

2. Edit the 'inputs.conf' file on the Splunk server:

   vi /opt/splunk/etc/apps/<Name of the app, where the configuration is saved>/local/inputs.conf

3. Update the file to use TLS:

   [SSL]
   serverCert = <full path to server PEM file>
   sslPassword = <challenge password>
   requireClientCert = true
   [tcp-ssl://<port>]
   index = <index>
   

4. Save the change in the file and close it.

5. Edit the 'server.conf' file on the Splunk server:

   vi /opt/splunk/etc/system/local/server.conf

6. Update the file to use the relevant CA PEM file:

   [sslConfig]
   sslRootCAPath = <full path to CA PEM file>
   
   [SSL]
   cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH
   

7. Save the change in the file and close it.

8. Restart the Splunk daemon:

   /opt/splunk/bin/splunk restart

QRadar

1. In the Authentication Mode field, choose TLS And Client Authentication.

2. Upload the Check Point certificate and private key to QRadar.

3. Provide the absolute path to those under the Provide Certificate option.

Notes:

• When using Client Authentication, you must provide the absolute path to the client certificate.
• Make sure the "Common Name" is unique in every certificate.

1. Delete the OPSEC application object from the GUI, if you only use it for the OPSEC application. Alternatively, remove the LEA client entity from it, if you do not use it:

   

2. If this is the only OPSEC LEA client, configure the $FWDIR/conf/fwopsec.conf file to not allow LEA (if this is not the only OPSEC LEA client, skip this step):

   Change these lines:

   From:	To:
   # lea_server auth_port 18184 lea_server port 0 #	# #lea_server auth_port 18184 #lea_server port 0 #

3. Install the Log Exporter according to the installation guide above.

1. Uninstall the CPLogToSyslog package using CPUSE. See the instructions in sk92449 - section 4-C.
2. Install the Log Exporter according to the installation guide above.

• If you want to change this environment, you must first consult your Check Point partner or vendor. This release is built for the specific environment. Upgrades or other changes can overwrite Hotfix functionality and environment customizations.
• When exporting logs from a certain domain using UDP protocol, the exported log's header will contain the MDS IP and not the CMA/CLM's IP address.

Special log fields

Syslog-NG Listener configuration

When configuring a source on a Syslog NG server it is recommended to use the syslog-protocol flag.

For example:

source s_network { network(transport("tcp") port(514) flags(syslog-protocol) ); };

Splunk Listener configuration

It is recommended to add these time settings to your source type:

TIME_FORMAT = %s
TIME_PREFIX = time=
MAX_TIMESTAMP_LOOKAHEAD = 15

ArcSight Listener configuration

The Log Exporter solution does not work with the OPSEC LEA connector. Instead, you must install the ArcSight Syslog-NG connector.

ArcSight Common Event Format (CEF) Mapping

CEF is an extensible, text-based format designed to support multiple device types by offering the most relevant information. Message syntaxes are reduced to work with ESM normalization. Specifically, CEF defines a syntax for log records comprised of a standard header and a variable extension, formatted as key-value pairs. The CEF format can be used with on-premise devices by implementing the ArcSight Syslog SmartConnector. CEF can also be used by cloud-based service providers by implementing the SmartConnector for ArcSight Common Event Format REST.

CEF Header format

QRadar Log Event Extended Format (LEEF) Mapping

The Log Event Extended Format (LEEF) is a customized event format for IBM Security QRadar.

LEEF Header format

Note: The time format is not compliant with the official LEEF format.

Until such a time as IBM will add support for epoch time format Log Exporter with LEEF format is only partially supported.

Note: Takes listed in the table below are related to the Log Exporter, and not to Jumbo Hotfix Accumulators.