Log Exporter is a multi-threaded daemon service, running on a Management Server / Log Server. The Log Exporter daemon reads each log, transforms it into the desired format and mapping, and sends it to the configured target. Therefore, we recommend to deploy the Log Exporter on every server that contains logs to be exported.

On a Multi-Domain Security Management Server / Multi-Domain Log Server, if the Log Exporter is deployed on several Domains, each Domain Management Server has its Log Exporter daemon . If you are exporting the logs to several targets, each target have its Log Exporter daemon.

The Log Exporter is implemented as the "ETL" procedure:

• Extract - Reads incoming logs from the Security Gateway, stored in local files.

• Transform - Changes the logs according to configuration files (both exported format and field name/values, removing irrelevant fields).

• Load - Sends the logs to the configured target server over the TCP Syslog / UDP Syslog (takes into consideration the filter configuration, if it exists).

• Data integrity - Log Exporter stops exporting when disconnected from the 3rd party server and remembers the last position exported. After the connection is established again, the Log Exporter automatically starts exporting logs from the last known position.

The Log Exporter is exporting both online and offline (if any) logs in parallel. In case the 3rd party server is slow, the Log Exporter reduces the offline exporting rate to prioritize the online logs over the offline logs.

These are the ways to configure the Log Exporter:

Syntax

cp_log_export <Command Name> [<Command Arguments>]

To see the built-in help, run:

cp_log_export <Command Name> help

Commands

Name	Description
add	Deploys a new Log Exporter instance.
set	Updates an existing Log Exporter instance configuration.
delete	Removes an existing Log Exporter instance.
show	Prints the current configurations of the existing Log Exporter instances.
status	Prints the overview statuses of the existing Log Exporter instances.
start	Starts the Log Exporter instance.
stop	Stops the Log Exporter instance.
restart	Restarts the Log Exporter instance.
reexport	Resets the current read position and re-exports all logs per the Log Exporter instance configuration.

Parameters

Parameter Name	Description	add	set	delete	show, status, start, stop, restart	reexport
name	Configure a unique name for the Log Exporter configuration.	Mandatory	Mandatory	Mandatory	Optional Default is "all"	Mandatory
domain-server	Configure the name or IP address of the relevant Domain.	Mandatory	Mandatory	Mandatory	Optional Default is "all"	Mandatory
target-server	Configure the IP address of the target server (to which the Log Exporter sends the logs).	Mandatory	Optional	N/A	N/A	N/A
target-port	Configure the port on which the target server is listening.	Mandatory	Optional	N/A	N/A	N/A
protocol	Configure the transport protocol to use (TCP or UDP).	Mandatory	Optional	N/A	N/A	N/A
format	Configure the format in which the logs are exported.	Optional	Optional	N/A	N/A	N/A
read-mode	Configure the mode in which the log files are read and exported.	Optional	Optional	N/A	N/A	N/A
enabled	Start the Log Exporter when running the cpstart or mdsstart command.	Optional	Optional	N/A	N/A	N/A
encrypted	Use TLS (SSL) encryption for exporting the logs.	Optional	Optional	N/A	N/A	N/A
ca-cert	Configure the full path to the Certificate Authority PEM certificate file. Relevant only when the value of 'encrypted' is 'true'.	Optional	Optional	N/A	N/A	N/A
client-cert	Configure the full path to the client P12 certificate file. Relevant only when the value of 'encrypted' is 'true'.	Optional	Optional	N/A	N/A	N/A
client-secret	Configure the challenge phrase that was used to create the client P12 certificate. Relevant only when the value of 'encrypted' is 'true'.	Optional	Optional	N/A	N/A	N/A
filter-action-in	Export all logs with a specific action. The value must be surrounded by double quotes (""). Multiple values are supported and must be separated by a comma.	Optional	Optional	N/A	N/A	N/A
filter-origin-in	Export all logs from a specific origin. The value must be surrounded by double quotes (""). Multiple values are supported and must be separated by a comma.	Optional	Optional	N/A	N/A	N/A
filter-blade-in	Export all logs that belong to a specific blade. The value must be surrounded by double quotes (""). Multiple values are supported and must be separated by a comma. Predefined blade families can be selected (TP, Access, Endpoint, Mobile).	Optional	Optional	N/A	N/A	N/A
--apply-now	Apply any change that was done in the "add" immediately.	Optional	Optional	Mandatory	N/A	Mandatory
export-link	Add a field to the exported log that represents a link to SmartView that shows the log card.	Optional	Optional	N/A	N/A	N/A
export-attachment-link	Add a field to the exported log that represents a link to SmartView that shows the log card and automatically opens the attachment.	Optional	Optional	N/A	N/A	N/A
export-link-ip	Make the 'export-link' and the 'export-attachment-link' use a customized IP address (for example, for a Log Server behind NAT).	Optional	Optional	N/A	N/A	N/A
export-attachment-ids	Add a field to the exported log that represents the ID of log's attachment (if exists).	Optional	Optional	N/A	N/A	N/A
reconnect-interval	Schedule a reconnection to the target server. The log exporter will reconnect to the remote server according to the specified time intervals.	Optional	Optional	N/A	N/A	N/A
export-log-position	Send the position of the exported log.	Optional	Optional	N/A	N/A	N/A

Important Note - Using the 'filter-action-in' / 'filter-origin-in' / 'filter-blade-in' replaces any other filter configuration that was declared earlier on these fields directly in the filtering XML. Other field filters are not overridden.

After deploying a new instance of Log Exporter, all related files to that deployment can be found in:
$EXPORTERDIR/targets/<Name of Log Exporter Configuration>

On a Multi-Domain Security Management Server / Multi-Domain Log Server, the environment variable EXPORTERDIR exists in each Domain, and its value is changed automatically when you switch between context of Domains with the "mdsenv" command.

Note - You must restart the Log Exporter instance for the new settings to take effect. Run the "cp_log_export restart" command.

If you customized your configuration files in the log exporter instance, then after upgrade, you will not get the updated configuration of the latest version. To get the latest configuration files, do these steps: 

Target Configuration XML

The Log Exporter configuration for the target server is saved in:
$EXPORTERDIR/targets/<Name of Log Exporter Configuration>/targetConfiguration.xml

These are some configuration options:

Parameter	Description	Valid / Default Values
<version></version>	The current Log Exporter version - used for upgrades.
<is_enabled></is_enabled>	Determines whether the process is monitored by the watch dog.	true false

Destination Parameters

Parameter	Description	Valid / Default Values
type	Reserved for future use
<ip></ip>	The IP address or FQDN of the target server	Any IPv4 address or FQDN
<port></port>	The port on which the target server is listening	Any valid port number
<protocol></protocol>	The Layer 4 protocol to use	TCP or UDP
<reconnect_interval></reconnect_interval>	Determines how frequently to start the connection to the target server after it is lost	Number of minutes

Security Parameters

Discussed in more detail in the "TLS Configuration" section.

Parameter	Description	Valid / Default Values
<security></security>	Determines whether the connection data is sent in clear text or encrypted.	clear (clear text - this is the default) tls (encrypted)
<pem_ca_file></pem_ca_file>	The location of the root Certificate Authority PEM file.
<p12_certificate_file></p12_certificate_file>	The location of the client key pair in the P12 format.
<client_certificate_challenge_phrase></client_certificate_challenge_phrase>	The challenge phrase that was used to create the P12 certificate. The value is hashed after restarting the process.

Source Parameters

Parameter	Description	Valid / Default Values
<folder></folder>	The path where the log files are located	The default location is $FWDIR/log/
<log_files></log_files>	Determines which log records to export or how far back to read the log records from the $FWDIR/log/fw.log file	<Number> - reads logs from the specific number (default=1) of days back (recommended) <Specific File Name> - reads logs from the specified file on-line If no value is specified, uses 'on-line'
<log_types></log_types>	Determines which logs to export based on their type	all (default) log audit
<read_mode></read_mode>	Determines whether to export complete logs or only their delta.	semi-unified (default since R81) raw

Resolver Parameters

Parameter	Description	Valid / Default Values
<mappingConfiguration></mappingConfiguration>	Configures the XML file that contains the log field mapping scheme. If left empty, uses the default settings.	Default values are based on the 'format'.
<exportAllFields>true</exportAllFields>	When this field is set to 'true', all log fields are sent regardless of whether they appear in the mapping scheme, except for specifically black-listed fields in the relevant log format mapping file (<exported>false</exported>). When this field is set to 'false', only those fields which appear in the relevant log format mapping file are sent (with exported flag set to 'true': <exported>true</exported>)	true false

Format Parameters

Parameter	Description	Valid / Default Values
<formatHeaderFile></formatHeaderFile>	Configures the XML file that contains the log header format scheme. If left empty, uses the default settings.	Default values are based on the 'format'.

General Filter Configuration Path

Parameter	Description	Valid / Default Values
<dynamicFilter></dynamicFilter>	Configures the XML file that contains the filtering configuration. If left empty, uses the default settings.	The default path is: conf/FilterConfiguration.xml

SmartView links parameters

Parameter	Description	Valid / Default Values
export_log_link	Adds a field to the exported log that represents a link to SmartView that shows the log card.	true false (default)
export_attachment_link	Adds a field to the exported log that represents a link to SmartView that shows the log card and automatically opens the attachment.	true false (default)
export_link_ip	Makes the 'export_log_link' and the 'export_attachment_link' use a customized IP address (for example, for a Log Server behind NAT).	empty (default) IPv4 Address

Parameters to filter out the Security Gateway connections

This configuration allows Log Exporter instance to filter out the Security Gateway traffic logs for several Software Blades ('Firewall-1 & VPN-1', 'HTTPS Inspection' and 'Security Gateway/Management').

Note - Security Gateway session logs are still exported (generated by tracking a Security Gateway rule per session).

Parameter	Description	Valid / Default Values
<filter filter_out_by_connection="false">	Determines whether to filter out the access logs. Note: No other Software Blade filters are currently supported. This is planned in future releases.	true (filters out the connection logs) false

Limitation: HTTPS Inspection logs, Security Gateway logs generated not from rules, and a few NAT update logs are still exported.

Every format has its own predefined format configuration file that configures the format of the exported logs, the delimiters, fields that are part of the header, and so on.

These files are located in:
$EXPORTERDIR/targets/<Name of Log Exporter Configuration>/conf/*FormatDefinition.xml

Note: Do not edit the original *FormatDefinition.xml files. Doing so causes a data loss after an upgrade. Instead, create a copy of the file and modify the copied file, while leaving the original intact. After modifying the copied file, refer to it (using a full path) in the <formatHeaderFile> element in the relevant targetConfiguration.xml file.

Body

Parameter	Description	Syslog	Splunk	RSA	CEF	LEEF	LogRhythm	Generic
<start_message_body></start_message_body>	The character preceding the log data payload.	[
<end_message_body></end_message_body>	The character following the log data payload.	]
<message_separator></message_separator>	The delimiter that separates logs.	&#10; (&#10;=='\n')	&#10; ('\n')	&#10; ('\n')	&#10; ('\n')	&#10; ('\n')	&#10; ('\n')	('\n')
<fields_separatator></fields_separatator>	The delimiter that separates log fields.	'; ' (semicolon + space)	| (pipe)	' ' (space)	' ' (space)	&#09 (<TAB>)	| (pipe)	' ' (space)
<field_value_separatator></field_value_separatator>	The assignment operator.	:	=	=	=	=	=	=
<value_encapsulation_start>&quot;</value_encapsulation_start>	The value encapsulation operator (start)	"				"		"
<value_encapsulation_end>&quot;</value_encapsulation_end>	The value encapsulation operator (end)	"				"		"
<escape_chars> <char> <orig></orig> <escaped></escaped> </char> </escape_chars>	Escaping unwanted characters. The escape functionality replaces the string that's encapsulated by the 'orig' tags with the one encapsulated by the 'escaped' tags	;\ --> \\ " --> \" &#10; --> ' ' ] --> \]	| --> ; = --> \= &#10; --> ' '	= --> \= &#10; --> ' '	;\ --> \\ = --> \= &#10; --> ' ' | --> \|	= --> \= &#10; --> ' '	| --> ; = --> \= &#10; --> ' '	\ --> \\ " --> ' &#10; --> ' '

Header

Parameter	Description	Default values<br/ >for Syslog	Default values<br/ >for Splunk	Default values<br/ >for RSA	Default values<br/ >for CEF	Default values<br/ >for LogRhythm
<header_format></header_format>	The delimiter between the header values and the number of values. Every {} is replaced with one value.	' ' (space)	time={}|hostname={}|	<134>	| ;	LOGV2 {}|

Notes:

• To add a constant string to the header, add the string to the <header_format> tag value.
• To add a new field to the header, add a new header format replacement string (for example: {}) to the <header_format> and add the relevant information in the <headers> tag.

Every format has its own predefined fields configuration file that allow to change the name / value of the exported field, filter out irrelevant fields, and so on.

These files are located under each deployment directory:
$EXPORTERDIR/targets/<Name of Log Exporter Configuration>/conf/*FieldsMapping.xml

Note: Do not edit the original *FieldsMapping.xml files. Doing so causes a data loss after an upgrade. Instead, create a copy of the file and modify the copied file, while leaving the original intact. After modifying the copied file, refer to it (using a full path) in the <mappingConfiguration> element in the relevant targetConfiguration.xml file.

How to set a secured connection between the Log Exporter and the Syslog server

Log Exporter can export logs over an encrypted connection using the TLS protocol.

When using TLS, it is important to know that only mutual authentication is allowed.

For mutual authentication, the Log Exporter needs these certificates:

• CA certificate (in the PEM format) that signed both the client (Log Exporter side) and server (syslog server side) certificates
• Client certificate (in the P12 format) on the Management Server / Log Server with Log Exporter

Notes:

• The Management Server / Log Server with Log Exporter must be able to connect to CA.
• In addition to these two certificates, a third certificate should be installed on the Syslog server (based on the server requirement). It is also possible to use self-signed certificates.

The procedure below uses the openssl commands on a non-Check Point server.

Creating a CA Certificate:

1. Create a CA key:

   openssl genrsa -out ca.key 2048

2. Create a CA Certificate:

   openssl req -x509 -new -nodes -key ca.key -days 2048 -out ca.pem

   You are prompted to provide information regarding the certificate. Apart from the Common Name (it is recommended to use the device IP address as the Common Name), all other fields are optional and can be skipped. If you are purchasing an SSL certificate from a certificate authority, it often requires these additional fields.

Creating a Log Exporter Certificate:

1. Create a Log Exporter key:

   openssl genrsa -out cp_client.key 2048

2. Create a Log Exporter CSR file:

   openssl req -new -key cp_client.key -out cp_client.csr

3. Create a Log Exporter CRT file:

   openssl x509 -req -in cp_client.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out cp_client.crt -days 2048 -sha256

4. Create Log Exporter P12 file:

   openssl pkcs12 -inkey cp_client.key -in cp_client.crt -export -out cp_client.p12

Creating a Syslog Server Certificate:

1. Create a Target Server key:

   openssl genrsa -out server.key 2048

2. Create a Target Server CSR file:

   openssl req -new -key server.key -out server.csr

3. Create a Target Server CRT file:

   openssl x509 -req -in server.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out server.crt -days 2048 -sha256

Note: Several SIEM applications require the Server certification to be in a specific format. For more information, refer to the section "SIEM Specific instructions".

The Log Exporter can filter logs based on the field values.

Because field mapping operation is done before the actual filtering, make sure to use the dst name / value of the fields, if configured specifically in the *FieldsMapping.xml file (based on the format). Otherwise, use the name / value as written in the raw log file.

It is possible to configure what to export or what not to export.

The filter configuration file is located in:
$EXPORTERDIR/targets/<Name of Log Exporter Configuration>/conf/FilterConfiguration.xml

These are the ways to configure the filtering feature:

<filters>
  <filterGroup operator="and">
    <field name="action" operator="and">
    </field>
    <field name="origin" operator="and">
    </field>
    <field name="product" operator="or">
    </field>
    <field name="severity" operator="or">
      <value operation="eq">3</value>
      <value operation="eq">4</value>
    </field>
    <field name="service_id" operator="or"> 
        <value operation="eq">https</value>
        <value operation="eq">http</value>
    </field>
    <field name="src" operator="or"> 
        <value operation="eq">1.2.1.2</value>
        <value operation="eq">3.2.3.2</value>
    </field>
    <field name="dst" operator="or"> 
        <value operation="eq">1.2.1.2</value>
        <value operation="eq">3.2.3.2</value>
    </field>
    <field name="rule_uid" operator="or"> 
        <value operation="eq">a24163d2-f2f2-1426-52d1-fa42133d04bd</value>
        <value operation="eq">b54152f1-a3c2-7242-52c1-dc86434a28ac</value>
    </field>
    <field name="NAT_rule_uid" operator="and"> 
        <value operation="eq">a24163d2-f2f2-1426-52d1-fa42133d04ac</value>
    </field>
    </filterGroup>
</filters>
     

Limitations:

• The relation between the values of the same operation is only logical "OR".

  Example:

  cp_log_export set name <target-name> filter-action-in "accept,drop"

  Only logs with action = "accept" OR action = "drop" are exported.

• Filtering is not supported for any of these fields:

  • app_category
  • app_desc
  • app_properties
  • app_risk
  • app_rule_name
  • appi_name
  • category
  • cvpn_category
  • cvpn_resource
  • desc
  • HTTPS_inspection_rule_name
  • matched_category
  • name
  • properties
  • time
  • UUID

• Filtering for a certain field with the a double "NOT" condition "not equal(value1) OR not equal(value2)" is not supported.
  When editing the filtering XML, make sure to have a maximum of one line of "neq" operation in each field.

Check Point Infinity solution includes multiple log fields, representing the diversity of Check Point's products.
The log fields mapping helps you understand security threats, logs language, complex queries and SIEM.

For information on Check Point's Log Fields Mapping, refer to sk144192.

Rsyslog

ArcSight

Splunk

QRadar

Notes:

• When using Client Authentication, you must provide the absolute path to the client certificate.
• Make sure the "Common Name" is unique in every certificate.

1. Delete the OPSEC application object from SmartConsole R80.10 and higher / SmartDashboard R77.30, if you only use it for the OPSEC application.
   Alternatively, remove the LEA client entity from it, if you do not use it:

   

2. If this is the only OPSEC LEA client, configure the $FWDIR/conf/fwopsec.conf file (on the Check Point Management Server / Log Server) to not allow LEA (if this is not the only OPSEC LEA client, skip this step):

   Change these lines:

   From:	To:
   # lea_server auth_port 18184 lea_server port 0 #	# #lea_server auth_port 18184 #lea_server port 0 #

3. Install the Log Exporter. See the Installation section.

1. Uninstall the CPLogToSyslog package using CPUSE. See the instructions in sk92449 - section 4-C.

2. Install the Log Exporter. See the Installation section.

• If you want to change this environment, you must first consult your Check Point partner or vendor. This release is built for the specific environment. Upgrades or other changes can overwrite Hotfix functionality and environment customizations.

• On a Multi-Domain Security Management Server / Multi-Domain Log Server, when exporting logs from a certain Domain using the UDP protocol, the exported log's header contains the IP address of the Multi-Domain Server and not the IP address of the Domain Management Server / Domain Log Server.

• Log Exporter is only supported for Quantum appliances and not Quantum Spark. For Quantum Spark appliances, use the WebUI to set up a syslog server to send logs to.

Special log fields

Field	Description
loguid	Some Check Point logs are updated over time. Update logs have the same loguid value. Check Point SmartLog client correlates those updates into a single unified log. When sending update logs to 3rd party (SIEM) servers, in raw read-mode, they arrive as distinct logs. Best use the semi-unified read-mode, which sends few instances of the log, but each instance contains the entire event chain until this update. Administrators can alternatively use the loguid field to correlate update logs and get the full event chain themselves. Note: All related log & log-updates share the same initial time as the 1st log (in semi-unified mode). Example of update logs includes the total number of bytes sent and received over time or the severity field which is updated over time as more information becomes available.
hll_key	Stands for High-Level Log key. This concept was introduced in R80.10, where Multiple connection logs can comprise one session with one shared hll_key. For example, when browsing a webpage, you might have multiple connection logs which are related to the same session. Connection logs which are part of the same session share the same hll_key value.

Syslog-NG Listener configuration

When configuring a source on a Syslog NG server it is recommended to use the syslog-protocol flag.

For example:

source s_network { network(transport("tcp") port(514) flags(syslog-protocol) ); };

Splunk Listener configuration

It is recommended to add these time settings to your source type:

TIME_FORMAT = %s
TIME_PREFIX = time=
MAX_TIMESTAMP_LOOKAHEAD = 15

ArcSight Listener configuration

The Log Exporter solution does not work with the OPSEC LEA connector. Instead, you must install the ArcSight Syslog-NG connector.

ArcSight Common Event Format (CEF) Mapping

CEF is an extensible, text-based format designed to support multiple device types by offering the most relevant information. Message syntaxes are reduced to work with ESM normalization. Specifically, CEF defines a syntax for log records comprised of a standard header and a variable extension, formatted as key-value pairs. The CEF format can be used with on-premise devices by implementing the ArcSight Syslog SmartConnector. CEF can also be used by cloud-based service providers by implementing the SmartConnector for ArcSight Common Event Format REST.

CEF Header format

	Version	Device Vendor	Device Product	Device Version	Device Event Class ID	Name	Severity
Default	CEF:0	Check Point	Log Update	Check Point	Log	Log	0
Values	-	-	Product Name (Blade)	-	Attack Name Protection Type Verdict Matched Category DLP Data Type Application Category Application Properties	Protection Name Application Name Message Info Service ID Service	Application Risk Risk Severity

QRadar Log Event Extended Format (LEEF) Mapping

The Log Event Extended Format (LEEF) is a customized event format for IBM Security QRadar.

LEEF Header format

	LEEF:Version	Vendor	Product	Version	EventID
Default	LEEF:2.0	Check Point	Log Update	1.0	Check Point Log
Values	-	-	Product Name (Blade)	-	Protection Name Application Name Action

Note: The time format is not compliant with the official LEEF format.

Until the time IBM adds support for Epoch time format, Log Exporter with LEEF format is only partially supported.

Note: Takes listed in the table below are related to the Log Exporter, and not to Jumbo Hotfix Accumulators.