Support Center > Search Results > SecureKnowledge Details
ATRG: Threat Extraction
Solution

Table of Contents:

  • Introduction
  • Introduction to the Threat Extraction Solution
  • Supported Configuration and Requirements
  • Threat Extraction Workflow
  • User Space
  • Debug
  • FAQ
  • Related Resources
  • Revision History

Introduction

To challenge today's malware landscape, Check Point's comprehensive Threat Prevention solution offers a multi-layered, pre and post-infection defense approach and a consolidated platform that enables enterprise security to deal with modern malware:

Software Blade Introduced in Description Reference
Threat Emulation
  • Gateway mode: R77
  • VSX mode: R77.20

Stops unknown malware, targeted attacks, and zero-day attacks.

Works by:

  1. Identifying files in email attachments (SMTP & SMTP/TLS) and
    downloads over the web (HTTP & HTTPS)
  2. Uploading the suspicious files to a virtual sandbox
    (in the cloud, or on a local appliance) for further emulation and analysis.
  3. Emulating the suspicious files in various OS environments
    by opening the files and monitoring abnormal behavior (related to
    file system, system registry, network connections, system processes, etc.)
  4. Stopping the malicious files, and preventing them from getting to the end user.
  5. Sharing (in real-time) the data about the detected malicious files with ThreatCloud.
sk114806 - ATRG: Threat Emulation
Threat Extraction
  • Gateway mode: R77.30
  • VSX mode: supported from R80.10
Pro-actively cleans potential threats from incoming documents Described in this article
Anti-Bot
  • Gateway mode: R75.40
  • VSX mode: R77.40VS
Post-infection bot detection, prevention, and threat visibility sk92264 - ATRG: Anti-Bot and Anti-Virus
Anti-Virus
  • NG AI R57

Pre-infection blocking of known viruses and file transfers.

Works by:

  • Looking for specific patterns.
  • Enforcing compliance of protocols to standards.
  • Detecting variations from the protocols.
sk92264 - ATRG: Anti-Bot and Anti-Virus

Each Threat Prevention Software Blade gives unique network protections and they can be combined to supply a strong malware solution.

Data from malicious attacks are shared between the Threat Prevention Software Blades and help keep your network safe.
For example, the signature from a threat that is identified by Threat Emulation is added to the Anti-Virus database.

The Threat Prevention Software Blades use a separate policy installation to minimize risk and operational impact.
They are also integrated with other Software Blades on the Security Gateway to detect and stop threats.

Introduction to the Threat Extraction Solution

Threat Extraction is a new Software Blade in the Threat Prevention family that pro-actively cleans potential threats from incoming documents.

SandBlast Threat Extraction prevents both known and unknown threats before they arrive at the organization, thus providing better protection against zero-day threats. Threat Extraction gives organizations the necessary protection against unknown threats in files that are downloaded from the Internet, or attached to emails.

Supported Configuration and Requirements

  1. In R77.30 release: Requires the R77.30 Add-on to be installed and enabled on the Security Management Server / Multi-Domain Security Management Server.

  2. Scanning attachements from incoming emails requires MTA configuration

  3. Scanning  files downloaded from the internent is supported using SandBlast Agent for Browsers

  4. Threat extraction is supported in VSX mode from R80.10.

  5. CXL support for Threat Extraction is available in from R80.20.

Threat Extraction Workflow

In MTA mode:

  1. A PostFix server receives and handles the emails.

  2. Emails are forwarded to the in.emaild.mta daemon, which:
    1. Parses the emails (For example, Base64 decode)
    2. Passes the attachments to the scrubd process, if needed (based on the configuration of supported file types).

  3. The scrubd process handles the file and sends it to the scrub_cp_file_convertd process with the relevant details (according to the policy).

  4. scrub_cp_file_convertd process
    1. Converts the file / extracts potentialy malicious content from it. 
    2. Returns a Safe copy of the file to scrubd.

  5. The scrubd process returns the Safe copy to the in.emaild.mta daemon

  6. The in.emaild.mta daemon:
     
    1. Replaces the original attachment with the Safe Copy version.

    2. Forwards the email to its destination.

Note: For environments with MTA bundle R80.10 jhf or R80.20, in.emaild.mta is replaced with mtad daemon

User Space

Daemon Section Description / Paths / Notes / Stop and Start Commands / Debug

scrub

Description The CLI client for the scrubd daemon (this process runs only when it is called explicitly).
Path $FWDIR/bin/scrub

scrubd

Description Main Threat Extraction daemon.
Path $FWDIR/bin/scrubd
Log file $FWDIR/log/scrubd.elg
/var/log/scrub/scrubd_messages
Configuration file $FWDIR/conf/scrub_debug.conf
To Stop [Expert@HostName]# cpstop
To Start [Expert@HostName]# cpstart
Debug
  1. Start Threat Extraction debug:
    scrub debug on
    scrub debug set all all
  2. Verify Threat Extraction debug is enabled:
    scrub debug stat
  3. Start debug of scrubd daemon:
    scrub debug set all all
  4. Replicate the issue.
  5. Stop debug of scrubd daemon:
    scrub debug off
    scrub debug reset
  6. Verify Threat Extraction debug is disabled:
    scrub debug stat
  7. Analyze:
    $FWDIR/log/scrubd.elg*

scrub_cp_file_convertd

Description Used to convert-to-PDF, or extract potentialy malicious content from various file formats. 
Path $FWDIR/bin/cp_file_convert
Log file /var/log/jail/$FWDIR/log/scrub_cp_file_convertd.elg
Configuration file

 $FWDIR/conf/file_convert.conf

After changing this conf, follow these steps:

1. Install policy.

2. ssh to the gateway

3. kill -9 `pidof scrub_cp_file_convertd`

To Stop [Expert@HostName]# cpstop
To Start [Expert@HostName]# cpstart
Debug
  1. Start debug:
    for PROC in $(pgrep scrub_cp_file_convertd) ; do fw debug $PROC on TDERROR_ALL_ALL=5 ; done
  2. Replicate the issue.
  3. Stop debug:
    for PROC in $(pgrep scrub_cp_file_convertd) ; do fw debug $PROC off TDERROR_ALL_ALL=0 ; done
  4. Analyze:
    /var/log/jail/$FWDIR/log/scrub_cp_file_convertd.elg*

in.emaild.mta
mtad (MTA bundle R80.10 jhf, R80.20)

Description EMail Security Server that receives emails sent by user and sends them to their destinations.
Path

$FWDIR/bin/fwssd

$FWDIR/bin/mtad (R80.20 and above)

Log file

$FWDIR/log/emaild.mta.elg
/var/log/scrub/in.emaild.mta_messages

$FWDIR/log/mtad.elg (R80.20 and above)

Configuration file $FWDIR/conf/mail_security_config
To Stop [Expert@HostName]# cpstop
To Start [Expert@HostName]# cpstart
Debug

Refer to sk60387:

  1. Start debug:
    fw debug in.emaild.mta off
    fw debug in.emaild.mta on TDERROR_ALL_ALL=5
  2. Replicate the issue.
  3. Stop debug:
    fw debug in.emaild.mta off TDERROR_ALL_ALL=0
  4. Analyze:
    $FWDIR/log/emaild.mta.elg
    $FWDIR/log/mtad.elg (R80.20 and above)

usrchkd

Description Main UserCheck daemon that deals with UserCheck requests (from CLI / from the user) that are sent from the UserCheck Web Portal.
Path $FWDIR/bin/usrchkd
Log file $FWDIR/log/usrchkd.elg
Configuration file $FWDIR/conf/usrchkd.conf
Notes
  • This daemon is not monitored by Check Point WatchDog ("cpwd_admin list")
  • This daemon is spawned by the FWD daemon
To Stop [Expert@HostName]# cpstop
To Start [Expert@HostName]# cpstart
To Restart [Expert@HostName]# killall usrchkd
Debug

Note: It might also be required to collect the relevant kernel debug.

  1. Start debug:
    usrchk debug set all all
  2. Verify:
    usrchk debug stat
  3. Replicate the issue.
  4. Stop debug:
    usrchk debug off
  5. Analyze:
    $FWDIR/log/usrchkd.elg*

usrchk

Description The CLI client for the UserCheck daemon USRCHKD (this process runs only when it is called explicitly).
  Path $FWDIR/bin/usrchk
  Log file $FWDIR/log/usrchk.elg
usercheckportal_php Description UserCheck Web Portal
  Log file /opt/CPUserCheckPortal/log/PortalLog.log
  To Stop [Expert@HostName]# cpstop
  To Start [Expert@HostName]# cpstart
  To Restart [Expert@HostName]# mpclient restart UserCheck
  Configuration Files

/opt/CPUserCheckPortal/phpincs/conf/proxy.ini

/opt/CPUserCheckPortal/phpincs/conf/httpd.conf

/opt/CPUserCheckPortal/phpincs/conf/L10N/portal_en.php

  Debug
  1. Start debug:
    Run /opt/CPUserCheckPortal/scripts/logs_conf_server
    select option 1
  2. Replicate the issue.
  3. Stop Debug:
    Run /opt/CPUserCheckPortal/scripts/logs_conf_server select option 2
  4. Analyze:
    /opt/CPUserCheckPortal/log/PortalLog.log

 

Debug

  1. Add a mark to the log files:

    [Expert@HostName:0]# echo "==debug_start==" >> $FWDIR/log/emaild.mta.elg
    [Expert@HostName:0]# echo "==debug_start==" >> $FWDIR/log/mtad.elg

    [Expert@HostName:0]# echo "==debug_start==" >> $FWDIR/log/scrubd.elg
    [Expert@HostName:0]# echo "==debug_start==" >> /var/log/jail/$FWDIR/log/scrub_cp_file_convertd.elg
  2. Start the debug of the IN.EMAILD.MTA process:

    [Expert@HostName:0]# fw_debug in.emaild.mta off

    [Expert@HostName:0]# fw_debug in.emaild.mta on TDERROR_ALL_ALL=5
  3. Set the SCRUBD debug options:

    [Expert@HostName:0]# scrub debug set all all
  4. Enable the debug of the SCRUBD process:

    [Expert@HostName:0]# scrub debug on
  5. Check the debug status of the SCRUBD process:

    [Expert@HostName:0]# scrub debug stat
  6. Start the debug of the SCRUB_CP_FILE_CONVERTD process:

    [Expert@HostName:0]# for PROC in $(pgrep scruscrub_cp_file_convertdb_cp_file_convertd) ; do fw debug $PROC on TDERROR_ALL_ALL=5 ; done
  7. Replicate the issue.

    Make sure the issue was replicated - save all the relevant outputs, take all the relevant screenshots.
  8. Stop the debug of the IN.EMAILD.MTA process:

    [Expert@HostName:0]# fw_debug in.emaild.mta off TDERROR_ALL_ALL=0
  9. Disable the debug of the SCRUBD process:

    [Expert@HostName:0]# scrub debug off
  10. Reset the SCRUBD debug options to their defaults:

    [Expert@HostName:0]# scrub debug reset
  11. Check the debug status of the SCRUBD process:

    [Expert@HostName:0]# scrub debug stat
  12. Stop the debug of the SCRUB_CP_FILE_CONVERTD process:

    [Expert@HostName:0]# for PROC in $(pgrep scrub_cp_file_convertd) ; do fw debug $PROC off TDERROR_ALL_ALL=0 ; done
  13. Add a mark to the log files:

    [Expert@HostName:0]# echo "==debug_stop==" >> $FWDIR/log/emaild.mta.elg
    [Expert@HostName:0]# echo "==debug_stop==" >> $FWDIR/log/mtad.elg

    [Expert@HostName:0]# echo "==debug_stop==" >> $FWDIR/log/scrubd.elg
    [Expert@HostName:0]# echo "==debug_stop==" >> /var/log/jail/$FWDIR/log/scrub_cp_file_convertd.elg
  14. Collect these files from the Security Gateway:

    • /var/log/maillog
    • $FWDIR/log/emaild.mta.elg*
    • $FWDIR/log/mtad.elg* (R80.20 and above)
    • $FWDIR/log/scrubd.elg*
    • /var/log/scrub/scrubd_messages*
    • $CPDIR/log/scrub_plg.log*
    • /var/log/jail/$FWDIR/log/scrub_cp_file_convertd.elg*
    • /var/log/messages*
    • all the relevant outputs
    • all the relevant screenshots
    • CPinfo file
    In addition, collect the CPinfo file from the Security Management Server / Domain Management Server that manages this Security Gateway

FAQ

Click Here to Show the Entire FAQ

Mail Disclaimer

How to change the email signature of Threat Extraction?
In order to change the default value of the signature, for example:

Connect to the Security Management Server with SmartDashboard/SmartConsule.
Go to 'Threat prevention > Advanced > Engine settings > Threat Extraction settings > Configure mail signatures'.
Link to original file in the email is not recognized as URL, and Link cannot be accessed directly by clicking it. Why does this happen?
This is mail-client dependent. The mail that the user sees is plain text. That is the reason the link isn't seen as clickable. The mail being either html or plain text is determined by the sender's mail-client (or by the recipient client if he chooses all incoming mails to be seen as plain text), and TEX adds the disclaimer to both sections if they exist. This cannot be resolved from the gateway side. It can only be resolved by changing the sender's mail client definitions.
How to change the mail disclaimer text?
On any email handled by Threat Extraction, the gateway adds a disclaimer for each attachment, describing how the attachment was handled (for example: "foo.doc: file(s) were stripped"). To change the default text of these disclaimers, in the gateway go to $FWDIR/conf and open the scrub_debug.conf file. Edit the value of the disclaimer you wish to change. Save and install policy.

MTA incoming files

Change information related to Safe copy files

How to change the filename suffix of the Safe copy file?
See "Granular control of cleaned file name" section in sk114613

what can be done when extracted file size (when using "Convert to PDF") is extremely large?
Change the value of scrub_convertdoc_graphic_output_dpi from 300 to a lower value (200 for example) - it will create smaller pictures in the converted file leading to smaller files.

This parameter is in $FWDIR/conf/file_convert.conf.
Need to install policy after the change, and to kill scrub_cp_file_convertd

Note that a too-low value (25 for example) may lead to corruption of QR/barcode in documents - so keeping a value as high as the customer can tolerate is preferred.
What is the Maximum File Size that Threat Extraction can subscribe in the real world?
The max file size supported is 100M.
Please note it is not only a function of file size, but also a process time (which make be high for complex files). Check Point limits the conversion operation to 30 seconds. When using convert to PDF, the converted file size can be even larger (which may be a limitation on the next-hop mail server). In addition keeping original files depends on disk space. When you increase the supported file size, consider reducing the number of days to keep the original files.

The max file size supported for WEB API and Sandblast for browsers is 15MB. It cannot be changed.
Can you change the time-out value of Threat Extraction?
  1. For timeout bigger or equal to 2 minutes, change in GUIDBedit all values of cpfc_async_timeout_in_seconds to the desired value (in sec). Maximum is 3600 sec = 1 hour
  2. In $FWDIR/conf/file_convert.conf :
    change the value of thread_inactivity_timeout to the desired value [seconds]. Default value is 30.
  3. Install Policy
  4. Run in ssh on gateway:
    kill -9 `pidof scrub_cp_file_convertd`
Note: Increasing the timeout value may increase the used resources and may expose the gateway to Dos Attacks.

Other issues

Threat Extraction has stopped processing files, although the MTA service is working and scrubd and scrub_cp_file_convertd daemons are up. What can I do?
Enable debug on in.emaild.mta and replicate the problem.
Open $FWDIR/log/in.emaild.mta.elg and search for:
"cp_md5_file_sig_create: Failed to open file '/tmp/scrub/<event id>' ".
If found, in the gateway go to /tmp/scrub/. If the folder contains more than 4000 or so email files, the Threat Extraction process may have failed due to lack of temporary storage.
Delete or move to external storage any files no longer needed.
Extraction of additional file types (see sk112240 - How to add support for new file types in Threat Extraction) is not working. What can I do?
First, make sure the instructions in the sk were followed.
In the gateway, open, (NOT in vi) the file $FWDIR/conf/scrub_fixed_file_types and remove any extra linebreaks or lines not in the correct format. Save and install policy.
In the gateway, open, (NOT in vi) the file /var/log/$FWDIR/conf/scrub_supported_file_types and remove any extra linebreaks or lines not in the correct format. Save, install policy and in the gateway run kill -9 'pidof scrub_cp_file_convertd' .

NOTE: Opening any file in vi will automatically add a new line break at the end of the file. Use notepad++ or similar instead.

Related Resources

 

Revision History

Show / Hide the revision history
Date Description
06 Sep 2018
  • First release of this article.

Give us Feedback
Please rate this document
[1=Worst,5=Best]
Comment