Support Center > Search Results > SecureKnowledge Details
ATRG: Threat Extraction Technical Level
Solution

Table of Contents:

  • Introduction
  • Introduction to the Threat Extraction Solution
  • Supported Configuration and Requirements
  • Threat Extraction Workflow
  • User Space
  • Debug
  • FAQ
  • Related Resources
  • Revision History

Introduction

To challenge today's malware landscape, Check Point's comprehensive Threat Prevention solution offers a multi-layered, pre and post-infection defense approach and a consolidated platform that enables enterprise security to deal with modern malware:

Software Blade Introduced in Description Reference
Threat Emulation
  • Gateway mode: R77
  • VSX mode: R77.20

Stops unknown malware, targeted attacks, and zero-day attacks.

Works by:

  1. Identifying files in email attachments (SMTP & SMTP/TLS) and
    downloads over the web (HTTP & HTTPS)
  2. Uploading the suspicious files to a virtual sandbox
    (in the cloud, or on a local appliance) for further emulation and analysis.
  3. Emulating the suspicious files in various OS environments
    by opening the files and monitoring abnormal behavior (related to
    file system, system registry, network connections, system processes, etc.)
  4. Stopping the malicious files, and preventing them from getting to the end user.
  5. Sharing (in real-time) the data about the detected malicious files with ThreatCloud.
sk114806 - ATRG: Threat Emulation
Threat Extraction
  • Gateway mode: R77.30
  • VSX mode: supported from R80.10
Pro-actively cleans potential threats from incoming documents Described in this article
Anti-Bot
  • Gateway mode: R75.40
  • VSX mode: R77.40VS
Post-infection bot detection, prevention, and threat visibility sk92264 - ATRG: Anti-Bot and Anti-Virus
Anti-Virus
  • NG AI R57

Pre-infection blocking of known viruses and file transfers.

Works by:

  • Looking for specific patterns.
  • Enforcing compliance of protocols to standards.
  • Detecting variations from the protocols.
sk92264 - ATRG: Anti-Bot and Anti-Virus

Each Threat Prevention Software Blade gives unique network protections and they can be combined to supply a strong malware solution.

Data from malicious attacks are shared between the Threat Prevention Software Blades and help keep your network safe.
For example, the signature from a threat that is identified by Threat Emulation is added to the Anti-Virus database.

The Threat Prevention Software Blades use a separate policy installation to minimize risk and operational impact.
They are also integrated with other Software Blades on the Security Gateway to detect and stop threats.

Introduction to the Threat Extraction Solution

Threat Extraction is a new Software Blade in the Threat Prevention family that pro-actively cleans potential threats from incoming documents.

SandBlast Threat Extraction prevents both known and unknown threats before they arrive at the organization, thus providing better protection against zero-day threats. Threat Extraction gives organizations the necessary protection against unknown threats in files that are downloaded from the Internet, or attached to emails.

Supported Configuration and Requirements

  1. In R77.30 release: Requires the R77.30 Add-on to be installed and enabled on the Security Management Server / Multi-Domain Security Management Server.

  2. Scanning attachements from incoming emails requires MTA configuration

  3. Scanning  files downloaded from the internent is supported using SandBlast Agent for Browsers

  4. Threat extraction is supported in VSX mode from R80.10. Refer to the "Using Threat Extraction with VSX" section in the Threat Prevention R80.10 Administration Guide

  5. CXL support for Threat Extraction is available in from R80.20.

Threat Extraction Workflow

In MTA mode:

  1. A PostFix server receives and handles the emails.

  2. Emails are forwarded to the in.emaild.mta daemon, which:
    1. Parses the emails (For example, Base64 decode)
    2. Passes the attachments to the scrubd process, if needed (based on the configuration of supported file types).

  3. The scrubd process handles the file and sends it to the scrub_cp_file_convertd process with the relevant details (according to the policy).

  4. scrub_cp_file_convertd process
    1. Converts the file / extracts potentialy malicious content from it. 
    2. Returns a Safe copy of the file to scrubd.

  5. The scrubd process returns the Safe copy to the in.emaild.mta daemon

    1. The in.emaild.mta daemon:
       
      1. Replaces the original attachment with the Safe Copy version.

      2. Forwards the email to its destination.

    Note: For environments with MTA bundle R80.10 jhf or R80.20, in.emaild.mta is replaced with mtad daemon

    User Space

      Daemon Section Description / Paths / Notes / Stop and Start Commands / Debug

      scrub

      Description The CLI client for the scrubd daemon (this process runs only when it is called explicitly).
      Path $FWDIR/bin/scrub

      scrubd

      Description Main Threat Extraction daemon.
      Path $FWDIR/bin/scrubd
      Log file $FWDIR/log/scrubd.elg
      /var/log/scrub/scrubd_messages
      Configuration file $FWDIR/conf/scrub_debug.conf
      To Stop [Expert@HostName]# cpstop
      To Start [Expert@HostName]# cpstart
      Debug
      1. Start Threat Extraction debug:
        scrub debug on
        scrub debug set all all
      2. Verify Threat Extraction debug is enabled:
        scrub debug stat
      3. Start debug of scrubd daemon:
        scrub debug set all all
      4. Replicate the issue.
      5. Stop debug of scrubd daemon:
        scrub debug off
        scrub debug reset
      6. Verify Threat Extraction debug is disabled:
        scrub debug stat
      7. Analyze:
        $FWDIR/log/scrubd.elg*

      scrub_cp_file_convertd

      Description Used to convert-to-PDF, or extract potentialy malicious content from various file formats. 
      Path $FWDIR/bin/cp_file_convert
      Log file /var/log/jail/$FWDIR/log/scrub_cp_file_convertd.elg
      Configuration file

       $FWDIR/conf/file_convert.conf

      After changing this conf, follow these steps:

      1. Install policy.

      2. ssh to the gateway

      3. kill -9 `pidof scrub_cp_file_convertd`

      To Stop [Expert@HostName]# cpstop
      To Start [Expert@HostName]# cpstart
      Debug
      1. Start debug:
        for PROC in $(pgrep scrub_cp_file_convertd) ; do fw debug $PROC on TDERROR_ALL_ALL=5 ; done
      2. Replicate the issue.
      3. Stop debug:
        for PROC in $(pgrep scrub_cp_file_convertd) ; do fw debug $PROC off TDERROR_ALL_ALL=0 ; done
      4. Analyze:
        /var/log/jail/$FWDIR/log/scrub_cp_file_convertd.elg*

      in.emaild.mta
      mtad (MTA bundle R80.10 jhf, R80.20)

      Description EMail Security Server that receives emails sent by user and sends them to their destinations.
      Path

      $FWDIR/bin/fwssd

      $FWDIR/bin/mtad (R80.20 and higher)

      Log file

      $FWDIR/log/emaild.mta.elg
      /var/log/scrub/in.emaild.mta_messages

      $FWDIR/log/mtad.elg (R80.20 and higher)

      Configuration file $FWDIR/conf/mail_security_config
      To Stop [Expert@HostName]# cpstop
      To Start [Expert@HostName]# cpstart
      Debug

      Refer to sk60387:

      1. Start debug:
        fw debug in.emaild.mta off
        fw debug in.emaild.mta on TDERROR_ALL_ALL=5
      2. Replicate the issue.
      3. Stop debug:
        fw debug in.emaild.mta off TDERROR_ALL_ALL=0
      4. Analyze:
        $FWDIR/log/emaild.mta.elg
        $FWDIR/log/mtad.elg (R80.20 and higher)

      usrchkd

      Description Main UserCheck daemon that deals with UserCheck requests (from CLI / from the user) that are sent from the UserCheck Web Portal.
      Path $FWDIR/bin/usrchkd
      Log file $FWDIR/log/usrchkd.elg
      Configuration file $FWDIR/conf/usrchkd.conf
      Notes
      • This daemon is not monitored by Check Point WatchDog ("cpwd_admin list")
      • This daemon is spawned by the FWD daemon
      To Stop [Expert@HostName]# cpstop
      To Start [Expert@HostName]# cpstart
      To Restart [Expert@HostName]# killall usrchkd
      Debug

      Note: It might also be required to collect the relevant kernel debug.

      1. Start debug:
        usrchk debug set all all
      2. Verify:
        usrchk debug stat
      3. Replicate the issue.
      4. Stop debug:
        usrchk debug off
      5. Analyze:
        $FWDIR/log/usrchkd.elg*

      usrchk

      Description The CLI client for the UserCheck daemon USRCHKD (this process runs only when it is called explicitly).
        Path $FWDIR/bin/usrchk
        Log file $FWDIR/log/usrchk.elg
      usercheckportal_php Description UserCheck Web Portal
        Log file /opt/CPUserCheckPortal/log/PortalLog.log
        To Stop [Expert@HostName]# cpstop
        To Start [Expert@HostName]# cpstart
        To Restart [Expert@HostName]# mpclient restart UserCheck
        Configuration Files

      /opt/CPUserCheckPortal/phpincs/conf/proxy.ini

      /opt/CPUserCheckPortal/phpincs/conf/httpd.conf

      /opt/CPUserCheckPortal/phpincs/conf/L10N/portal_en.php

        Debug
      1. Start debug:
        Run /opt/CPUserCheckPortal/scripts/logs_conf_server
        select option 1
      2. Replicate the issue.
      3. Stop Debug:
        Run /opt/CPUserCheckPortal/scripts/logs_conf_server select option 2
      4. Analyze:
        /opt/CPUserCheckPortal/log/PortalLog.log

       

      Debug

      1. Add a mark to the log files:

        [Expert@HostName:0]# echo "==debug_start==" >> $FWDIR/log/emaild.mta.elg
        [Expert@HostName:0]# echo "==debug_start==" >> $FWDIR/log/mtad.elg

        [Expert@HostName:0]# echo "==debug_start==" >> $FWDIR/log/scrubd.elg
        [Expert@HostName:0]# echo "==debug_start==" >> /var/log/jail/$FWDIR/log/scrub_cp_file_convertd.elg
      2. Start the debug of the IN.EMAILD.MTA process:

        [Expert@HostName:0]# fw_debug in.emaild.mta off

        [Expert@HostName:0]# fw_debug in.emaild.mta on TDERROR_ALL_ALL=5
      3. Set the SCRUBD debug options:

        [Expert@HostName:0]# scrub debug set all all
      4. Enable the debug of the SCRUBD process:

        [Expert@HostName:0]# scrub debug on
      5. Check the debug status of the SCRUBD process:

        [Expert@HostName:0]# scrub debug stat
      6. Start the debug of the SCRUB_CP_FILE_CONVERTD process:

        [Expert@HostName:0]# for PROC in $(pgrep scruscrub_cp_file_convertdb_cp_file_convertd) ; do fw debug $PROC on TDERROR_ALL_ALL=5 ; done
      7. Replicate the issue.

        Make sure the issue was replicated - save all the relevant outputs, take all the relevant screenshots.
      8. Stop the debug of the IN.EMAILD.MTA process:

        [Expert@HostName:0]# fw_debug in.emaild.mta off TDERROR_ALL_ALL=0
      9. Disable the debug of the SCRUBD process:

        [Expert@HostName:0]# scrub debug off
      10. Reset the SCRUBD debug options to their defaults:

        [Expert@HostName:0]# scrub debug reset
      11. Check the debug status of the SCRUBD process:

        [Expert@HostName:0]# scrub debug stat
      12. Stop the debug of the SCRUB_CP_FILE_CONVERTD process:

        [Expert@HostName:0]# for PROC in $(pgrep scrub_cp_file_convertd) ; do fw debug $PROC off TDERROR_ALL_ALL=0 ; done
      13. Add a mark to the log files:

        [Expert@HostName:0]# echo "==debug_stop==" >> $FWDIR/log/emaild.mta.elg
        [Expert@HostName:0]# echo "==debug_stop==" >> $FWDIR/log/mtad.elg

        [Expert@HostName:0]# echo "==debug_stop==" >> $FWDIR/log/scrubd.elg
        [Expert@HostName:0]# echo "==debug_stop==" >> /var/log/jail/$FWDIR/log/scrub_cp_file_convertd.elg
      14. Collect these files from the Security Gateway:

        • /var/log/maillog
        • $FWDIR/log/emaild.mta.elg*
        • $FWDIR/log/mtad.elg* (R80.20 and higher)
        • $FWDIR/log/scrubd.elg*
        • /var/log/scrub/scrubd_messages*
        • $CPDIR/log/scrub_plg.log*
        • /var/log/jail/$FWDIR/log/scrub_cp_file_convertd.elg*
        • /var/log/messages*
        • all the relevant outputs
        • all the relevant screenshots
        • CPinfo file
        In addition, collect the CPinfo file from the Security Management Server / Domain Management Server that manages this Security Gateway

      FAQ

      Click Here to Show the Entire FAQ

      Mail Disclaimer

      How to change the email signature of Threat Extraction?
      In order to change the default value of the signature, for example:

      Connect to the Security Management Server with SmartDashboard/SmartConsule.
      Go to 'Threat prevention > Advanced > Engine settings > Threat Extraction settings > Configure mail signatures'.
      Link to original file in the email is not recognized as URL, and Link cannot be accessed directly by clicking it. Why does this happen?
      This is mail-client dependent. The mail that the user sees is plain text. That is the reason the link isn't seen as clickable. The mail being either html or plain text is determined by the sender's mail-client (or by the recipient client if he chooses all incoming mails to be seen as plain text), and TEX adds the disclaimer to both sections if they exist. This cannot be resolved from the gateway side. It can only be resolved by changing the sender's mail client definitions.
      How to change the mail disclaimer text?
      On any email handled by Threat Extraction, the gateway adds a disclaimer for each attachment, describing how the attachment was handled (for example: "foo.doc: file(s) were stripped"). To change the default text of these disclaimers, in the gateway go to $FWDIR/conf and open the scrub_debug.conf file. Edit the value of the disclaimer you wish to change. Save and install policy.

      MTA incoming files

      Change information related to Safe copy files

      How to change the filename suffix of the Safe copy file?
      See "Granular control of cleaned file name" section in sk114613

      what can be done when extracted file size (when using "Convert to PDF") is extremely large?
      Change the value of scrub_convertdoc_graphic_output_dpi from 300 to a lower value (200 for example) - it will create smaller pictures in the converted file leading to smaller files.

      This parameter is in $FWDIR/conf/file_convert.conf.
      Need to install policy after the change, and to kill scrub_cp_file_convertd

      Note that a too-low value (25 for example) may lead to corruption of QR/barcode in documents - so keeping a value as high as the customer can tolerate is preferred.
      What is the Maximum File Size that Threat Extraction can subscribe in the real world?
      The max file size supported is 100M.
      Please note it is not only a function of file size, but also a process time (which make be high for complex files). Check Point limits the conversion operation to 30 seconds. When using convert to PDF, the converted file size can be even larger (which may be a limitation on the next-hop mail server). In addition keeping original files depends on disk space. When you increase the supported file size, consider reducing the number of days to keep the original files.

      The max file size supported for WEB API and Sandblast for browsers is 15MB. It cannot be changed.
      Can you change the time-out value of Threat Extraction?
      1. For timeout bigger or equal to 2 minutes, change in GUIDBedit all values of cpfc_async_timeout_in_seconds to the desired value (in sec). Maximum is 3600 sec = 1 hour
      2. In $FWDIR/conf/file_convert.conf :
        change the value of thread_inactivity_timeout to the desired value [seconds]. Default value is 30.
      3. Install Policy
      4. Run in ssh on gateway:
        kill -9 `pidof scrub_cp_file_convertd`
      Note: Increasing the timeout value may increase the used resources and may expose the gateway to Dos Attacks.

      Other issues

      Threat Extraction has stopped processing files, although the MTA service is working and scrubd and scrub_cp_file_convertd daemons are up. What can I do?
      Enable debug on in.emaild.mta and replicate the problem.
      Open $FWDIR/log/in.emaild.mta.elg and search for:
      "cp_md5_file_sig_create: Failed to open file '/tmp/scrub/<event id>' ".
      If found, in the gateway go to /tmp/scrub/. If the folder contains more than 4000 or so email files, the Threat Extraction process may have failed due to lack of temporary storage.
      Delete or move to external storage any files no longer needed.
      Extraction of additional file types (see sk112240 - How to add support for new file types in Threat Extraction) is not working. What can I do?
      First, make sure the instructions in the sk were followed.
      In the gateway, open, (NOT in vi) the file $FWDIR/conf/scrub_fixed_file_types and remove any extra linebreaks or lines not in the correct format. Save and install policy.
      In the gateway, open, (NOT in vi) the file /var/log/$FWDIR/conf/scrub_supported_file_types and remove any extra linebreaks or lines not in the correct format. Save, install policy and in the gateway run kill -9 'pidof scrub_cp_file_convertd' .

      NOTE: Opening any file in vi will automatically add a new line break at the end of the file. Use notepad++ or similar instead.

      Related Resources

       

      Revision History

      Show / Hide the revision history
      Date Description
      06 Sep 2018
      • First release of this article.

      Give us Feedback
      Please rate this document
      [1=Worst,5=Best]
      Comment