Support Center > Search Results > SecureKnowledge Details
High CPU / process crashes / timeout due to large database / first time operations / load on the machine
Solution

Table of Contents 

  • Overview
  • Common Solution
  • Various Scenarios
    • Scenario 1: High CPU load at around 90%-100% caused by fw daemon on Security Gateway
    • Scenario 2: Performance and policy installation issues on Multi-Domain Management
    • Scenario 3: Policy installation or Global policy assignment/re-assignment fails
    • Scenario 4: Policy installation takes a long time
    • Scenario 5: "Operation incomplete due to timeout" error on policy installation
    • Scenario 6: When trying to install policy to a large number of gateways at the same time, the installation times out
    • Scenario 7: In Multi-Domain Management / Provider-1, assigning a Global Policy from the Multi-Domain Server (MDS) GUI to a CMA fails with errors
    • Scenario 8: Policy installation on Correlation Unit fails
    • Scenario 9: Policy compilation for Edge device is failing with error: "Can't contact database"
    • Scenario 10: User Activity Report is always limited to 42 Pages
    • Scenario 11: Synchronization Status for all HA Domains are flapping between Synced and Not Synced
    • Scenario 12: Status of managed Security Gateways in SmartView Monitor changes every few minutes, from OK to Disconnected and back to OK
    • Scenario 13: "Machine is not configured as 'SmartEvent' on Management. Exiting..." error when connecting to SmartEvent NGSE client
    • Scenario 14: CPSTAT or 3rd Part OPSEC client cannot read more than 3000 routes from the routing table
    • Scenario 15: ROBO gateways disappear from SmartLSM
Click Here to Show the Entire Article

Overview

The size of the Communication Buffer is hard-coded to 1MB-4MB depending on the version (For R77.10 and above: 4MB).

Under some circumstances, this size might not be enough.

In order to control the size of this buffer, a new environment variable called FWASYNC_MAXBUF was added to the code.

Support for the new environment variable FWASYNC_MAXBUF is included from Check Point R77.10

Check Point recommends to always upgrade to the most recent version.

For lower / other versions, Check Point can supply a Hotfix. Contact Check Point Support to get a Hotfix for this issue. A Support Engineer will make sure the Hotfix is compatible with your environment before providing the Hotfix.

Note: There are cases in which FWASYNC_MAXBUF is modified on the Security Gateway.

Common Solution

  • On Security Management Server

    1. Stop Check Point services:

      [Expert@HostName]# cpstop

    2. vi $CPDIR/tmp/.CPprofile.sh:

      Add this line:

      FWASYNC_MAXBUF=new_size_in_bytes ; export FWASYNC_MAXBUF

      Under this line

      INFODIR=/opt/CPinfo-10 ; export INFODIR

      Notes:
      • It is strongly suggested to set FWASYNC_MAXBUF=40000000 (or more, according to what the allocation failed on, i.e. if it failed on 40MB, then set to 80MB)

      • The line "INFODIR=/opt/CPinfo-10 ; export INFODIR" may not exist exactly as written. In later versions, the line appears like this: "_cpprof_add INFODIR "/opt/CPinfo-10" -1 0"

        So if you see this format, you would just edit the $CPDIR/tmp/.CPprofile.sh file, adding the new line after the "infodir" line to look like this:

        _cpprof_add INFODIR "/opt/CPinfo-10" -1 0
        FWASYNC_MAXBUF=40000000 ; export FWASYNC_MAXBUF

    3. Start Check Point services from another shell:

      [Expert@HostName]# cpstart

    4. Execute the shell script:

      [Expert@HostName]# . $CPDIR/tmp/.CPprofile.sh

      Note: Include the leading period and space before $CPDIR. (Refreshing the shell is required before the new value will appear. Without doing so, the output will likely be a blank line to indicate the default.)

    5. Check that the value was set to 40000000:

      [Expert@HostName]# echo $FWASYNC_MAXBUF

  • On Provider-1 Server / Multi-Domain Security Management Server

    1. Stop Provider-1 Server / Domain Management Server:

      [Expert@HostName]# mdsstop

    2. Edit the shell script:

      [Expert@HostName]# vi $CPDIR/tmp/.CPprofile.sh:

      Add this line:

      FWASYNC_MAXBUF=new_size_in_bytes ; export FWASYNC_MAXBUF

      Under this line

      INFODIR=/opt/CPinfo-10 ; export INFODIR

      Notes:
      • It is strongly suggested to set FWASYNC_MAXBUF=40000000 (or more according to what the allocation failed on, i.e. if it failed on 40MB, then set to 80MB)

      • The line "INFODIR=/opt/CPinfo-10 ; export INFODIR" may not exist exactly as written. In later versions, the line appears like this: "_cpprof_add INFODIR "/opt/CPinfo-10" -1 0"

        So if you see this format, you would just edit the $CPDIR/tmp/.CPprofile.sh file, adding the new line after the "infodir" line to look like this:

        _cpprof_add INFODIR "/opt/CPinfo-10" -1 0
        FWASYNC_MAXBUF=40000000 ; export FWASYNC_MAXBUF


        After doing so, you would have to perform mdsstop, then mdsstart.

    3. Start Provider-1 Server/ Domain Management Server from another shell:

      [Expert@HostName]# mdsstart

    4. Execute the shell script:

      [Expert@HostName]# . $CPDIR/tmp/.CPprofile.sh

      Note: Include the leading period and space before $CPDIR. (Refreshing the shell is required before the new value will appear. Without doing so, the output will likely be a blank line to indicate the default.)

    5. Check that the value was set to 40000000:

      [Expert@HostName]# echo $FWASYNC_MAXBUF

Various Scenarios

There are quite a number of scenarios, in which you may need to modify FWASYNC_MAXBF to increase the Communication Buffer size. The scenarios that we have encountered and dealt with are detailed below.

Scenario 1: High CPU load at around 90%-100% caused by fw daemon on Security Gateway

Product: Security Gateway, Security Management, Multi-Domain Management / Provider-1

Symptoms:

  • CPU load on Security Gateway is around 90%-100%. According to output of top command, the load is caused by fwdaemon (which in fact means fwd daemon).

  • If the fwd daemon on Security Gateway is killed manually and then restarted by CP WatchDog, the CPU load goes back to normal.

  • FWD daemon debug (per sk86321) shows:
    fwasync_connbuf_realloc: Connection buffer overflow. Cannot allocate X bytes of memory. Limit is Y.
    fwasync_conn_pre_send_check: Internal error. Failed to enlarge connection buffer.

Cause:

The size of the Communication Buffer is hard-coded to 1MB-4MB depending on the version (R77.10 and above 4MB).

Under some circumstances, this size might not be enough.

Solution:

In order to control the size of this buffer, a new environment variable called FWASYNC_MAXBUF was added to the code.

Hotfix installation instructions:

  1. The hotfix has to be installed on the Security Management Server / Multi-Domain Security Management Server.

    Note: In a Management HA environment, this procedure must be performed on both Management Servers.

  2. Transfer the hotfix package to the machine (e.g., into /some_path_to_fix/).

  3. Unpack the hotfix package:

    [Expert@HostName]# cd /some_path_to_fix/
    [Expert@HostName]# tar -xvfz fw1_HOTFIX_NAME.tgz
  4. Install the hotfix:

    [Expert@HostName]# ./fw1_HOTFIX_NAME

  5. Do NOT reboot the machine yet.
  6. Configure the desired buffer size:

    Show / Hide buffer size configuration instructions for Security Manag Server

      Refer to Common Solution (as performed on Security Management Server)


  7. Reboot the machine.

Scenario 2: Performance and policy installation issues on Multi-Domain Management

Product: Multi-Domain Management / Provider-1

Symptoms:

  • Performance of the MDS is reduced:
    • It takes a lot of time to install the policy (if it possible at all)
    • It takes a lot of time to launch the SmartDashboard of a specific CMA / Domain
    • CLM cannot be accessed with SmartLog orSmartView Tracker, but mdsstat shows state as "up"

  • Every several days, the Multi-Domain Management machine (MDS) cannot perform some functions, such as:
    • Policy installation to specific CMA
    • Reassigning of the Global Policy on MDS

Cause:

The fwm process does not get enough memory from the system in order to complete its tasks.
There is a certain buffer that is in charge of the memory allocation for fwm operations:

  • Receiving updates from the Security Gateway it manages
  • Communicating with the MDS it is hosted on - sending notifications to the MDS
  • Communicating with the MDS it is hosted on - answering and processing queries from the MDS
  • Communicating with the GUI (policy verifications, objects management, etc.)
  • Communicating with the different CP processes (CPD, FWD, CPCA, etc.)


By default, the system allocates a hard-coded size to this buffer for the fwm process. This buffer is used for memory allocation to all of the functions running for that matter.

In case of large databases, this buffer is exhausted over time. After a few days, the rate of allocations overcomes the rate of de-allocations, so the buffer gets filled up. Then you experience timeouts, GUI delays, allocation errors and policy installation failures.

Solution:

Refer to Common Solution

Scenario 3: Policy installation or Global policy assignment/re-assignment fails

Product: Security Management, Multi-Domain Management / Provider-1

Symptoms:

  • "Failed to send reply on session" error on policy installation failure.

  • Debug of fwm process shows:

    fwasync_connbuf_realloc: Connection buffer overflow. Cannot allocate 1106190 bytes of memory. Limit is 1048576.
    fwasync_conn_pre_send_check: Internal error. Failed to enlarge connection buffer.
    OPSEC_SET_ERRNO: err = 12 Failed to malloc (pre = 4)
    opsec_comm_send_datagram: COMM b18fa68 switched to congested state
    OPSEC_SET_ERRNO: err = 0 NO Error (pre = 12)
    COMM b18fa68 is queuing frag 17/18 of DG_TYPE=f01
    OPSEC_SET_ERRNO: err = 18 Communication path is congested (pre = 12)
    fwasync_connected: 39: getpeername: Transport endpoint is not connected
    cpmi_request_send: send datagram failed on session



  • Global policy assignment/re-assignment fails for one or several CMAs

  • cpca process down when trying to create secondary CMA.

  • Creation of seconday CMA fails with "The Mirror CMA has been created, but SIC has not been initialized between the Primary and the Mirror CMAs. Please initialize SIC manually" message.

Refer to Common Solution

Scenario 4: Policy installation takes a long time

Product: Security Management, Multi-Domain Management / Provider-1

Symptoms:

  • Debug of fwm daemon (per sk86186) when policy installation takes a long time shows:
    fwasync_conn_get: get max buffer size (1048576)
    fwasync_connbuf_realloc: Connection buffer overflow. Cannot allocate 1059399 bytes of memory. Limit is 1048576.

Cause:

There are several possible reasons for this problem to take place:

  1. The new software blades use additional processes. This causes higher memory consumption on the Security Gateway. As a result, the policy installation time increases.
  2. The size of the Communication Buffer may be too small.

Solution:

Follow these steps:

  1. In SmartDashboard: open the Security Gateway object and disable unused Software Blades (make sure that only essential Software Blades are enabled on the Security Gateway) and install the policy.
  2. If the issue persists, increase the size of the Communication Buffer on the Security Management Server / Multi-Domain Security Management Server.

    The size of the special buffer is controlled through an environment variable for the operating system - FWASYNC_MAXBUF:

    In order to set the value permanently, it has to be added manually to all Check Point shell scripts that define environment variables for the given operating system:

    • Show / Hide instructions for Security Management Server, which runs on Gaia / SecurePlatform / Linux / IPSO OS

      1. In the $CPDIR/tmp/.CPprofile.sh script:

        Add this line:
        FWASYNC_MAXBUF=new_size_in_bytes ; export FWASYNC_MAXBUF

        Under this line
        INFODIR=/opt/CPinfo-10 ; export INFODIR

        Note: The line "INFODIR=/opt/CPinfo-10 ; export INFODIR" may not exist exactly as written. In later versions, the line appears like this: "_cpprof_add INFODIR "/opt/CPinfo-10" -1 0"

        So if you see this format, you would just edit the $CPDIR/tmp/.CPprofile.sh file, adding the new line after the "infodir" line to look like this:

        _cpprof_add INFODIR "/opt/CPinfo-10" -1 0
        FWASYNC_MAXBUF=40000000 ; export FWASYNC_MAXBUF


        After doing so, you would have to perform cpstop, then cpstart.
      2. In the  $CPDIR/tmp/.CPprofile.csh script:

      3. Add this line:
        setenv FWASYNC_MAXBUF new_size_in_bytes

        Under this line:
        setenv INFODIR "/opt/CPinfo-10"
      4. Restart all Check Point services:

        [Expert@HostName]# cpstop ; cpstart

        Important Note: Any change in the value of FWASYNC_MAXBUF (either on-the-fly, or via the scripts) will take effect only after the fwd daemon is restarted. If you are not able to restart the Check Point services at this time, then run these commands:

        • In the Bash shell (default shell for Expert mode):

          [Expert@HostName]# export FWASYNC_MAXBUF=new_size_in_bytes
          [Expert@HostName]# echo $FWASYNC_MAXBUF
          [Expert@HostName]# cpwd_admin stop -name FWD -path "$FWDIR/bin/fw" -command "fw kill fwd"
          [Expert@HostName]# cpwd_admin start -name FWD -path "$FWDIR/bin/fw" -command "fwd"
        • In the Csh shell:

          [admin@HostName ~]# setenv FWASYNC_MAXBUF new_size_in_bytes
          [admin@HostName ~]# echo $FWASYNC_MAXBUF
          [admin@HostName ~]# cpwd_admin stop -name FWD -path "$FWDIR/bin/fw" -command "fw kill fwd"
          [admin@HostName ~]# cpwd_admin start -name FWD -path "$FWDIR/bin/fw" -command "fwd"

      Notes:

      • To unset this variable permanently, remove it from the above shell scripts and restart Check Point services with 'cpstop;cpstart' commands.
      • To unset this variable on-the-fly (in the current shell session):

        • In the Bash shell (default shell for Expert mode):

          [Expert@HostName]# unset FWASYNC_MAXBUF
          [Expert@HostName]# echo $FWASYNC_MAXBUF
          [Expert@HostName]# cpwd_admin stop -name FWD -path "$FWDIR/bin/fw" -command "fw kill fwd"
          [Expert@HostName]# cpwd_admin start -name FWD -path "$FWDIR/bin/fw" -command "fwd"
        • In the Csh shell:

          [admin@HostName ~]# unsetenv FWASYNC_MAXBUF
          [admin@HostName ~]# echo $FWASYNC_MAXBUF
          [admin@HostName ~]# cpwd_admin stop -name FWD -path "$FWDIR/bin/fw" -command "fw kill fwd"
          [admin@HostName ~]# cpwd_admin start -name FWD -path "$FWDIR/bin/fw" -command "fwd"


    • Show / Hide instructions for Security Management Server, which runs on Windows OS

      1. In Start menu, run... - "%WINDIR%\system32\rundll32.exe" sysdm.cpl,EditEnvironmentVariables and click "OK"

        Alternatively:

        1. Desktop - right-click on the 'My Computer' icon and click on 'Properties'
        2. Windows (Vista / 7 / 2012): In the left pane, click on 'Advanced system settings'.
        3. Go to 'Advanced' tab.
        4. At the bottom, click on the "Environment Variables..." button.


      2. Under 'System Variables', click on "New..."

        • name: FWASYNC_MAXBUF
        • value: integer between 1 and 10
        • and click 'OK' to close the 'New System Variable' window.


      3. Click "OK" to close the 'Environment Variables' window.
      4. Reboot the machine.

      Notes:

      • To unset this variable permanently, under 'System Variables' click on this variable, click "Delete" and reboot the machine.
      • To unset this variable on-the-fly (in the current shell session):

        Run this command in Windows Command Prompt:
        C:\> set FWASYNC_MAXBUF=

        And restart the FWD daemon:
        C:\> cpwd_admin stop -name FWD -path "%FWDIR%\bin\fw" -command "fw kill fwd"
        C:\> cpwd_admin start -name FWD -path "%FWDIR%\bin\fw" -command "fwd"


    • Show / Hide instructions for Multi-Domain Security Management Server, which runs on Gaia / SecurePlatform / Linux OS

      1. Go to context of MDS:

        [Expert@HostName]# mdsenv
        [Expert@HostName]# mcd
      2. In $MDSDIR/scripts/mdsstart_customer script:

        Add this line:
        setenv FWASYNC_MAXBUF new_size_in_bytes

        Above the last line:
        exit 0
      3. In $CPDIR/tmp/.CPprofile.sh (/var/opt/CPshrd-Rxx/tmp/.CPprofile.sh) script:

        Add this line:
        FWASYNC_MAXBUF=new_size_in_bytes ; export FWASYNC_MAXBUF

        Under this line
        INFODIR=/opt/CPinfo-10 ; export INFODIR

        Note: The line "INFODIR=/opt/CPinfo-10 ; export INFODIR" may not exist exactly as written. In later versions, the line appears like this: "_cpprof_add INFODIR "/opt/CPinfo-10" -1 0"

        So if you see this format, you would just edit the $CPDIR/tmp/.CPprofile.sh file, adding the new line after the "infodir" line to look like this:

        _cpprof_add INFODIR "/opt/CPinfo-10" -1 0
        FWASYNC_MAXBUF=40000000 ; export FWASYNC_MAXBUF


        After doing so, you would have to perform mdsstop, then mdsstart.
      4. In $CPDIR/tmp/.CPprofile.csh (/var/opt/CPshrd-Rxx/tmp/.CPprofile.csh) script:

        Add this line:
        setenv FWASYNC_MAXBUF new_size_in_bytes

        Under this line:
        setenv INFODIR "/opt/CPinfo-10"

      5. Restart all Check Point services on Multi-Domain Security Management Server:

        [Expert@HostName]# mdsstop
        [Expert@HostName]# mdsstart

        Important Note: Any change in the value of FWASYNC_MAXBUF (either on-the-fly, or via the scripts) will take effect only after the fwd daemon is restarted. If you are not able to restart the Check Point services at this time, then run these commands:

        • In the Bash shell (default shell for Expert mode):

          [Expert@HostName]# export FWASYNC_MAXBUF=new_size_in_bytes
          [Expert@HostName]# echo $FWASYNC_MAXBUF
          [Expert@HostName]# cpwd_admin stop -name FWD -path "$FWDIR/bin/fw" -command "fw kill fwd"
          [Expert@HostName]# cpwd_admin start -name FWD -path "$FWDIR/bin/fw" -command "fwd"
        • In the Csh shell:

          [admin@HostName ~]# setenv FWASYNC_MAXBUF new_size_in_bytes
          [admin@HostName ~]# echo $FWASYNC_MAXBUF
          [admin@HostName ~]# cpwd_admin stop -name FWD -path "$FWDIR/bin/fw" -command "fw kill fwd"
          [admin@HostName ~]# cpwd_admin start -name FWD -path "$FWDIR/bin/fw" -command "fwd"

      Notes:

      • To unset this variable permanently, remove it from the above shell scripts and restart Check Point services with 'mdsstop;mdsstart' commands.
      • To unset this variable on-the-fly (in the current shell session):

        • In the Bash shell (default shell for Expert mode):

          [Expert@HostName]# unset FWASYNC_MAXBUF
          [Expert@HostName]# echo $FWASYNC_MAXBUF
          [Expert@HostName]# cpwd_admin stop -name FWD -path "$FWDIR/bin/fw" -command "fw kill fwd"
          [Expert@HostName]# cpwd_admin start -name FWD -path "$FWDIR/bin/fw" -command "fwd"
        • In the Csh shell:

          [admin@HostName ~]# unsetenv FWASYNC_MAXBUF
          [admin@HostName ~]# echo $FWASYNC_MAXBUF
          [admin@HostName ~]# cpwd_admin stop -name FWD -path "$FWDIR/bin/fw" -command "fw kill fwd"
          [admin@HostName ~]# cpwd_admin start -name FWD -path "$FWDIR/bin/fw" -command "fwd"

Related Scenarios:

  • Scenario 1: High CPU load at around 90%-100% caused by fw daemon on Security Gateway

Scenario 5: "Operation incomplete due to timeout" error on policy installation

Product: Security Management, Multi-Domain Management / Provider-1

Symptoms:

  • Policy install fails with "Operation incomplete due to timeout" error and SIC is broken with the Security Gateway.

  • When installing Policy on a group of Security Gateways, some of the Security Gateways do receive the updated policy.

  • SIC is restored after removing the policy using fw unloadlocal

Cause:

There are possible several reasons for this problem:

  1. Excessive memory use by fwm and fw_loader processes due to large number of configured satellite gateways.

  2. Anti-Spoofing is incorrectly configured.
    Security Management traffic, such as SIC, is accepted by implied rules, preceeding rules created in the firewall policy. Anti-Spoofing however will drop the traffic, if not configured correctly. Running the fw ctl debug + drop kernel debug will show drops due to Anti-spoofing.

  3. The connection to the Security Management server times-out since the "install_policy_timeout" value in the Management database is not high enough.
    When installing policy on a group of gateways, if one gateway reached the time-out connection limit, the install will show error for all other gateways, even if some installs completed successfully.

  4. An incorrect 'Hardware' or 'Version' is selected in the 'Platform' field in the Security Gateway/Cluster object in SmartDashboard.

Solution:

  1. For the excessive memory usage problem, this problem was fixed. The fix is included in:

    Check Point recommends to always upgrade to the most recent version (upgrade Security Gateway / upgrade Security Management Server / upgrade Multi-Domain Security Management Server).

    If you do not wish to upgrade, contact Check Point Support to get a Hotfix for this issue.
    A Support Engineer will make sure the Hotfix is compatible with your environment before providing the Hotfix.
    For faster resolution and verification please collect CPinfo files from the Security Management and Security Gateways involved in the case.



  2. Define Anti-Spoofing correctly to avoid any subsequent drops.

    To install the new policy with the correct Anti-Spoofing definition, the current policy needs to be removed by using the fw unloadlocal command.


  3. for the install_policy_timeout value problem, there are two possible solutions; try them one by one:

    A. On the problematic Security Management Server:

    1. Close all SmartConsole windows (SmartDashboard, SmartView Tracker, SmartView Monitor, etc.).
    2. Connect to the Security Management server database with GuiDBedit (see sk13009)
    3. Search for the value install_policy_timeout and change it from "600" to "2000" (value in seconds)
    4. Save all and exit.
    5. Remove all GUI cache files:
      # cpstop
      # mv $FWDIR/conf/CPMIL* /var/tmp/
      # cpstart
    6. Log back into SmartDashboard and install policy.

     

    B. Increase the size of the Communication Buffer. 

    Refer to Common Solution

    C. On Windows GUI client machine perform:

    1. Go to 'Start Menu > Computer > Local Disk (C:) > Program files (x86) > CheckPoint > SmartConsole > Rxx (Management version) > PROGRAM > data > CPMICache'
    2. Delete the content of CPMICache folder.
    3. Log back into SmartDashboard and install Security policy.

    If this does not resolve the issue:

    • Increase the fwasync_maxbuf value according to 

      Scenario 4: Policy installation takes a long time

    • Edit the Management Database according to sk38893 
    • Edit commit_func_timeout on the gateway according to sk41736


  4. Modify the 'Platform' field to reflect the correct 'Hardware' and 'Version'.

Scenario 6: When trying to install policy to a large number of gateways at the same time, the installation times out

Product: Security Management, Multi-Domain Management / Provider-1

Symptoms:

  • When trying to install policy to a large number of gateways at the same time, the installation times out.

  • When a policy is installed on a cluster, it installs successfully on the primary member. For the secondary member, however, the following error message is displayed:

    policy installation failed for Module xxx... Reason: Connection failed - Session timeout

  • fwm debug shows lines similar to these:

    fwasync_connbuf_realloc: Connection buffer overflow. Cannot allocate 1078522 bytes of memory. Limit is 1048576.
    fwasync_conn_pre_send_check: Internal error. Failed to enlarge connection buffer.
    OPSEC_SET_ERRNO: err = 12 Failed to malloc (pre = 12)

Solution:

Refer to Common Solution

Scenario 7: In Multi-Domain Management / Provider-1, assigning a Global Policy from the Multi-Domain Server (MDS) GUI to a CMA fails with errors

Product: Multi-Domain Management / Provider-1

OS: SecurePlatform

Symptoms:

  • In Multi-Domain Management / Provider-1, assigning a Global Policy on the MDS GUI to a CMA fails. The following error is displayed in the assignment window: "opsec_errno=Communication path is congested"
  • fwm mds fwmconnect process crashes when assigning a Global Policy through the command line interface (CLI).

Cause:

The default size of the Connection Buffer size is too small to handle the thousands of objects that are configured in the Global Policy.

Solution:

Check Point can supply a Hotfix. Contact Check Point Support to get a Hotfix for this issue.
A Support Engineer will make sure the Hotfix is compatible with your environment before providing the Hotfix

If you choose not to get the Hotfix, you can use this workaround:

Increase the buffer size on the Multi Domain Server (MDS) level 

Refer to Common Solution MDS


After completing this procedure, reassigning a Global Policy should succeed.

Scenario 8: Policy installation on Correlation Unit fails

Product: SmartEvent / Eventia Analyzer

Symptoms:

  • Security policy installation from the Eventia Analyzer server to the Correlation Unit fails.
  • Errors:

    • Unable to resolve the network object <object_name>: object is empty

    • Failed to build correlation policy for event <event_name>

  • These error messages appears only for group objects that are actually not empty.

Cause:

Large database causes a buffer overflow on the Eventia Server.

Solution:

To resolve the problem, increase the buffer size limit on the Eventia server:

Refer to Common Solution (as performed on Security Management Server)

Scenario 9: Policy compilation for Edge device is failing with error: "Can't contact database"

Product: Edge

Symptoms:

  • Compilation is failing with the following error message:
    "Error: Can't contact database".

Cause:

SofawareLoader binary fails to communicate with fwm daemon.

Solution:

Refer to Common Solution (as performed on Security Management Server)

Scenario 10: User Activity Report is always limited to 42 Pages

Product: SmartEvent / Eventia Analyzer, SmartReporter / Eventia Reporter, Security Management

Symptoms:

  • User Activity Report is always limited to 42 pages.

Solution:

Follow these steps:

  1. Close all SmartConsole windows (SmartDashboard, SmartView Tracker, SmartView Monitor, etc.).
  2. Connect with GuiDBedit Tool:

    • to SmartEvent Server - if SmartEvent product and Security Management are installed on separate machines

    • to Security Management Server - if SmartEvent product is installed on the same machine as the Security Management Server
  3. In the upper left pane, go to 'Table' - 'Reporting' - 'reporting_reports'.
  4. In the upper right pane, select/search for the "OBJ_582E2CBE_B209_4BE9_B4A0_783D87DFCD98" object.
  5. Press CTRL+F (or go to 'Search' menu - 'Find') - paste max_rows_count - click on "Find Next".
  6. In the lower pane, right-click on the max_rows_count - 'Edit...' - change the value from "1250" to "-1" - click on "OK"
  7. Save the changes: go to 'File' menu - 'Save All'.
  8. Close the GuiDBedit Tool.
  9. Recommended step:

    1. Connect with SmartDashboard to Security Management Server (regradless of where the SmartEvent product is installed).

    2. Go to 'Policy' menu - click on 'Install Database...' - select all objects.

Important: This might affect performance generating Large Reports ( Like GUI Unresponsive)

On the SmartEvent server, clear GUI cache and increase buffer.

Refer to Common Solution (as performed on Security Management Server)

Important Note: It is forbidden to delete the applications.C* files on Provider-1 / Multi-Domain Security Management Server.

The "OBJ_582E2CBE_B209_4BE9_B4A0_783D87DFCD98" stated above is for a specific report.  This process will need to be performed on the specific object for the report that is being generated.

To find the specific object to modify, search under the reporting_reports table for the exact report name.
Once GuiDBedit finds the specific value for the name of the report, you are in the correct object. Scroll to the top of the bottom pane of options and then search for "max_rows_count" and change the value from "1250" to "-1".

Scenario 11: Synchronization Status for all HA Domains are flapping between Synced and Not Synced

Product: SmartEvent / Eventia Analyzer, SmartReporter / Eventia Reporter, Security Management

Symptoms:

  • After upgrade or installation of hotfixes, the Synchronization status for all Domains start flapping between Synced and Not Synced.

  • There are no Domain synchronization issues, i.e. one can force synchronization per Domain without errors.

Cause: The Communication Buffer on the Primary MDS is too small.

Solution:

The solution is to enlarge FWASYNC_MAXBUF as follows:

On MDS level for all MDS servers

Refer to Common Solution MDS

Manually sync all Domains via SmartDashboard

 

Related Solution:

sk106313 - SmartDomain Manager shows that status of Domains is flapping between "Synchronized" / "Not Synchronized"

Scenario 12: Status of managed Security Gateways in SmartView Monitor changes every few minutes, from OK to Disconnected and back to OK

Product: SmartView Monitor, Security Management, Multi-Domain Management / Provider-1

Symptoms:

  • Status of managed Security Gateways in SmartView Monitor changes every few minutes, from 'OK' to 'Disconnected' and back to 'OK' (cyclical connects and disconnects).

  • Debug of fwm daemon on Management Server (per sk86186) shows:

    ReportConnectionFailureToHolders: failed to establish AMON session due to a sic error 'SIC Error for amon: timeout elapsed during authentication protocol.' error code 148[
    amon_reply_get_error: Invalid Argument AmonReply (0)
    fwasync_conn_pre_send_check: Internal error. Failed to enlarge connection buffer.

  • Debug of cpd daemon on Security Gateway (per sk86320) shows:

    fwasync_connbuf_realloc: Connection buffer overflow. Cannot allocate 1054563 bytes of memory. Limit is xxxxxxx
    fwasync_conn_pre_send_check: Internal error. Failed to enlarge connection buffer.
    CP Status extractor: Failed to send reply.

Cause: Insufficient size of communication buffer on Management Server.

Solution:

In order to control the size of this buffer, a new environment variable called FWASYNC_MAXBUF was added to the code.

Support for the new environment variable FWASYNC_MAXBUF is included from Check Point R77.10

Check Point recommends to always upgrade to the most recent version.

For lower / other versions, Check Point can supply a Hotfix. Contact Check Point Support to get a Hotfix for this issue. A Support Engineer will make sure the Hotfix is compatible with your environment before providing the Hotfix.
For faster resolution and verification, please collect CPinfo files from the Security Management and Security Gateways involved in the case.

Hotfix installation instructions:

  1. Hotfix has to be installed on Security Management Server / Multi-Domain Security Management Server.

    Note: In Management HA environment, this procedure must be performed on both Management Servers.

  2. Transfer the hotfix package to the machine (e.g., into /some_path_to_fix/).

  3. Unpack the hotfix package:

    [Expert@HostName]# cd /some_path_to_fix/
    [Expert@HostName]# tar -xvfz fw1_HOTFIX_NAME.tgz
  4. Install the hotfix:

    [Expert@HostName]# ./fw1_HOTFIX_NAME

  5. Do NOT reboot the machine yet.
  6. Configure the desired buffer size:

    Show / Hide buffer size configuration instructions for Security Manag Server

      Refer to Common Solution (as performed on Security Management Server)


  7. Reboot the machine.

Scenario 13: "Machine is not configured as 'SmartEvent' on Management. Exiting..." error when connecting to SmartEvent NGSE client

Product: SmartEvent / Eventia Analyzer

Symptoms:

  • "Machine is not configured as 'SmartEvent' on Management. Exiting..." error pops up when trying to log in to SmartEvent NGSE:


  • Many "fwasync_conn_get: get max buffer size (1048576)" messages in fwm.elg, cpsemd.elg and GUI debugs output files.

Cause: The maximum buffer size value is too small.

Solution:

Increase the value of FWASYNC_MAXBUF on SmartEvent NGSE. To do so:

  1. Edit the $CPDIR/tmp/.CPprofile.sh file with VI editor and add the line to bottom: 

    FWASYNC_MAXBUF=12582912 ; export FWASYNC_MAXBUF

  2. Restart processes:

    # cpstop;cpstart

  3. Connect to SmartEvent with the SmartEvent NGSE client.

Scenario 14: CPSTAT or 3rd Part OPSEC client cannot read more than 3000 routes from the routing table

Product: Security Gateway

Symptoms:

  • Running: cpstat os -f routing provides only a maximum of 3000 routes, even if the routing table is bigger than that.

  • 3rd Pary OPSEC clients (such as AlgoSec, etc.) are not able to retrieve the entire routing table.

Cause: CPSTAT has a hard-coded limit of "3000" on the number of routes that can be queried, (for the sake of performance) as the CPU usage of CPD spikes when it queries the routing table.

Solution:

The limit can be raised by using the following procedure:

The change involves two modifications:

  1. Communication buffer allocated by CPD / FWD for responding to requests - such as via CPSTAT.
  2. Maximum number of routes to query

Procedure:

  1. Run the following lines to add the environment variables to $CPDIR/tmp/.CPprofile.sh:
    # echo "MAX_SIZE_ROUTING_TABLE=XXX; export MAX_SIZE_ROUTING_TABLE" >> $CPDIR/tmp/.CPprofile.sh
    # echo "MAX_LIMIT_ROUTING_TABLE=XXX; export MAX_LIMIT_ROUTING_TABLE" >> $CPDIR/tmp/.CPprofile.sh
    # echo "FWASYNC_MAXBUF=7316116 ; export FWASYNC_MAXBUF" >> $CPDIR/tmp/.CPprofile.sh


    The XXX should be replaced with a value that would leave enough "room" for future changes in the routing table (especially in cases where dynamic routing is used), but not too high.

  2. Run a cpstop / cpstart cycle - In some cases, a reboot might be necessary.

  3. Verify the environment variable are in place:
    # echo $MAX_SIZE_ROUTING_TABLE
    # echo $MAX_LIMIT_ROUTING_TABLE
    # echo $FWASYNC_MAXBUF

Note: This change will cause CPD to spike in CPU usage when pulling the routing table. It is advised to not query the routing table frequently.

Scenario 15: ROBO gateways disappear from SmartLSM

Product: SmartProvisioning / LSM

Symptoms:

  • When enter SmartLSM, see that all ROBO gateways disappear. This configuration worked before.

Cause: CPMI buffer overflow.

Solution:

Refer to Common Solution (as performed on Security Management Server
Applies To:
  • The following solutions were merged into sk109236:
    sk65746, sk34571, sk62871, sk97826, sk108552, sk33678, sk34336, sk40892, sk43911, sk93845, sk100495, sk33645, sk105599, sk103096, sk42093

Give us Feedback
Please rate this document
[1=Worst,5=Best]
Comment