Support Center > Search Results > SecureKnowledge Details
Kernel debug 'fw ctl debug' command is not applied to all CoreXL FW instances in R77.10
Symptoms
  • Kernel debug is disabled only for CoreXL FW instance 0 (and not for all instances), when running the 'fw ctl debug 0' command:
    output of 'fw -i INSTANCE_NUMBER ctl debug' shows that the debugs are disabled for instance 0, and are still enabled for all other instances.
    Show / Hide example
    [Expert@GW:0]# fw ctl debug -buf 32000
    Initialized kernel debugging buffer to size 32000K
    
    [Expert@Gaia-R77.10:0]# fw ctl debug -m fw + conn drop
    Updated kernel's debug variable for module fw
    
    [Expert@Gaia-R77.10:0]# fw -i 0 ctl debug -m fw
    Kernel debugging buffer size: 32000KB
    Module: fw
    Enabled Kernel debugging options: error warning conn drop
    
    [Expert@Gaia-R77.10:0]# fw -i 1 ctl debug -m fw
    Kernel debugging buffer size: 50KB
    Module: fw
    Enabled Kernel debugging options: error warning conn drop
    
    [Expert@Gaia-R77.10:0]# fw -i 2 ctl debug -m fw
    Kernel debugging buffer size: 50KB
    Module: fw
    Enabled Kernel debugging options: error warning conn drop
    
    [Expert@Gaia-R77.10:0]# fw ctl debug 0
    Defaulting all kernel debugging options
    
    [Expert@Gaia-R77.10:0]# fw -i 0 ctl debug -m fw
    Kernel debugging buffer size: 50KB
    Module: fw
    Enabled Kernel debugging options: error warning
    
    [Expert@Gaia-R77.10:0]# fw -i 1 ctl debug -m fw
    Kernel debugging buffer size: 50KB
    Module: fw
    Enabled Kernel debugging options: error warning conn drop
    
    [Expert@Gaia-R77.10:0]# fw -i 2 ctl debug -m fw
    Kernel debugging buffer size: 50KB
    Module: fw
    Enabled Kernel debugging options: error warning conn drop
    
  • Kernel debug flags 'all' are enabled only for CoreXL FW instance 0 (and not for all instances), when running the 'fw ctl debug -m MODULE all' command / 'fw ctl zdebug all' command:
    output of 'fw -i INSTANCE_NUMBER ctl debug' shows that the debugs are enabled only for instance 0, and are disabled for all other instances.
    Show / Hide example
    [Expert@GW:0]# fw ctl debug -buf 32000
    Initialized kernel debugging buffer to size 32000K
    
    [Expert@Gaia-R77.10:0]# fw ctl debug -m fw all
    Updated kernel's debug variable for module fw
    
    [Expert@Gaia-R77.10:0]# fw -i 0 ctl debug -m fw
    Kernel debugging buffer size: 32000KB
    Module: fw
    Enabled Kernel debugging options: error warning cookie crypt domain ex driver filter hold if install ioctl kbuf ld log machine memory misc packet q xlate xltrc conn synatk media sip vm chain bridge tcpstr scv highavail ipv6 packval sync ipopt link nat cifs drop route citrix misp portscan leaks mgcp sock mail spii chainfwd msnms wire balance dynlog smtp wap content mrtsync sam sock malware cmi aspii dos advp multik netquota monitor monitorall dfilter integrity epq cvpnd cptls ftp nac span ucd acct dlp ua icmptun dnstun ips rad te zeco user shmem utest qos context
    
    [Expert@Gaia-R77.10:0]# fw -i 1 ctl debug -m fw
    Kernel debugging buffer size: 50KB
    Module: fw
    Enabled Kernel debugging options: error warning
    
    [Expert@Gaia-R77.10:0]# fw -i 2 ctl debug -m fw
    Kernel debugging buffer size: 50KB
    Module: fw
    Enabled Kernel debugging options: error warning
    
  • Kernel debug specified flags are enabled only for CoreXL FW instance 0 (and not for all instances), when running the 'fw ctl debug -m MODULE FLAG' command / 'fw ctl zdebug -m MODULE FLAG' command:
    output of 'fw -i INSTANCE_NUMBER ctl debug -m MODULE' shows that the specified debug flags are enabled only for instance 0, and default debug flags are enabled for all other instances.
    Show / Hide example
    [Expert@Gaia-R77.10:0]# fw -i 0 ctl debug -m fw
    Kernel debugging buffer size: 50KB
    Module: fw
    Enabled Kernel debugging options: error warning
    
    [Expert@GW:0]# fw ctl debug -buf 32000
    Initialized kernel debugging buffer to size 32000K
    
    [Expert@Gaia-R77.10:0]# fw ctl debug -m fw conn drop
    Updated kernel's debug variable for module fw
    
    [Expert@Gaia-R77.10:0]# fw -i 0 ctl debug -m fw
    Kernel debugging buffer size: 32000KB
    Module: fw
    Enabled Kernel debugging options: conn drop
    
    [Expert@Gaia-R77.10:0]# fw -i 1 ctl debug -m fw
    Kernel debugging buffer size: 50KB
    Module: fw
    Enabled Kernel debugging options: error warning
    
    [Expert@Gaia-R77.10:0]# fw -i 2 ctl debug -m fw
    Kernel debugging buffer size: 50KB
    Module: fw
    Enabled Kernel debugging options: error warning
    
  • The CPU consumption may remain high after running the 'fw ctl debug 0' command, due to the fact that the debugs are not disabled on some of the CoreXL FW instances.

  • Kernel crash may occur some time after enabling kernel debugs.

    Show / Hide crash stack
    crash> bt
    PID: 0      TASK: ...  CPU: 0   COMMAND: "swapper"
    #0 [...] crash_kexec at ...
    #1 [...] kdb_main_loop at ...
    #2 [...] kdb_save_running at ...
    #3 [...] kdba_main_loop at ...
    #4 [...] kdb at ...
    #5 [...] die at ...
    #6 [...] do_page_fault at ...
    #7 [...] error_code (via page_fault) at ...
    ... ... ...
    #8 [...] misp_if_watch_dog(opq=Unavailable) at misp.c
    #9 [...] cptim_timer_expired(opaque=Unavailable) at cptimer.c
    #10 [...] run_timer_softirq(h=Unavailable) at timer.c
    #11 [...] __do_softirq() at softirq.c
    #12 [...] do_softirq() at softirq.c
    #13 [...] smp_apic_timer_interrupt(regs=Unavailable) at apic.c
    #14 [...] apic_timer_interrupt() at ...
    #15 [...] cpu_idle() at process.c
    
Cause

Incorrect parsing of the 'fw ctl debug' command / 'fw ctl zdebug' command causes every debug command that does not contain "-" or "+" to be handled only by CoreXL FW instance 0.


Solution

This problem was fixed. The fix is included in:

Check Point recommends to always upgrade to the most recent version (upgrade Security Gateway / upgrade Security Management Server / upgrade Multi-Domain Security Management Server).


For lower versions, Check Point can supply a Hotfix. Contact Check Point Support to get a Hotfix for this issue. A Support Engineer will make sure the Hotfix is compatible with your environment before providing the Hotfix.
For faster resolution and verification please collect CPinfo files from the Security Management and Security Gateways involved in the case.

If you do not wish to install a hotfix, workarounds are available.

 

Table of Contents:

  • Hotfix installation instructions
  • Hotfix uninstall instructions
  • Workarounds
  • Kernel debug syntax

 

Hotfix installation instructions

Show / Hide hotfix installation instructions

Contact Check Point Support for any assistance.

Note: In cluster environment, this procedure must be performed on all members of the cluster.

  • Show / Hide instructions - Gaia OS using CPUSE (Check Point Update Service Engine)

    We recommend using CPUSE to install this hotfix.

    Note: Hotfix has to be installed on Security Gateway / each cluster member.

    • In Gaia Portal:

      Important Note for VSX mode: Gaia Portal is not supported on Security Gateway in VSX mode. Users must use the Clish.

      1. Connect to the Gaia Portal on your machine.

      2. Obtain the lock over the configuration database (click on the lock icon at the top - near 'Sign Out').

      3. Navigate to the 'Software Updates' - 'Status and Actions' pane.

      4. Go to the 'Updates' tab to see the published hotfixes available for download.

      5. Select the Check_Point_R77.10_Hotfix_Gaia_sk98625.tgz package - right-click on it - click on 'Download' (this will download the hotfix to your machine).

      6. Right-click on the Check_Point_R77.10_Hotfix_Gaia_sk98625.tgz package - click on 'Install' (this will install the hotfix on the machine and display the installation status).

      7. When prompted for reboot (a pop up window appears), confirm to reboot the machine.


    • In Clish:

      Important Note for VSX mode: Gaia Portal is not supported on Security Gateway in VSX mode. Users must use the Clish.

      1. Connect to Gaia command line (over SSH, or console).

      2. Log in to Clish shell.

      3. See the list of available packages for download:

        HostName> show installer available_packages

      4. Download this hotfix:

        HostName> installer download Check_Point_R77.10_Hotfix_Gaia_sk98625.tgz

      5. Check the download progress by repeatedly running this command:

        HostName> show installer package_status
        Outputs for example:
        Check_Point_R77.10_Hotfix_Gaia_sk98625.tgz - Downloading (2.95 MB/s)   - Progress: 6%
        Check_Point_R77.10_Hotfix_Gaia_sk98625.tgz - Available for install
        
      6. See the list of available packages for install:

        HostName> show installer available_local_packages

      7. Install this hotfix:

        HostName> installer install Check_Point_R77.10_Hotfix_Gaia_sk98625.tgz

      8. Check the installation progress by repeatedly running this command:

        HostName> show installer package_status
        Outputs for example:
        Check_Point_R77.10_Hotfix_Gaia_sk98625.tgz - Installing                - Progress: 3%
        Check_Point_R77.10_Hotfix_Gaia_sk98625.tgz - installed
        
      9. Machine will be rebooted automatically.

    Contact Check Point Support for any assistance.



  • Show / Hide instructions - Gaia / SecurePlatform / Linux OS

    Contact Check Point Support for any assistance.

    1. Hotfix has to be installed on Security Gateway / each cluster member.

    2. Download the relevant hotfix package:

      Platform R77.10
      Gaia / SecurePlatform / Linux (TGZ)


    3. Transfer the hotfix package to the machine (into some directory, e.g., /some_path_to_fix/).

    4. Unpack the hotfix package:

      [Expert@HostName]# cd /some_path_to_fix/
      [Expert@HostName]# tar zxvf Check_Point_R77.10_Hotfix_Linux_sk98625.tgz

    5. Install the hotfix:

      [Expert@HostName]# ./fw1_wrapper_HOTFIX_NAME

      Note: The script will stop all of Check Point services (cpstop) - read the output on the screen.

    6. Reboot the machine.


  • Show / Hide instructions - IPSO OS

    Contact Check Point Support for any assistance.

    1. Hotfix has to be installed on Security Gateway / each cluster member.

    2. Download the relevant hotfix package:

      Platform R77.10
      IPSO (TGZ)


    3. Transfer the hotfix package to the machine (into some directory, e.g., /some_path_to_fix/).

    4. Unpack the hotfix package:

      [Expert@HostName]# cd /some_path_to_fix/
      [Expert@HostName]# tar zxvf Check_Point_R77.10_Hotfix_IPSO6_sk98625.tgz

    5. Install the hotfix:

      [Expert@HostName]# ./fw1_wrapper_HOTFIX_NAME

      Note: The script will stop all of Check Point services (cpstop) - read the output on the screen.

    6. Reboot the machine.

 

Hotfix uninstall instructions

Check Point offers a hotfix for this issue.

Show / Hide hotfix uninstall instructions

Contact Check Point Support for any assistance.

Note: In cluster environment, this procedure must be performed on all members of the cluster.

  1. Hotfix has to be uninstalled from Security Gateway / each cluster member.

  2. Uninstall the hotfix:

    [Expert@HostName]# cd /opt/CPsuite-R77
    [Expert@HostName]# ./uninstall_fw1_wrapper_HOTFIX_GULLI_HF_031_002

    Note: The script will stop all of Check Point services (cpstop) - read the output on the screen.

  3. Reboot the machine.

 

Workarounds

If you do not wish to install a hotfix, workarounds are available.

Show / Hide workaround instructions

  • Kernel debugs of all kernel modules can be disabled manually for all CoreXL FW instances by running the following command:
    [Expert@HostName]# fw ctl debug 0 -
    
    Note: Pay attention to the minus sign at the end of the line.

  • Kernel debugs with flag 'all' can be enabled for all CoreXL FW instances by running the 'fw ctl debug -m MODULE + all' command / 'fw ctl zdebug -m MODULE + all' command (with "+" sign).

  • Kernel debugs with specified flags for all CoreXL FW instances should be enabled by running the 'fw ctl debug -m MODULE + FLAG1 FLAG2 FLAG3 ...' command / 'fw ctl zdebug -m MODULE + FLAG1 FLAG2 FLAG3 ...' command (the specified flags will be enabled in addition to the default flags).
    Note: Pay attention to the plus sign before the flags.

 

Kernel debug syntax

Show / Hide kernel debug syntax

  • Usage:
    [Expert@HostName]# fw ctl debug -h
    
  • Default (clear) all current kernel debugging options:
    [Expert@HostName]# fw ctl debug 0
    
  • Disable all kernel debugging options (de-allocates the buffer automatically kills "fw ctl debug" process):
    [Expert@HostName]# fw ctl debug -x
    
  • Allocate the debugging buffer (to catch debug messages):
    [Expert@HostName]# fw ctl debug -buf 32000
    
  • Enable desired debug flags (in addition to the default flags):
    [Expert@HostName]# fw ctl debug -m MODULE_NAME + FLAG1 FLAG2 FLAG3
    
    Note: Pay attention to the "+" sign and space before the flags.

  • Enable only the specified debug flags (all other flags will be overwritten):
    [Expert@HostName]# fw ctl debug -m MODULE_NAME FLAG4 FLAG5
    
    Note: Pay attention that there is no "+" sign before the flags.

  • Disable undesired debug flags:
    [Expert@HostName]# fw ctl debug -m MODULE_NAME - FLAG6 FLAG7
    
    Note: Pay attention to the "-" sign and space before the flags.

  • Display all kernel modules and their flags that were turned on:
    [Expert@HostName]# fw ctl debug
    
  • Display all kernel modules and their flags that Security Gateway "understands":
    [Expert@HostName]# fw ctl debug -m
    
  • Display the flags for specific module that were turned on:
    [Expert@HostName]# fw ctl debug -m MODULE_NAME
    
  • Print the timestamp in debug output (t = seconds ; T = microseconds):
    [Expert@HostName]# fw ctl kdebug -t
      or
    [Expert@HostName]# fw ctl kdebug -T
    
  • Save the debug messages from debugging buffer into a file:
    [Expert@HostName]# fw ctl kdebug -T -f > /var/log/debug.txt
    
  • To stop the debug - press CTRL+C
Applies To:
  • 01365460 , 01365459 , 01403930

Give us Feedback
Please rate this document
[1=Worst,5=Best]
Comment