Support Center > Search Results > SecureKnowledge Details
Best Practices - Security Gateway Performance
Solution

Click Here to Show the Entire Article

 

Table of Contents (click on section titles to see sub-sections):

  • (1) Background
  • (2) Introduction and Limitations
    • (2-1) SecureXL
    • (2-2) CoreXL
    • (2-3) SMT (HyperThreading)
    • (2-4) Multi-Queue
  • (3) Best practices
    • (3-1) Network interface cards
    • (3-2) Throughput
    • (3-3) SecureXL
    • (3-4) CoreXL
    • (3-5) SecureXL with CoreXL
    • (3-6) SMT (HyperThreading)
    • (3-7) Multi-Queue
    • (3-8) Rulebase optimization
    • (3-9) IPS optimization
    • (3-10) Application Control & URL Filtering optimization
    • (3-11) Anti-Virus & Anti-Bot optimization
  • (4) Initial diagnostics
    • (4-1) CPU
    • (4-2) Memory
    • (4-3) Network interface cards
    • (4-4) SecureXL
    • (4-5) CoreXL
  • (5) Advanced diagnostics
    • (5-1) CPU
    • (5-2) Memory
    • (5-3) Network interface cards
    • (5-4) SecureXL
    • (5-5) CoreXL
  • (6) Command Line syntax
    • (6-1) SecureXL
      • (6-1-A) 'fwaccel' command
      • (6-1-B) 'sim' command
    • (6-2) CoreXL
      • (6-2-A) Gateway mode
      • (6-2-B) VSX mode
    • (6-3) Multi-Queue
  • (7) Examples
  • (8) Related documentation
  • (9) Related solutions
  • (10) Revision history

 

Organization of this article:

  • Chapter 1 "Background" - provides a short background on performance of Security Gateway.

  • Chapter 2 "Introduction" - lists the relevant definitions, supported configurations, limitations, and commands specific to a product.

  • Chapter 3 "Best practices" - provides the recommendations and guidelines for achieving the optimal performance.

  • Chapter 4 "Initial diagnostics" - lists the basic commands and guidelines for checking the current utilization of machine's resources.

  • Chapter 5 "Advanced diagnostics" - lists the advanced commands and guidelines for checking the current utilization of machine's resources.

  • Chapter 6 "Command Line syntax" - provides the complete list of commands and their options specific to a product.

  • Chapter 7 "Examples" - provides real-life examples.

  • Chapter 8 "Related documentation" - lists the relevant documents.

  • Chapter 9 "Related solutions" - lists the relevant solutions.

     

 


 

(1) Background

Performance of Security Gateway depends on:

  • CPU - utilization / saturation / errors
  • Memory - utilization / saturation / errors
  • Network Interfaces - utilization / saturation / errors
  • Storage device I/O, capacity, controller - utilization / saturation / errors
  • Throughput (packet rate * packet size)

Check Point solutions for improving the performance of Security Gateway:

 


 

(2) Introduction and Limitations

 

  • (2-1) Introduction and Limitations - SecureXL

  • Show / Hide introduction information

    Definitions:

    • Performance Pack is a software acceleration product installed on Security Gateways. Performance Pack uses SecureXL technology and other innovative network acceleration techniques to deliver wire-speed performance for Security Gateways. SecureXL is implemented either in software, or in hardware (SAM cards on Check Point 21000 appliances; ADP cards on IP Series appliances).

    • Affinity - Association of a particular network interface with a CPU core (either 'Automatic' (default), or 'Static' / 'Manual'). Interfaces are bound to CPU cores via SMP IRQ affinity settings (refer to sk61962 - SMP IRQ Affinity on Check Point Security Gateway).
      Note: The default SIM Affinity setting for all interfaces is 'Automatic' - the affinity for each interface is automatically reset every 60 seconds, and balanced between all available CPU cores based on the current CPU load.

    • FWACCEL - FireWall Accelerator (acceleration feature).

    • SIM - SecureXL Implementation Module (acceleration device).

    • Connection offload - Firewall kernel passes the relevant information about the connection from Firewall Connections Table to SecureXL Connections Table.
      Note: In ClusterXL High Availability, the connections are not offloaded to SecureXL on Standby member.

    • Connection notification - SecureXL passes the relevant information about the accelerated connection from SecureXL Connections Table to Firewall Connections Table.

    • Partial connection - Connection that exists in the Firewall Connections Table, but not in the SecureXL Connections Table (versions R70 and above).

      • In Cluster HA - partial connections are offloaded when member becomes Active
      • In Cluster LS - partial connections are offloaded upon post-sync (only for NAT / VPN connections)
      Such connections must be offloaded to SecureXL, since packets in these connections must not be dropped.
      If a packet matched a partial connection in the outbound, then it should be dropped.
    • Delayed connection - Connection created from SecureXL Connection Templates without notifying the Firewall for a predefined period of time. The notified connections are deleted by the Firewall.

    • Anticipated connection - Connection that is anticipated by SecureXL based on policy rules to avoid dropping it by Drop Template.

      In the following firewall policy:
      NO. SOURCE DESTINATION VPN SERVICE ACTION
      1 Any Any Any Traffic ftp Accept
      2 Any Any Any Traffic Any Drop
      1. Any FTP connection that will be opened will be accepted.
      2. A SYN packet of a Telnet connection will be dropped and a Drop Template will be offloaded to SecureXL.
      3. The FTP Data connection FTP might match this Drop Template and will be dropped.
      Therefore, to prevent the drop of a legitimate connection:
      1. SecureXL will offload such connections (in this example, FTP Data) and will mark them as anticipated.
      2. SecureXL will try to match an anticipated connection to an existing connection or an existing Accept Template.
      3. If such match was found, the packet will be forwarded to the firewall and will not be matched to a Drop Template.
    • Accept Template - Feature that accelerates the speed, at which a connection is established by matching a new connection to a set of attributes. When a new connection matches the Accept Template, subsequent connections are established without performing a rule match and therefore are accelerated. Accept Templates are generated from active connections according to policy rules. Currently, Accept Template acceleration is performed only on connections with the same destination port (using wildcards for source ports).
      Note: Size of Templates table (cphwd_tmpl, id 8111) is limited to 1/4 of the size of Firewall Connections Table (connections, id 8158).

    • Drop Template - Feature that accelerates the speed, at which a connection is dropped by matching a new connection to a set of attributes. When a new connection matches the Drop Template, subsequent connections are dropped without performing a rule match and therefore are accelerated. Currently, Drop Template acceleration is performed only on connections with the same destination port (does not use wildcards for source ports). Drop Templates are generated from policy rules by special algorithm:

      • Analyze the rulebase
      • Produce mutually exclusive ranges
      • Offload the ranges to SecureXL
      • Once a packet is dropped, offload a Drop Template
      • All subsequent packets matching that range will be dropped by SecureXL
    • Accelerated path - Packet flow when the packet is completely handled by the SecureXL device. It is processed and forwarded to the network.

    • Medium path (PXL) - Packet flow when the packet is handled by the SecureXL device, except for IPS (some protections) / VPN (in some configurations) / Application Control / Content Awareness / Anti-Virus / Anti-Bot / HTTPS Inspection / Proxy mode / Mobile Access / VoIP / Web Portals. The CoreXL layer passes the packet to one of the CoreXL FW instances to perform the processing (even when CoreXL is disabled, the CoreXL infrastructure is used by SecureXL device to send the packet to the single FW instance that still functions).

    • Firewall path / Slow path (F2F) - Packet flow when the SecureXL device is unable to process the packet (refer to sk32578 - SecureXL Mechanism). The packet is passed on to the CoreXL layer and then to one of the CoreXL FW instances for full processing. This path also processes all packets when SecureXL is disabled.

    • Active Streaming (CPAS) - Technology that sends streams of data to be inspected in the kernel, since more than a single packet at a time is needed in order to understand the application that is running (such as HTTP data). Active Streaming is Read- and Write-enabled, and works as a transparent proxy. Connections that pass through Active Streaming can not be accelerated by SecureXL.

    • Passive Streaming - Technology that sends streams of data to be inspected in the kernel, since more than a single packet at a time is needed in order to understand the application that is running (such as HTTP data). Passive Streaming is Read-only and it cannot hold packets, but the connections are accelerated by SecureXL.

    • Passive Streaming Library (PSL) - IPS infrastructure, which transparently listens to TCP traffic as network packets, and rebuilds the TCP stream out of these packets. Passive Streaming can listen to all TCP traffic, but process only the data packets, which belong to a previously registered connection. For more details, refer to sk95193 - ATRG: IPS.

    • PXL - Technology name for combination of SecureXL and PSL.

    • QXL - Technology name for combination of SecureXL and QoS (R77.10 and above).

    • F2F / F2Fed - Packets that can not be accelerated by SecureXL (refer to sk32578 - SecureXL Mechanism) are Forwarded to Firewall.

    • F2P - Forward to PSL/Applications. Feature that allows to perform the PSL processing on the CPU cores, which are dedicated to the Firewall.

    • SAM card - Security Acceleration Module card (Acceleration Ready card). Connections that use SAM card, are accelerated by SecureXL and are processed by the SAM card's CPU instead of the main CPU (refer to 21000 Appliance Security Acceleration Module Getting Started Guide)).

    • ADP card - Accelerated Data Path card. Connections that use ADP card, are accelerated by SecureXL and are processed by network processors (NP) instead of the main CPU (refer to sk60508 - How to Configure ADP & SecureXL on IPSO).

    • IRQ Swizzling - Traditionally, in a PCIe bus, all PCIe ports are mapped to one interrupt. Swizzling allows the PCIe slots to be balanced across four interrupts instead of one (enabling IRQ Swizzling requires a BIOS update).

    Supported operating systems:

    • Gaia OS
    • Gaia Embedded OS
    • SecurePlatform OS
    • SecurePlatform Embedded OS
    • IPSO OS
    • Crossbeam XOS
    • Crossbeam COS

    Traffic flow:

    Packet flow:

    • Flow logic:

    • Accelerated packet:

    • Packet matched to a template:

    • Non-accelerated packet:

    Limitations:

    • For limitations of traffic acceleration and templating, refer to sk32578 - SecureXL Mechanism.
    • SIM Affinity is applicable only to physical interfaces (VLAN / Bond interfaces are influenced by the "physical" decision).
      Note: Some systems may require BIOS update to enable IRQ Swizzling and EIRQ technologies. To determine whether a specific system supports the required technology, contact your hardware vendor.
    • On VSX Gateway, WARP interfaces can not be assigned to a CPU core.
    • SecureXL NAT Templates are supported only in R75.40 and above (sk71200).
    • SecureXL Drop Templates are supported only in R76 and above (sk66402).
    • SecureXL Optimized Drops feature (used for DoS/DDoS protection) is supported only in R76 and above (sk90861).
    • SecureXL Penalty box feature (used for DoS/DDoS protection) is supported only in R75.40VS and above (on VSX Gateway, the penalty box would only be enforced on Virtual System 0) (sk74520).
    • SecureXL does not support Point-to-Point interfaces (PPP, PPTP, PPPoE).
    • In R80.20, SAM is supported only for non-accelerated usage. Traffic connected to the Acceleration-ready 10G Interface Card (CPAC-ACCL-4-10F-21000) will be handled by the host. 10G Ports on the CPAC-ACCL-4-10F-21000 cannot be assigned as SAM ports in R80.20. 
      Note: In case a PPP-interface is detected, SecureXL disables itself on that interface (sk79880).
    • ClusterXL Sticky Decision Function (SDF) disables SecureXL.
    • QoS disables SecureXL (for R77.10 and above, refer to sk98229).
    • Delayed Synchronization in cluster:
      • Applies only to TCP services whose 'Protocol Type' is set to 'HTTP' or 'None'.
      • Delayed Synchronization is disabled if the 'Track' option in the rule is set to 'Log' or 'Account'.
      • Delayed Synchronization is performed only for connections matching a SecureXL Connection Template.
    • It is possible that a connection will exist in the Firewall Connections Table, but not in the SecureXL Connections Table (partial connection).
      This situation can occur:
      • After policy installation
      • After cluster failover
      • If user turned off ('fwaccel off' command) and turned on ('fwaccel on' command) acceleration

    Documentation:

    • Performance Pack Administration Guide (R75, R75.20, R75.40, R75.40VS).
    • Performance Tuning Administration Guide (R76, R77) - Chapter 1 'Performance Pack'.

    Command Line syntax:

    • FWACCEL (controls acceleration feature)

      [Expert@HostName]# fwaccel <parameter> [-h]
      [Expert@HostName]# fwaccel6 <parameter> [-h]
      
    • SIM (controls acceleration device)

      [Expert@HostName]# sim <parameter> [-h]
      [Expert@HostName]# sim6 <parameter> [-h]
      

     

  • (2-2) Introduction and Limitations - CoreXL

  • Show / Hide introduction information

    Definitions:

    • CoreXL - A performance-enhancing technology for Security Gateways on multi-core processing platforms. CoreXL enhances Security Gateway performance by enabling the CPU processing cores to concurrently perform multiple tasks.

    • Secure Network Distributor (SND) - Traffic entering network interface cards (NICs) is directed to a processing CPU core running the SND, which is responsible for:

      • Processing incoming traffic from the network interfaces
      • Securely accelerating authorized packets (if SecureXL is enabled)
      • Distributing non-accelerated packets among Firewall kernel instances (SND maintains global dispatching table - which connection was assigned to which instance)
    • Firewall Instance / FW Instance - On a Security Gateway with CoreXL enabled, the Firewall kernel is replicated multiple times. Each replicated copy, or Firewall Instance, runs on one CPU processing core. These FW instances handle traffic concurrently, and each FW instance is a complete and independent Firewall inspection kernel. When CoreXL is enabled, all the Firewall kernel instances on the Security Gateway process traffic through the same interfaces and apply the same security policy.

    • Affinity - Association of a particular network interface / FW kernel instance / daemon with a CPU core (either 'Automatic' (default), or 'Manual').
      Note: The default CoreXL interface affinity setting for all interfaces is 'Automatic' when SecureXL is installed and enabled.

    • Accelerated path - Packet flow when the packet is completely handled by the SecureXL device. It is processed and forwarded to the network.

    • Medium path (PXL) - Packet flow when the packet is handled by the SecureXL device, except for IPS (some protections) / VPN (in some configurations) / Application Control / Content Awareness / Anti-Virus / Anti-Bot / HTTPS Inspection / Proxy mode / Mobile Access / VoIP / Web Portals. The CoreXL layer passes the packet to one of the CoreXL FW instances to perform the processing. This path is available only when CoreXL is enabled.

    • Firewall path / Slow path (F2F) - Packet flow when the SecureXL device is unable to process the packet (refer to sk32578 - SecureXL Mechanism). The packet is passed on to the CoreXL layer and then to one of the CoreXL FW instances for full processing. This path also processes all packets when SecureXL is disabled.

    • Passive Streaming Library (PSL) - IPS infrastructure, which transparently listens to TCP traffic as network packets, and rebuilds the TCP stream out of these packets. Passive Streaming can listen to all TCP traffic, but process only the data packets, which belong to a previously registered connection. For more details, refer to sk95193 - ATRG: IPS.

    • PXL - Technology name for combination of SecureXL and PSL.

    Supported operating systems:

    • Gaia OS
    • SecurePlatform OS
    • IPSO OS
    • Crossbeam XOS
    • Crossbeam COS

    Architecture:

    Example of default configuration for machine with 8 CPU cores:

    Default number of CoreXL IPv4 FW instances:

    Note: The real number of CoreXL FW instances depends on the current CoreXL license.

    Number of
    CPU
    cores
    Default number of
    CoreXL IPv4
    FW instances
    Default number of
    Secure Network Distributors
    (SNDs)
    1 1
    Note: CoreXL is disabled
    0
    Note: CoreXL is disabled
    2 2 2
    4 3 1
    6 - 20 [Number of CPU cores] - 2 2
    More than 20 [Number of CPU cores] - 4
    Note: However, no more than 40 (*).
    4

    (*) Note: For Maximal number of CoreXL IPv4 FW instances, check sk98737 - ATRG: CoreXL

    Relation between CoreXL IPv4 FW instances and CoreXL IPv6 FW instances:
    (run the 'fw ctl multik stat' command and 'fw6 ctl multik stat' command on Security Gateway)

    • The number of IPv4 FW instances - from a minimum of 2 to a number equal to the maximal number of CPU cores on the Security Gateway.
    • The number of IPv6 FW instances - from a minimum of 2 to a number equal to the number of IPv4 FW instances.
    • The number of IPv6 FW instances cannot exceed the number of IPv4 FW instances.
    • The total number of IPv4 FW instances and IPv6 FW instances together cannot exceed 32.

    CoreXL and ClusterXL:

    • Number of CoreXL FW instances must be identical on all members of the cluster because the state synchronization between members is performed per CoreXL FW instance (e.g., Instance #2 on Member_A can synchronize only with Instance #2 on Member_B).
      Note: Member with higher number of CoreXL FW instances will enter the 'Ready' state. Refer to sk42096 - Cluster member is stuck in 'Ready' state.

    Limitations:

    Documentation:

    • Firewall Administration Guide (R75, R75.20, R75.40, R75.40VS) - Chapter 'CoreXL Administration'.
    • Performance Tuning Administration Guide (R76, R77) - Chapter 'CoreXL Administration'.

    Command Line syntax:

    [Expert@HostName]# fw ctl multik <parameter>
    [Expert@HostName]# fw ctl affinity [-flag]
    

     

  • (2-3) Introduction and Limitations - SMT (HyperThreading)

  • Show / Hide introduction information

    Definitions:

    • SMT (Simultaneous Multi-Threading - IntelĀ® HyperThreading, or IntelĀ® HT) is a feature that is supported on Check Point appliances running Gaia OS. When enabled, SMT doubles the number of logical CPUs on the Security Gateway, which enhances physical processor utilization. When SMT is disabled, the number of logical CPUs equals the number of physical cores.
      SMT improves performance up to 30% in NGFW software blades such as IPS, Application & URL Filtering and Threat Prevention by increasing the number of CoreXL FW instances based on the number of logical CPUs.

    Supported configurations:

    • SMT is supported by R77 release and later.
    • SMT is supported only on Security Gateways running Gaia OS with 64-bit kernel.
    • SMT is supported only on Check Point appliances.

    Limitations:

    • SMT is relevant only if CoreXL is enabled.

    Notes:

    • When SMT (HyperThreading) is enabled, the number of available CPU cores is not linear.

       

      Example:

      CPU core 0-5 - socket 1
      CPU core 6-11 - socket 2
      CPU core 12-15 - socket 1 HyperThreading
      CPU core 16-23 - socket 2 HyperThreading
      

    Documentation:

     

  • (2-4) Introduction and Limitations - Multi-Queue

  • Show / Hide introduction information

    Background:

    Today, each network interface card has one traffic queue that is handled by one CPU at a time. Since the Secure Network Distributor (SND) - SecureXL and CoreXL Distributor is running on the CPU cores that handle the traffic queues, user cannot use more CPU cores for acceleration than the number of network interface cards passing the traffic.

    Definitions:

    • Multi-Queue is an acceleration feature that lets the user configure more than one traffic queue for each network interface card, which allows using more CPU cores for acceleration.

    • rx queue - Receive packet queue.

    • tx queue - Transmit packet queue.

    • Secure Network Distributor (SND) - Traffic entering network interface cards (NICs) is directed to a processing CPU core running the SND, which is responsible for:

      • Processing incoming traffic from the network interfaces
      • Securely accelerating authorized packets (if SecureXL is enabled)
      • Distributing non-accelerated packets among Firewall kernel instances
    • IRQ affinity - Process of binding a network interface card's IRQ to one or more CPU cores.

    • Accelerated path - Packet flow when the packet is completely handled by the SecureXL device. It is processed and forwarded to the network.

    • Medium path (PXL) - Packet flow when the packet is handled by the SecureXL device, except for IPS (some protections) / VPN (in some configurations) / Application Control / Content Awareness / Anti-Virus / Anti-Bot / HTTPS Inspection / Proxy mode / Mobile Access / VoIP / Web Portals. The CoreXL layer passes the packet to one of the CoreXL FW instances to perform the processing. This path is available only when CoreXL is enabled.

    • Firewall path / Slow path - Packet flow when the SecureXL device is unable to process the packet (refer to sk32578 - SecureXL Mechanism). The packet is passed on to the CoreXL layer and then to one of the CoreXL FW instances for full processing. This path also processes all packets when SecureXL is disabled.

    • Passive Streaming Library (PSL) - IPS infrastructure, which transparently listens to TCP traffic as network packets, and rebuilds the TCP stream out of these packets. Passive Streaming can listen to all TCP traffic, but process only the data packets, which belong to a previously registered connection. For more details, refer to sk95193 - ATRG: IPS.

    • PXL - Technology name for combination of SecureXL and PSL.

    Supported configurations:

    • Multi-Queue is integrated into R76, R77 and above. For lower versions, a Multi-Queue hotfix has to be installed (refer to sk80940).
    • Multi-Queue is supported on Check Point Appliances (including IP Series Appliances) and on Open Servers.
    • Multi-Queue is supported only on machines that run SecurePlatform OS or Gaia OS.
    • Multi-Queue is supported only for network interface cards that use igb (1 GbE) and ixgbe (10 GbE) drivers.

    Limitations:

    • Multi-Queue is relevant only if SecureXL is enabled.
    • Multi-Queue is relevant only if CoreXL is enabled.
    • Multi-queue is not supported on machines with a single CPU core.
    • Multi-Queue allows to configure a maximum of 5 interfaces (due to IRQ limitations).
    • The number of traffic queues is limited by the number of CPU cores and the type of network interface card driver:
      • igb driver - up to 4 RX queues
      • ixgbe driver - up to 16 RX queues

    Documentation:

    Command Line syntax:

    [Expert@HostName]# cpmq <parameter>
    

     


     

    (3) Best practices

     

  • (3-1) Best practices - Network interface cards

  • Show / Hide best practices information

    • Refer to the list of Certified Network Interfaces.
    • You should use the PCI Express (PCIe) cards, because they have better performance than PCI-X cards.
    • If you are using a motherboard with multiple PCI or PCI-X buses, make sure that each Network Interface Card is installed in a slot connected to a different bus.
    • If you are using more than two Network Interface Cards in a system with only two 64-bit/66Mhz PCI buses, make sure that the least-used cards are installed in slots connected to the same bus.
    • If the traffic rate is very high and causes high amount of SoftIRQ (and traffic drops on interfaces), consider increasing the sizes of receive/transmit buffer on network interface cards as described in sk42181 - How to increase sizes of buffer on SecurePlatform/Gaia for Intel NIC and Broadcom NIC.

     

  • (3-2) Best practices - Throughput

  • Show / Hide best practices information

    Setting the maximal number of concurrent connections:

    • Versions R75.40VS, R75.45 and above

      1. SmartDashboard - open Security Gateway object.
      2. Go to 'Optimizations' pane.
      3. The 'Calculate the maximum limit for concurrent connections' should be set to 'Automatically'.
        Note: Manual limit should be set only for security reasons.
      4. Enter the desired value.
      5. The 'Calculate connections hash table size and memory pool' should be set to 'Automatically'.
      6. Click on 'OK'.
      7. Install the policy.
    • Versions R75.40 and lower

      1. SmartDashboard - open Security Gateway object.
      2. Go to 'Optimizations' pane.
      3. In the 'Calculate the maximum limit for concurrent connections', select the 'Manually' (recommended setting is 'Automatically').
      4. Enter the desired value.
      5. The 'Calculate connections hash table size and memory pool' should be set to 'Automatically'.
      6. Click on 'OK'.
      7. Install the policy.

    Notes:

    • You should ensure that the total number of concurrent connections is appropriate to the TCP end timeout.
      Too many concurrent connections will adversely affect the Security Gateway's performance.
    • You can calculate the maximum number of concurrent connections by multiplying the session establishment rate by the TCP session timeout:
      [MAXIMAL number of concurrent connections] = [MAXIMAL session establishment rate] x [TCP entire session timeout]
      Notes:
      • This formula is used to understand what number of concurrent connections a session rate test will generate.
      • It is very hard to predict the maximal connections capacity of a Security Gateway because of multiple varying factors.
      • Administrator should monitor the memory utilization on Security Gateway after changing these settings.
      • Only the maximal (possible) session rate should be considered.
      • This formula will not predict connections capacity, which is stated in Check Point datasheet documents.
      • Average session rate can be obtained using the sk101878 - CPView Utility - from 'Traffic' tab.
      • Additional information can be obtained based on sk67560 - How to export History Report from SmartView Monitor - from 'Traffic' view.
      • TCP timeout varies highly between applications and protocols (e.g., SSH is usually long, HTTP is usually short).
    • By reducing the following timeouts, you increase the capacity of actual TCP and UDP connections (SmartDashboard - 'Policy' menu - 'Global Properties' - 'Stateful Inspection'):
      • TCP end timeout - determines the amount of time a TCP connection will stay in the FireWall Connections Table (id 8158) after a TCP session has ended.
      • UDP virtual session timeout - determines the amount of time a UDP connection will stay in the FireWall Connections Table (id 8158) after the last UDP packet was seen by the Security Gateway.

    Optimal connection/sec rate:

    1. SmartDashboard - go to 'Policy' menu - click on 'Global Properties'.
    2. Go to 'FireWall' pane.
    3. Uncheck the following boxes:
      • Accept RIP
      • Accept Domain Name over UDP (Queries)
      • Accept Domain Name over TCP (Zone Transfer)
      • Accept ICMP requests
    4. Click on 'OK'.
    5. Install the policy.

    ARP cache table:

    Increase the number of entries in the ARP cache table:

    NAT session rate:

    • For Security Gateway versions R75.40 and above:
      Enable SecureXL NAT Templates per sk71200 - SecureXL NAT Templates.
    • For Security Gateway versions R75.30 and lower:
      1. Disable SecureXL.
      2. And change one of the following:
        • Either decrease the TCP end timeout to 2 seconds (SmartDashboard - 'Policy' menu - 'Global Properties' - 'Stateful Inspection').
        • Or increase the hash size of SND's Connection Table by increasing (permanently per sk26202) the value of the kernel parameter fwmultik_gconn_tab_hsize from the default 524288 to 8388608 and rebooting the machine.
          Important Note: This change reduces the capacity for the maximum number of concurrent connections.

     

  • (3-3) Best practices - SecureXL

  • Show / Hide best practices information

    Note: This section does not take the CoreXL into consideration.

    BIOS settings:

    • If your BIOS supports CPU clock setting, then make sure that the BIOS is set to the actual CPU speed (no over-clocking).
    • If your BIOS supports Hyper-Threading, then refer to "SMT (HyperThreading)" section.

    Network Interface Cards:

    • Refer to the list of Certified Network Interfaces.
    • You should use the PCI Express (PCIe) cards, because they have better performance than PCI-X cards.
    • If you are using a motherboard with multiple PCI or PCI-X buses, make sure that each Network Interface Card is installed in a slot connected to a different bus.
    • If you are using more than two Network Interface Cards in a system with only two 64-bit/66Mhz PCI buses, make sure that the least-used cards are installed in slots connected to the same bus.

    Network interface affinity:

    • Each NIC should be bound (affined) to a separate CPU core using the 'Static' affinity mode (run the 'sim affinity -s' command).
    • Pairs of interfaces carrying significant data flows (based on network topology) should be assigned to pairs of CPU cores on the same physical CPU processor.
    • Pairs of interfaces that serve the same connections (based on network topology) should be assigned to pairs of CPU cores on the same physical CPU core.
    • For systems with 4 CPU cores and Dual Port NICs, the IRQ Swizzling technology should be enabled to properly distribute IRQs among 4 CPU cores.
      Note: Applies only to Dual Port NICs. IRQ Swizzling is not required with Quad Port NICs.
    • For systems with 8 CPU cores and Quad Port NICs, the EIRQ technology should be enabled to properly distribute IRQs among all 8 CPU cores.
      Note: At the time of this writing, there is no certified platform with this ability.

    NAT session rate:

    • For Security Gateway versions R75.40 and above:
      Enable SecureXL NAT Templates per sk71200 - SecureXL NAT Templates.
    • For Security Gateway versions R75.30 and lower:
      1. Disable SecureXL.
      2. And change one of the following:
        • Either decrease the TCP end timeout to 2 seconds (SmartDashboard - 'Policy' menu - 'Global Properties' - 'Stateful Inspection').
        • Or increase the hash size of SND's Connection Table by increasing (permanently per sk26202) the value of the kernel parameter fwmultik_gconn_tab_hsize from the default 524288 to 8388608 and rebooting the machine.
          Important Note: This change reduces the capacity for the maximum number of concurrent connections.

    Accelerated Drop Rules:

    • Available in R75.40 and above.
    • Accelerated Drop Rules protect the Security Gateway and site from Denial of Service attacks by dropping packets at the acceleration layer (SecureXL).
    • The drop rules are configured in a file on the Security Gateway (using the 'sim dropcfg <options>' command), which is then offloaded to the SecureXL device for enforcement.
      Note: There is no relation between the rules being configured in the file and the rules configured in SmartDashboard.
    • Refer to sk67861 - Accelerated Drop Rules Feature in R75.40 and above.

    Optimized Drops:

    • Available in R76 and above.
    • Allows SecureXL to accelerate dropped traffic.
    • Once the feature is enabled, dropped traffic is accelerated depending on traffic drop rate per second. Once drop rate matches the thresholds, feature is dynamically activated/deactivated.
    • Refer to sk90861 - Optimized Drops feature in R76 and above.

    Delayed Synchronization in cluster:

    • To decrease the load on CPU in cluster environment:

      • Either disable the synchronization of non-critical connections (e.g., UDP DNS, ICMP).
      • Or (if connection must be synchronized) start synchronizing the connection only some time after its initiation (right-click on the service - 'Edit...' - 'Advanced...' - check the box 'Start synchronizing [X] seconds after connection initiation' - install policy).

      Notes:

      • Applies only to TCP services whose 'Protocol Type' is set to 'HTTP' or 'None'.
      • Delayed Synchronization is disabled if the 'Track' option in the rule is set to 'Log' or 'Account'.
      • Delayed Synchronization is performed only for connections matching a SecureXL Connection Template.

     

  • (3-4) Best practices - CoreXL

  • Show / Hide best practices information

    Note: This section does not take the SecureXL into consideration.

    Notes:

    • CoreXL improves performance with almost linear scalability in the following scenarios:

      • Much of the traffic can not be accelerated by SecureXL
      • Many IPS features enabled, which disable SecureXL functionality
      • Large rulebase
      • NAT rules
    • CoreXL will not improve performance in the following scenarios:

      • SecureXL accelerates much of the traffic
      • Traffic mostly consists of VPN
      • Traffic mostly consists of VoIP
    • It is very important to verify that the CPU cores are equally utilized (run the 'top' command). If this is not the case, you should consider changing the distribution of the Secure Network Distributors (SNDs) and CoreXL FW instances.

    • Under normal circumstances, it is not recommended for the SND and a CoreXL FW instance to share the same CPU core.
      However, it is necessary in the following cases:

      • When using a machine with exactly two cores. It is better for both SNDs and CoreXL FW instances to share CPU cores, instead of allocating only one CPU core to each.
      • When you know that almost all of the packets are being processed in the Accelerated path, and you want to assign all CPU cores to this path. If the CoreXL FW instances do not process significant amount of traffic, then it is appropriate to share the CPU cores.
    • The interfaces affinity should be configured only to to CPU cores that are running as SNDs (CPU cores that are not running CoreXL FW instances), with these exceptions:

      • On machines with exactly two CPU cores (both SNDs and CoreXL FW instances use the same CPU cores).
      • For tests, in which traffic is accelerated by SecureXL (if it is enabled).
    • If you replaced the CPU on the machine to a CPU with more/less CPU cores than the previous CPU, then you must reconfigure the number of CoreXL FW instances in the 'cpconfig' menu.

    • In a cluster environment, changing the number of CoreXL FW instances should be treated as a version upgrade - member with higher number of CoreXL FW instances will enter the 'Ready' state. Refer to sk42096 - Cluster member is stuck in 'Ready' state.
      To change the number of CoreXL FW instances, schedule a maintenance window and follow either a Minimal Effort Upgrade procedure, or a Zero Downtime Upgrade procedure from the Installation and Upgrade Guide (R75, R75.20, R75.40, R75.40VS, R76, R77 Gaia, R77 Non-Gaia).

    Changing the distribution of the SND, CoreXL FW instances, and daemons among the CPU cores:

    Note: Run the 'fw ctl affinity -l -r -a -v' command to see the current distribution.

    • To change the distribution of the SND, CoreXL FW instances, and daemons, change the current affinities of interfaces and/or of daemons.

      Notes:

      • To make the affinity settings persistent, edit the '$FWDIR/conf/fwaffinity.conf' file on Security Gateway (see the contents of the file for the correct syntax).
      • To apply the configuration from the '$FWDIR/conf/fwaffinity.conf' file on-the-fly (without reboot), execute the '$FWDIR/scripts/fwaffinity_apply' shell script on Security Gateway (see the contents of the script for available flags).
    • To ensure CoreXL's efficiency, all traffic must be directed to CPU cores that are running as SNDs (CPU cores that are not running CoreXL FW instances).
      Therefore, if you change affinities of interfaces and/or daemons, you will need to accordingly set the number of CoreXL FW instances and ensure that the CoreXL FW instances run on other CPU cores.

    • It is recommended to allocate an additional CPU core to the SND only if all of the following conditions are met:

      • Your platform has at least 8 CPU cores.
      • The 'idle' value (run 'top' command and press 1 to display all CPU cores) for the CPU core currently running the SND is in the 0%-5% range.
      • The sum of the 'idle' values (run the 'top' command and press 1 to display all CPU cores) for the CPU cores running CoreXL FW instances is significantly higher than 100%.

      If any of the above conditions are not met, the default configuration of one processing core allocated to the SND is sufficient, and no further configuration is necessary.

    • It is recommended to allocate an additional CPU core to the FWD daemon if the Security Gateway is performing heavy logging.
      Note: Avoiding the CPU cores that are running the SND is important only if these CPU cores are explicitly defined as affinities of interfaces. If interface affinities are set to 'Automatic', any CPU core that is not running a CoreXL FW instance can be used for the FWD daemon, and interface traffic will be automatically diverted to other CPU cores.

      Example of configuration for machine with 8 CPU cores:

     

  • (3-5) Best practices - SecureXL with CoreXL

  • Show / Hide best practices information

    Background:

    When most of the traffic is accelerated by SecureXL (run 'fwaccel stats -s' command), the load on CPU cores that run as Secure Network Distributor (SND) can be very high, while the load on CPU cores that run CoreXL FW instances can be very low. This is an inefficient utilization of CPU capacity.

    Notes:

    • Traffic is processed by the CoreXL FW instances only when the traffic is not accelerated by SecureXL (if SecureXL is installed and enabled).

    • With CoreXL, there are cases when performance without SecureXL is better than with it, even when SecureXL does manage to accelerate part of the traffic.

    Packet flow:

    When SecureXL is enabled, a packet enters the Security Gateway and first reaches the SecureXL device. The packet will be handled in one of three ways:

    1. Accelerated path - The packet is completely handled by the SecureXL device. It is processed and forwarded to the network.

    2. Medium path (PXL) - Packet flow when the packet is handled by the SecureXL device, except for IPS (some protections) / VPN (in some configurations) / Application Control / Content Awareness / Anti-Virus / Anti-Bot / HTTPS Inspection / Proxy mode / Mobile Access / VoIP / Web Portals. The CoreXL layer passes the packet to one of the CoreXL FW instances to perform the processing.
      This path is available only when CoreXL is enabled.

    3. Firewall path / Slow path - The SecureXL device is unable to process the packet (refer to sk32578 - SecureXL Mechanism). The packet is passed on to the CoreXL layer and then to one of the CoreXL FW instances for full processing.
      This path also processes all packets when SecureXL is disabled.

    Default affinity settings for interfaces:

    • If SecureXL is enabled - the default affinities of all interfaces are 'Automatic' - the affinity for each interface is automatically reset every 60 seconds, and balanced between available CPU cores based on the current load.

    • If SecureXL is disabled - the default affinities of all interfaces are with available CPU cores - those CPU cores that are not running a CoreXL FW instance or not defined as the affinity for a daemon.

    Setting interface affinities:

    • If SecureXL is enabled - the affinities of all interfaces are handled by the 'sim affinity' command. Interface affinities are automatically distributed among CPU cores that are not running CoreXL FW instances and that are not set as the affinity for any daemon.

    • If SecureXL is disabled - interface affinities are loaded at boot based on the $FWDIR/conf/fwaffinity.conf configuration file (if SecureXL is enabled, then lines beginning with "i" in this file are ignored).

    • To balance the load on the CPU cores that are running as SNDs (CPU cores that are not running CoreXL FW instances):

      1. Check the amount of interrupts from each interface - run:

        [Expert@HostName]# cat /proc/interrupts
      2. Check, which CPU cores (that run as SNDs) are most utilized - run:

        [Expert@HostName]# top

        Note: Press digit 1 (above the letter Q) to display all CPU cores and press Shift+W to save this configuration.

      3. Assign the interfaces accordingly across the CPU cores that run as SNDs - run:

        [Expert@HostName]# sim affinity -s

     

  • (3-6) Best practices - SMT (HyperThreading)

  • Show / Hide best practices information

    Notes:

    • SMT is not recommended if only FireWall/VPN blades are used, because performance improvement by SMT is achieved on NGFW software blades.

    • SMT is not recommended for environments that have high memory utilization.

      Reason: Firewall consumes memory for traffic inspection. If the memory utilization is already very high before enabling the SMT, the performance will decrease noticeably because SMT adds more CoreXL FW instances.

      To check the extent of memory utilization on the Security Gateway, refer to:

      • Initial diagnostics - Memory
      • Advanced diagnostics - Memory
    • SMT is not recommended if these blades/features are enabled:

      • Data Loss Prevention blade
      • Anti-Virus in Traditional Mode
      • Using Services with Resources in Firewall policy

      Reason: Each of these blades might have high memory consumption. These blades run Security Servers that are executed per CoreXL FW instance. Since SMT adds more CoreXL FW instances, overall memory consumption on Security Gateway might increase considerably.

      To check the extent of memory utilization on the Security Gateway, refer to:

      • Initial diagnostics - Memory
      • Advanced diagnostics - Memory
    • SMT is not recommended for environments that use Hide NAT extensively.

      Reason: An entire range of ports for Hide NAT is divided between the current CoreXL FW instances. The more CoreXL FW instances are running, the less ports for Hide NAT will be available for each CoreXL FW instance. As a result, if one CoreXL FW instance is handling a high number of NATed connections, its port range may get exhausted, while at the same time, other CoreXL FW instances may have enough available ports for Hide NAT.

      The port distribution is based on the following factors:

      • Number of CoreXL FW instances
      • Whether cluster is enabled
      • Whether SecureXL is enabled
      • Whether VPN blade is enabled

      To check the extent of NAT on the Security Gateway, use the cpsizeme tool (refer to sk88160):

      1. Run the tool for at least 24 hours.
      2. Run the 'cpsizeme -S' command.
      3. Select "Show summary of last successful session" (to see the summary of the collected statistics).
      4. Check the "Estimated average of NAT connections:" counter.
    • If you have two ports on 12600/12700/12800, 13500/13800 and 21000 appliances to handle most of the traffic, it is also recommended to enable Multi-Queue feature to increase SMT performance (refer to Performance Tuning Administration Guide (R76, R77) - Chapter 3 'Multi-queue').

    Documentation:

     

  • (3-7) Best practices - Multi-Queue

  • Show / Hide best practices information

    When Multi-Queue will not help:

    • When most of the processing is done in CoreXL - either in the Medium path, or in the Firewall path (Slow path).
    • All current CoreXL FW instances are highly loaded, so there are no CPU cores that can be reassigned to SecureXL.
    • When IPS, or other deep inspection Software Blades are heavily used.
    • When all network interface cards are processing the same amount of traffic.
    • When all CPU cores that are currently used by SecureXL are congested.
    • When trying to increase traffic session rate.
    • When there is not enough diversity of traffic flows. In the extreme case of a single flow, for example, traffic will be handled only by a single CPU core. (Clarification: The more traffic is passing to/from different ports/IP addresses, the more you benefit from Multi-Queue. If there is a single traffic flow from a single Client to a single Server, then Multi-Queue will not help.)

    Guidelines for configuring Multi-Queue:

    • Network interface cards must support Multi-Queue.
    • Multi-Queue is relevant only if SecureXL is enabled.
    • Multi-Queue is relevant only if CoreXL is enabled.
    • Examine the CPU affinity (for interfaces (SND) and for CoreXL FW instances).
    • Examine the CPU utilization.
    • Decide if more CPU cores can be allocated to the Secure Network Distributor (SND).

    Multi-Queue is recommended when all these conditions occur:

    • Load on CPU cores that run as SND is high (idle < 20%).
    • Load on CPU cores that run CoreXL FW instances is low (idle > 50%).
    • There are no CPU cores left to be assigned to the SND by changing interface affinity.

    Number of active RX queues:

    • By default on a Security Gateway, the number of active RX queues is calculated by:
      Number of Active RX queues = [Total Number of CPU cores] - [Number of CoreXL FW instances]

    • By default on a VSX gateway, the number of active RX queues is calculated by:
      Number of Active RX queues = The lowest CPU ID, to which the fwk process on VSX is assigned

    Notes:

    • For best performance, it is not recommended to assign both SND and a CoreXL FW instance to the same CPU core.
    • Do not change the IRQ affinity of queues manually. Changing the IRQ affinity of the queues manually can adversely affect performance.

    Documentation:

     

  • (3-8) Best practices - Rulebase optimization

  • Show / Hide best practices information

    1. Refer to sk32578 - SecureXL Mechanism to allow more connections to be accelerated by SecureXL.

    2. Place most used rules at the top - use the Hit Count in the SmartDashboard (R75.40 and above) and SmartView Monitor ('Top' view).
      Related solution: sk72860 - How to reset the 'Hit Count' in SmartDashboard.

      Note: the new column-based matching of Gateways of version R80.10 and above eliminates this need.

    3. In addition, you can provide a full Connections Table from your Security Gateway to Check Point Support for thorough analysis (run the command 'fw tab -t connections -u > /var/log/Connections_Table.txt').

     

  • (3-9) Best practices - IPS optimization

  • Show / Hide best practices information

    1. Create a dedicated IPS profile for each Security Gateway (to fine-tune the performance of each individual Gateway).

    2. Note that protections are categorized by performance impact. As such, enabling protections with "Critical" performance impact will highly increase CPU consumption.

    3. Enable / Disable protections depending on your network devices and your needs (do not enable all the available protections).

    4. Avoid setting protections to run in "Detect" mode - it might increase CPU consumption (without increasing the security).

    5. Identify the protections that consume most of the CPU resources - follow sk43733 - How to measure CPU time consumed by IPS protections.

    6. Disable protections that are not needed in your environment. To check, which protections are needed, you may use vulnerability tools (such as Nessus).

    7. Set Protection Scope to "Protect internal hosts only" (SmartDashboard - Security Gateway properties - IPS pane).

      Note: this optimization is no longer needed with the improved IPS mechanism for Gateways of version R80.10 and above.

    Related solutions:

     

  • (3-10) Best practices - Application Control & URL Filtering optimization

  • Show / Hide best practices information

    Application Control & URL Filtering policy:

    1. Create as specific rules as possible, to prevent the unwanted traffic from hitting the wrong rule.

      Limit the scope of the rule by selecting only the relevant services (instead of the default 'Any') - right-click on the column titles - click on the 'Service' column:

    2. Avoid using "Any" in "Source" and "Destination" columns.

    3. In the "Destination" column, use the default "Internet", unless a specific destination has to be explicitly selected (e.g., in case of internal servers).

    4. There is no need for "Clean Up" rule at the bottom (because the behavior of Application Control & URL Filtering Blades is different from that of the Firewall Blade).
      Such rule should only be used for short period of time and only for logging purposes.

    5. Remove the "Any - Any - Allow" rule, if such rule exists (because the behavior of Application Control & URL Filtering Blades is different from that of the Firewall Blade).
      Such rule should only be used for short period of time and only for logging purposes.

    Advanced - Engine Settings:

    1. Go to "Check Point Online Web Service" section.

    2. In the "Website categorization mode" section, select "Background" (to prevent latency due to packet holding until categorization is completed).

    Related solutions:

     

  • (3-11) Best practices - Anti-Virus & Anti-Bot optimization

  • Show / Hide best practices information

    Anti-Virus & Anti-Bot policy:

    1. Create as specific rules as possible, to prevent the unwanted traffic from hitting the wrong rule.

    2. Avoid using "Any" in "Source" and "Destination" columns.

    3. There is no need for "Clean Up" rule at the bottom (because the behavior of Anti-Virus & Anti-Bot Blades is different from that of the Firewall Blade).
      Such rule should only be used for short period of time and only for logging purposes.

    Advanced - Engine Settings:

    1. Go to "Check Point Online Web Service" section.

    2. In the "Website categorization mode" section, select "Background" (to prevent latency due to packet holding until categorization is completed).

    Decreasing performance impact:

    1. Edit the Anti-Virus & Anti-Bot profile:

      1. Go to 'Profiles' pane.

      2. Select the relevant profile - click on 'Edit...' button.

      3. Go to 'General Properties' pane.

      4. In the 'Performance Impact' field, select 'Low'.

      5. Click on 'OK'.

      6. Save the changes: go to 'File' menu - click on 'Save'.

      7. Install the policy onto the relevant Security Gateway / Cluster object.

    2. Exclude networks:

      • Consider excluding networks, whose traffic does not have to be inspected - follow sk92515 - How to configure Anti-Virus Exceptions.
      • Note: Standard exceptions are still being inspected (i.e., CPU is consumed), however the traffic will be allowed. The exceptions per sk92515 are completely excluded from the Anti-Virus & Anti-Bot engine inspection (i.e., decrease the load on CPU).
    3. Decrease the CPU consumption by RAD process and DLPU process:

    4. Disable scanning of archive file:

      Consider disabling 'Archive Scanning' because it requires high amount of CPU resources:

      1. Go to 'Profiles' pane.

      2. Select the relevant profile - click on 'Edit...' button.

      3. Go to 'Anti-Virus Settings' pane.

      4. In the 'Archives' section, uncheck the box 'Enable Archive scanning (impacts performance).'.

      5. Click on 'OK'.

      6. Save the changes: go to 'File' menu - click on 'Save'.

      7. Install the policy onto the relevant Security Gateway / Cluster object.

    Related solutions:

     


     

    (4) Initial diagnostics

     

  • (4-1) Initial diagnostics - CPU

  • Show / Hide initial diagnostics information

    1. cpview

      Background:

      Diagnostics:

      • Monitor the CPU utilization during the problem

      Analysis:

      • On the 'Overview' tab, refer to 'CPU:' section - look at counter 'Used'
      • On the 'I/S' tab, go to 'CPU' menu - go to 'Overview' menu - refer to section 'CPU'
    2. cpstat -f multi_cpu os

      Background:

      • Displays internal statistics for OS about all CPU cores as collected by Check Point

      Diagnostics:

      • Collect this output continuously during the problem

      Analysis:

      • Look at all the columns

      Example from machine with 8 CPU cores:

      Processors load
      ---------------------------------------------------------------------------------
      |CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
      ---------------------------------------------------------------------------------
      |   1|           1|             2|          96|       4|        ?|             0|
      |   2|           2|             4|          94|       6|        ?|             0|
      |   3|           1|             1|          98|       2|        ?|             0|
      |   4|           2|             3|          94|       6|        ?|             0|
      ---------------------------------------------------------------------------------
      
    3. top

      Background:

      Usage:

      • When running the command for the first time:

        1. Press '1' to display all CPU cores
        2. Press 'd' to set update interval - type 2 and press Enter
        3. Press 'n' to set maximum tasks displayed - type 15 and press Enter
        4. Press Shift+W to save the top's configuration
      • When running the command for diagnostics:

        • Press Shift+P to sort the output by CPU utilization
        • Press Shift+M to sort the output by Memory utilization

      Diagnostics:

      • Collect this output continuously during the problem

      Analysis:

      • Look at the load in "User Space" - counter us
        High CPU consumption in "User Space" can be caused by processes that perform heavy tasks (e.g., too much logging by fwd, reloading the configuration during policy installation, etc.)
      • Look at the load in "System (kernel) Space" - counter sy
        High CPU consumption in "System (kernel) Space" can be caused by heavy tasks (e.g., deep inspection of packets, enabling of all blades, enabling of all IPS protections in Prevent mode, etc.)
      • Look at the amount of "Idle" - counter id
        The more CPU is idle, the better the machine's performance is
      • Look at the amount of "I/O waiting" - counter wa
        High amount of "I/O waiting" is caused by heavy reading from/writing to hard disk (e.g., during policy installation, heavy logging, insufficient RAM, etc.)
      • Look at the amount of "SoftIRQ" - counter si
        High amount of "SoftIRQ" is usually caused by high amount of traffic
      • Look at the amount of "Hardware IRQs" - counter hi
      • Look at the CPU consumption by the processes - column %CPU
        Constant high CPU consumption by a process can be caused by numerous factors - function stack should be collected from the process using a special Check Point shell script ('pstack') - refer to "(5-1) Advanced diagnostics - CPU" section
        Related solution: sk116740 - "top" command on VSX Gateway shows that FWK process consumes CPU at more than 100%.
      • Look at the Memory consumption by the processes - column %MEM
        Constant increase in memory consumption by a process might suggest some memory leak - valgrind tool should be used to collect the necessary information from the process - refer to "(5-2) Advanced diagnostics - Memory" section

      Example from machine with 2 CPU cores:

      top - 15:18:47 up 14:06,  1 user,  load average: 0.55, 0.37, 0.35
      Tasks: 213 total,   1 running, 212 sleeping,   0 stopped,   0 zombie
      Cpu0  :  0.7%us,  0.7%sy,  0.0%ni, 97.3%id,  0.0%wa,  0.0%hi,  1.4%si,  0.0%st
      Cpu1  :  0.7%us,  0.7%sy,  0.0%ni, 97.3%id,  0.0%wa,  0.0%hi,  1.4%si,  0.0%st
      Cpu2  :  0.0%us,  0.7%sy,  0.0%ni, 98.6%id,  0.0%wa,  0.0%hi,  0.7%si,  0.0%st
      Cpu3  :  2.0%us,  3.4%sy,  0.0%ni, 93.3%id,  0.0%wa,  0.0%hi,  1.3%si,  0.0%st
      Mem:   2005908k total,  1907412k used,    98496k free,   103952k buffers
      Swap:  4225084k total,   323268k used,  3901816k free,   198288k cached
      
        PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
      16948 admin     24   0 71692  10m 4744 S    2  0.6  15:14.00 DAService
       4103 admin     15   0     0    0    0 S    1  0.0   8:38.37 fw_worker_0
       4105 admin     15   0     0    0    0 S    1  0.0  10:46.19 fw_worker_2
       4104 admin     15   0     0    0    0 S    1  0.0   4:05.49 fw_worker_1
          1 admin     15   0  2040  720  624 S    0  0.0   0:01.44 init
          2 admin     RT  -5     0    0    0 S    0  0.0   0:01.82 migration/0
          3 admin     15   0     0    0    0 S    0  0.0   0:00.02 ksoftirqd/0
      ... ... ...
      
    4. ps auxwwwf

      Background:

      Usage:

      • Refer to the manual page
      • By default, output is sorted by PID column
      • Simplest sorting can be done by pipelining the output of ps command to sort command - for example, to sort the CPU usage, run:
        [Expert@HostName]# ps auxww | sort -nrk 3

      Diagnostics:

      • Collect this output continuously during the problem

      Analysis:

      • Look at the amount of "CPU", "MEM", "VSZ", "RSS", "TIME" consumed by the daemons
      • Constant increase in memory consumption might suggest some memory leak - valgrind tool should be used to collect the necessary information from the process - refer to "(5-2) Advanced diagnostics - Memory" section
      • Constant high CPU consumption can be caused by numerous factors - function stack should be collected from the process using a special Check Point shell script ('pstack') - refer to "(5-1) Advanced diagnostics - CPU" section

      Example (excerpt):

      USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
      admin        1  0.0  0.0   2040   720 ?        Ss   01:12   0:01 init [3]
      admin        2  0.0  0.0      0     0 ?        S<   01:12   0:01 [migration/0]
      admin        3  0.0  0.0      0     0 ?        S    01:12   0:00 [ksoftirqd/0]
      admin        4  0.0  0.0      0     0 ?        S<   01:12   0:00 [watchdog/0]
      ..................
      admin     4103  1.0  0.0      0     0 ?        S    01:13   8:41 [fw_worker_0]
      admin     4104  0.4  0.0      0     0 ?        S    01:13   4:07 [fw_worker_1]
      admin     4105  1.2  0.0      0     0 ?        S    01:13  10:50 [fw_worker_2]
      admin     4115  0.0  0.2  13976  4184 ?        Ss   01:13   0:02 /opt/CPshrd-R77/bin/cpwd
      admin     5098  0.2  2.4 342988 49860 ?        Ssl  01:13   2:07  \_ cpd
      admin     5136  0.0  0.3 153612  6216 ?        Ss   01:13   0:00  \_ mpdaemon /opt/CPshrd-R77/log/
      ..................
      admin     5374  0.1  5.6 610672 113844 ?       Ssl  01:13   1:32  \_ fwd
      admin     5449  0.0  0.4  27856  9232 ?        S    01:13   0:00  |   \_ cpca
      admin     5720  0.0  2.0 207256 41100 ?        S    01:13   0:01  |   \_ syslog 514 all
      admin     6030  0.0  2.7 295536 54336 ?        Sl   01:14   0:06  |   \_ vpnd 0
      admin     6033  0.0  2.5 239748 52012 ?        Sl   01:14   0:15  |   \_ pdpd 0 -t
      admin     6034  0.0  2.5 211336 50924 ?        S    01:14   0:11  |   \_ pepd 0 -t
      admin     6036  0.0  2.1 233164 43804 ?        Sl   01:14   0:02  |   \_ fwpushd 0
      admin     6038  0.0  0.6  35016 13232 ?        S    01:14   0:00  |   \_ wstlsd 0 0
      ..................
      admin     5382  0.5  7.5 453272 151396 ?       Ssl  01:13   4:18  \_ fwm
      admin     5390  0.0  1.0  46240 21504 ?        Ss   01:13   0:00  \_ status_proxy
      admin     5750  0.0  0.8  35228 17264 ?        Ss   01:13   0:01  \_ rad
      admin     5827  0.0  1.1  42500 22748 ?        Ss   01:13   0:00  \_ cpstat_monitor
      admin     5996  0.0  0.5  57176 11516 ?        Ssl  01:14   0:02  \_ fwucd
      admin     6022  0.0  1.0 132424 21652 ?        Ssl  01:14   0:03  \_ dlpu -i 0 0
      ..................
      admin     4636  0.0  0.3  25020  7784 ?        Ss   01:13   0:00 /bin/pm
      admin     4653  0.1  0.5  31620 11908 ?        Ss   01:13   1:02  \_ /bin/confd
      admin     4654  0.1  0.7  38040 14312 ?        SNsl 01:13   0:57  \_ /bin/searchd -niceboost 10
      admin     4656  0.0  0.4  99740  9880 ?        Ssl  01:13   0:01  \_ /bin/rconfd /etc/actions_mapping.xml
      admin     4657  0.0  0.1   6472  3824 ?        Ss   01:13   0:18  \_ /bin/monitord
      admin     4683  0.0  0.3  25252  7168 ?        Ss   01:13   0:00  \_ /bin/cloningd
      admin     4685  0.0  0.1  10048  2348 pts/0    Ss+  01:13   0:00  \_ /bin/clishd default server
      admin     4687  0.0  0.1  19032  2796 ?        Ssl  01:13   0:00  \_ /bin/clish -p
      admin     4688  0.0  0.0   2948   868 ?        Ss   01:13   0:00  \_ /usr/bin/tclsh /usr/libexec/netflowd
      admin     4689  0.0  0.3  30360  7368 ?        Ss   01:13   0:07  \_ /usr/sbin/snmpd -f -c /etc/snmp/userDefinedSettings.conf
      admin     5041  0.0  0.3  32768  6912 ?        Ss   01:13   0:00  \_ /bin/routed -N
      admin     5047  0.0  0.3  33128  7820 ?        S    01:13   0:00  |   \_ /bin/routed -i default -f /etc/routed0.conf -h 0
      ..................
      admin     9438  0.0  0.0   1652   496 tty1     Ss+  01:14   0:00 /sbin/agetty 9600 tty1
      admin     9439  0.0  0.0   1648   500 tty2     Ss+  01:14   0:00 /sbin/agetty 9600 tty2
      admin     9440  0.0  0.0   1652   504 tty3     Ss+  01:14   0:00 /sbin/agetty 9600 tty3
      admin     9441  0.0  0.0   2416  1052 ?        Ss   01:14   0:00 /bin/bash /bin/console_agetty
      admin     9463  0.0  0.0   1652   508 tty4     Ss+  01:14   0:00  \_ /sbin/agetty 9600 tty4 vt100
      admin    16948  2.1  0.5  71692 11072 ?        Sl   03:15  15:21 /opt/CPda/bin/DAService
      
    5. cat /proc/interrupts

      Background:

      Usage:

      • Refer to the manual page
      • By default, output is sorted by IRQ number
      • The only relevant devices are network interfaces
      • To shorten the output, use the 'grep' command - for example, run:
        [Expert@HostName]# cat /proc/interrupts | grep -E "CPU|eth"
      • To collect this output continuously, use the 'watch' command - for example, run:
        [Expert@HostName]# watch -d -n 1 'cat /proc/interrupts | grep -E "CPU|eth"'

      Diagnostics:

      • Collect this output continuously during the problem

      Analysis:

      • Look at the general trend - which CPU receives more interrupts and from which interfaces
      • If some CPU cores receive more interrupts than others, then affinity of interfaces to CPU cores should be optimized - interface should be redistributed better

      Example from machine with 4 CPU cores and 4 interfaces:

                 CPU0       CPU1       CPU2       CPU3
        0:   51254849          0          0          0    IO-APIC-edge  timer
        1:          4          0          0          5    IO-APIC-edge  i8042
        6:          2          3          0          0    IO-APIC-edge  floppy
        7:          0          0          0          0    IO-APIC-edge  parport0
        8:          3          0          0          0    IO-APIC-edge  rtc
        9:          0          0          0          0   IO-APIC-level  acpi
       12:          0          0          0        115    IO-APIC-edge  i8042
       15:        302       1043        168       1127    IO-APIC-edge  ide1
       59:     320145       6732       5944       6189   IO-APIC-level  ioc0, eth3
       67:   15765753          0          0          0   IO-APIC-level  eth0
       75:      24271       1285         36          0   IO-APIC-level  eth1
       83:      24272          0       1322          0   IO-APIC-level  eth2
      NMI:          0          0          0          0
      LOC:   50950084   50945459   50949418   50945128
      ERR:          0
      MIS:          0
      
    6. cat /proc/cpuinfo

      Background:

      • Manual page - http://linux.die.net/man/5/proc
      • Displays a collection of CPU and system architecture dependent items about CPU (on multi-CPU (SMP) machines will show information for each CPU)

      Diagnostics:

      • Collect this output to see the information about CPU (architecture, vendor, number)

      Analysis:

      • Look at the processor number
      • Look at the processor model
      • Look at the CPU clock frequency
      • Look at the supported flags (e.g., pae)

      Example (excerpt):

      processor       : 0
      vendor_id       : GenuineIntel
      cpu family      : 6
      model           : 44
      model name      : Intel(R) Xeon(R) CPU           E5645  @ 2.40GHz
      stepping        : 2
      cpu MHz         : 2399.483
      cache size      : 12288 KB
      physical id     : 0
      siblings        : 2
      core id         : 0
      cpu cores       : 2
      fdiv_bug        : no
      hlt_bug         : no
      f00f_bug        : no
      coma_bug        : no
      fpu             : yes
      fpu_exception   : yes
      cpuid level     : 11
      wp              : yes
      flags           : fpu vme de pse tsc msr pae mce ...
      bogomips        : 4806.12
      
    7. dmesg

      Background:

      Diagnostics:

      • Collect this output to search for errors from various kernel modules and hardware components

      Analysis:

      • Check each line for kernel boot parameters (e.g., vmalloc), errors, failures, components/features being disabled

     

  • (4-2) Initial diagnostics - Memory

  • Show / Hide initial diagnostics information

    Related solution: sk115072 - How to determine amount of installed RAM on Check Point Appliance.

    1. cpview

      Background:

      Diagnostics:

      • Monitor the Memory utilization during the problem

      Analysis:

      • On the 'Overview' tab, refer to 'Memory:' section - look at counters 'Free MB'
      • On the 'I/S' tab, go to 'Memory' menu - go to 'FW-Kernel' menu - refer to section 'Firewall kernel memory usage summary:' - look at counter '%Usage'
    2. fw ctl pstat

      Background:

      • Displays FireWall internal statistics about memory and traffic

      Diagnostics:

      • Collect the output before and after the suspected problem
      • Use different flags to get more data:
        • for HMEM: fw ctl pstat -h
        • for SMEM: fw ctl pstat -s
        • for KMEM: fw ctl pstat -k
        • for Handles (kbufs): fw ctl pstat -l
      • Counters are reset when Check Point Services are stopped (with 'cpstop' command)
      • Under memory, 'allocations' counter always grows, may wrap around

      Analysis:

      • No single field that indicates a problem - need to interpret all counters together
      • HMEM
        • failures under HMEM - no real memory problem, just mean HMEM is full ; HMEM should have been configured larger
        • "failed allocations" under HMEM (only) do not indicate any problem
      • SMEM
        • failures under SMEM - reached Check Point memory limit , exhausted OS memory, large non-sleep allocation , indicate some shortage
        • "failed allocations" under SMEM may not mean that a user's allocation failed, maybe HMEM extension failed
        • "failed free" under SMEM means an overrun or freeing an invalid pointer - indicates a bug
      • KMEM
        • failures under KMEM - application asked for memory and could not get it , usually, it is a memory problem
        • "failed allocations" under KMEM means that the application didn't get memory

      Example:

      System Capacity Summary:
        Memory used: 12% (183 MB out of 1515 MB) - below watermark
        Concurrent Connections: 79 (Unlimited)
        Aggressive Aging is not active
      
      Hash kernel memory (hmem) statistics:
        Total memory allocated: 155189248 bytes in 37888 (4096 bytes) blocks using 37 pools
        Total memory bytes  used: 33563132   unused: 121626116 (78.37%)   peak: 53675664
        Total memory blocks used:     9082   unused:    28806 (76%)   peak:    13407
        Allocations: 96170689 alloc, 0 failed alloc, 95799050 free
      
      System kernel memory (smem) statistics:
        Total memory  bytes  used: 280891432   peak: 297288060
        Total memory bytes wasted: 12156438
          Blocking  memory  bytes   used:   902804   peak:   933368
          Non-Blocking memory bytes used: 279988628   peak: 296354692
        Allocations: 10526 alloc, 1 failed alloc, 8437 free, 0 failed free
        vmalloc bytes  used:  7340032 expensive: yes
      
      Kernel memory (kmem) statistics:
        Total memory  bytes  used: 159039668   peak: 185833572
        Allocations: 96179738 alloc, 1 failed alloc
                     95807459 free, 0 failed free
        External Allocations: 0 for packets, 63652 for SXL
      
      Cookies:
              17293941 total, 0 alloc, 0 free,
              19087 dup, 17386724 get, 117090 put,
              32037742 len, 0 cached len, 0 chain alloc,
              0 chain free
      
      Connections:
              82980 total, 31766 TCP, 50795 UDP, 7 ICMP,
              412 other, 0 anticipated, 0 recovered, 79 concurrent,
              568 peak concurrent
      
      Fragments:
              0 fragments, 0 packets, 0 expired, 0 short,
              0 large, 0 duplicates, 0 failures
      
      NAT:
              0/0 forw, 0/0 bckw, 0 tcpudp,
              0 icmp, 0-0 alloc
      
      Sync: off
      
    3. cpstat -f memory os

      Background:

      • Displays internal statistics for OS about memory as collected by Check Point

      Diagnostics:

      • Collect this output continuously during the problem (usually, over some time period)

      Analysis:

      Example:

      Total Virtual Memory (Bytes):  6380535808
      Active Virtual Memory (Bytes): 1882390528
      Total Real Memory (Bytes):     2054049792
      Active Real Memory (Bytes):    1643986944
      Free Real Memory (Bytes):      410062848
      Memory Swaps/Sec:              -
      Memory To Disk Transfers/Sec:  -
      
    4. cat /proc/meminfo

      Background:

      • Manual page - http://linux.die.net/man/5/proc
      • Displays the amount of free and used memory (both physical and swap) on the system as well as the shared memory and buffers used by the kernel

      Diagnostics:

      • Collect this output continuously during the problem (usually, over some time period)

      Analysis:

      Example:

      MemTotal:      2005908 kB
      RawMemTotal:   2097152 kB
      MemFree:         95092 kB
      Buffers:        107136 kB
      Cached:         198520 kB
      SwapCached:      90452 kB
      Active:        1335364 kB
      Inactive:       121220 kB
      HighTotal:      262016 kB
      HighFree:          808 kB
      LowTotal:      1743892 kB
      LowFree:         94284 kB
      SwapTotal:     4225084 kB
      SwapFree:      3901816 kB
      Dirty:             864 kB
      Writeback:           0 kB
      AnonPages:     1137312 kB
      Mapped:         108660 kB
      Slab:            82164 kB
      PageTables:      19132 kB
      NFS_Unstable:        0 kB
      Bounce:              0 kB
      CommitLimit:   5228036 kB
      Committed_AS:  4025512 kB
      VmallocTotal:   247800 kB
      VmallocUsed:    114732 kB
      VmallocChunk:   117952 kB
      HugePages_Total:     0
      HugePages_Free:      0
      HugePages_Rsvd:      0
      Hugepagesize:     2048 kB
      
    5. dmesg

      Background:

      Diagnostics:

      • Collect this output to search for errors from various kernel modules and hardware components

      Analysis:

      • Check each line for kernel boot parameters (e.g., vmalloc), errors, failures, components/features being disabled

     

  • (4-3) Initial diagnostics - Network interface cards

  • Show / Hide initial diagnostics information

    Important related solution: sk110351 - Traffic latency on VSX Gateway if MTU larger than 4096 (Jumbo Frames) is configured on an interface.

    1. cpview

      Background:

      Diagnostics:

      • Monitor the traffic and interfaces during the problem

      Analysis:

      • On the 'Overview' tab, refer to 'Traffic counters:' section
      • On the 'SysInfo' tab, refer to 'Hardware Information:' section
      • On the 'Traffic' tab, go to 'Overview' menu - refer to 'Traffic Rate:' section
      • On the 'Traffic' tab, go to 'Overview' menu - refer to 'Concurrent Connections:' section
      • On the 'Traffic' tab, go to 'Overview' menu - refer to 'Interface drops:' section
      • On the 'Traffic' tab, go to 'Interfaces' menu - refer to 'Errors and Drops:' section
    2. netstat -ni

      Background:

      Diagnostics:

      • Collect this output continuously during the problem (usually, over some time period)

      Analysis:

      • RX-OK - number of packets that have been received error-free
      • RX-ERR - number of packets that have been received damaged (refer to sk61922)
      • RX-DRP - number of received packets that have been dropped (refer to sk61922)
      • RX-OVR - number of received packets that have been lost because of an overrun (number of times the receiver hardware was unable to hand received data to a hardware buffer - the internal FIFO buffer of the chip is full, but is still tries to handle incoming traffic ; most likely, the input rate of traffic exceeded the ability of the receiver to handle the data)
      • TX-OK - number of packets that have been transmitted error-free
      • TX-ERR - number of packets that have been transmitted damaged
      • TX-DRP - number of transmitted packets that have been dropped
      • TX-OVR - number of transmitted packets that have been lost because of an overrun
      • Flg - shows the flags that have been set for this interface - these characters are one-character versions of the long flag names that are displayed in the output of 'ifconfig' command:
        • A = this interface will receive all Multicast addresses
        • B = a Broadcast address has been set
        • D = debugging is turned on
        • L = this interface is a loopback device
        • M = all packets are received (promiscuous mode)
        • m = master
        • N = trailers are avoided
        • O = ARP is turned off for this interface
        • P = this is a Point-to-Point connection
        • R = interface is running
        • s = slave
        • U = interface is up

      Example:

      Kernel Interface table 
      Iface       MTU  Met      RX-OK RX-ERR  RX-DRP RX-OVR      TX-OK TX-ERR TX-DRP TX-OVR Flg
      br0        1500    0    4384062      0       0      0         28      0      0      0 BMRU
      eth0       1500    0   12505484      0       0      0   20359171      0      0      0 BMRU
      eth1       1500    0   11882710      0      62      0  449677832      0      0      0 BMRU
      eth2       1500    0 2592575900 362803 3626586      0 1158425513      0      0      0 BMRU
      eth3       1500    0 1189952933     30 9382997      0 2575575506      0      0      0 BMRU
      lo        16436    0     174524      0       0      0     174524      0      0      0 LRU
      
    3. ifconfig name_of_interface

      Background:

      Diagnostics:

      • Collect this output continuously during the problem (usually, over some time period)

      Analysis:

      • UP - indicates that the kernel modules related to the Ethernet interface has been loaded
      • BROADCAST - indicates that interface supports broadcasting - a necessary characteristic to obtain IP address via DHCP
      • NOTRAILERS - indicates that trailer encapsulation is disabled (Linux usually ignore trailer encapsulation so this value has no effect at all)
      • RUNNING - indicates that interface is ready to accept data
      • MULTICAST - indicates that the interface supports multicasting
      • MTU - Maximum Transmission Unit is the size of each packet received by the interface (default value is 1500; setting MTU to a higher value could hazard packet fragmentation or buffer overflows)
      • Metric - used to compute the cost of a route - it tells the OS, to which interface a packet should be forwarded, when multiple interfaces could be used to reach the packet's destination
        Takes values of 0,1,2,3,... The lower the value, the more leverage it has. This parameter has significance only while routing packets. For example, if you have two Ethernet cards and you want to forcibly make your machine use one card over the other in sending the data. Then you can set the Metric value of the Ethernet card which you favor lower than that of the other Ethernet card.
      • RX packets - total number of packets received via the interface
      • RX errors - number of damaged packets received (refer to sk61922)
      • RX dropped - number of dropped packets due to reception errors (refer to sk61922)
      • RX overruns - number of received packets that experienced data overruns (number of times the receiver hardware was unable to hand received data to a hardware buffer)
      • RX frame - number received packets that experienced frame errors
      • TX packets - total number of packets transmitted via the interface
      • TX errors - number of packets that experienced transmission error
      • TX dropped - number of dropped transmitted packets due to transmission errors
      • TX overruns - number of transmitted packets that experienced data overruns (number of times the transmitter hardware was unable to hand received data to a hardware buffer)
      • TX carriers - number received packets that experienced loss of carriers
      • TX collisions - number of transmitted packets that experienced Ethernet collisions (a nonzero value of this field indicates possibility of network congestion)
      • TX txqueuelen - configured length of transmission queue
      • RX bytes - total bytes received over this interface
      • TX bytes - total total bytes transmitted over this interface

      Example:

      eth0        Link encap:Ethernet  HWaddr 00:50:56:AA:69:33
                  inet addr:172.30.41.90  Bcast:172.30.255.255  Mask:255.255.0.0
                  UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
                  RX packets:21363169 errors:0 dropped:0 overruns:0 frame:0
                  TX packets:128400 errors:0 dropped:0 overruns:0 carrier:0
                  collisions:0 txqueuelen:1000
                  RX bytes:1608073037 (1.4 GiB)  TX bytes:33703584 (32.1 MiB)
      
    4. netstat -anp

      Background:

      Diagnostics:

      • Collect this output continuously during the problem (usually, over some time period)

      Analysis:

      • Under "Active Internet connections" look at "Recv-Q" and at "Send-Q"
      • Recv-Q - data (in bytes), which has not yet been pulled from the socket buffer by the application (value should be as close to 0 as possible)
      • Send-Q - data (in bytes), which the sending application has given to the transport, but has yet to be ACKnowledged by the receiving TCP (value should be as close to 0 as possible - a large number may indicate a network bottleneck)

      Example:

      [Expert@FW]# netstat -anp
      Active Internet connections (servers and established)
      Proto Recv-Q Send-Q Local Address           Foreign Address         State          PID/Program name
      tcp        0   2368 172.30.20.69:22         172.30.20.228:3676      ESTABLISHED 16179/0
      udp   256956      0 0.0.0.0:9282            0.0.0.0:*
      [Expert@FW]#
      
    5. ethtool name_of_interface

      Background:

      Diagnostics:

      • Collect this output to check the current status

      Analysis:

      • Check every line of the output
      • Configuration on both ends of the cable should be identical (speed /duplex / auto-negotiation)

      Example:

      Settings for eth0:
              Supported ports: [ TP ]
              Supported link modes:   10baseT/Half 10baseT/Full
                                      100baseT/Half 100baseT/Full
                                      1000baseT/Full
              Supports auto-negotiation: Yes
              Advertised link modes:  10baseT/Half 10baseT/Full
                                      100baseT/Half 100baseT/Full
                                      1000baseT/Full
              Advertised auto-negotiation: Yes
              Speed: 1000Mb/s
              Duplex: Full
              Port: Twisted Pair
              PHYAD: 1
              Transceiver: internal
              Auto-negotiation: on
              Supports Wake-on: d
              Wake-on: d
              Current message level: 0x00000007 (7)
              Link detected: yes
      
    6. ethtool -i name_of-interface

      Background:

      Diagnostics:

      • Collect this output to check the current status

      Analysis:

      • Use driver with NAPI
      • Use the latest version of the driver

      Example:

      driver: e1000
      version: 7.6.15.5-NAPI
      firmware-version: N/A
      bus-info: 0000:02:00.0
      
    7. ethtool -g name_of-interface

      Background:

      Diagnostics:

      • Collect this output to check the current status

      Analysis:

      Example:

      Ring parameters for eth0:
      Pre-set maximums:
      RX:             4096
      RX Mini:        0
      RX Jumbo:       0
      TX:             4096
      Current hardware settings:
      RX:             256
      RX Mini:        0
      RX Jumbo:       0
      TX:             1024
      
    8. arp -an | wc -l

      Background:

      Diagnostics:

      • Collect this output to check the current status

      Analysis:

      Example:

      [Expert@HostName]# arp -an | wc -l
      1
      [Expert@HostName]# arp -an
      ? (172.30.1.1) at 00:1C:EB:1C:B4:43 [ether] on eth0
      [Expert@HostName]# arp -en
      Address                  HWtype  HWaddress           Flags Mask            Iface
      172.30.1.1               ether   00:1C:EB:1C:B4:43   C                     eth0
      [Expert@HostName]# 
      
    9. dmesg

      Background:

      Diagnostics:

      • Collect this output to search for errors from various kernel modules and hardware components

      Analysis:

      • Check each line for kernel boot parameters (e.g., vmalloc), errors, failures, components/features being disabled

     

  • (4-4) Initial diagnostics - SecureXL

  • Show / Hide initial diagnostics information

    1. cpview

      Background:

      Diagnostics:

      • Monitor the SecureXL performance during the problem

      Analysis:

      • On the 'SysInfo' tab, refer to 'Configuration Information:' section - look at 'PPack Status'
      • On the 'Traffic' tab, go to 'Overview' menu - refer to section 'Templates:'
      • On the 'I/S' tab, go to 'SXL' menu - go to 'Overview' menu
    2. fwaccel stat

      Background:

      • Displays the status of SecureXL and SecureXL Templates

      Diagnostics:

      • Collect this output to check the current status

      Analysis:

      Example:

      Accelerator Status : on
      Accept Templates   : disabled by Firewall
                                     disabled from rule #31
      Drop Templates     : disabled
      NAT Templates      : disabled by user
      
      Accelerator Features : Accounting, NAT, Cryptography, Routing,
                             HasClock, Templates, Synchronous, IdleDetection,
                             Sequencing, TcpStateDetect, AutoExpire,
                             DelayedNotif, TcpStateDetectV2, CPLS, WireMode,
                             DropTemplates, NatTemplates, Streaming,
                             MultiFW, AntiSpoofing, ViolationStats,
                             Nac, AsychronicNotif, ERDOS
      Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
                              3DES, DES, CAST, CAST-40, AES-128, AES-256,
                              ESP, LinkSelection, DynamicVPN, NatTraversal,
                              EncRouting, AES-XCBC, SHA256
      
    3. fwaccel stats -s

      Background:

      • Displays summary of SecureXL acceleration statistics

      Diagnostics:

      • Collect this output continuously during the problem

      Analysis:

      • If the ratio of "Accelerated/Total" is less than 80%, it means that acceleration is poor, and the optimization is required
      • The higher the ratio of "F2Fed pkts/Total pkts" is, the more traffic is Forwarded to Firewall for inspection (the poorer the performance is) - see the fwaccel conns command below
      • The higher the ratio of "PXL pkts/Total pkts" is, the more packets passed in Medium path. Meaning, packets were handled by the SecureXL device, except for IPS / Application Control / Anti-Virus / Anti-Bot processing. The CoreXL layer passed the packets to one of the CoreXL FW instances to perform the processing. This path is available only when CoreXL is enabled.
      • The higher the ratio of "QXL pkts/Total pkts" is, the more packets were processed by QoS.

      Example:

      Accelerated conns/Total conns : 364/13215 (2%)
      Delayed conns/(Accelerated conns + PXL conns) : 48/12023 (0%)
      Accelerated pkts/Total pkts   : 18252/564927 (3%)
      F2Fed pkts/Total pkts   : 36776/564927 (6%)
      PXL pkts/Total pkts   : 509899/564927 (90%)
      QXL pkts/Total pkts   : 0/564927 (0%)
      
    4. fwaccel conns

      Background:

      • Displays entries from SecureXL connections table

      Diagnostics:

      • Collect this output continuously during the problem (Important Note: This command consumes high amount of memory)

      Analysis:

      • Look at the Flags column:
        • F = Forward to Firewall - the connection is not accelerated
          • To quickly determine the amount of F2F connections:
            1. Check the total number of connections in SecureXL connections table:
              [Expert@HostName]# fwaccel conns -s
            2. Check the total number of F2F connections:
              [Expert@HostName]# fwaccel conns | grep ' F' | grep -v 'Flags' | wc -l
          • To see why specific connections are Forwarded to Firewall, run kernel debug. Refer to "(5-4) Advanced diagnostics - SecureXL" section
        • U = Unidirectional - the connection can pass data on either C2S or S2C - data packets from the opposite direction will be F2F'ed
        • N = NAT is being performed on the connection by the device
        • A = Accounting is performed on the connection (the connection is viewed by either rulebase accounting or SmartView Monitor)
        • C = Encryption is done on the connection by the device
        • W = the connection is in wire mode
        • P = Partial (versions R70 and higher)
        • S = Streaming - PXL (versions R70 and higher)

      Example (all IP addresses were replaced by "X"):

      Source        SPort Destination    DPort PR Flags   C2S i/f   S2C i/f
      ------------- ----- -------------- ----- -- ------- --------- ---------
         X.X.X.X     61242    X.X.X.X      80  6 ..N.... eth5/eth0 eth0/eth5
         X.X.X.X      6000    X.X.X.X    3842  6 ....... eth0/eth4 eth4/eth0
         X.X.X.X      1620    X.X.X.X      88 17 ....... eth4/eth2 eth2/eth4
         X.X.X.X     50829    X.X.X.X      80  6 ..N.... eth5/eth0 eth0/eth5
         X.X.X.X      4285    X.X.X.X      80  6 F.N.... eth5/eth0 eth0/eth5
         X.X.X.X        80    X.X.X.X   49312  6 F.N.... eth5/eth0 eth0/eth5
         X.X.X.X        80    X.X.X.X   11450  6 ..N.... eth5/eth0 eth0/eth5
         X.X.X.X     58562    X.X.X.X      80  6 F.N.... eth5/eth0 eth0/eth5
         X.X.X.X       161    X.X.X.X    5002 17 F...... eth2/eth2 -/-
         X.X.X.X     21891    X.X.X.X       0  1 F...... eth2/eth4 eth4/eth2
         X.X.X.X     34303    X.X.X.X     389  6 F.N.... eth2/eth2 eth2/-
      
    5. fwaccel templates

      Background:

      • Displays SecureXL Connection Templates
        Note: Size of Templates table (cphwd_tmpl, id 8111) is limited to 1/4 of the size of Firewall Connections Table (connections, id 8158).

      Diagnostics:

      • Collect this output continuously during the problem (Important Note: This command consumes high amount of memory)

      Analysis:

      • Check whether templates are created:
        [Expert@HostName]# fwaccel templates -s
      • Check whether templates were created for relevant connections (search for relevant IP addresses using the grep command)
      • Look at the Flags column:
        • You should not see connections with F, N, or C flags
        • F = Forward to Firewall - the connection is not accelerated
        • U = Unidirectional - the connection can pass data on either C2S or S2C - data packets from the opposite direction will be F2F'ed
        • N = NAT is being performed on the connection by the device
        • A = Accounting is performed on the connection (the connection is viewed by either rulebase accounting or SmartView Monitor)
        • C = Encryption is done on the connection by the device
        • W = the connection is in wire mode
        • P = Partial (versions R70 and higher)
        • S = Streaming - PXL (versions R70 and higher)
        • D = Drop Template
        • L = Log drop action
      • The LCT column - shows how many seconds ago a connection was created by the template
      • The DLY column - shows Delayed Synchronization value for the template

      Example (all IP addresses were replaced by "X"):

      Source       SPort Destination  DPort PR  Flags     LCT DLY C2S i/f S2C i/f Inst Identity
      ------------ ----- ------------ ----- -- --------- ---- --- ------- ------- ---- --------
        X.X.X.X     *    X.X.X.X        161 17 ...A...S.   65   0 36/21   21/36    0        0
        X.X.X.X     *    X.X.X.X        161 17 ...A...S.    5   0 36/8    8/36     1        0
        X.X.X.X     *    X.X.X.X       1437 17 ...A...S.   27   0 36/8    8/36     1        0
        X.X.X.X     *    X.X.X.X        161 17 ...A...S.   23   0 36/21   21/36    2        0
        X.X.X.X     *    X.X.X.X         88 17 ...A...S.   11   0 8/36    36/8     4        0
      
    6. sim if

      Background:

      • Displays the list of interfaces used and seen by the SecureXL

      Diagnostics:

      • Collect this output to see the current status

      Analysis:

      • Look at the Name column and Address column
      • Look at the F column (refer to "(6-1-B) 'sim' command" section - 'sim if' command)

      Example:

       Name      | Address         | MTU  | F | SIM F | IRQ | Dev        | Output
       --------------------------------------------------------------------------------
       eth0      | 172.30.168.38   | 1500 | 039 | 00000 |  67 | 0x85b1b000 | 0xffffffff
       eth1      | 10.20.30.38     | 1500 | 029 | 00008 |  75 | 0xbc5d3000 | 0xffffffff
       eth2      | 10.5.0.1        | 1500 | 029 | 00000 |  83 | 0xbcbbb000 | 0xffffffff
      
    7. sim affinity -l

      Background:

      • Displays the current affinity of network interfaces to CPU cores

      Diagnostics:

      • Collect this output to see the current status

      Check the SIM Affinity mode:

      • If SIM Affinity was configured in Static mode, then the following configuration file should exist and should not be empty:

        [Expert@HostName]# cat $PPKDIR/boot/modules/sim_aff.conf
      • If SIM Affinity was configured in Automatic mode, then the following Task should exist in CPD Scheduler:

        [Expert@HostName]# cpd_sched_config print | grep -A 5 "Sim_Affinity"
        Task: "Sim_Affinity"
                Command: sim
                Arguments: affinity -c
                Interval: 60
                Active: true
                RunAtStart: false 
        

      Example:

      eth0 : 0
      eth1 : 0
      eth2 : 1
      eth3 : 1
      

     

  • (4-5) Initial diagnostics - CoreXL

  • Show / Hide initial diagnostics information

    1. cpview

      Background:

      Diagnostics:

      • Monitor the CoreXL performance during the problem

      Analysis:

      • On the 'SysInfo' tab, refer to 'Configuration Information:' section - look at 'CoreXL Status'
      • On the 'SysInfo' tab, refer to 'Configuration Information:' section - look at 'CoreXL instances'
      • On the 'I/S' tab, go to 'CoreXL' menu - go to 'General' menu - refer to 'Queues:' section - look at counter 'Enqueue fail'
      • On the 'I/S' tab, go to 'CoreXL' menu - go to 'Instances' menu - activate the statistics (!may affect performance) - go to each CoreXL FW instance ('FW-Instance<N>'):
        • refer to 'FW Stats' section
        • refer to 'Top FW-Lock consumers:' section
    2. fw ctl multik stat

      Background:

      • Displays status of CoreXL instances and summary for traffic that passes through each CoreXL FW instance (current number and peak number of concurrent connections)

      Diagnostics:

      • Collect this output to see the current status

      Analysis:

      • Check the number and the allocation of FW instances to CPU cores (in cluster, the output must be identical on all members)
      • Check the peak number connections on FW instances (instances should be loaded as equally as possible)

      Example from machine with 4 CPU cores (1 SND + 3 CoreXL FW instances):

      ID | Active  | CPU    | Connections | Peak
      ----------------------------------------------
       0 | Yes     | 3      |          26 |       66
       1 | Yes     | 2      |          36 |       64
       2 | Yes     | 1      |          39 |       53
      
    3. fw ctl affinity -l -r -v -a

      Background:

      • Displays affinity of interfaces, processes and CoreXL FW instances to CPU cores

      Diagnostics:

      • Collect this output to see the current status

      Analysis:

      • Check the affinity settings (of interfaces, processes and FW instances) to CPU cores (in cluster, the output must be identical on all members)
      • The interfaces affinity should be configured only to to CPU cores that are running as SNDs (CPU cores that are not running CoreXL FW instances)
      • Under normal circumstances, it is not recommended for the SND and a CoreXL FW instance to share the same CPU core

      Example from machine with 4 CPU cores and 4 interfaces (1 SND + 3 CoreXL FW instances):

      CPU 0:  eth0 (irq 67) eth3 (irq 59)
      CPU 1:  eth1 (irq 75)
              fw_2
      CPU 2:  eth2 (irq 83)
              fw_1
      CPU 3:  fw_0
      All:    mpdaemon fwucd cpca vpnd fwm ... cpd cprid
      
    4. cat /proc/interrupts

      Background:

      Usage:

      • Refer to the manual page
      • By default, output is sorted by IRQ number
      • The only relevant devices are network interfaces
      • To shorten the output, use the 'grep' command - for example, run:
        [Expert@HostName]# cat /proc/interrupts | grep -E "CPU|eth"
      • To collect this output continuously, use the 'watch' command - for example, run:
        [Expert@HostName]# watch -d -n 1 'cat /proc/interrupts | grep -E "CPU|eth"'

      Diagnostics:

      • Collect this output continuously during the problem

      Analysis:

      • Look at the general trend - which CPU receives more interrupts and from which interfaces
      • If some CPU cores receive more interrupts than others, then affinity of interfaces to CPU cores should be optimized - interface should be redistributed better

      Example from machine with 8 CPU cores and 5 interfaces:

                 CPU0       CPU1       CPU2       CPU3
        0:   51254849          0          0          0    IO-APIC-edge  timer
        1:          4          0          0          5    IO-APIC-edge  i8042
        6:          2          3          0          0    IO-APIC-edge  floppy
        7:          0          0          0          0    IO-APIC-edge  parport0
        8:          3          0          0          0    IO-APIC-edge  rtc
        9:          0          0          0          0   IO-APIC-level  acpi
       12:          0          0          0        115    IO-APIC-edge  i8042
       15:        302       1043        168       1127    IO-APIC-edge  ide1
       59:     320145       6732       5944       6189   IO-APIC-level  ioc0, eth3
       67:   15765753          0          0          0   IO-APIC-level  eth0
       75:      24271       1285         36          0   IO-APIC-level  eth1
       83:      24272          0       1322          0   IO-APIC-level  eth2
      NMI:          0          0          0          0
      LOC:   50950084   50945459   50949418   50945128
      ERR:          0
      MIS:          0
      

     


     

    (5) Advanced diagnostics

     

  • (5-1) Advanced diagnostics - CPU

  • Show / Hide advanced diagnostics information

    1. cpview

      Background:

      Diagnostics:

      • Monitor the CPU utilization during the problem

      Analysis:

      • On the 'I/S' tab, go to 'CPU' menu - go to 'Contexts' menu - refer to section 'Contexts'
    2. vmstat [-n] [delay_in_sec [number_of_samples]]

      Background:

      Usage:

      • Refer to the manual page
      • By default, the output's header (with names of columns) is displayed every 20 samples. If output is redirected to a file, then it will be very difficult to analyze it using such commands as 'grep' / 'awk'. Therefore, if output is redirected to a file, use the '-n' flag to display the header only once (at the very top) - run:
        [Expert@HostName]# vmstat -n [delay_in_sec [number_of_samples]]
      • By default, only 1 sample is printed (with the average values since boot). Therefore, to collect this output continuously, specify the delay between the samples - run:
        [Expert@HostName]# vmstat [-n] delay_in_sec [number_of_samples]
      • If you specified the delay between the samples, then the output will be collected indefinitely (until the command is stopped or killed). If you want to limit the output, then specify the total number of samples - run:
        [Expert@HostName]# vmstat [-n] delay_in_sec number_of_samples

      Diagnostics:

      • Collect this output continuously during the problem

      Analysis:

      • Look at the "procs" section - number of processes waiting for CPU (counter r)
      • Look at the "swap" section - reading from swap file (si) and writing to swap file (so)
      • Look at the "io" section - reading from hard disk (bi) and writing to hard disk (bo)
      • Look at the "system" section - number of Context Switches (cs)
      • Look at the "cpu" section - at all counters:
        • "User Space" - counter us
        • "System (kernel) Space" - counter sy
        • "Idle" - counter id
        • "I/O waiting" - counter wa

      Example:

      procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
       r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
       2  0 323280 100128 103192 198132    1    2     8    35  117  130  2  3 96  0  0
       2  0 323280 100400 103196 198132    0    0     0   260 1297 6904  1  4 95  0  0
       1  0 323280 100404 103196 198132    0    0     0     0 1289 7100  1  4 95  0  0
       7  0 323280 100252 103200 198132    0    0     0    32 1308 6971  0  2 97  0  0
       1  0 323280 100428 103200 198132    0    0     0     0 1265 7111  1  3 97  0  0
      
    3. sar [-u] [-P { <cpu> | ALL }] [interval_in_sec [number_of_samples]]

      Background:

      Usage:

      Diagnostics:

      • Collect this output during the problem

      Analysis:

      • Look at the load in "User Space" - counter user
        High CPU consumption in "User Space" can be caused by processes that perform heavy tasks (e.g., too much logging by fwd, reloading the configuration during policy installation, etc.)
      • Look at the load in "System (kernel) Space" - counter system
        High CPU consumption in "System (kernel) Space" can be caused by heavy tasks (e.g., deep inspection of packets, enabling of all blades, enabling of all IPS protections in Prevent mode, etc.)
      • Look at the amount of "Idle" - counter idle
        The more CPU is idle, the better the machine's performance is
      • Look at the amount of "I/O waiting" - counter iowait
        High amount of "I/O waiting" is caused by heavy reading from/writing to hard disk (e.g., during policy installation, heavy logging, insufficient RAM, etc.)
      • Look at the counter steal - Percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.

      Example:

      [Expert@R77-GW:0]# sar -u
      Linux 2.6.18-92cp (R77-GW)      08/04/16
      
      16:10:01          CPU     %user     %nice   %system   %iowait    %steal     %idle
      16:20:01          all      0.50      0.00      1.49      0.43      0.00     97.58
      16:30:01          all      0.53      0.00      1.78      0.36      0.00     97.32
      16:40:01          all      0.53      0.00      1.55      0.24      0.00     97.68
      16:50:01          all      0.51      0.00      1.76      0.26      0.00     97.46
      17:00:01          all      0.51      0.21      1.71      0.33      0.00     97.25
      17:10:01          all      0.48      0.00      1.57      0.35      0.00     97.59
      17:20:01          all      0.49      0.00      1.64      0.31      0.00     97.57
      17:30:01          all      0.48      0.00      1.60      0.48      0.00     97.44
      17:40:01          all      0.48      0.00      1.61      0.31      0.00     97.60
      17:50:01          all      0.50      0.00      1.68      0.32      0.00     97.51
      18:00:01          all      0.49      0.18      1.44      0.29      0.00     97.60
      18:10:01          all      0.48      0.00      1.76      0.45      0.00     97.31
      18:20:01          all      0.49      0.00      1.71      0.32      0.00     97.48
      18:30:01          all      0.49      0.00      1.53      0.41      0.00     97.56
      18:40:01          all      0.53      0.00      1.88      0.32      0.00     97.26
      18:50:01          all      0.49      0.00      1.56      0.29      0.00     97.66
      Average:          all      0.50      0.02      1.64      0.34      0.00     97.49
      [Expert@R77-GW:0]#
      
      [Expert@R77-GW:0]# sar -P 1
      Linux 2.6.18-92cp (R77-GW)      08/04/16
      
      16:10:01          CPU     %user     %nice   %system   %iowait    %steal     %idle
      16:20:01            1      0.51      0.00      1.28      0.20      0.00     98.01
      16:30:01            1      0.52      0.00      1.41      0.03      0.00     98.04
      16:40:01            1      0.51      0.00      1.35      0.04      0.00     98.10
      16:50:01            1      0.47      0.00      1.38      0.01      0.00     98.14
      17:00:01            1      0.52      0.20      1.37      0.05      0.00     97.86
      17:10:01            1      0.47      0.00      1.39      0.08      0.00     98.06
      17:20:01            1      0.51      0.00      1.38      0.11      0.00     98.01
      17:30:01            1      0.50      0.00      1.36      0.20      0.00     97.94
      17:40:01            1      0.49      0.00      1.30      0.08      0.00     98.14
      17:50:01            1      0.49      0.00      1.37      0.11      0.00     98.03
      Average:            1      0.50      0.02      1.36      0.09      0.00     98.03
      [Expert@R77-GW:0]#
      
    4. sh pstack_sym.sh <PID_of_problematic_daemon>

      Background:

      • Manual page - http://linux.die.net/man/1/pstack
      • Displays a stack trace of a running process
      • Check Point R&D have created a special shell script to collect the necessary information based on the 'pstack' utility

      Usage:

      1. Download the package that contains the 'pstack' utility and special Check Point shell script that collects the necessary information.

      2. Transfer the package to the problematic machine (into some directory, e.g., /some_path_to_pstack/).

      3. Navigate to the directory with the package:

        [Expert@HostName]# cd /some_path_to_pstack/
      4. Unpack the package:

        [Expert@HostName]# tar -xvf pstack_sym.tar
      5. Install the 'pstack' utility:

        [Expert@HostName]# rpm -ihv --force --nodeps pstack-1.1-7.i386.rpm

        Note: there is no need to restart any service or to reboot the machine.
      6. Determine the PID of the problematic daemon (that consumes the CPU at high level) - run the top command and look at the left-most column 'PID'.

        Alternatively, if you already know the name of the daemon, run this command:

        [Expert@HostName]# ps auxw | grep -v grep | grep -i NAME_OF_PROBLEMATIC_DAEMON_THAT_CONSUMES_CPU | awk '{print $2}'
      7. During the problem, run the special shell script at least 5-7 times:

        [Expert@HostName]# sh pstack_sym.sh <PID_of_problematic_daemon>

        Important Note: It might take 10-15 minutes for the script to complete the analysis - please wait patiently.

        1. run the script
        2. wait until the script finishes
        3. follow the instructions on the screen
        4. run the script
        5. wait until the script finishes
        6. follow the instructions on the screen
        7. etc., etc., etc.
      8. Send the following to Check Point Support for analysis:

        • Outputs collected by the script (location is shown on the screen)
        • Log files from the problematic daemon (should be located either in $CPDIR/log/, or in $FWDIR/log/)
        • CPinfo file from the problematic machine (just to see the versions and the configuration)

      Analysis:

      • All the outputs will be analyzed by Check Point Support and R&D

      Example:

      [Expert@FW]# sh pstack_sym.sh 30477
       
      **************************************************
      Starting the analysis - please wait patiently
      **************************************************
       
      
      30477: cpd
      (No symbols found)
      0x77751db7 : select + 57 (libc-2.3.2.so) (112, 7f9b7de0, 7f9b85e0, 7f9b8de0, 779c3244, 0) + 1850
      0x779acaac : T_event_mainloop_iter_select + 1ec (libComUtils.so) (87f8b90, 779c3244, 96, 779a6259, 779a4560, 779c3738) + 20
      0x779ad23f : T_event_mainloop_iter + 16f (libComUtils.so) (87f8b90, 77f95218, 0, 885dad8, 7f9b9678, 77fb5250) + 10
      0x779ad298 : T_event_mainloop_e + 28 (libComUtils.so) (87f8b90, 805359c)
      0x77f12402 : opsec_mainloop + 22 (libopsec.so) (885dc88, 0, 80516f2, 770d, 0, 0) + 9a0
      0x0804cfd8 : /opt/CPshrd-R71/bin/cpd + 4fd8 (1, 7f9ba0c4, 7f9ba0cc, 0, 777b1898, 77fbe020) + 40
      0x776917fd : __libc_start_main + ed (libc-2.3.2.so) (804cb10, 1, 7f9ba0c4, 80509e0, 8050a28, 77fb5be0) + 80645f48
       
      **************************************************
      This script has completed the analysis
      **************************************************
       
       
      Going to TAR the output files :
      ----------------------------------
      tar: Removing leading `/' from member names
      tmp/pstack_30477.out
      tmp/pstack_30477_FINAL_OUTPUT.txt
      tmp/pstack_30477_maps.out
      tmp/pstack_30477_objdump.out
      tmp/pstack_30477_proc-stat.txt
      tmp/pstack_30477_ps_auxxxwwf.txt
      tmp/pstack_30477_top.txt
      opt/CPshrd-R71/log/cpwd.elg
      opt/CPshrd-R71/log/cpwd.elg.1
      opt/CPshrd-R71/log/cpwd.elg.2
      var/log/messages
      var/log/messages.1
      var/log/messages.2
      var/log/messages.3
      var/log/messages.4
      
       
      Please send the following file to Check Point Support...
      ----------------------------------
      /tmp/pstack_30477.tar
      ----------------------------------
       
      [Expert@FW]#
      

     

  • (5-2) Advanced diagnostics - Memory

  • Show / Hide advanced diagnostics information

    1. cpview

      Background:

      Diagnostics:

      • Monitor the Memory utilization during the problem

      Analysis:

      • On the 'I/S' tab, go to 'Memory' menu - go to 'FW-Kernel' menu - refer to section 'KMEM:' - look at counter 'Failed'
      • On the 'I/S' tab, go to 'Memory' menu - go to 'FW-Kernel' menu - refer to section 'HMEM:' - look at counter 'Failed'
      • On the 'I/S' tab, go to 'Memory' menu - go to 'FW-Kernel' menu - refer to section 'SMEM:' - look at counter 'Failed'
      • On the 'I/S' tab, go to 'Memory' menu - go to 'SMEM-Failures' menu
      • On the 'I/S' tab, go to 'Memory' menu - go to 'Contexts' menu
    2. vmstat [-n] [delay_in_sec [number_of_samples]]

      Background:

      Usage:

      • Refer to the manual page
      • By default, the output's header (with names of columns) is displayed every 20 samples. If output is redirected to a file, then it will be very difficult to analyze it using such commands as 'grep' / 'awk'. Therefore, if output is redirected to a file, use the '-n' flag to display the header only once (at the very top) - run:
        [Expert@HostName]# vmstat -n [delay_in_sec [number_of_samples]]
      • By default, only 1 sample is printed (with the average values since boot). Therefore, to collect this output continuously, specify the delay between the samples - run:
        [Expert@HostName]# vmstat [-n] delay_in_sec [number_of_samples]
      • If you specified the delay between the samples, then the output will be collected indefinitely (until the command is stopped or killed). If you want to limit the output, then specify the total number of samples - run:
        [Expert@HostName]# vmstat [-n] delay_in_sec number_of_samples

      Diagnostics:

      • Collect this output continuously during the problem

      Analysis:

      • Look at the "procs" section - number of processes waiting for CPU (r)
      • Look at the "memory" section - sum of these counters should be compared to the total amount of RAM
      • Look at the "swap" section - reading from swap file (si) and writing to swap file (so)

      Example:

      procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
       r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
       2  0 323280 100128 103192 198132    1    2     8    35  117  130  2  3 96  0  0
       2  0 323280 100400 103196 198132    0    0     0   260 1297 6904  1  4 95  0  0
       1  0 323280 100404 103196 198132    0    0     0     0 1289 7100  1  4 95  0  0
       7  0 323280 100252 103200 198132    0    0     0    32 1308 6971  0  2 97  0  0
       1  0 323280 100428 103200 198132    0    0     0     0 1265 7111  1  3 97  0  0
      
    3. sar [-q][-r][-B][-W] [interval_in_sec [number_of_samples]]

      Background:

      Usage:

      Diagnostics:

      • Collect this output during the problem

      Analysis:

      • Look at the number of tasks waiting for run time - counter runq-sz
      • Look at the system load average - counters ldavg-1, ldavg-5, ldavg-15
      • Look at the percentage of used memory - counter memused
      • Look at the percentage of used swap memory - counter swpused
      • Look at the total number of kB the system paged in from disk per second - counter pgpgin/s
      • Look at the total number of kB the system paged out to disk per second - counter pgpgout/s
      • Look at the number of page faults (major + minor) made by the system per second - counter fault/s
      • Look at the number of major faults the system has made per second - counter majflt/s
      • Look at the total number of swap pages - counter pswpin/s and counter pswpout/s

      Example:

      [Expert@R77-GW:0]# sar -q
      Linux 2.6.18-92cp (R77-GW)      08/04/16
      
      16:10:01      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
      16:20:01            1       114      1.11      1.04      1.06
      16:30:01            0       115      1.07      1.13      1.09
      16:40:01            0       115      1.15      1.11      1.09
      16:50:01            1       114      1.03      1.04      1.06
      17:00:01            0       114      1.10      1.09      1.08
      17:10:01            1       114      1.08      1.08      1.08
      17:20:01            1       112      1.01      1.10      1.09
      17:30:01            0       112      1.23      1.11      1.09
      17:40:01            1       112      1.15      1.12      1.09
      17:50:01            0       114      1.14      1.10      1.09
      Average:            0       114      1.11      1.09      1.08
      [Expert@R77-GW:0]#
      
      [Expert@R77-GW:0]# sar -r
      Linux 2.6.18-92cp (R77-GW)      08/04/16
      
      16:10:01    kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
      16:20:01      1221768    784220     39.09     43328    381364   2128604         0      0.00         0
      16:30:01      1220728    785260     39.15     44648    381408   2128604         0      0.00         0
      16:40:01      1219280    786708     39.22     45880    381432   2128604         0      0.00         0
      16:50:01      1218016    787972     39.28     47300    381456   2128604         0      0.00         0
      17:00:01      1216792    789196     39.34     48628    381420   2128604         0      0.00         0
      17:10:01      1215504    790484     39.41     50076    381444   2128604         0      0.00         0
      17:20:01      1214964    791024     39.43     51272    381472   2128604         0      0.00         0
      17:30:01      1213444    792544     39.51     52524    381492   2128604         0      0.00         0
      17:40:01      1212624    793364     39.55     53568    381436   2128604         0      0.00         0
      17:50:01      1210840    795148     39.64     54940    381488   2128604         0      0.00         0
      Average:      1216396    789592     39.36     49216    381441   2128604         0      0.00         0
      [Expert@R77-GW:0]#
      
      [Expert@R77-GW:0]# sar -B
      Linux 2.6.18-92cp (R77-GW)      08/04/16
      
      16:10:01     pgpgin/s pgpgout/s   fault/s  majflt/s
      16:20:01         0.00     34.64   4417.09      0.00
      16:30:01         0.03     37.03   4506.64      0.00
      16:40:01         0.01     35.28   4402.80      0.00
      16:50:01         0.00     36.98   4462.80      0.00
      17:00:01         0.00     36.63   4519.38      0.00
      17:10:01         0.00     37.15   4489.80      0.00
      17:20:01         0.00     35.27   4440.20      0.00
      17:30:01         0.01     35.86   4487.54      0.00
      17:40:01         0.01     35.87   4460.36      0.00
      Average:         0.01     36.08   4464.99      0.00
      [Expert@R77-GW:0]#
      
      [Expert@R77-GW:0]# sar -W
      Linux 2.6.18-92cp (R77-GW)      08/04/16
      
      16:10:01     pswpin/s pswpout/s
      16:20:01         0.00      0.00
      16:30:01         0.00      0.00
      16:40:01         0.00      0.00
      16:50:01         0.00      0.00
      17:00:01         0.00      0.00
      17:10:01         0.00      0.00
      17:20:01         0.00      0.00
      17:30:01         0.00      0.00
      17:40:01         0.00      0.00
      Average:         0.00      0.00
      [Expert@R77-GW:0]#
      
    4. valgrind

      Background:

      Installation:

      Note: This tool is for 32-bit OS - if running Gaia OS in 64-bit, then first you have to switch to 32-bit kernel and reboot.

      1. Contact Check Point Support to get the 'valgrind' utility and relevant instructions.

      2. Transfer the package to the problematic machine (into some directory, e.g., /some_path_to_valgrind/).

      3. Navigate to the directory with the package:

        [Expert@HostName]# cd /some_path_to_valgrind/
      4. Unpack the package:

        [Expert@HostName]# tar -xvf VALGRIND_v3.7.0.tar
      5. Unpack the 'valgrind' tool:

        [Expert@HostName]# tar -xvzf valgrind.tar.gz
      6. Copy the 'valgrind' files to relevant directories:

        [Expert@HostName]# cd valgrind_install_files
        [Expert@HostName]# cp -r bin/* /usr/bin
        [Expert@HostName]# cp -r lib/* /usr/lib
      7. Test the 'valgrind' tool:

        [Expert@HostName]# valgrind --version

      Important Notes:

      1. After collecting the required information with the 'valgrind' tool (wait for the confirmation from Check Point Support), this tool has to be removed from the machine for security reasons (because this tool is able to collect internal information from the processes).

      2. When a command / daemon is started under 'valgrind', 'valgrind' loads all the relevant library files to read the relevant information from them (to know how to collect the memory data). This process (loading of library files) takes time and might cause the CPU load to increase up to 100%.

        Therefore, schedule a maintenance window to start the 'valgrind'.

        After 'valgrind' loads all the library files, it does not cause the additional load on CPU.

      3. Do NOT KILL the 'valgrind' - it will NOT be able to save the collected data.

        You have to TERMINATE the 'valgrind':

        [Expert@HostName]# kill -TERM $(pidof valgrind)

      4. If VPND daemon has to be started under 'valgrind', then first you need to run this command (to prevent the VPND daemon from breaking the pipe to FWD daemon used to send the PID of the VPND process back to FWD daemon for monitoring purposes):

        [Expert@HostName]# export FWD_NO_CHILD_UP_DBG=1

      Usage:

      Analysis:

      • All the outputs will be analyzed by Check Point Support and R&D

      Example for FWM daemon:

      1. Stop the FWM daemon:

        [Expert@HostName]# cpwd_admin list
        [Expert@HostName]# cpwd_admin stop -name FWM -path "$FWDIR/bin/fw" -command "fw kill fwm"
        [Expert@HostName]# cpwd_admin list
      2. Note: If VPND daemon has to be started under 'valgrind', then first you need to run this command (to prevent the VPND daemon from breaking the pipe to FWD daemon used to send the PID of the VPND process back to FWD daemon for monitoring purposes):

        [Expert@HostName]# export FWD_NO_CHILD_UP_DBG=1
      3. Start FWM daemon under 'valgrind':

        [Expert@HostName]# valgrind --log-file=/var/log/valgrind_output.txt --verbose --trace-children=yes --track-fds=yes --time-stamp=yes --leak-check=full --leak-resolution=high --show-reachable=yes --error-limit=no --smc-check=stack --demangle=yes $FWDIR/bin/fwm
      4. Replicate the issue.

      5. Terminate (do NOT kill) the valgrind:

        [Expert@HostName]# kill -TERM $(pidof valgrind)

        Note: Wait for several minutes for 'valgrind' to write its summary report.
      6. Start the FWM daemon:

        [Expert@HostName]# cpwd_admin start -name FWM -path "$FWDIR/bin/fwm" -command "fwm"
        [Expert@HostName]# cpwd_admin list
      7. Sent for analysis:

        • /var/log/valgrind_output*
        • $FWDIR/log/fwm.el*
        • $CPDIR/log/cpwd.el*
    5. slabtop [-d refresh_in_sec] [-sSort_Criteria]

      Background:

      Usage:

      • Refer to the manual page
      • By default, output is refreshed every 3 seconds, unless you use the "-d refresh_in_sec" option
      • By default, output is sorted by the number of objects (column "OBJS", which is equal to using the "-so" option).
      • Recommended output sort is by the cache size of the slab (column "CACHE SIZE", use the "-sc" option).

      Diagnostics:

      • Collect this output continuously during the problem

      Analysis:

      • Provide the output(s) to Check Point Support (involvement of R&D is required)
      • The most interesting information is the amount of resources a certain kernel module is using (if this amount seems too high, then there may be something wrong with this module)
      • Some of the more commonly used statistics from /proc/slabinfo file that are included in the output of /usr/bin/slabtop:

        • OBJS - The total number of objects (memory blocks), including those in use (allocated), and some spares not in use
        • ACTIVE - The number of objects (memory blocks) that are in use (allocated)
        • USE - Percentage of total objects that are active [(ACTIVE/OBJS)*100%]
        • OBJ SIZE - The size of the objects
        • SLABS - The total number of slabs
        • OBJ/SLAB - The number of objects that fit into a slab
        • CACHE SIZE - The cache size of the slab
        • NAME - The name of the slab

      Example of 'slabtop' output (excerpt):

       Active / Total Objects (% used)    : 160565 / 174730 (91.9%)
       Active / Total Slabs (% used)      : 5005 / 5005 (100.0%)
       Active / Total Caches (% used)     : 94 / 141 (66.7%)
       Active / Total Size (% used)       : 19571.55K / 20754.12K (94.3%)
       Minimum / Average / Maximum Object : 0.01K / 0.12K / 128.00K
      
        OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
       85536  82456  96%    0.05K   1188       72      4752K buffer_head
       12876  10385  80%    0.13K    444       29      1776K dentry_cache
        9516   9498  99%    0.05K    122       78       488K selinux_inode_security
        7571   6711  88%    0.03K     67      113       268K size-32
        7524   6869  91%    0.09K    171       44       684K vm_area_struct
        7056   7056 100%    0.48K    882        8      3528K ext3_inode_cache
        5936   5703  96%    0.27K    424       14      1696K radix_tree_node
        4524   4453  98%    0.05K     58       78       232K sysfs_dir_cache
      

      Example of 'cat /proc/slabinfo' output (excerpt):

      # name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables ...
      bridge_fdb_cache       0      0     64   59    1 : tunables  120   60    8 : slabdata      0      0      0
      jbd_1k                 0      0   1024    4    1 : tunables   54   27    8 : slabdata      0      0      0
      jbd_4k                 1      1   4096    1    1 : tunables   24   12    8 : slabdata      1      1      0
      ......
      scsi_io_context        0      0    104   37    1 : tunables  120   60    8 : slabdata      0      0      0
      ext3_inode_cache    4596   4888    492    8    1 : tunables   54   27    8 : slabdata    611    611      0
      ext3_xattr             0      0     48   78    1 : tunables  120   60    8 : slabdata      0      0      0
      journal_handle        89    169     20  169    1 : tunables  120   60    8 : slabdata      1      1      0
      journal_head         205    432     52   72    1 : tunables  120   60    8 : slabdata      6      6      0
      ......
      size-32(DMA)           0      0     32  113    1 : tunables  120   60    8 : slabdata      0      0      0
      size-64             3724   4071     64   59    1 : tunables  120   60    8 : slabdata     69     69      0
      size-32            17449  18306     32  113    1 : tunables  120   60    8 : slabdata    162    162     60
      kmem_cache           141    165    256   15    1 : tunables  120   60    8 : slabdata     11     11      0
      
    6. fw ctl failmem PERCENTAGE_of_ALLOCATIONS_to_FAIL

      Background:

      • Administrator can test the behaviour of the Security Gateway during a memory shortage by failing a certain percentage of all memory allocations

      Usage:

      Diagnostics:

      • Run this test to check how the Security Gateway is going to behave during a memory shortage

      Analysis:

      • Monitor Security Gateway's resources utilization during this test - Memory, CPU, Interfaces, how traffic is passing, etc.

     

  • (5-3) Advanced diagnostics - Network interface cards

  • Show / Hide advanced diagnostics information

    1. netstat -s

      Background:

      Diagnostics:

      • Collect this output continuously during the problem (usually, over some time period)

      Analysis:

      • In the "Ip" section, look at "incoming packets discarded"
      • In the "Icmp" section, look at "ICMP messages failed"
      • In the "Tcp" section, look at "bad segments received"
      • In the "Udp" section, look at "packet receive errors"
      • Search for lines with "error", "fail", "timeout", "loss", "lost"

      Example:

      Ip:
          17862464 total packets received
          0 forwarded
          0 incoming packets discarded
          948002 incoming packets delivered
          788408 requests sent out
          287 outgoing packets dropped
          1 dropped because of missing route
      Icmp:
          297 ICMP messages received
          0 input ICMP message failed.
      ......
      IcmpMsg:
              InType0: 7
              InType3: 290
              OutType3: 290
              OutType8: 11
      Tcp:
          66034 active connections openings
          11704 passive connection openings
          23572 failed connection attempts
      ......
      Udp:
          15488 packets received
          287 packets to unknown port received.
      ......
      TcpExt:
          1 packets pruned from receive queue because of socket buffer overrun
          1142 TCP sockets finished time wait in fast timer
      ......
      IpExt:
          InMcastPkts: 420
          InBcastPkts: 277064
      
    2. ethtool -S name_of_interface

      Background:

      Diagnostics:

      • Collect this output continuously during the problem (usually, over some time period)

      Analysis:

      Example:

      NIC statistics:
           rx_packets: 21610536
           tx_packets: 130262
           rx_bytes: 1712923726
           tx_bytes: 34001412
      ......
           rx_errors: 0
           tx_errors: 0
           tx_dropped: 0
      ......
           rx_length_errors: 0
           rx_over_errors: 0
           rx_crc_errors: 0
           rx_frame_errors: 0
           rx_no_buffer_count: 0
           rx_missed_errors: 0
           tx_aborted_errors: 0
           tx_carrier_errors: 0
           tx_fifo_errors: 0
      ......
      
    3. sar [-n { DEV | EDEV }] [interval_in_sec [number_of_samples]]

      Background:

      Usage:

      Diagnostics:

      • Collect this output during the problem

      Analysis:

      • Look at all the counters

      Example:

      [Expert@R77-GW:0]#  sar -n DEV 1 2
      Linux 2.6.18-92cp (R77-GW)      08/04/16
      
      19:12:02        IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s   rxcmp/s   txcmp/s  rxmcst/s
      19:12:03           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
      19:12:03         eth0      1.01      2.02    107.07    193.94      0.00      0.00      0.00
      19:12:03         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00
      
      19:12:03        IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s   rxcmp/s   txcmp/s  rxmcst/s
      19:12:04           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
      19:12:04         eth0      6.06      9.09    595.96   1076.77      0.00      0.00      0.00
      19:12:04         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00
      
      Average:        IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s   rxcmp/s   txcmp/s  rxmcst/s
      Average:           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
      Average:         eth0      3.54      5.56    351.52    635.35      0.00      0.00      0.00
      Average:         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00
      [Expert@R77-GW:0]#
      
      [Expert@R77-GW:0]#  sar -n EDEV 1 2
      Linux 2.6.18-92cp (R77-GW)      08/04/16
      
      19:12:36        IFACE   rxerr/s   txerr/s    coll/s  rxdrop/s  txdrop/s  txcarr/s  rxfram/s  rxfifo/s  txfifo/s
      19:12:37           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
      19:12:37         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
      19:12:37         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
      
      19:12:37        IFACE   rxerr/s   txerr/s    coll/s  rxdrop/s  txdrop/s  txcarr/s  rxfram/s  rxfifo/s  txfifo/s
      19:12:38           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
      19:12:38         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
      19:12:38         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
      
      Average:        IFACE   rxerr/s   txerr/s    coll/s  rxdrop/s  txdrop/s  txcarr/s  rxfram/s  rxfifo/s  txfifo/s
      Average:           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
      Average:         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
      Average:         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
      [Expert@R77-GW:0]#
      

     

  • (5-4) Advanced diagnostics - SecureXL

  • Show / Hide advanced diagnostics information

    Refer to sk98722 - ATRG: SecureXL.

    1. cpview

      Background:

      Diagnostics:

      • Monitor the SecureXL performance during the problem

      Analysis:

      • On the 'Traffic' tab, go to 'Overview' menu - refer to section 'Templates:'
      • On the 'Traffic' tab, go to 'Overview' menu - refer to section 'Logged drops:' - look at counter 'SXL'
      • On the 'I/S' tab, go to 'SXL' menu - go to 'F2F-Reasons' menu
      • On the 'I/S' tab, go to 'SXL' menu - go to 'Drop-Reasons' menu
    2. Forwarded to Firewall (F2F) traffic

      Background:

      • Some connections cannot be accelerated by design - refer to sk32578 - SecureXL Mechanism

        Example output of 'fwaccel conns' command (all IP addresses were replaced by "X"):

        Source          SPort Destination     DPort PR Flags   C2S i/f   S2C i/f
        --------------- ----- --------------- ----- -- ------- --------- ---------
            X.X.X.X     61242        X.X.X.X     80  6 ..N.... eth5/eth0 eth0/eth5
            X.X.X.X      6000        X.X.X.X   3842  6 ....... eth0/eth4 eth4/eth0
            X.X.X.X      1620        X.X.X.X     88 17 ....... eth4/eth2 eth2/eth4
            X.X.X.X     50829        X.X.X.X     80  6 ..N.... eth5/eth0 eth0/eth5
            X.X.X.X      4285        X.X.X.X     80  6 F.N.... eth5/eth0 eth0/eth5
            X.X.X.X        80        X.X.X.X  49312  6 F.N.... eth5/eth0 eth0/eth5
            X.X.X.X        80        X.X.X.X  11450  6 ..N.... eth5/eth0 eth0/eth5
            X.X.X.X     58562        X.X.X.X     80  6 F.N.... eth5/eth0 eth0/eth5
            X.X.X.X       161        X.X.X.X   5002 17 F...... eth2/eth2 -/-
            X.X.X.X     21891        X.X.X.X      0  1 F...... eth2/eth4 eth4/eth2
            X.X.X.X     34303        X.X.X.X    389  6 F.N.... eth2/eth2 eth2/-
        

      Diagnostics:

      • Collect this information during the problem (Important Note: These commands might increase CPU and memory consumption)

      Analysis:

      • To quickly determine the amount of F2F connections:
        1. Check the total number of connections in SecureXL connections table:
          [Expert@HostName]# fwaccel conns -s
        2. Check the total number of F2F connections:
          [Expert@HostName]# fwaccel conns | grep ' F' | grep -v 'Flags' | wc -l
      • To see why specific connections are Forwarded to Firewall, run the following kernel debug (debug will apply only to new connections):
        [Expert@HostName]# fw ctl debug 0
        [Expert@HostName]# fwaccel dbg resetall
        [Expert@HostName]# sim dbg resetall
        [Expert@HostName]# fw ctl debug -buf 32000
        [Expert@HostName]# fwaccel dbg + offload
        [Expert@HostName]# sim dbg + f2f
        [Expert@HostName]# fwaccel dbg list
        [Expert@HostName]# sim dbg list
        [Expert@HostName]# fw ctl kdebug -T -f > /var/log/debug.txt
        
    3. fwaccel stats

      Background:

      • Displays SecureXL acceleration statistics

      Diagnostics:

      • Collect this output continuously during the problem

      Analysis:

      • Refer to Performance Pack Administration Guide (R70, R71, R75, R75.20, R75.40, R75.40VS) - Chapter 'Command Line' - fwaccel stats and fwaccel6 stats.
      • Refer to Performance Tuning Administration Guide (R76, R77) - Chapter 'Performance Pack' - Command Line - fwaccel stats and fwaccel6 stats.

      Example:

      Name                  Value              Name                  Value
      --------------------  ---------------    --------------------  ---------------
      
      Accelerated Path
      ------------------------------------------------------------------------------
      accel packets                       0    accel bytes                         0
      conns created                   60665    conns deleted                   60606
      ... ...
      
      Accelerated VPN Path
      ------------------------------------------------------------------------------
      C crypt conns                       0    enc bytes                           0
      dec bytes                           0    ESP enc pkts                        0
      ... ...
      
      Medium Path
      ------------------------------------------------------------------------------
      PXL packets                         0    PXL async packets                   0
      PXL bytes                           0    C PXL conns                         0
      C PXL templates                     0
      
      Accelerated QoS Path
      ------------------------------------------------------------------------------
      QXL packets                         0    QXL async packets                   0
      QXL bytes                           0    C QXL conns                         0
      C QXL templates                     0
      
      Firewall Path
      ------------------------------------------------------------------------------
      F2F packets                  14870123    F2F bytes                   966432478
      C F2F conns                        59    TCP violations                      0
      C partial conns                     0    C anticipated conns                 0
      port alloc f2f                      0
      
      General
      ------------------------------------------------------------------------------
      memory used                         0    free memory                         0
      C used templates                    0    pxl tmpl conns                      0
      ... ...
      
      (*) Statistics marked with C refer to current value, others refer to total value
      
    4. cat /proc/ppk/stats

      Background:

      • Displays total number of packets that passed through interfaces

      Diagnostics:

      • Collect this output to see the current status

      Example:

      IRQ | Stats           | Interface
      ----------------------------------------------
       67         793651087   eth0
       75                 0   eth1
       83                 0   eth2
       59                 0   eth3
      
      Ifn1 | Ifn2 | Stats
      -----------------------------
      
    5. cat /proc/ppk/conf

      Background:

      • Displays SecureXL configuration and basic statistics

      Diagnostics:

      • Collect this output to see the current status

      Analysis:

      • For Flags line, refer to "(6-1-B) 'sim' command" section - 'sim if' command
      • For TmplQuota lines, refer to "(6-1-B) 'sim' command" section - 'sim tmplquota' command
      • For Debug flags section, refer to "(6-1-B) 'sim' command" section - 'sim dbg' command
        (Note: the numbers in the left column correspond to order of debug modules in the output of 'sim dbg list' command)

      Example:

      Flags                          : 0x00009a16
      Accounting Update Interval     : 60
      Conn Refresh Interval          : 512
      SA Sync Notification Interval  : 0
      UDP Encapsulation Port         : 0
      Min TCP MSS                    : 0
      TCP Auto-Expire Timeout        : 20
      Connection Limit               : 18446744073709551615
      TmplQuota Enabled              : 0
      TmplQuota Quota (rate)         : 512
      TmplQuota Drop Dduration       : 300
      TmplQuota Monitor only         : 0
      TmplQuota Dropped pkts         : 0
      
      Total Number of conns          : 54
      Number of F2F conns            : 54
      Number of Crypt conns          : 0
      Number of TCP conns            : 18
      Number of Non-TCP conns        : 36
      Number of Delayed TCP conns    : 0
      Number of Delayed Non-TCP conns: 0
      
      Debug flags :
       0     : 0x0
       1     : 0x0
       2     : 0x0
       3     : 0x0
       4     : 0x0
       5     : 0x0
       6     : 0x0
       7     : 0x0
       8     : 0x0
       9     : 0x0
       10     : 0x0
       11     : 0x0
       12     : 0x0
       13     : 0x0
       14     : 0x0
       15     : 0x0
      
    6. cat /proc/ppk/ifs

      Background:

      • Displays the list of interfaces used and seen by the SecureXL

      Diagnostics:

      • Collect this output to see the current status

      Analysis:

      • Look at the Name column and Address column
      • Look at the F column (refer to "(6-1-B) 'sim' command" section - 'sim if' command)

      Example:

       No | Interface | Address         | IRQ |  F  | SIM F | Dev                | Output Func        | Features
      -------------------------------------------------------------------------------------------------------------
        2 | eth0      |    172.30.41.90 |  67 |  39 |     0 | 0xeffc2000         | 0xf0a862e0         | 0x000013a0
        3 | eth1      |     10.10.10.90 |  75 |  29 |     0 | 0xefc33000         | 0xf0a862e0         | 0x000013a0
        4 | eth2      |     20.20.20.90 |  83 |  29 |     0 | 0xef7fd000         | 0xf0a862e0         | 0x000013a0
        5 | eth3      |     30.30.30.90 |  59 |  29 |     0 | 0xef496000         | 0xf0a862e0         | 0x000013a0
      
    7. cat /proc/ppk/affinity

      Background:

      • Displays the status and the thresholds for New Affinity (feature will be activated only if there is no massive VPN traffic and the packets-per-second rate (cut-through) is high enough to benefit from the new affinity; feature will be activated only if CPU strength is greater than 3 GHz)

      Diagnostics:

      • Collect this output to see the current status

      Example:

      New affinity enabled           : no
      Min CPU speed to act (MHz)     : 3000
      Min accelerated PPS to act     : 250000
      Max enc. bytes rate to act     : 10000
      Current accelerated PPS        : 0
      Current enc. bytes rate        : 0
      Currently new sim affinity set : no
      
    8. cat /proc/ppk/cpls

      Background:

      • Displays the SecureXL configuration for ClusterXL Load Sharing support

      Diagnostics:

      • Collect this output to see the current status

      Analysis:

      • The most important data in this output is the value of fwha_conf_flags - it should include the value of 0x1 (i.e., ACTIVE).

      Example:

      fwha_conf_flags: 0
      fwha_df_type: 0
      fwha_member_id: 255
      fwha_port: 0
      fwha_src_mac: 00 00 00 00 00 00
      fwha_src_mac_mask: 00 00 00 00 00 00
      udp_enc_port: 0
      selection table size: 0
      selection table:
      00 00 00 00 00 00 00 00 00 00
      
    9. cat /proc/ppk/statistics

      Background:

      • Displays SecureXL statistics - the same output as from 'fwaccel stats -l' command

      Diagnostics:

      • Collect this output to see the current status

      Example:

      Name                  Value              Name                  Value
      --------------------  ---------------    --------------------  ---------------
      conns created                   67518    conns deleted                   67475
      temporary conns                     0    templates                           0
      nat conns                           0    accel packets                       0
      accel bytes                         0    F2F packets                  16599213
      ESP enc pkts                        0    ESP enc err                         0
      ESP dec pkts                        0    ESP dec err                         0
      ESP other err                       0    espudp enc pkts                     0
      espudp enc err                      0    espudp dec pkts                     0
      espudp dec err                      0    espudp other err                    0
      AH enc pkts                         0    AH enc err                          0
      AH dec pkts                         0    AH dec err                          0
      AH other err                        0    memory used                         0
      free memory                         0    acct update interval               60
      current total conns                43    TCP violations                      0
      conns from templates                0    TCP conns                          15
      delayed TCP conns                   0    non TCP conns                      28
      delayed nonTCP conns                0    F2F conns                          43
      F2F bytes                  1076600709    crypt conns                         0
      enc bytes                           0    dec bytes                           0
      partial conns                       0    anticipated conns                   0
      dropped packets                     0    dropped bytes                       0
      nat templates                       0    port alloc templates                0
      conns from nat templ                0    port alloc conns (tm                0
      port alloc f2f                      0    PXL templates                       0
      PXL conns                           0    PXL packets                         0
      PXL bytes                           0    PXL async packets                   0
      conns auto expired                  0    C used templates                    0
      pxl tmpl conns                      0    C conns from tmpl                   0
      C non TCP F2F conns                28    C tcp handshake conn               14
      C tcp established co                1    C tcp closed conns                  0
      C tcp f2f handshake                14    C tcp f2f establishe                1
      C tcp f2f closed con                0    C tcp pxl handshake                 0
      C tcp pxl establishe                0    C tcp pxl closed con                0
      QXL templates                       0    QXL conns                           0
      QXL packets                         0    QXL bytes                           0
      QXL async packets                   0    outbound packets                    0
      outbound pxl packets                0    outbound f2f packets           124862
      outbound bytes                      0    outbound pxl bytes                  0
      outbound f2f bytes           33552839    trimmed pkts  0
      
    10. cat /proc/ppk/drop_statistics

      Background:

      • Displays SecureXL drop statistics

      Diagnostics:

      • Collect this output to see the current status

      Example:

      Reason                Packets            Reason                Packets
      --------------------  ---------------    --------------------  ---------------
      general reason                      0    PXL decision                        0
      fragment error                      0    hl - spoof viol                     0
      F2F not allowed                     0    hl - TCP viol                       0
      corrupted packet                    0    hl - new conn                       0
      clr pkt on vpn                      0    partial conn                        0
      encrypt failed                      0    drop template                       0
      decrypt failed                      0    outb - no conn                      0
      interface down                      0    cluster error                       0
      XMT error                           0    template quota                      0
      anti spoofing                       0    Attack mitigation                   0
      local spoofing                      0    sanity error                        0
      monitored spoofed                   0    QXL decision                        0
      
    11. cat /proc/ppk/viol_statistics

      Background:

      • Displays violations statistics

      Diagnostics:

      • Collect this output to see the current status

      Example:

      Violation             Packets            Violation             Packets
      --------------------  ---------------    --------------------  ---------------
      pkt is a fragment                   0    pkt has IP options                406
      ICMP miss conn                    932    TCP-SYN miss conn                 169
      TCP-other miss conn                17    UDP miss conn                10305933
      other miss conn                     0    VPN returned F2F                    0
      ICMP conn is F2Fed                  7    TCP conn is F2Fed               23665
      UDP conn is F2Fed             6275273    other conn is F2Fed                 0
      uni-directional viol                0    possible spoof viol                 0
      TCP state viol                      0    out if not def/accl                 0
      bridge, src=dst                     0    routing decision err                0
      sanity checks failed                0    temp conn expired                   0
      fwd to non-pivot                    0    broadcast/multicast                 0
      cluster message                     0    partial conn                        0
      PXL returned F2F                    0    cluster forward                     0
      chain forwarding                    0    general reason                      0
      
    12. cat /proc/ppk/mcast_statistics

      Background:

      • Displays SecureXL multicast statistics

      Diagnostics:

      • Collect this output to see the current status

      Example:

      Name                  Value              Name                  Value
      --------------------  ---------------    --------------------  ---------------
      in packets                        406    out packets                         0
      if restricted                       0    conns with down if                  0
      f2f packets                         0    f2f bytes                           0
      dropped packets                     0    dropped bytes                       0
      accel packets                       0    accel bytes                         0
      mcast conns                         1
      

     

  • (5-5) Advanced diagnostics - CoreXL

  • Show / Hide advanced diagnostics information

    Refer to sk98737 - ATRG: CoreXL.

    1. fw -i FW_INSTANCE_ID tab -t connections [flags]

      Background:

      • Displays the Connections Table for specified CoreXL FW instance

      Diagnostics:

      • Collect this output to see the current status

      Example:

      localhost:
      -------- connections --------
      dynamic, id 8158, attributes: keep, sync, aggressive aging, kbufs 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34, expires 25, refresh, , hashsize 1048576, unlimited
      <00000001, ac1e295a, 0000b47d, 54279820, 00000050, 00000006; 00020001, 00046000, 49000000, 00000164, 00000000, 531e03e8, 00000000, 5a291eac, c0000000, ffffffff, ffffffff, ffffffff, 00000001, 00000000, 00000000, 00008000, 49196800, 00000000, 00000000, 00000000, 00000000, 00000000, fb13c000, 00000000, 0f13d800, 00000000, 00000000, 00000000, df175800, 00000000, 00000000; 25/25>
      <00000000, ac1e7560, 0000008a, ac1effff, 0000008a, 00000011; 00010001, 00004000, 00000001, 0000018b, 00000106, 531e03da, 00000000, 5a291eac, c0000000, 00000001, ffffffff, ffffffff, ffffffff, 02000000, 00000000, 00000000, 99163800, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000, 7a179800, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000; 23/40>
    2. cat $FWDIR/conf/fwaffinity.conf

      Background:

      • Displays the CoreXL affinity configuration

      Diagnostics:

      • Collect this output to see the current status

      Analysis:

      • Check the affinity settings

      Example:

      i eth0 1
      n fwd 3
      
    3. cat /etc/fw.boot/boot.conf

      Background:

      • Displays the CoreXL FW instances configuration

      Diagnostics:

      • Collect this output to see the current status if CoreXL configuration is not saved correctly / does not survive reboot

      Analysis:

      • Check each line under the 'DEFAULT_FILTER_PATH'

      Example:

      CTL_IPFORWARDING        1
      DEFAULT_FILTER_PATH     /etc/fw.boot/default.bin
      KERN_INSTANCE_NUM       3
      COREXL_INSTALLED        1
      KERN6_INSTANCE_NUM      2
      IPV6_INSTALLED  0
      
    4. cpview

      Background:

      Diagnostics:

      • Monitor the CoreXL performance during the problem

      Analysis:

      • On the 'I/S' tab, go to 'CoreXL' menu - go to 'General' menu
      • On the 'I/S' tab, go to 'CoreXL' menu - go to 'Global' menu
      • On the 'I/S' tab, go to 'CoreXL' menu - go to 'Instances' menu - activate the statistics (!may affect performance)

     


     

    (6) Command Line syntax

     

  • (6-1) Command Line syntax - SecureXL

    • (6-1-A) FWACCEL (controls acceleration feature)
    Show / Hide SecureXL 'fwaccel' syntax

    [Expert@HostName]# fwaccel <parameter> [-h]

    [Expert@HostName]# fwaccel6 <parameter> [-h]

    Command Description
    fwaccel help Prints the general help message with available parameters
    fwaccel ehelp Prints the extended help message with available parameters
    fwaccel <parameter> -h Prints the specific help message for given <parameter>
    fwaccel on [-a] [-q]

    Starts acceleration on-the-fly (if Performance Pack is installed)

    • "-q" flag suppresses the output
    • "-a" flag means to start acceleration on all Virtual Systems

    Returned strings:

    • SecureXL device is enabled.
    • Failed to start SecureXL.
    • No license for SecureXL.
    • SecureXL is disabled by the firewall. Please try again later.
    • The installed SecureXL device is not compatible with the installed firewall (version mismatch).
    • The SecureXL device is in the process of being stopped. Please try again later.
    • SecureXL cannot be started while "flows" are active.
    • SecureXL is already started.
    • SecureXL will be started after a policy is loaded.
    • fwaccel: Failed to check FloodGate-1 status. Acceleration will not be started.
    • FW-1: SecureXL acceleration cannot be started while QoS is running in express mode.
      Please disable FloodGate-1 express mode or SecureXL.
    • FW-1: SecureXL acceleration cannot be started while QoS is running with citrix printing rule.
      Please remove the citrix printing rule to enable SecureXL.
    • FW-1: SecureXL acceleration cannot be started while QoS is running with UAS rule.
      Please remove the UAS rule to enable SecureXL.
    • FW-1: SecureXL acceleration cannot be started while QoS is running.
      Please remove the QoS blade to enable SecureXL.
    • Failed to enable SecureXL device
    • fwaccel_on: failed to set process context <VSID>
    fwaccel off [-a] [-q]

    Stops acceleration on-the-fly

    • "-q" flag suppresses the output
    • "-a" flag means to stop acceleration on all Virtual Systems

    Returned strings:

    • SecureXL device disabled
    • SecureXL device is not active
    • Failed to disable SecureXL device
    • fwaccel_on: failed to set process context <VSID>
    fwaccel ver

    Shows SecureXL FWAccel and FireWall version

    Example:

    Firewall version: R77.10 - Build 243
    Acceleration Device: Performance Pack
    Accelerator Version 2.1
    Firewall API version: 2.90NG (13/6/2013)
    Accelerator API version: 2.90NG (13/6/2013)
    
    fwaccel stat [-a]

    Shows SecureXL status

    • "-a" flag means to show SecureXL status for all Virtual Systems

    Example:

    Accelerator Status : on
    Accept Templates   : enabled
    Drop Templates     : disabled
    NAT Templates      : disabled by user
    

    Returned strings:

    • Accelerator Status:

      • Accelerator Status : on
      • Accelerator Status : off
      • Accelerator Status : disabled by Firewall
      • Accelerator Status : waiting for policy load
      • Accelerator Status : no license for SecureXL
    • Accept Templates:

      • Accept Templates : enabled
      • Accept Templates : not supported by the SecureXL device
      • Accept Templates : disabled by Firewall
      • Accept Templates : disabled by user
    • Drop Templates (refer to sk66402):

      • Drop Templates : enabled
      • Drop Templates : disabled
      • Drop Templates : not supported by the SecureXL device
      • Drop Templates : disabled by Firewall
      • Drop Templates : disabled by user
    • NAT Templates (refer to sk71200):

      • NAT Templates : enabled
      • NAT Templates : not supported by the SecureXL device
      • NAT Templates : disabled by Firewall
      • NAT Templates : disabled by user
    fwaccel stats Prints all acceleration statistics (output is divided into relevant sections)
       fwaccel stats -h Prints the help message with available flags for 'stats' parameter
       fwaccel stats -s Prints the summary of all acceleration statistics
       fwaccel stats -d Prints the acceleration statistics for dropped packets
       fwaccel stats -n

    Prints the acceleration statistics for Network Access Control (NAC)

    Note: Refer to Identity Awareness Administration Guide (R75, R75.20, R75.40, R75.40VS, R76, R77)
       fwaccel stats -p Prints the acceleration statistics for SecureXL violations (F2F packets)
       fwaccel stats -l Prints all acceleration statistics in Legacy mode (output is not divided into sections)
       fwaccel stats -m Prints the acceleration statistics for multicast traffic
       fwaccel stats -r Resets all acceleration statistics
    fwaccel conns

    Prints the SecureXL Connections Table ('cphwd_db')

    Note: Depending on the number of concurrent connections, might consume memory at very high level

    Returned strings:

    • The SecureXL connections table is empty
    • Failed to read the SecureXL connections table
    • There are VALUE connections in SecureXL connections table
      ...
      Total number of connections: VALUE
       fwaccel conns -h Prints the help message with available flags for 'conns' parameter
       fwaccel conns -s

    Prints the summary of SecureXL Connections Table (number of connections)

    Note: Depending on the number of concurrent connections, might consume memory at very high level
       fwaccel conns -m max_entries Prints the SecureXL Connections Table limited to the number of max_entries
       fwaccel conns -f <filter> Prints the SecureXL Connections Table entries based on <filter> (run 'fwaccel conns -h');
    Filtering flag is a single letter (either capital, or small)
    fwaccel templates

    Prints the SecureXL Connections Templates ('cphwd_tmpl')

    Note: Depending on the number of current templates, might consume memory at very high level

    Returned strings:

    • The SecureXL templates table is empty
    • Failed to read the SecureXL templates table
    • There are VALUE templates in SecureXL templates table
      ...
      Total number of templates: VALUE
       fwaccel templates -h Prints the help message with available flags for 'templates' parameter
       fwaccel templates -s

    Prints the summary of SecureXL Connections Templates (number of templates)

    Note: Depending on the number of current templates, might consume memory at very high level
       fwaccel templates -d Prints the summary of SecureXL Connections Templates for dropped packets
       fwaccel templates -m max_entries Prints the SecureXL Connections Templates limited to the number of max_entries
    fwaccel identities

    Prints the SecureXL Identities Table ('cphwd_dev_identity_table')

    Returned strings:

    • The SecureXL identity table is empty
    • Failed to read the SecureXL identity table - no memory
    • There are VALUE identities in SecureXL identities table
      ...
      Total number of identities: VALUE

    Notes:

       fwaccel identities -h Prints the help message with available flags for 'identities' parameter
       fwaccel identities -s Prints the summary of SecureXL Identities Table (number of entries)
       fwaccel identities -m max_entries Prints the SecureXL Identities Table limited to the number of max_entries
    fwaccel revoked_ips

    Prints the SecureXL Revoked IPs Table ('cphwd_dev_revoked_ips_table')

    Returned strings:

    • The SecureXL templates' revoked IPs table is empty
    • Failed to read the SecureXL templates' revoked IPs table - no memory
    • There are VALUE revoked IPs in SecureXL templates' revoked IPs table
      ...
      Total number of templates' revoked IPs:: VALUE

    Notes:

       fwaccel revoked_ips -h Prints the help message with available flags for 'revoked_ips' parameter
       fwaccel revoked_ips -s Prints the summary of SecureXL Revoked IPs Table (number of entries)
       fwaccel revoked_ips -m max_entries Prints the SecureXL Revoked IPs Table limited to the number of max_entries
    fwaccel dbg Controls SecureXL debugging (run 'fwaccel dbg -h')

    By default, debug messages will be printed to /var/log/messages file, therefore you must set the usual kernel debugging options with:

    [Expert@HostName]# fw ctl debug 0
    [Expert@HostName]# fwaccel dbg resetall
    [Expert@HostName]# sim dbg resetall
    [Expert@HostName]# fw ctl debug -buf 32000
    [Expert@HostName]# fwaccel dbg -m MODULE + FLAG1 FLAG2 ... FLAGn
    [Expert@HostName]# fwaccel dbg list
    [Expert@HostName]# sim dbg list
    [Expert@HostName]# fw ctl kdebug -T -f > /var/log/debug.txt
       fwaccel dbg -h Prints the help message with available options, list of debug modules and their flags
       fwaccel dbg list Prints all currently enabled debug flags for all modules
       fwaccel dbg resetall Resets all debug flags for all modules to their default (none)
       fwaccel dbg -m MODULE reset Resets all debug flags for specified module to their default (none)
       fwaccel dbg -m MODULE all Enables all supported debug flags for specified module
       fwaccel dbg -m MODULE + FLAG1 FLAG2 ... FLAGn Enables specified debug flags for specified module
       fwaccel dbg -m MODULE - FLAG1 FLAG2 ... FLAGn Disables specified debug flags for specified module
       fwaccel dbg -f Source_IP,Source_Port,Dest_IP,Dest_Port,Proto Sets debugging filter - only the specified connection will be printed in the debug output

    Notes:

    • Only 1 filter can be set at one time
    • You can use the asterisk "*" as a wildcard for IP/port/proto

    Example for SSH connection:

    [Expert@HostName]# fwaccel dbg -f 172.30.1.1,*,172.30.41.90,22,6
       fwaccel dbg -f reset Resets the debugging filter

    New 'fwaccel' commands added in R80.20:
    Command Description
       fwaccel dos

     Interface to DOS mitigation techniques in SecureXL.

    Commands:

    • blacklist - Interface to the IP blacklist in SecureXL.
    • config - Interface to DOS mitigation configuration in SecureXL.
    • pbox -  Interface to the penalty box policy in SecureXL.
    • rate - Interface to the rate limiting policy in SecureXL.
    • stats - Interface to DOS real-time statistics.
    • tab -  Manage DOS tables.
    • whitelist - Interface to the IP whitelist in SecureXL.
       fwaccel dos blacklist Interface to the IP blacklist in SecureXL..

    Commands:

    • add - Add IPs to the blacklist.
    • clear - Remove all IPs from the blacklist.
    • get - Print the blacklist.
    • remove - Remove IPs from the blacklist.
       fwaccel dos config Interface to DOS mitigation configuration in SecureXL.

    Commands:

    • get - View configuration parameters.
    • set - Modify configuration parameters.
       fwaccel dos pbox Interface to the penalty box policy in SecureXL.

    Commands:

    • whitelist -Interface to the penalty box source IP whitelist in SecureXL.
       fwaccel dos rate Interface to the rate limiting policy in SecureXL.

    Commands:

    • get - View rule information in the rate limiting policy.
    • install - Manually install a rate limiting policy via stdin. E.g.:

      fw samp get -l -k req_type -t in -v quota | fwaccel dos rate install


    Note: Installing a new rate limiting policy with more than one rule will automatically enable the rate limiting feature. To manually disable the feature after this command, run:

    fwaccel dos config set --disable-rate-limit

    In order to delete the current policy, simply install a new policy with zero rules.

       fwaccel dos stats

     Interface to DOS real-time statistics.

    Commands:

    • clear - Clear real-time statistics.
    • get - View real-time statistics.
       fwaccel dos tab

     Manage DOS tables.

    Commands:

    • get - Print contents of the given DOS table.
       fwaccel dos whitelist

    Interface to the IP whitelist in SecureXL.

    Flags:

    • -a addr/mask   - Add entry to whitelist (/mask is optional)
    • -d addr/mask   - Delete entry from whitelist (/mask is optional)
    • -l filename  - Load whitelist from filename
    • -L                               - Load whitelist from $FWDIR/conf/dos-whitelist-v4.conf
    • -s                              - Show current whitelist entries
    • -F                              - Flush all whitelist entries
    • This whitelist overrides entries in the blacklist. Before using 3rd-party or automatic blacklists, add trusted networks and hosts to the whitelist to avoid outages.
    • This whitelist also unblocks ip options and fragments from trusted sources when the --enable-drop-opts or --enable-drop-frags features are active.
    • For whitelisting of rate limiting policy, refer to the bypass action for the fw samp_policy command. I.e.: "fw samp -a b ..."
    • To replace the current whitelist with the contents of a new file, use both the -F and -l (or -L) options on the same command line.
    • The whitelist file should contain one entry per line in the format of the -a option. Lines beginning with '#' and blank lines are ignored.
    • fwaccel dos whitelist -L is automatically run at boot time.
    • The file that the -L option loads, does not exist by default.
    • See also:

      fwaccel dos pbox whitelist
      fwaccel synatk whitelist
      fw samp -a b




    • (6-1-B) SIM (controls acceleration device)
    Show / Hide SecureXL 'sim' syntax

    [Expert@HostName]# sim <parameter> [-h]

    [Expert@HostName]# sim6 <parameter> [-h]

    Command Description
    sim help Prints the general help message with available parameters
    sim ver [-k] Shows SecureXL SIM version
    sim if

    Prints the list of interfaces used and seen by the SecureXL implementation (Performance Pack)

    Configuration flags
    (sum of the following values in the "F" column):

    • 0x0001 - If set, the packet should be dropped at the end of the inbound processing, if the packet is "cut-through" packet. In outbound, all the packets should ne forwarded to the network.
    • 0x0002 - If set, and the SIM "tcp" feature is enabled (on), then an appropriate notification should be sent whenever a TCP state change occurs (connection established/teared down).
    • 0x0004 - If set, then when encapsulating an encrypted packet (UDP encapsulation), the UDP header's checksum field should be set correctly. If flag is not set, then the UDP header's checksum field should be set to zero. It is safe to ignore this flag if it is set to 0 (i.e., still calculate the checksum).
    • 0x0008 - If set, then when the Connections Table has reached the specified limit, new connections that match a template should not be created and the packet matching the template should be dropped. If flag is not set, the packet should be forwarded to the firewall.
    • 0x0010 - If set, then fragments should be forwarded to the firewall.
    • 0x0020 - If set, then connection creation from templates should be disabled. Connection can still be offloaded to the device. This flag disables only the creation of TCP templates.
    • 0x0040 - Accelerated connections should periodically be refreshed in the firewall through the notification. The refreshing should be done only if this global flag is set.
    • 0x0080 - If set, then connection creation from templates should be disabled. Connection can still be offloaded to the device. This flag disables only the creation of non-TCP templates.
    • 0x0100 - If set, then sequence verification violations should be allowed for connections that did not complete the 3-way handshake process (instead of F2F-ing violating packets).
    • 00x200 - If set, then sequence verification violations should be allowed for connections that completed the 3-way handshake process (instead of F2F-ing violating packets).
    • 0x0400 - If set, then RST TCP packets should be forwarded to firewall.
    • 0x0800 - If set, then Path MTU Discovery should not be enforced for IP multicast packets.
    • 0x1000 - If set, then SIM "drop_templates" feature should be disabled.
    • 0x2000 - Indicates whether Link Selection Load Sharing feature was enabled by the user.
    • 0x4000 - If set, then SecureXL asynchronic notification feature should be disabled.
    • 0x8000 - Indicates that Firewall Connections Table capacity is unlimited.

    Examples:

    F=039 means the sum of the following flags:

    • 0x0001
    • 0x0008
    • 0x0010
    • 0x0020

    F=0x00009a16 means the sum of the following flags:

    • 0x0002
    • 0x0004
    • 0x0010
    • 00x200
    • 0x0800
    • 0x1000
    • 0x8000
    sim vpn <on | off> Enables/Disables acceleration of VPN traffic:
    • sim vpn on = enable acceleration of VPN traffic
    • sim vpn off = disable acceleration of VPN traffic
    sim ranges Prints the loaded ranges
       sim ranges -l Prints the list of loaded ranges
       sim ranges -a Prints all loaded ranges
       sim ranges range_id Prints specified loaded range
       sim ranges -s range_id Prints summary for specified loaded range
    sim affinity Controls network interfaces' affinity settings
    Note: Command is available only on Linux-based OS
       sim affinity -h Prints the help message with available options for 'affinity' parameter
       sim affinity -l Prints the current network interfaces' affinity
       sim affinity -a Sets the network interfaces' affinity in 'Automatic' mode
       sim affinity -s Sets the network interfaces' affinity in 'Static' ('Manual') mode
    sim tmplquota Controls SIM module's template quota
    Note: Command is available only on Linux-based OS
       sim tmplquota -h Prints the help message with available options for 'tmplquota' parameter
       sim tmplquota -e <1 | 0> Enables/Disables template quota feature:
    • sim tmplquota -e 1 = enable the template quota feature
    • sim tmplquota -e 0 = disable the template quota feature
       sim tmplquota -q <quota> Sets maximum of connections per second per template (quota is allowed)
       sim tmplquota -d <drop_duration> Sets drop duration (in seconds) for drop state
       sim tmplquota -m <0 | 1> Controls monitor only mode:
    • sim tmplquota -m 0 = disable monitor only mode (default)
    • sim tmplquota -m 1 = enable monitor only mode
       sim tmplquota -r Resets SIM module template quota values to their defaults
       sim tmplquota -d <file_name> Loads exclusion list of Source IP addresses / Source Subnets from the file
    sim tab -d templates Prints only templates in drop state (output is printed into /var/log/messages files and into Linux kernel ring buffer (output of 'dmesg' command))
    sim affinityload Applies SIM Affinity in 'Automatic' mode
    Note: Command is available only on Linux-based OS
    sim dropcfg Configures drop parameters (run 'sim dropcfg')
    Notes:
       sim dropcfg -h Prints the help message with available options for 'dropcfg' parameter
       sim dropcfg -l Prints current drop configuration
       sim dropcfg -f </path_to/file_name> Sets drop configuration file
    Note: The drop rules configuration does not survive the reboot. Therefore, in order to apply the configured drop rules after the reboot, use a startup script (e.g., /etc/rc.d/rc.local) to run the 'sim dropcfg -f </path_to/file_name>' command automatically during each boot)
       sim dropcfg -e Enforces drop configuration on the external interface only
       sim dropcfg -y Avoids confirmation
       sim dropcfg -r Resets drop rules
    sim dbg Note: From R80.20 "sim dbg" is deprecated, replaced by "fwaccel dbg".

    Controls SecureXL SIM debugging (run 'sim dbg -h')
    By default, debug messages will be printed to /var/log/messages file, therefore you must set the usual kernel debugging options with:
    [Expert@HostName]# fw ctl debug 0
    [Expert@HostName]# sim dbg resetall
    [Expert@HostName]# fwaccel dbg resetall
    [Expert@HostName]# fw ctl debug -buf 32000
    [Expert@HostName]# sim dbg -m MODULE + FLAG1 FLAG2 ... FLAGn
    [Expert@HostName]# sim dbg list
    [Expert@HostName]# fwaccel dbg list
    [Expert@HostName]# fw ctl kdebug -T -f > /var/log/debug.txt
       sim dbg -h Prints the help message with available options, list of debug modules and their flags
       sim dbg list Prints all currently enabled debug flags for all modules
       sim dbg resetall Resets all debug flags for all modules to their default (none)
       sim dbg -m MODULE reset Resets all debug flags for specified module to their default (none)
       sim dbg -m MODULE all Enables all supported debug flags for specified module
       sim dbg -m MODULE + FLAG1 FLAG2 ... FLAGn Enables specified debug flags for specified module
       sim dbg -m MODULE - FLAG1 FLAG2 ... FLAGn Disables specified debug flags for specified module
       sim dbg -f Source_IP,Source_Port,Dest_IP,Dest_Port,Proto Sets debugging filter - only the specified connection will be printed in the debug output

    Notes:

    • Only 1 filter can be set at one time
    • You can use the asterisk "*" as a wildcard for IP/port/proto

    Example for SSH connection:

    [Expert@HostName]# sim dbg -f 172.30.1.1,*,172.30.41.90,22,6
       sim dbg -f reset Resets the debugging filter
    sim feature Controls SIM module features
       sim feature -h Prints the help message with available options for 'feature' parameter
       sim feature <feature_name> <on | off> Enables/Disables the specified SIM module feature
    Available features:
    • anti_spoofing
    • delayed
    • drop_templates
    • dynamic_vpn
    • linksel
    • linksel_ls
    • mcast_route
    • mcast_route_v2
    • nac
    • qos
    • routing
    • streaming
    • tcp
    • vpn
    • wire

     

  • (6-2) Command Line syntax - CoreXL

  • (6-2-A) Show / Hide CoreXL syntax - in Gateway mode

    [Expert@HostName]# fw ctl multik <parameter>

    [Expert@HostName]# fw ctl affinity [-flag1] [-flag2] ...

    [Expert@HostName]# fw6 ctl multik <parameter>

    [Expert@HostName]# fw6 ctl affinity [-flag1] [-flag2] ...

    Command Description
    fw ctl multik Controls CoreXL FW instances
       fw ctl multik Prints the general help message with available parameters
       fw ctl multik stat Prints the summary table for CPU cores and CoreXL FW instances
       fw ctl multik start Starts CoreXL
       fw -i Instance_ID ctl multik start Starts specific CoreXL FW instance
       fw ctl multik stop Stops CoreXL
       fw -i Instance_ID ctl multik stop Stops specific CoreXL FW instance
    fw ctl affinity <options> Controls CoreXL affinities of interfaces/processes/CoreXL FW instances to CPU cores
       fw ctl affinity Prints the help message with available options
       fw -d ctl affinity -corelicnum Prints the number of system CPU cores allowed by CoreXL license
       fw ctl affinity -l Prints the current CoreXL affinities - output shows affinities of interfaces/processes/CoreXL FW instances to CPU cores
       fw ctl affinity -l -r Prints the current CoreXL affinities in reverse order - output shows CPU cores and which interface/process/CoreXL FW instance is affined to each CPU core
       fw ctl affinity -l -a Prints all current CoreXL affinities - output shows affinities of interfaces/processes/CoreXL FW instances to CPU cores, and also shows targets without specific affinity
       fw ctl affinity -l -v Prints the current CoreXL affinities - verbose output shows affinities of interfaces/processes/CoreXL FW instances to CPU cores (targets are shown as 'Interface' (with IRQ), 'Kernel', 'Process'
       fw ctl affinity -l -q Prints the current CoreXL affinities - output shows affinities of interfaces/processes/CoreXL FW instances to CPU cores, and suppresses errors
       fw ctl affinity -l -r -a -v Prints the current CoreXL affinities - verbose output that combines all possible outputs (shows all targets in reverse order)
       fw ctl affinity -l -p PID [-r] [-a] [-v] Prints the current CoreXL affinity of the specified process (by PID) to CPU cores
       fw ctl affinity -l -n Daemon_Name [-r] [-a] [-v] Prints the current CoreXL affinity of the specified process (by name [maximal length = 255 characters]) to CPU cores
       fw ctl affinity -l -k Instance_ID [-r] [-a] [-v] Prints the current CoreXL affinity of the specified CoreXL FW instance to CPU cores
       fw ctl affinity -l -i Interface_Name [-r] [-a] [-v] Prints the current CoreXL affinity of the specified interface to CPU cores
    fw ctl affinity -s <target> { CPU_ID [ CPU_ID ... ] | all } Sets CoreXL Affinity
       fw ctl affinity -s -p PID { CPU_ID [ CPU_ID ... ] | all } Sets CoreXL affinity of the specified process (by PID) to CPU cores
       fw ctl affinity -s -n Daemon_Name { CPU_ID [ CPU_ID ... ] | all } Sets CoreXL affinity of the specified process (by name [maximal length = 255 characters]) to CPU cores
       fw ctl affinity -s -k Instance_ID { CPU_ID [ CPU_ID ... ] | all } Sets CoreXL affinity of the specified CoreXL FW instance to CPU cores
       fw ctl affinity -s -i Interface_Name { CPU_ID [ CPU_ID ... ] | all } Sets CoreXL affinity of the specified interface to CPU cores


    (6-2-B) Show / Hide CoreXL syntax - in VSX mode

    [Expert@HostName]# fw ctl multik <parameter>

    [Expert@HostName]# fw ctl affinity [-flag1] [-flag2] ...

    [Expert@HostName]# fw6 ctl multik <parameter>

    [Expert@HostName]# fw6 ctl affinity [-flag1] [-flag2] ...

    Command Description
    fw ctl multik Controls CoreXL FW instances
       fw ctl multik stat Prints the summary table for CPU cores and CoreXL FW instances
       fw ctl multik stat Prints the general help message with available parameters
       fw ctl multik start Starts CoreXL
       fw -i Instance_ID ctl multik start Starts specific CoreXL FW instance
       fw ctl multik stop Stops CoreXL
       fw -i Instance_ID ctl multik stop Stops specific CoreXL FW instance
    fw ctl affinity <options> Controls CoreXL affinities of Virtual Devices/interfaces/processes/CoreXL FW instances to CPU cores
       fw ctl affinity Prints the help message with available options
       fw -d ctl affinity -corelicnum Prints the number of system CPU cores allowed by CoreXL license
       fw ctl affinity -l Prints the current CoreXL affinities
       fw ctl affinity -l -r Prints the current CoreXL affinities in reverse order
       fw ctl affinity -l -a Prints all current CoreXL affinities
       fw ctl affinity -l -v Prints the current CoreXL affinities - verbose output
       fw ctl affinity -l -q Prints the current CoreXL affinities and suppresses errors
       fw ctl affinity -l -r -a -v Prints the current CoreXL affinities - verbose output that combines all possible outputs (shows all targets in reverse order)
       fw ctl affinity -l -x [-vsid VSID_ranges] [-cpu CPU_ID_ranges] [-flags e|k|t|n|h] [-r] [-a] [-v] Prints the current CoreXL affinities - extended output
    Notes:
    • If "-vsid" flag is omitted, the current context will be used (in which the command was issued)
    • The "-vsid" flag accepts either a single VSID (e.g., '-vsid 7'), or a range of VSID numbers (e.g., '-vsid 0-5'), or a combination (e.g., '-vsid 0-2 4')
    • The "-cpu" flag accepts either a single CPU ID (e.g., '-cpu 7'), or a range of CPU ID numbers (e.g., '-cpu 0-5'), or a combination (e.g., '-cpu 0-2 4')
    • The "-flags" requires at least one of the following arguments (multiple arguments must be specified together):
      • e - do not print exception processes
      • k - do not print kernel threads
      • t - print all process threads
      • n - print process name instead of /proc/<PID>/cmdline
      • h - print CPU mask in Hex format
      • o - print output into the '/tmp/affinity_list_output' file (VSX R77.10 and above)
    fw ctl affinity -s Sets CoreXL Affinity
       fw ctl affinity -s -i Interface_Name CPU_IDs | all Sets affinities of the specified interface to CPU cores
       fw ctl affinity -s -p PID CPU_IDs | all Sets affinities of the specified process (by PID) to CPU cores
       fw ctl affinity -s -d -pname Daemon_Name [-vsid VSID_ranges] -cpu CPU_ID_ranges Sets affinity of the specified process (by name) to CPU cores
    Notes:
    • If "-vsid" flag is omitted, the current context will be used (in which the command was issued)
    • The "-vsid" flag accepts either a single VSID (e.g., '-vsid 7'), or a range of VSID numbers (e.g., '-vsid 0-5'), or a combination (e.g., '-vsid 0-2 4')
    • The "-cpu" flag accepts either a single CPU ID (e.g., '-cpu 7'), or a range of CPU ID numbers (e.g., '-cpu 0-5'), or a combination (e.g., '-cpu 0-2 4')
       fw ctl affinity -s -d [-vsid VSID_ranges] -cpu CPU_ID_ranges Sets affinities of Virtual Devices (VS, VR, VSW) to CPU cores
    Notes:
    • If "-vsid" flag is omitted, the current context will be used (in which the command was issued)
    • The "-vsid" flag accepts either a single VSID (e.g., '-vsid 7'), or a range of VSID numbers (e.g., '-vsid 0-5'), or a combination (e.g., '-vsid 0-2 4')
    • The "-cpu" flag accepts either a single CPU ID (e.g., '-cpu 7'), or a range of CPU ID numbers (e.g., '-cpu 0-5'), or a combination (e.g., '-cpu 0-2 4')
       fw ctl affinity -s -d -inst Instances_ranges -cpu CPU_ID_ranges Sets affinities of the specified FWK instances to CPU cores
    Notes:
    • The "-inst" flag accepts either a single FWK_ID (e.g., '-inst 7'), or a list of FWK_ID numbers (e.g., '-inst 0 2 4')
    • The "-cpu" flag accepts either a single CPU ID (e.g., '-cpu 7'), or a range of CPU ID numbers (e.g., '-cpu 0-5'), or a combination (e.g., '-cpu 0-2 4')
       fw ctl affinity -s -d -fwkall Number_of_CPUs Sets affinities of all FWK instances to CPU cores (where Number_of_CPUs is an integer number)
    fw ctl affinity -vsx_factory_defaults Resets all VSX affinity settings (prompts the user) (VSX R77 and above)
    fw ctl affinity -vsx_factory_defaults_no_prompt Resets all VSX affinity settings (does not prompt the user) (VSX R77 and above)

     

  • (6-3) Command Line syntax - Multi-Queue

  • Show / Hide Multi-Queue 'cpmq' syntax

    [Expert@HostName]# cpmq <parameter>

    Command Description
    cpmq get Shows Multi-Queue status of active supported interfaces
       cpmq get -a Shows Multi-Queue status of all supported interfaces (those with enabled Multi-Queue and those with disabled Multi-Queue)
       cpmq get -v Shows Multi-Queue status of active supported interfaces with IRQ affinity information
       cpmq get rx_num igb Shows the number of active RX queues for interfaces that use igb driver
       cpmq get rx_num ixgbe Shows the number of active RX queues for interfaces that use ixgbe driver
    cpmq set [-f] Enables/Disables Multi-Queue per interface (the "-f" flag forces the operation)
       cpmq set rx_num all default [-f] Sets the number of active RX queues to the number of CPU cores that run as CoreXL SND (CPU cores that are not used by CoreXL FW instances) - for all interfaces (those that use igb driver and those that use ixgbe driver)
    Note: This is recommended configuration
       cpmq set rx_num igb default [-f] Sets the number of active RX queues to the number of CPU cores, which are not used by CoreXL FW instances (recommended) - for interfaces that use igb driver
       cpmq set rx_num ixgbe default [-f] Sets the number of active RX queues to the number of CPU cores, which are not used by CoreXL FW instances (recommended) - for interfaces that use ixgbe driver
       cpmq set rx_num all <number> [-f] Sets the number of active RX queues to the number between 2 and the number of CPU cores - for all interfaces (those that use igb driver and those that use ixgbe driver)
       cpmq set rx_num igb <number> [-f] Sets the number of active RX queues to the number between 2 and the number of CPU cores - for interfaces that use igb driver
       cpmq set rx_num ixgbe <number> [-f] Sets the number of active RX queues to the number between 2 and the number of CPU cores - for interfaces that use ixgbe driver
       cpmq set affinity Sets the IRQ affinity for Multi-Queue interfaces after the following occurs (in the given order):
    1. Multi-Queue is enabled on an interface
    2. The interface status is changed to 'down'
    3. The machine is rebooted
    4. The interface status is changed back to 'up'
    Run this command after the interface status is changed back to 'up' to reset the IRQ affinity for this interface.
       cpmq reconfigure [-q] Reconfigures the Multi-Queue (the "-q" flag suppresses the output):
    • After changing the number of CoreXL FW instances
    • After changing the physical interfaces on the machine
    • After changing the number of CPU cores, to which the fwk processes on VSX are assigned

     


     

    (7) Examples

    Show / Hide example #1

    1. Output of 'top' command shows that 'fw_worker_X' processes constantly consume the CPU at 100%:

      Cpu0 : 3.3%us, 9.6%sy, 0.0%ni, 0.6%id, 0.0%wa, 0.0%hi, 86.5%si, 0.0%st
      Cpu1 : 7.6%us, 16.6%sy, 0.0%ni, 0.5%id, 0.0%wa, 1.0%hi, 74.2%si, 0.0%st
      Cpu2 : 3.0%us, 14.6%sy, 0.0%ni, 0.6%id, 0.0%wa, 0.0%hi, 81.8%si, 0.0%st
      Cpu3 : 3.0%us, 11.3%sy, 0.0%ni, 0.5%id, 0.0%wa, 0.0%hi, 85.2%si, 0.0%st
      Cpu4 : 1.0%us, 7.6%sy, 0.0%ni, 0.7%id, 0.0%wa, 0.0%hi, 90.7%si, 0.0%st
      
        PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
       4083 admin     15   0     0    0    0 S    2  100   101:01.47 fw_worker_1
       4019 admin     15   0     0    0    0 S    2  100   101:01.47 fw_worker_2
       4023 admin     15   0     0    0    0 S    2  100   101:01.47 fw_worker_5
      
    2. Output of 'fwaccel stats -s' command shows very poor acceleration ratios and high percentage of packets that are passing through Medium path (PXL):

      Accelerated conns/Total conns : 364/13215 (2%)
      Delayed conns/(Accelerated conns + PXL conns) : 48/12023 (0%)
      Accelerated pkts/Total pkts   : 18252/564927 (3%)
      F2Fed pkts/Total pkts   : 36776/564927 (6%)
      PXL pkts/Total pkts   : 509899/564927 (90%)
      QXL pkts/Total pkts   : 0/564927 (0%)
      
    3. Output of 'fwaccel stat' command shows that 'Accept Templates' are disabled after specific rule:

      Accept Templates   : disabled by Firewall
                           disabled from rule #251
      
    4. Thorough analysis of Firewall's Connections Table (using 'connstat' utility from sk85780) during the issue (~99800 concurrent connections) showed that most matched rules are lower than Rule #251 (after which SecureXL Accept Templates are disabled):

      Top 10 Rules: 
      =============
      Rule: 342       Hits: 71463
      Rule: 343       Hits: 16732
      Rule: 356       Hits: 4488
      Rule: 354       Hits: 3126
      Rule: 216       Hits: 2249
      Rule: 361       Hits: 2003
      Rule: 357       Hits: 1536
      Rule: 51        Hits: 722
      Rule: 25        Hits: 666
      Rule: 362       Hits: 426
      
    5. Rulebase was optimized per:

    6. Performance has improved greatly after this change.

    7. Output of 'fwaccel stats -s' command showed that 'Accept Templates' are now disabled after much lower rule:

      Accept Templates   : disabled by Firewall
                           disabled from rule #396
      
    8. The following steps were taken to decrease the percentage of packets that are passing through Medium path (PXL):

      1. Current IPS profile:

        • 2400 protections in 'Detect' mode
        • 870 protections in 'Prevent' mode
      2. Created new IPS profile according to the environment:

        • 15 protections in 'Detect' mode
        • 570 protections in 'Prevent' mode
        • All other protections were set to 'Inactive'
      3. In the Application Control & URL Filtering policy, disabled the "Any - Any - Allow" rule.

      4. In the Anti-Virus & Anti-Bot configuration, excluded networks, whose traffic does not have to be inspected per sk92515 - How to configure Anti-Virus Exceptions.
    9. Performance has improved greatly after the overall optimization:

      • Before the optimization:
        Accelerated pkts/Total pkts   : 18252/564927 (3%)
        PXL pkts/Total pkts   : 509899/564927 (90%)
        
      • After the optimization:
        Accelerated pkts/Total pkts   : 2454970439/2821805103 (87%)
        PXL pkts/Total pkts   : 282180510/2821805103 (10%)
        


    Show / Hide example #2

    Relevant outputs were collected during a period of time from Security Gateway with the help of the shell script that runs relevant commands every 1 second. All outputs were analyzed and correlated to each other.

    1. Output of 'vmstat' command showed that CPU time is consumed by:

      • IOWait : min 0% , average 36.6% , max 82%
      • Kernel Space : min 11% , average 16.2% , max 45%
      • User Space : min 0% , average 1.8% , max 36%
      This leaves IDLE at min 0% , average 45.3% , max 89%

      Example:

      procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
       r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
      ......
       1  7 334816 708020    820  37388 2152    0  3280     4 18012 17602  0 13  7 80  0
      ......
       0  9 334348 652800    988  29420 1664    0  3944     4 21575 19725  1 13  5 80  0
      ......
       1 12 291604 723400    884  29696  196    0  4272    12 20799 18273  5 18  0 77  0  
      
    2. Output of 'cat /proc/stat' command showed that CPU time is consumed by:

      • User Space : min 0% , average 1% , max 7%
      • Kernel Space : min 0% , average 1% , max 3%
      • Idle : min 16% , average 34% , max 54%
      • IOWait : min 32% , average 48% , max 69%
      • IRQ : min 0% , average 1% , max 1%
      • SoftIRQ : min 11% , average 14% , max 19
    3. Output of 'top' command showed:

      • fw_worker_0 / fw_worker_1 / fw_worker_2 - constantly consume CPU at ~15-20%
      • fw_worker_X spike to 20-25%
      • 'fw' spikes up to 50, 60, 80%
      • fwssd , in.asessiond , in.aufpd , in.ufclnt , mdq , igwd , sds , stormd , dtps

      Example:

        PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
       6283 root      19   0 54688  16m  12m D   87  0.8   0:00.56 fw                 
       2283 root      15   0     0    0    0 R   30  0.0   4844:09 fw_worker_2        
       2209 root      15   0     0    0    0 R   16  0.0   5254:57 fw_worker_1        
       2144 root      15   0     0    0    0 S   14  0.0   5227:59 fw_worker_0
      
    4. Output of 'ps auxwf' command correlated to the output of 'top' command showed that during the spikes of CPU load by 'fw', the CPD daemon calls the 'sim affinity -c' command, which calls a series of Check Point shell script that calculate the current load by running various 'fw ctl' commands. These commands process high amount of data - and this causes the spike in CPU load.
      The CPD daemon calling the 'sim affinity -c' command shows that SIM Affinity is configured in Automatic mode (refer to sk63330 - Explanation about 'sim affinity -c' , 'fwaffinity_used_cpus' , 'fw ctl affinity -l -v'):

      root     15829  0.1  1.2 231704 25748 ?        Ssl  Jul23  49:45  \_ cpd
      root      5153  0.0  0.0   2832  1088 ?        S    12:19   0:00  |   \_ sim affinity -c
      root      5212  0.0  0.0   1236   564 ?        S    12:19   0:00  |       \_ sh -c $FWDIR/scripts/fwaffinity_used_cpus > /tmp/sim_fw_cpus.tmp
      root      5213  0.0  0.0   1236   564 ?        S    12:19   0:00  |           \_ /bin/sh /opt/CPsuite-R75.20/fw1/scripts/fwaffinity_used_cpus
      root      5214  0.0  0.0   1236   424 ?        S    12:19   0:00  |               \_ /bin/sh /opt/CPsuite-R75.20/fw1/scripts/fwaffinity_used_cpus
      root      5215  0.0  0.0   1236   372 ?        S    12:19   0:00  |                   \_ /bin/sh /opt/CPsuite-R75.20/fw1/scripts/fwaffinity_used_cpus
      root      5216  0.0  0.8  55636 18080 ?        S    12:19   0:00  |                       \_ fw ctl affinity -l -v
      root      5226  0.0  0.0   1236   552 ?        S    12:19   0:00  |                       |   \_ sh -c /bin/netstat -n 2>&1
      root      5227  0.0  0.0   1668   560 ?        R    12:19   0:00  |                       |       \_ /bin/netstat -n
      root      5217  0.0  0.0   1644   476 ?        S    12:19   0:00  |                       \_ grep -E -v ^Interface.*:
      root      5218  0.0  0.0   2028   644 ?        S    12:19   0:00  |                       \_ awk -F: {print $2}
      root      5219  0.0  0.0   1628   464 ?        S    12:19   0:00  |                       \_ sed -e s/CPU//
      root      5220  0.0  0.0   1604   372 ?        S    12:19   0:00  |                       \_ tr [:space:] \n
      root      5221  0.0  0.0   1644   468 ?        S    12:19   0:00  |                       \_ grep -v ^[:space:]*$
      root      5222  0.0  0.0  35424   508 ?        S    12:19   0:00  |                       \_ sort
      root      5223  0.0  0.0   1592   384 ?        S    12:19   0:00  |                       \_ uniq
      
    5. SIM Affinity was configured from Auto Mode to Static Mode:

      [Expert@HostName]# sim affinity -s
    6. To minimize the load caused by IO Waiting, Linux kernel was fine-tuned per sk60703 - IOWait consumes 100% CPU after Security policy installation.

     


     

    Show / Hide related documentation

     


     

    Show / Hide related solutions

     


     

    (10) Revision history

    Show / Hide revision history

    Date Description
    02 Oct 2017 Added additional related solution (sk120131)
    12 Sep 2017 Updated instructions for Valgrind tool (to get this tool, need to contact Check Point Support)
    19 June 2017 Updated definition of "Medium path (PXL)"
    18 June 2017 Added additional related solution (sk115072)
    20 Apr 2017 Added additional related solution (sk110351)
    15 Apr 2017 Added additional related solution
    08 Feb 2017 Added additional related solutions
    21 Nov 2016 Improved HTML design
    04 Aug 2016 Added instructions for the sar command
    11 Jan 2016 Corrected the syntax for using the pstack script
    11 Oct 2015 Updated definition of "Medium path (PXL)"
    04 July 2015 Added additional related solution
    11 June 2015 Added additional related solution
    13 May 2015 Added additional related solution
    25 Apr 2015 Updated instructions for Valgrind tool
    08 Apr 2015 Added additional related solution
    02 Mar 2015 Updated the best practices for Application Control & URL Filtering optimization
    10 Feb 2015 Added additional related solutions
    14 Jan 2015 Corrected the the formula for calculating the maximum number of concurrent connections
    20 Nov 2014 Corrected the best practices for Application Control & URL Filtering optimization
    05 Nov 2014 Added additional related solutions
    08 Sep 2014 Added CPView Utility
    02 July 2014 Added additional related solutions
    19 May 2014 Added additional related solutions
    13 May 2014 Added additional related solutions
    12 Apr 2014 Added additional related solutions
    16 Mar 2014 Added additional related solutions
    Added SecureXL Penalty box feature to SecureXL Limitations
    12 Mar 2014 Added additional related solutions
    Added instructions for 'valgrind' tool
    Added instructions for 'pstack' shell script
    11 Mar 2014 First release of this document

    Give us Feedback
    Please rate this document
    [1=Worst,5=Best]
    Comment