Performance Pack is a software acceleration product installed on Security Gateways. Performance Pack uses SecureXL technology and other innovative network acceleration techniques to deliver wire-speed performance for Security Gateways. SecureXL is implemented either in software, or in hardware (SAM cards on Check Point 21000 appliances; ADP cards on IP Series appliances).
Affinity - Association of a particular network interface with a CPU core (either 'Automatic' (default), or 'Static' / 'Manual'). Interfaces are bound to CPU cores via SMP IRQ affinity settings (refer to sk61962 - SMP IRQ Affinity on Check Point Security Gateway).
Notes:
The default SIM Affinity setting for all interfaces is 'Automatic' - the affinity for each interface is automatically reset every 60 seconds, and balanced between all available CPU cores based on the current CPU load.
Connection offload - Firewall kernel passes the relevant information about the connection from Firewall Connections Table to SecureXL Connections Table. Note: In ClusterXL High Availability, the connections are not offloaded to SecureXL on Standby member.
Connection notification - SecureXL passes the relevant information about the accelerated connection from SecureXL Connections Table to Firewall Connections Table.
Partial connection - Connection that exists in the Firewall Connections Table, but not in the SecureXL Connections Table (versions R70 and above).
In Cluster HA - partial connections are offloaded when member becomes Active
In Cluster LS - partial connections are offloaded upon post-sync (only for NAT / VPN connections)
Such connections must be offloaded to SecureXL, since packets in these connections must not be dropped. If a packet matched a partial connection in the outbound, then it should be dropped.
Delayed connection - Connection created from SecureXL Connection Templates without notifying the Firewall for a predefined period of time. The notified connections are deleted by the Firewall.
Anticipated connection - Connection that is anticipated by SecureXL based on policy rules to avoid dropping it by Drop Template.
In the following firewall policy:
NO.
SOURCE
DESTINATION
VPN
SERVICE
ACTION
1
Any
Any
Any Traffic
ftp
Accept
2
Any
Any
Any Traffic
Any
Drop
Any FTP connection that will be opened will be accepted.
A SYN packet of a Telnet connection will be dropped and a Drop Template will be offloaded to SecureXL.
The FTP Data connection FTP might match this Drop Template and will be dropped.
Therefore, to prevent the drop of a legitimate connection:
SecureXL will offload such connections (in this example, FTP Data) and will mark them as anticipated.
SecureXL will try to match an anticipated connection to an existing connection or an existing Accept Template.
If such match was found, the packet will be forwarded to the firewall and will not be matched to a Drop Template.
Accept Template - Feature that accelerates the speed, at which a connection is established by matching a new connection to a set of attributes. When a new connection matches the Accept Template, subsequent connections are established without performing a rule match and therefore are accelerated. Accept Templates are generated from active connections according to policy rules. Currently, Accept Template acceleration is performed only on connections with the same destination port (using wildcards for source ports). Note: Size of Templates table (cphwd_tmpl, id 8111) is limited to 1/4 of the size of Firewall Connections Table (connections, id 8158).
Drop Template - Feature that accelerates the speed, at which a connection is dropped by matching a new connection to a set of attributes. When a new connection matches the Drop Template, subsequent connections are dropped without performing a rule match and therefore are accelerated. Currently, Drop Template acceleration is performed only on connections with the same destination port (does not use wildcards for source ports). Drop Templates are generated from policy rules by special algorithm:
Analyze the rulebase
Produce mutually exclusive ranges
Offload the ranges to SecureXL
Once a packet is dropped, offload a Drop Template
All subsequent packets matching that range will be dropped by SecureXL
Accelerated path - Packet flow when the packet is completely handled by the SecureXL device. It is processed and forwarded to the network.
Medium path (PXL) - Packet flow when the packet is handled by the SecureXL device, except for IPS (some protections) / VPN (in some configurations) / Application Control / Content Awareness / Anti-Virus / Anti-Bot / HTTPS Inspection / Proxy mode / Mobile Access / VoIP / Web Portals. The CoreXL layer passes the packet to one of the CoreXL FW instances to perform the processing (even when CoreXL is disabled, the CoreXL infrastructure is used by SecureXL device to send the packet to the single FW instance that still functions).
Firewall path / Slow path (F2F) - Packet flow when the SecureXL device is unable to process the packet (refer to sk32578 - SecureXL Mechanism). The packet is passed on to the CoreXL layer and then to one of the CoreXL FW instances for full processing. This path also processes all packets when SecureXL is disabled.
Active Streaming (CPAS) - Technology that sends streams of data to be inspected in the kernel, since more than a single packet at a time is needed in order to understand the application that is running (such as HTTP data). Active Streaming is Read- and Write-enabled, and works as a transparent proxy. Connections that pass through Active Streaming can not be accelerated by SecureXL.
Passive Streaming - Technology that sends streams of data to be inspected in the kernel, since more than a single packet at a time is needed in order to understand the application that is running (such as HTTP data). Passive Streaming is Read-only and it cannot hold packets, but the connections are accelerated by SecureXL.
Passive Streaming Library (PSL) - IPS infrastructure, which transparently listens to TCP traffic as network packets, and rebuilds the TCP stream out of these packets. Passive Streaming can listen to all TCP traffic, but process only the data packets, which belong to a previously registered connection. For more details, refer to sk95193 - ATRG: IPS.
PXL - Technology name for combination of SecureXL and PSL.
QXL - Technology name for combination of SecureXL and QoS (R77.10 and above).
F2F / F2Fed - Packets that can not be accelerated by SecureXL (refer to sk32578 - SecureXL Mechanism) are Forwarded to Firewall.
F2P - Forward to PSL/Applications. Feature that allows to perform the PSL processing on the CPU cores, which are dedicated to the Firewall.
IRQ Swizzling - Traditionally, in a PCIe bus, all PCIe ports are mapped to one interrupt. Swizzling allows the PCIe slots to be balanced across four interrupts instead of one (enabling IRQ Swizzling requires a BIOS update).
SIM Affinity has been deprecated in R80.40 and higher versions, refer to sk170012.
SIM Affinity is applicable only to physical interfaces (VLAN / Bond interfaces are influenced by the "physical" decision). Note: Some systems may require BIOS update to enable IRQ Swizzling and EIRQ technologies. To determine whether a specific system supports the required technology, contact your hardware vendor.
On VSX Gateway, WARP interfaces can not be assigned to a CPU core.
SecureXL NAT Templates are supported only in R75.40 and above (sk71200).
SecureXL Drop Templates are supported only in R76 and above (sk66402).
SecureXL Optimized Drops feature (used for DoS/DDoS protection) is supported only in R76 and above (sk90861).
SecureXL Penalty box feature (used for DoS/DDoS protection) is supported only in R75.40VS and above (on VSX Gateway, the penalty box would only be enforced on Virtual System 0) (sk74520).
SecureXL does not support Point-to-Point interfaces (PPP, PPTP, PPPoE).
In R80.20, SAM is supported only for non-accelerated usage. Traffic connected to the Acceleration-ready 10G Interface Card (CPAC-ACCL-4-10F-21000) will be handled by the host. 10G Ports on the CPAC-ACCL-4-10F-21000 cannot be assigned as SAM ports in R80.20. Note: In case a PPP-interface is detected, SecureXL disables itself on that interface (sk79880).
ClusterXL Sticky Decision Function (SDF) disables SecureXL.
QoS disables SecureXL (for R77.10 and above, refer to sk98229).
Delayed Synchronization in cluster:
Applies only to TCP services whose 'Protocol Type' is set to 'HTTP' or 'None'.
Delayed Synchronization is disabled if the 'Track' option in the rule is set to 'Log' or 'Account'.
Delayed Synchronization is performed only for connections matching a SecureXL Connection Template.
It is possible that a connection will exist in the Firewall Connections Table, but not in the SecureXL Connections Table (partial connection). This situation can occur:
After policy installation
After cluster failover
If user turned off ('fwaccel off' command) and turned on ('fwaccel on' command) acceleration
CoreXL - A performance-enhancing technology for Security Gateways on multi-core processing platforms. CoreXL enhances Security Gateway performance by enabling the CPU processing cores to concurrently perform multiple tasks.
Secure Network Distributor (SND) - Traffic entering network interface cards (NICs) is directed to a processing CPU core running the SND, which is responsible for:
Processing incoming traffic from the network interfaces
Securely accelerating authorized packets (if SecureXL is enabled)
Distributing non-accelerated packets among Firewall kernel instances (SND maintains global dispatching table - which connection was assigned to which instance)
Firewall Instance / FW Instance - On a Security Gateway with CoreXL enabled, the Firewall kernel is replicated multiple times. Each replicated copy, or Firewall Instance, runs on one CPU processing core. These FW instances handle traffic concurrently, and each FW instance is a complete and independent Firewall inspection kernel. When CoreXL is enabled, all the Firewall kernel instances on the Security Gateway process traffic through the same interfaces and apply the same security policy.
Affinity - Association of a particular network interface / FW kernel instance / daemon with a CPU core (either 'Automatic' (default), or 'Manual'). Note: The default CoreXL interface affinity setting for all interfaces is 'Automatic' when SecureXL is installed and enabled.
Accelerated path - Packet flow when the packet is completely handled by the SecureXL device. It is processed and forwarded to the network.
Medium path (PXL) - Packet flow when the packet is handled by the SecureXL device, except for IPS (some protections) / VPN (in some configurations) / Application Control / Content Awareness / Anti-Virus / Anti-Bot / HTTPS Inspection / Proxy mode / Mobile Access / VoIP / Web Portals. The CoreXL layer passes the packet to one of the CoreXL FW instances to perform the processing. This path is available only when CoreXL is enabled.
Firewall path / Slow path (F2F) - Packet flow when the SecureXL device is unable to process the packet (refer to sk32578 - SecureXL Mechanism). The packet is passed on to the CoreXL layer and then to one of the CoreXL FW instances for full processing. This path also processes all packets when SecureXL is disabled.
Passive Streaming Library (PSL) - IPS infrastructure, which transparently listens to TCP traffic as network packets, and rebuilds the TCP stream out of these packets. Passive Streaming can listen to all TCP traffic, but process only the data packets, which belong to a previously registered connection. For more details, refer to sk95193 - ATRG: IPS.
PXL - Technology name for combination of SecureXL and PSL.
Supported operating systems:
Gaia OS
SecurePlatform OS
IPSO OS
Crossbeam XOS
Crossbeam COS
Architecture:
Example of default configuration for machine with 8 CPU cores:
Default number of CoreXL IPv4 FW instances:
Note: The real number of CoreXL FW instances depends on the current CoreXL license.
Number of CPU cores
Default number of CoreXL IPv4 FW instances
Default number of Secure Network Distributors (SNDs)
1
1 Note: CoreXL is disabled
0 Note: CoreXL is disabled
2
2
2
4
3
1
6 - 20
[Number of CPU cores] - 2
2
More than 20
[Number of CPU cores] - 4 Note: However, no more than 40 (*).
Relation between CoreXL IPv4 FW instances and CoreXL IPv6 FW instances: (run the 'fw ctl multik stat' command and 'fw6 ctl multik stat' command on Security Gateway)
The number of IPv4 FW instances - from a minimum of 2 to a number equal to the maximal number of CPU cores on the Security Gateway.
The number of IPv6 FW instances - from a minimum of 2 to a number equal to the number of IPv4 FW instances.
The number of IPv6 FW instances cannot exceed the number of IPv4 FW instances.
The total number of IPv4 FW instances and IPv6 FW instances together cannot exceed 32.
CoreXL and ClusterXL:
Number of CoreXL FW instances must be identical on all members of the cluster because the state synchronization between members is performed per CoreXL FW instance (e.g., Instance #2 on Member_A can synchronize only with Instance #2 on Member_B). Note: Member with higher number of CoreXL FW instances will enter the 'Ready' state. Refer to sk42096 - Cluster member is stuck in 'Ready' state.
SMT (Simultaneous Multi-Threading - Intel® HyperThreading, or Intel® HT) is a feature that is supported on Check Point appliances running Gaia OS. When enabled, SMT doubles the number of logical CPUs on the Security Gateway, which enhances physical processor utilization. When SMT is disabled, the number of logical CPUs equals the number of physical cores. SMT improves performance up to 30% in NGFW software blades such as IPS, Application & URL Filtering and Threat Prevention by increasing the number of CoreXL FW instances based on the number of logical CPUs.
Supported configurations:
SMT is supported by R77 release and later.
SMT is supported only on Security Gateways running Gaia OS with 64-bit kernel.
SMT is supported on Check Point appliances. On Open Servers, SMT is supported starting R80.40 Jumbo HFA take #45.
Limitations:
SMT is relevant only if CoreXL is enabled.
Notes:
When SMT (HyperThreading) is enabled, the number of available CPU cores is not linear.
Example:
CPU core 0-5 - socket 1
CPU core 6-11 - socket 2
CPU core 12-15 - socket 1 HyperThreading
CPU core 16-23 - socket 2 HyperThreading
Today, each network interface card has one traffic queue that is handled by one CPU at a time. Since the Secure Network Distributor (SND) - SecureXL and CoreXL Distributor is running on the CPU cores that handle the traffic queues, user cannot use more CPU cores for acceleration than the number of network interface cards passing the traffic.
Definitions:
Multi-Queue is an acceleration feature that lets the user configure more than one traffic queue for each network interface card, which allows using more CPU cores for acceleration.
rx queue - Receive packet queue.
tx queue - Transmit packet queue.
Secure Network Distributor (SND) - Traffic entering network interface cards (NICs) is directed to a processing CPU core running the SND, which is responsible for:
Processing incoming traffic from the network interfaces
Securely accelerating authorized packets (if SecureXL is enabled)
Distributing non-accelerated packets among Firewall kernel instances
IRQ affinity - Process of binding a network interface card's IRQ to one or more CPU cores.
Accelerated path - Packet flow when the packet is completely handled by the SecureXL device. It is processed and forwarded to the network.
Medium path (PXL) - Packet flow when the packet is handled by the SecureXL device, except for IPS (some protections) / VPN (in some configurations) / Application Control / Content Awareness / Anti-Virus / Anti-Bot / HTTPS Inspection / Proxy mode / Mobile Access / VoIP / Web Portals. The CoreXL layer passes the packet to one of the CoreXL FW instances to perform the processing. This path is available only when CoreXL is enabled.
Firewall path / Slow path - Packet flow when the SecureXL device is unable to process the packet (refer to sk32578 - SecureXL Mechanism). The packet is passed on to the CoreXL layer and then to one of the CoreXL FW instances for full processing. This path also processes all packets when SecureXL is disabled.
Passive Streaming Library (PSL) - IPS infrastructure, which transparently listens to TCP traffic as network packets, and rebuilds the TCP stream out of these packets. Passive Streaming can listen to all TCP traffic, but process only the data packets, which belong to a previously registered connection. For more details, refer to sk95193 - ATRG: IPS.
PXL - Technology name for combination of SecureXL and PSL.
Supported configurations:
Multi-Queue is integrated into R76, R77 and above. For lower versions, a Multi-Queue hotfix has to be installed (refer to sk80940).
Multi-Queue is supported on Check Point Appliances (including IP Series Appliances) and on Open Servers.
Multi-Queue is supported only on machines that run SecurePlatform OS or Gaia OS.
Multi-Queue is supported only for network interface cards that use igb (1 GbE) and ixgbe (10 GbE) drivers.
Limitations:
Multi-Queue is relevant only if SecureXL is enabled.
Multi-Queue is relevant only if CoreXL is enabled.
Multi-queue is not supported on machines with a single CPU core.
In Gaia with kernel 2.6.18: Multi-Queue allows to configure a maximum of 5 interfaces (due to IRQ limitations).
The number of traffic queues is limited by the number of CPU cores and the type of network interface card driver:
igb driver:
Gaia with kernel 2.6.18: up to 4 RX queues
Gaia with kernel 3.10: 2-16 (depends on the interface)
You should use the PCI Express (PCIe) cards, because they have better performance than PCI-X cards.
If you are using a motherboard with multiple PCI or PCI-X buses, make sure that each Network Interface Card is installed in a slot connected to a different bus.
If you are using more than two Network Interface Cards in a system with only two 64-bit/66Mhz PCI buses, make sure that the least-used cards are installed in slots connected to the same bus.
Setting the maximal number of concurrent connections:
Versions R75.40VS, R75.45 and above
SmartDashboard - open Security Gateway object.
Go to 'Optimizations' pane.
The 'Calculate the maximum limit for concurrent connections' should be set to 'Automatically'. Note: Manual limit should be set only for security reasons.
Enter the desired value.
The 'Calculate connections hash table size and memory pool' should be set to 'Automatically'.
Click on 'OK'.
Install the policy.
Versions R75.40 and lower
SmartDashboard - open Security Gateway object.
Go to 'Optimizations' pane.
In the 'Calculate the maximum limit for concurrent connections', select the 'Manually' (recommended setting is 'Automatically').
Enter the desired value.
The 'Calculate connections hash table size and memory pool' should be set to 'Automatically'.
Click on 'OK'.
Install the policy.
Notes:
You should ensure that the total number of concurrent connections is appropriate to the TCP end timeout. Too many concurrent connections will adversely affect the Security Gateway's performance.
You can calculate the maximum number of concurrent connections by multiplying the session establishment rate by the TCP session timeout: [MAXIMAL number of concurrent connections] = [MAXIMAL session establishment rate] x [TCP entire session timeout] Notes:
This formula is used to understand what number of concurrent connections a session rate test will generate.
It is very hard to predict the maximal connections capacity of a Security Gateway because of multiple varying factors.
Administrator should monitor the memory utilization on Security Gateway after changing these settings.
Only the maximal (possible) session rate should be considered.
This formula will not predict connections capacity, which is stated in Check Point datasheet documents.
TCP timeout varies highly between applications and protocols (e.g., SSH is usually long, HTTP is usually short).
By reducing the following timeouts, you increase the capacity of actual TCP and UDP connections (SmartDashboard - 'Policy' menu - 'Global Properties' - 'Stateful Inspection'):
TCP end timeout - determines the amount of time a TCP connection will stay in the FireWall Connections Table (id 8158) after a TCP session has ended.
UDP virtual session timeout - determines the amount of time a UDP connection will stay in the FireWall Connections Table (id 8158) after the last UDP packet was seen by the Security Gateway.
Optimal connection/sec rate:
SmartDashboard - go to 'Policy' menu - click on 'Global Properties'.
Go to 'FireWall' pane.
Uncheck the following boxes:
Accept RIP
Accept Domain Name over UDP (Queries)
Accept Domain Name over TCP (Zone Transfer)
Accept ICMP requests
Click on 'OK'.
Install the policy.
ARP cache table:
Increase the number of entries in the ARP cache table:
If you are testing large subnets that are directly connected to the Security Gateway without a router.
If 'kernel: neighbour table overflow' appears repeatedly in /var/log/messages files and in the output of the 'dmesg' command.
Either decrease the TCP end timeout to 2 seconds (SmartDashboard - 'Policy' menu - 'Global Properties' - 'Stateful Inspection').
Or increase the hash size of SND's Connection Table by increasing (permanently per sk26202) the value of the kernel parameter fwmultik_gconn_tab_hsize from the default 524288 to 8388608 and rebooting the machine. Important Note: This change reduces the capacity for the maximum number of concurrent connections.
You should use the PCI Express (PCIe) cards, because they have better performance than PCI-X cards.
If you are using a motherboard with multiple PCI or PCI-X buses, make sure that each Network Interface Card is installed in a slot connected to a different bus.
If you are using more than two Network Interface Cards in a system with only two 64-bit/66Mhz PCI buses, make sure that the least-used cards are installed in slots connected to the same bus.
Network interface affinity:
Each NIC should be bound (affined) to a separate CPU core using the 'Static' affinity mode (run the 'sim affinity -s' command).
Note: 'sim affinity' has been deprecated in R80.40 (refer to sk170012). to set affinity use (3-4) Best Practices - CoreXL.
Pairs of interfaces carrying significant data flows (based on network topology) should be assigned to pairs of CPU cores on the same physical CPU processor.
Pairs of interfaces that serve the same connections (based on network topology) should be assigned to pairs of CPU cores on the same physical CPU core.
For systems with 4 CPU cores and Dual Port NICs, the IRQ Swizzling technology should be enabled to properly distribute IRQs among 4 CPU cores. Note: Applies only to Dual Port NICs. IRQ Swizzling is not required with Quad Port NICs.
For systems with 8 CPU cores and Quad Port NICs, the EIRQ technology should be enabled to properly distribute IRQs among all 8 CPU cores. Note: At the time of this writing, there is no certified platform with this ability.
Either decrease the TCP end timeout to 2 seconds (SmartDashboard - 'Policy' menu - 'Global Properties' - 'Stateful Inspection').
Or increase the hash size of SND's Connection Table by increasing (permanently per sk26202) the value of the kernel parameter fwmultik_gconn_tab_hsize from the default 524288 to 8388608 and rebooting the machine. Important Note: This change reduces the capacity for the maximum number of concurrent connections.
Accelerated Drop Rules:
Available in R75.40 and above.
Accelerated Drop Rules protect the Security Gateway and site from Denial of Service attacks by dropping packets at the acceleration layer (SecureXL).
The drop rules are configured in a file on the Security Gateway (using the 'sim dropcfg <options>' command), which is then offloaded to the SecureXL device for enforcement. Note: There is no relation between the rules being configured in the file and the rules configured in SmartDashboard.
Once the feature is enabled, dropped traffic is accelerated depending on traffic drop rate per second. Once drop rate matches the thresholds, feature is dynamically activated/deactivated.
To decrease the load on CPU in cluster environment:
Either disable the synchronization of non-critical connections (e.g., UDP DNS, ICMP).
Or (if connection must be synchronized) start synchronizing the connection only some time after its initiation (right-click on the service - 'Edit...' - 'Advanced...' - check the box 'Start synchronizing [X] seconds after connection initiation' - install policy).
Notes:
Applies only to TCP services whose 'Protocol Type' is set to 'HTTP' or 'None'.
Delayed Synchronization is disabled if the 'Track' option in the rule is set to 'Log' or 'Account'.
Delayed Synchronization is performed only for connections matching a SecureXL Connection Template.
Note: This section does not take the SecureXL into consideration.
Notes:
CoreXL improves performance with almost linear scalability in the following scenarios:
Much of the traffic can not be accelerated by SecureXL
Many IPS features enabled, which disable SecureXL functionality
Large rulebase
NAT rules
CoreXL will not improve performance in the following scenarios:
SecureXL accelerates much of the traffic
Traffic mostly consists of VPN
Traffic mostly consists of VoIP
It is very important to verify that the CPU cores are equally utilized (run the 'top' command). If this is not the case, you should consider changing the distribution of the Secure Network Distributors (SNDs) and CoreXL FW instances.
Under normal circumstances, it is not recommended for the SND and a CoreXL FW instance to share the same CPU core. However, it is necessary in the following cases:
When using a machine with exactly two cores. It is better for both SNDs and CoreXL FW instances to share CPU cores, instead of allocating only one CPU core to each.
When you know that almost all of the packets are being processed in the Accelerated path, and you want to assign all CPU cores to this path. If the CoreXL FW instances do not process significant amount of traffic, then it is appropriate to share the CPU cores.
The interfaces affinity should be configured only to to CPU cores that are running as SNDs (CPU cores that are not running CoreXL FW instances), with these exceptions:
On machines with exactly two CPU cores (both SNDs and CoreXL FW instances use the same CPU cores).
For tests, in which traffic is accelerated by SecureXL (if it is enabled).
If you replaced the CPU on the machine to a CPU with more/less CPU cores than the previous CPU, then you must reconfigure the number of CoreXL FW instances in the 'cpconfig' menu.
In a cluster environment, changing the number of CoreXL FW instances should be treated as a version upgrade - member with higher number of CoreXL FW instances will enter the 'Ready' state. Refer to sk42096 - Cluster member is stuck in 'Ready' state. To change the number of CoreXL FW instances, schedule a maintenance window and follow either a Minimal Effort Upgrade procedure, or a Zero Downtime Upgrade procedure from the Installation and Upgrade Guide (R75, R75.20, R75.40, R75.40VS, R76, R77 Gaia, R77 Non-Gaia).
Changing the distribution of the SND, CoreXL FW instances, and daemons among the CPU cores:
Note: Run the 'fw ctl affinity -l -r -a -v' command to see the current distribution.
To change the distribution of the SND, CoreXL FW instances, and daemons, change the current affinities of interfaces and/or of daemons.
Notes:
To make the affinity settings persistent, edit the '$FWDIR/conf/fwaffinity.conf' file on Security Gateway (see the contents of the file for the correct syntax).
To apply the configuration from the '$FWDIR/conf/fwaffinity.conf' file on-the-fly (without reboot), execute the '$FWDIR/scripts/fwaffinity_apply' shell script on Security Gateway (see the contents of the script for available flags).
To ensure CoreXL's efficiency, all traffic must be directed to CPU cores that are running as SNDs (CPU cores that are not running CoreXL FW instances). Therefore, if you change affinities of interfaces and/or daemons, you will need to accordingly set the number of CoreXL FW instances and ensure that the CoreXL FW instances run on other CPU cores.
It is recommended to allocate an additional CPU core to the SND only if all of the following conditions are met:
Your platform has at least 8 CPU cores.
The 'idle' value (run 'top' command and press 1 to display all CPU cores) for the CPU core currently running the SND is in the 0%-5% range.
The sum of the 'idle' values (run the 'top' command and press 1 to display all CPU cores) for the CPU cores running CoreXL FW instances is significantly higher than 100%.
If any of the above conditions are not met, the default configuration of one processing core allocated to the SND is sufficient, and no further configuration is necessary.
It is recommended to allocate an additional CPU core to the FWD daemon if the Security Gateway is performing heavy logging. Note: Avoiding the CPU cores that are running the SND is important only if these CPU cores are explicitly defined as affinities of interfaces. If interface affinities are set to 'Automatic', any CPU core that is not running a CoreXL FW instance can be used for the FWD daemon, and interface traffic will be automatically diverted to other CPU cores.
Example of configuration for machine with 8 CPU cores:
When most of the traffic is accelerated by SecureXL (run 'fwaccel stats -s' command), the load on CPU cores that run as Secure Network Distributor (SND) can be very high, while the load on CPU cores that run CoreXL FW instances can be very low. This is an inefficient utilization of CPU capacity.
Notes:
Traffic is processed by the CoreXL FW instances only when the traffic is not accelerated by SecureXL (if SecureXL is installed and enabled).
With CoreXL, there are cases when performance without SecureXL is better than with it, even when SecureXL does manage to accelerate part of the traffic.
Packet flow:
When SecureXL is enabled, a packet enters the Security Gateway and first reaches the SecureXL device. The packet will be handled in one of three ways:
Accelerated path - The packet is completely handled by the SecureXL device. It is processed and forwarded to the network.
Medium path (PXL) - Packet flow when the packet is handled by the SecureXL device, except for IPS (some protections) / VPN (in some configurations) / Application Control / Content Awareness / Anti-Virus / Anti-Bot / HTTPS Inspection / Proxy mode / Mobile Access / VoIP / Web Portals. The CoreXL layer passes the packet to one of the CoreXL FW instances to perform the processing. This path is available only when CoreXL is enabled.
Firewall path / Slow path - The SecureXL device is unable to process the packet (refer to sk32578 - SecureXL Mechanism). The packet is passed on to the CoreXL layer and then to one of the CoreXL FW instances for full processing. This path also processes all packets when SecureXL is disabled.
Default affinity settings for interfaces:
If SecureXL is enabled - the default affinities of all interfaces are 'Automatic' - the affinity for each interface is automatically reset every 60 seconds, and balanced between available CPU cores based on the current load.
If SecureXL is disabled - the default affinities of all interfaces are with available CPU cores - those CPU cores that are not running a CoreXL FW instance or not defined as the affinity for a daemon.
Setting interface affinities:
If SecureXL is enabled - the affinities of all interfaces are handled by the 'sim affinity' command. Interface affinities are automatically distributed among CPU cores that are not running CoreXL FW instances and that are not set as the affinity for any daemon.
Note: 'sim affinity' has been deprecated in R80.40 (refer to sk170012). to set affinity use (3-4) Best Practices - CoreXL.
If SecureXL is disabled - interface affinities are loaded at boot based on the $FWDIR/conf/fwaffinity.conf configuration file (if SecureXL is enabled, then lines beginning with "i" in this file are ignored).
To balance the load on the CPU cores that are running as SNDs (CPU cores that are not running CoreXL FW instances):
Check the amount of interrupts from each interface - run:
[Expert@HostName]# cat /proc/interrupts
Check, which CPU cores (that run as SNDs) are most utilized - run:
[Expert@HostName]# top
Note: Press digit 1 (above the letter Q) to display all CPU cores and press Shift+W to save this configuration.
Assign the interfaces accordingly across the CPU cores that run as SNDs - run:
SMT is not recommended if only FireWall/VPN blades are used, because performance improvement by SMT is achieved on NGFW software blades.
SMT is not recommended for environments that have high memory utilization.
Reason: Firewall consumes memory for traffic inspection. If the memory utilization is already very high before enabling the SMT, the performance will decrease noticeably because SMT adds more CoreXL FW instances.
To check the extent of memory utilization on the Security Gateway, refer to:
Initial diagnostics - Memory
Advanced diagnostics - Memory
SMT is not recommended if these blades/features are enabled:
Data Loss Prevention blade
Anti-Virus in Traditional Mode
Using Services with Resources in Firewall policy
Reason: Each of these blades might have high memory consumption. These blades run Security Servers that are executed per CoreXL FW instance. Since SMT adds more CoreXL FW instances, overall memory consumption on Security Gateway might increase considerably.
To check the extent of memory utilization on the Security Gateway, refer to:
Initial diagnostics - Memory
Advanced diagnostics - Memory
SMT is not recommended for environments that use Hide NAT extensively.
Reason: An entire range of ports for Hide NAT is divided between the current CoreXL FW instances. The more CoreXL FW instances are running, the less ports for Hide NAT will be available for each CoreXL FW instance. As a result, if one CoreXL FW instance is handling a high number of NATed connections, its port range may get exhausted, while at the same time, other CoreXL FW instances may have enough available ports for Hide NAT.
The port distribution is based on the following factors:
Number of CoreXL FW instances
Whether cluster is enabled
Whether SecureXL is enabled
Whether VPN blade is enabled
To check the extent of NAT on the Security Gateway, use the cpsizeme tool (refer to sk88160):
Run the tool for at least 24 hours.
Run the 'cpsizeme -S' command.
Select "Show summary of last successful session" (to see the summary of the collected statistics).
Check the "Estimated average of NAT connections:" counter.
If you have two ports on 12600/12700/12800, 13500/13800 and 21000 appliances to handle most of the traffic, it is also recommended to enable Multi-Queue feature to increase SMT performance (refer to Performance Tuning Administration Guide (R76, R77) - Chapter 3 'Multi-queue').
When most of the processing is done in CoreXL - either in the Medium path, or in the Firewall path (Slow path).
All current CoreXL FW instances are highly loaded, so there are no CPU cores that can be reassigned to SecureXL.
When IPS, or other deep inspection Software Blades are heavily used.
When all network interface cards are processing the same amount of traffic.
When all CPU cores that are currently used by SecureXL are congested.
When trying to increase traffic session rate.
When there is not enough diversity of traffic flows. In the extreme case of a single flow, for example, traffic will be handled only by a single CPU core. (Clarification: The more traffic is passing to/from different ports/IP addresses, the more you benefit from Multi-Queue. If there is a single traffic flow from a single Client to a single Server, then Multi-Queue will not help.)
Guidelines for configuring Multi-Queue:
Network interface cards must support Multi-Queue.
Multi-Queue is relevant only if SecureXL is enabled.
Multi-Queue is relevant only if CoreXL is enabled.
Examine the CPU affinity (for interfaces (SND) and for CoreXL FW instances).
Examine the CPU utilization.
Decide if more CPU cores can be allocated to the Secure Network Distributor (SND).
Multi-Queue is recommended when all these conditions occur:
Load on CPU cores that run as SND is high (idle < 20%).
Load on CPU cores that run CoreXL FW instances is low (idle > 50%).
There are no CPU cores left to be assigned to the SND by changing interface affinity.
Number of active RX queues:
By default on a Security Gateway, the number of active RX queues is calculated by: Number of Active RX queues = [Total Number of CPU cores] - [Number of CoreXL FW instances]
By default on a VSX gateway, the number of active RX queues is calculated by: Number of Active RX queues = The lowest CPU ID, to which the fwk process on VSX is assigned
Note: the new column-based matching of Gateways of version R80.10 and above eliminates this need.
In addition, you can provide a full Connections Table from your Security Gateway to Check Point Support for thorough analysis (run the command 'fw tab -t connections -u > /var/log/Connections_Table.txt').
Disable protections that are not needed in your environment. To check, which protections are needed, you may use vulnerability tools (such as Nessus).
Set Protection Scope to "Protect internal hosts only" (SmartDashboard - Security Gateway properties - IPS pane). Note: This step is not needed with the improved IPS mechanism for Gateways of version R80.10 and above.
Create as specific rules as possible, to prevent the unwanted traffic from hitting the wrong rule.
Limit the scope of the rule by selecting only the relevant services (instead of the default 'Any') - right-click on the column titles - click on the 'Service' column:
Avoid using "Any" in "Source" and "Destination" columns.
In the "Destination" column, use the default "Internet", unless a specific destination has to be explicitly selected (e.g., in case of internal servers).
There is no need for "Clean Up" rule at the bottom (because the behavior of Application Control & URL Filtering Blades is different from that of the Firewall Blade). Such rule should only be used for short period of time and only for logging purposes.
Remove the "Any - Any - Allow" rule, if such rule exists (because the behavior of Application Control & URL Filtering Blades is different from that of the Firewall Blade). Such rule should only be used for short period of time and only for logging purposes.
Advanced - Engine Settings:
Go to "Check Point Online Web Service" section.
In the "Website categorization mode" section, select "Background" (to prevent latency due to packet holding until categorization is completed).
Create as specific rules as possible, to prevent the unwanted traffic from hitting the wrong rule.
Avoid using "Any" in "Source" and "Destination" columns.
There is no need for "Clean Up" rule at the bottom (because the behavior of Anti-Virus & Anti-Bot Blades is different from that of the Firewall Blade). Such rule should only be used for short period of time and only for logging purposes.
Advanced - Engine Settings:
Go to "Check Point Online Web Service" section.
In the "Website categorization mode" section, select "Background" (to prevent latency due to packet holding until categorization is completed).
Decreasing performance impact:
Edit the Anti-Virus & Anti-Bot profile:
Go to 'Profiles' pane.
Select the relevant profile - click on 'Edit...' button.
Go to 'General Properties' pane.
In the 'Performance Impact' field, select 'Low'.
Click on 'OK'.
Save the changes: go to 'File' menu - click on 'Save'.
Install the policy onto the relevant Security Gateway / Cluster object.
Displays dynamic real-time view of a running system on Linux
Usage:
When running the command for the first time:
Press '1' to display all CPU cores
Press 'd' to set update interval - type 2 and press Enter
Press 'n' to set maximum tasks displayed - type 15 and press Enter
Press Shift+W to save the top's configuration
When running the command for diagnostics:
Press Shift+P to sort the output by CPU utilization
Press Shift+M to sort the output by Memory utilization
Diagnostics:
Collect this output continuously during the problem
Analysis:
Look at the load in "User Space" - counter us High CPU consumption in "User Space" can be caused by processes that perform heavy tasks (e.g., too much logging by fwd, reloading the configuration during policy installation, etc.)
Look at the load in "System (kernel) Space" - counter sy High CPU consumption in "System (kernel) Space" can be caused by heavy tasks (e.g., deep inspection of packets, enabling of all blades, enabling of all IPS protections in Prevent mode, etc.)
Look at the amount of "Idle" - counter id The more CPU is idle, the better the machine's performance is
Look at the amount of "I/O waiting" - counter wa High amount of "I/O waiting" is caused by heavy reading from/writing to hard disk (e.g., during policy installation, heavy logging, insufficient RAM, etc.)
Look at the amount of "SoftIRQ" - counter si High amount of "SoftIRQ" is usually caused by high amount of traffic
Look at the amount of "Hardware IRQs" - counter hi
Displays information about the current processes (daemons)
Usage:
Refer to the manual page
By default, output is sorted by PID column
Simplest sorting can be done by pipelining the output of ps command to sort command - for example, to sort the CPU usage, run: [Expert@HostName]# ps auxww | sort -nrk 3
Diagnostics:
Collect this output continuously during the problem
Analysis:
Look at the amount of "CPU", "MEM", "VSZ", "RSS", "TIME" consumed by the daemons
Constant high CPU consumption can be caused by numerous factors - function stack should be collected from the process using a special Check Point shell script ('pstack') - refer to "(5-1) Advanced diagnostics - CPU" section
Displays the number of interrupts on each CPU core from each IRQ
Usage:
Refer to the manual page
By default, output is sorted by IRQ number
The only relevant devices are network interfaces
To shorten the output, use the 'grep' command - for example, run: [Expert@HostName]# cat /proc/interrupts | grep -E "CPU|eth"
To collect this output continuously, use the 'watch' command - for example, run: [Expert@HostName]# watch -d -n 1 'cat /proc/interrupts | grep -E "CPU|eth"'
Diagnostics:
Collect this output continuously during the problem
Analysis:
Look at the general trend - which CPU receives more interrupts and from which interfaces
If some CPU cores receive more interrupts than others, then affinity of interfaces to CPU cores should be optimized - interface should be redistributed better
Example from machine with 4 CPU cores and 4 interfaces:
Total Virtual Memory (Bytes): 6380535808
Active Virtual Memory (Bytes): 1882390528
Total Real Memory (Bytes): 2054049792
Active Real Memory (Bytes): 1643986944
Free Real Memory (Bytes): 410062848
Memory Swaps/Sec: -
Memory To Disk Transfers/Sec: -
Collect this output continuously during the problem (usually, over some time period)
Analysis:
RX-OK - number of packets that have been received error-free
RX-ERR - number of packets that have been received damaged (refer to sk61922)
RX-DRP - number of received packets that have been dropped (refer to sk61922)
RX-OVR - number of received packets that have been lost because of an overrun (number of times the receiver hardware was unable to hand received data to a hardware buffer - the internal FIFO buffer of the chip is full, but is still tries to handle incoming traffic ; most likely, the input rate of traffic exceeded the ability of the receiver to handle the data)
TX-OK - number of packets that have been transmitted error-free
TX-ERR - number of packets that have been transmitted damaged
TX-DRP - number of transmitted packets that have been dropped
TX-OVR - number of transmitted packets that have been lost because of an overrun
Flg - shows the flags that have been set for this interface - these characters are one-character versions of the long flag names that are displayed in the output of 'ifconfig' command:
A = this interface will receive all Multicast addresses
Displays the status and traffic statistics for the currently active interfaces
Diagnostics:
Collect this output continuously during the problem (usually, over some time period)
Analysis:
UP - indicates that the kernel modules related to the Ethernet interface has been loaded
BROADCAST - indicates that interface supports broadcasting - a necessary characteristic to obtain IP address via DHCP
NOTRAILERS - indicates that trailer encapsulation is disabled (Linux usually ignore trailer encapsulation so this value has no effect at all)
RUNNING - indicates that interface is ready to accept data
MULTICAST - indicates that the interface supports multicasting
MTU - Maximum Transmission Unit is the size of each packet received by the interface (default value is 1500; setting MTU to a higher value could hazard packet fragmentation or buffer overflows)
Metric - used to compute the cost of a route - it tells the OS, to which interface a packet should be forwarded, when multiple interfaces could be used to reach the packet's destinationTakes values of 0,1,2,3,... The lower the value, the more leverage it has. This parameter has significance only while routing packets. For example, if you have two Ethernet cards and you want to forcibly make your machine use one card over the other in sending the data. Then you can set the Metric value of the Ethernet card which you favor lower than that of the other Ethernet card.
RX packets - total number of packets received via the interface
RX errors - number of damaged packets received (refer to sk61922)
RX dropped - number of dropped packets due to reception errors (refer to sk61922)
RX overruns - number of received packets that experienced data overruns (number of times the receiver hardware was unable to hand received data to a hardware buffer)
RX frame - number received packets that experienced frame errors
TX packets - total number of packets transmitted via the interface
TX errors - number of packets that experienced transmission error
TX dropped - number of dropped transmitted packets due to transmission errors
TX overruns - number of transmitted packets that experienced data overruns (number of times the transmitter hardware was unable to hand received data to a hardware buffer)
TX carriers - number received packets that experienced loss of carriers
TX collisions - number of transmitted packets that experienced Ethernet collisions (a nonzero value of this field indicates possibility of network congestion)
TX txqueuelen - configured length of transmission queue
RX bytes - total bytes received over this interface
TX bytes - total total bytes transmitted over this interface
Displays both listening and non-listening (for TCP this means established connections) sockets
Diagnostics:
Collect this output continuously during the problem (usually, over some time period)
Analysis:
Under "Active Internet connections" look at "Recv-Q" and at "Send-Q"
Recv-Q - data (in bytes), which has not yet been pulled from the socket buffer by the application (value should be as close to 0 as possible)
Send-Q - data (in bytes), which the sending application has given to the transport, but has yet to be ACKnowledged by the receiving TCP (value should be as close to 0 as possible - a large number may indicate a network bottleneck)
Example:
[Expert@FW]# netstat -anp
Active Internet connections (servers and established)
Proto Recv-QSend-Q Local Address Foreign Address State PID/Program name
tcp 0 2368 172.30.20.69:22 172.30.20.228:3676 ESTABLISHED 16179/0
udp 256956 0 0.0.0.0:9282 0.0.0.0:*
[Expert@FW]#
"Cryptography Features" section appears only if SecureXL device supports cryptography, or if a VPN Accelerator card is installed on the machine.
Example:
Accelerator Status : on
Accept Templates : disabled by Firewall
disabled from rule #31
Drop Templates : disabled
NAT Templates : disabled by user
Accelerator Features : Accounting, NAT, Cryptography, Routing,
HasClock, Templates, Synchronous, IdleDetection,
Sequencing, TcpStateDetect, AutoExpire,
DelayedNotif, TcpStateDetectV2, CPLS, WireMode,
DropTemplates, NatTemplates, Streaming,
MultiFW, AntiSpoofing, ViolationStats,
Nac, AsychronicNotif, ERDOS
Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
3DES, DES, CAST, CAST-40, AES-128, AES-256,
ESP, LinkSelection, DynamicVPN, NatTraversal,
EncRouting, AES-XCBC, SHA256
fwaccel stats -s
Background:
Displays summary of SecureXL acceleration statistics
Diagnostics:
Collect this output continuously during the problem
Analysis:
If the ratio of "Accelerated/Total" is less than 80%, it means that acceleration is poor, and the optimization is required
The higher the ratio of "F2Fed pkts/Total pkts" is, the more traffic is Forwarded to Firewall for inspection (the poorer the performance is) - see the fwaccel conns command below
The higher the ratio of "PXL pkts/Total pkts" is, the more packets passed in Medium path. Meaning, packets were handled by the SecureXL device, except for IPS / Application Control / Anti-Virus / Anti-Bot processing. The CoreXL layer passed the packets to one of the CoreXL FW instances to perform the processing. This path is available only when CoreXL is enabled.
The higher the ratio of "QXL pkts/Total pkts" is, the more packets were processed by QoS.
Displays SecureXL Connection Templates Note: Size of Templates table (cphwd_tmpl, id 8111) is limited to 1/4 of the size of Firewall Connections Table (connections, id 8158).
Diagnostics:
Collect this output continuously during the problem (Important Note: This command consumes high amount of memory)
Analysis:
Check whether templates are created: [Expert@HostName]# fwaccel templates -s
Check whether templates were created for relevant connections (search for relevant IP addresses using the grep command)
Look at the Flags column:
You should not see connections with F, N, or C flags
F = Forward to Firewall - the connection is not accelerated
U = Unidirectional - the connection can pass data on either C2S or S2C - data packets from the opposite direction will be F2F'ed
N = NAT is being performed on the connection by the device
A = Accounting is performed on the connection (the connection is viewed by either rulebase accounting or SmartView Monitor)
C = Encryption is done on the connection by the device
W = the connection is in wire mode
P = Partial (versions R70 and higher)
S = Streaming - PXL (versions R70 and higher)
D = Drop Template
L = Log drop action
The LCT column - shows how many seconds ago a connection was created by the template
The DLY column - shows Delayed Synchronization value for the template
Displays the CoreXL status and statistics (and many other counters). Refer to sk101878 - CPView Utility.
Diagnostics:
Monitor the CoreXL performance during the problem
Analysis:
On the 'SysInfo' tab, refer to 'Configuration Information:' section - look at 'CoreXL Status'
On the 'SysInfo' tab, refer to 'Configuration Information:' section - look at 'CoreXL instances'
On the 'I/S' tab, go to 'CoreXL' menu - go to 'General' menu - refer to 'Queues:' section - look at counter 'Enqueue fail'
On the 'I/S' tab, go to 'CoreXL' menu - go to 'Instances' menu - activate the statistics (!may affect performance) - go to each CoreXL FW instance ('FW-Instance<N>'):
refer to 'FW Stats' section
refer to 'Top FW-Lock consumers:' section
fw ctl multik stat
Background:
Displays status of CoreXL instances and summary for traffic that passes through each CoreXL FW instance (current number and peak number of concurrent connections)
Diagnostics:
Collect this output to see the current status
Analysis:
Check the number and the allocation of FW instances to CPU cores (in cluster, the output must be identical on all members)
Check the peak number connections on FW instances (instances should be loaded as equally as possible)
Example from machine with 4 CPU cores (1 SND + 3 CoreXL FW instances):
Displays the number of interrupts on each CPU core from each IRQ
Usage:
Refer to the manual page
By default, output is sorted by IRQ number
The only relevant devices are network interfaces
To shorten the output, use the 'grep' command - for example, run: [Expert@HostName]# cat /proc/interrupts | grep -E "CPU|eth"
To collect this output continuously, use the 'watch' command - for example, run: [Expert@HostName]# watch -d -n 1 'cat /proc/interrupts | grep -E "CPU|eth"'
Diagnostics:
Collect this output continuously during the problem
Analysis:
Look at the general trend - which CPU receives more interrupts and from which interfaces
If some CPU cores receive more interrupts than others, then affinity of interfaces to CPU cores should be optimized - interface should be redistributed better
Example from machine with 8 CPU cores and 5 interfaces:
Displays information about processes, memory, paging, block IO, and CPU activity
Usage:
Refer to the manual page
By default, the output's header (with names of columns) is displayed every 20 samples. If output is redirected to a file, then it will be very difficult to analyze it using such commands as 'grep' / 'awk'. Therefore, if output is redirected to a file, use the '-n' flag to display the header only once (at the very top) - run: [Expert@HostName]# vmstat -n [delay_in_sec [number_of_samples]]
By default, only 1 sample is printed (with the average values since boot). Therefore, to collect this output continuously, specify the delay between the samples - run: [Expert@HostName]# vmstat [-n] delay_in_sec [number_of_samples]
If you specified the delay between the samples, then the output will be collected indefinitely (until the command is stopped or killed). If you want to limit the output, then specify the total number of samples - run: [Expert@HostName]# vmstat [-n] delay_in_secnumber_of_samples
Diagnostics:
Collect this output continuously during the problem
Analysis:
Look at the "procs" section - number of processes waiting for CPU (counter r)
Look at the "swap" section - reading from swap file (si) and writing to swap file (so)
Look at the "io" section - reading from hard disk (bi) and writing to hard disk (bo)
Look at the "system" section - number of Context Switches (cs)
Look at the "cpu" section - at all counters:
"User Space" - counter us
"System (kernel) Space" - counter sy
"Idle" - counter id
"I/O waiting" - counter wa
Example:
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 323280 100128 103192 198132 1 2 8 35 117 130 2 3 96 0 0
2 0 323280 100400 103196 198132 0 0 0 260 1297 6904 1 4 95 0 0
1 0 323280 100404 103196 198132 0 0 0 0 1289 7100 1 4 95 0 0
7 0 323280 100252 103200 198132 0 0 0 32 1308 6971 0 2 97 0 0
1 0 323280 100428 103200 198132 0 0 0 0 1265 7111 1 3 97 0 0
sar [-u] [-P { <cpu> | ALL }] [interval_in_sec [number_of_samples]]
Look at the load in "User Space" - counter user High CPU consumption in "User Space" can be caused by processes that perform heavy tasks (e.g., too much logging by fwd, reloading the configuration during policy installation, etc.)
Look at the load in "System (kernel) Space" - counter system High CPU consumption in "System (kernel) Space" can be caused by heavy tasks (e.g., deep inspection of packets, enabling of all blades, enabling of all IPS protections in Prevent mode, etc.)
Look at the amount of "Idle" - counter idle The more CPU is idle, the better the machine's performance is
Look at the amount of "I/O waiting" - counter iowait High amount of "I/O waiting" is caused by heavy reading from/writing to hard disk (e.g., during policy installation, heavy logging, insufficient RAM, etc.)
Look at the counter steal - Percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.
Displays information about processes, memory, paging, block IO, and CPU activity
Usage:
Refer to the manual page
By default, the output's header (with names of columns) is displayed every 20 samples. If output is redirected to a file, then it will be very difficult to analyze it using such commands as 'grep' / 'awk'. Therefore, if output is redirected to a file, use the '-n' flag to display the header only once (at the very top) - run: [Expert@HostName]# vmstat -n [delay_in_sec [number_of_samples]]
By default, only 1 sample is printed (with the average values since boot). Therefore, to collect this output continuously, specify the delay between the samples - run: [Expert@HostName]# vmstat [-n] delay_in_sec [number_of_samples]
If you specified the delay between the samples, then the output will be collected indefinitely (until the command is stopped or killed). If you want to limit the output, then specify the total number of samples - run: [Expert@HostName]# vmstat [-n] delay_in_secnumber_of_samples
Diagnostics:
Collect this output continuously during the problem
Analysis:
Look at the "procs" section - number of processes waiting for CPU (r)
Look at the "memory" section - sum of these counters should be compared to the total amount of RAM
Look at the "swap" section - reading from swap file (si) and writing to swap file (so)
Example:
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 323280 100128 103192 198132 1 2 8 35 117 130 2 3 96 0 0
2 0 323280 100400 103196 198132 0 0 0 260 1297 6904 1 4 95 0 0
1 0 323280 100404 103196 198132 0 0 0 0 1289 7100 1 4 95 0 0
7 0 323280 100252 103200 198132 0 0 0 32 1308 6971 0 2 97 0 0
1 0 323280 100428 103200 198132 0 0 0 0 1265 7111 1 3 97 0 0
sar [-q][-r][-B][-W] [interval_in_sec [number_of_samples]]
Displays kernel slab cache information in real time (caches of frequently used objects in the Linux kernel - buffer heads, inodes, dentries, etc.)
Instead of manually parsing the highly verbose /proc/slabinfo file manually, it is recommended to use the /usr/bin/slabtop program.
Usage:
Refer to the manual page
By default, output is refreshed every 3 seconds, unless you use the "-d refresh_in_sec" option
By default, output is sorted by the number of objects (column "OBJS", which is equal to using the "-so" option).
Recommended output sort is by the cache size of the slab (column "CACHE SIZE", use the "-sc" option).
Diagnostics:
Collect this output continuously during the problem
Analysis:
Provide the output(s) to Check Point Support (involvement of R&D is required)
The most interesting information is the amount of resources a certain kernel module is using (if this amount seems too high, then there may be something wrong with this module)
Some of the more commonly used statistics from /proc/slabinfo file that are included in the output of /usr/bin/slabtop:
OBJS - The total number of objects (memory blocks), including those in use (allocated), and some spares not in use
ACTIVE - The number of objects (memory blocks) that are in use (allocated)
USE - Percentage of total objects that are active [(ACTIVE/OBJS)*100%]
OBJ SIZE - The size of the objects
SLABS - The total number of slabs
OBJ/SLAB - The number of objects that fit into a slab
CACHE SIZE - The cache size of the slab
NAME - The name of the slab
Example of 'slabtop' output (excerpt):
Active / Total Objects (% used) : 160565 / 174730 (91.9%)
Active / Total Slabs (% used) : 5005 / 5005 (100.0%)
Active / Total Caches (% used) : 94 / 141 (66.7%)
Active / Total Size (% used) : 19571.55K / 20754.12K (94.3%)
Minimum / Average / Maximum Object : 0.01K / 0.12K / 128.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
85536 82456 96% 0.05K 1188 72 4752K buffer_head
12876 10385 80% 0.13K 444 29 1776K dentry_cache
9516 9498 99% 0.05K 122 78 488K selinux_inode_security
7571 6711 88% 0.03K 67 113 268K size-32
7524 6869 91% 0.09K 171 44 684K vm_area_struct
7056 7056 100% 0.48K 882 8 3528K ext3_inode_cache
5936 5703 96% 0.27K 424 14 1696K radix_tree_node
4524 4453 98% 0.05K 58 78 232K sysfs_dir_cache
Displays SecureXL configuration and basic statistics
Diagnostics:
Collect this output to see the current status
Analysis:
For Flags line, refer to "(6-1-B) 'sim' command" section - 'sim if' command
For TmplQuota lines, refer to "(6-1-B) 'sim' command" section - 'sim tmplquota' command
For Debug flags section, refer to "(6-1-B) 'sim' command" section - 'sim dbg' command (Note: the numbers in the left column correspond to order of debug modules in the output of 'sim dbg list' command)
Example:
Flags : 0x00009a16
Accounting Update Interval : 60
Conn Refresh Interval : 512
SA Sync Notification Interval : 0
UDP Encapsulation Port : 0
Min TCP MSS : 0
TCP Auto-Expire Timeout : 20
Connection Limit : 18446744073709551615
TmplQuota Enabled : 0
TmplQuota Quota (rate) : 512
TmplQuota Drop Dduration : 300
TmplQuota Monitor only : 0
TmplQuota Dropped pkts : 0
Total Number of conns : 54
Number of F2F conns : 54
Number of Crypt conns : 0
Number of TCP conns : 18
Number of Non-TCP conns : 36
Number of Delayed TCP conns : 0
Number of Delayed Non-TCP conns: 0
Debug flags :
0 : 0x0
1 : 0x0
2 : 0x0
3 : 0x0
4 : 0x0
5 : 0x0
6 : 0x0
7 : 0x0
8 : 0x0
9 : 0x0
10 : 0x0
11 : 0x0
12 : 0x0
13 : 0x0
14 : 0x0
15 : 0x0
cat /proc/ppk/ifs
Background:
Displays the list of interfaces used and seen by the SecureXL
Diagnostics:
Collect this output to see the current status
Analysis:
Look at the Name column and Address column
Look at the F column (refer to "(6-1-B) 'sim' command" section - 'sim if' command)
Displays the status and the thresholds for New Affinity (feature will be activated only if there is no massive VPN traffic and the packets-per-second rate (cut-through) is high enough to benefit from the new affinity; feature will be activated only if CPU strength is greater than 3 GHz)
Diagnostics:
Collect this output to see the current status
Example:
New affinity enabled : no
Min CPU speed to act (MHz) : 3000
Min accelerated PPS to act : 250000
Max enc. bytes rate to act : 10000
Current accelerated PPS : 0
Current enc. bytes rate : 0
Currently new sim affinity set : no
cat /proc/ppk/cpls
Background:
Displays the SecureXL configuration for ClusterXL Load Sharing support
Diagnostics:
Collect this output to see the current status
Analysis:
The most important data in this output is the value of fwha_conf_flags - it should include the value of 0x1 (i.e., ACTIVE).
Displays SecureXL statistics - the same output as from 'fwaccel stats -l' command
Diagnostics:
Collect this output to see the current status
Example:
Name Value Name Value
-------------------- --------------- -------------------- ---------------
conns created 67518 conns deleted 67475
temporary conns 0 templates 0
nat conns 0 accel packets 0
accel bytes 0 F2F packets 16599213
ESP enc pkts 0 ESP enc err 0
ESP dec pkts 0 ESP dec err 0
ESP other err 0 espudp enc pkts 0
espudp enc err 0 espudp dec pkts 0
espudp dec err 0 espudp other err 0
AH enc pkts 0 AH enc err 0
AH dec pkts 0 AH dec err 0
AH other err 0 memory used 0
free memory 0 acct update interval 60
current total conns 43 TCP violations 0
conns from templates 0 TCP conns 15
delayed TCP conns 0 non TCP conns 28
delayed nonTCP conns 0 F2F conns 43
F2F bytes 1076600709 crypt conns 0
enc bytes 0 dec bytes 0
partial conns 0 anticipated conns 0
dropped packets 0 dropped bytes 0
nat templates 0 port alloc templates 0
conns from nat templ 0 port alloc conns (tm 0
port alloc f2f 0 PXL templates 0
PXL conns 0 PXL packets 0
PXL bytes 0 PXL async packets 0
conns auto expired 0 C used templates 0
pxl tmpl conns 0 C conns from tmpl 0
C non TCP F2F conns 28 C tcp handshake conn 14
C tcp established co 1 C tcp closed conns 0
C tcp f2f handshake 14 C tcp f2f establishe 1
C tcp f2f closed con 0 C tcp pxl handshake 0
C tcp pxl establishe 0 C tcp pxl closed con 0
QXL templates 0 QXL conns 0
QXL packets 0 QXL bytes 0
QXL async packets 0 outbound packets 0
outbound pxl packets 0 outbound f2f packets 124862
outbound bytes 0 outbound pxl bytes 0
outbound f2f bytes 33552839 trimmed pkts 0
cat /proc/ppk/drop_statistics
Background:
Displays SecureXL drop statistics
Diagnostics:
Collect this output to see the current status
Example:
Reason Packets Reason Packets
-------------------- --------------- -------------------- ---------------
general reason 0 PXL decision 0
fragment error 0 hl - spoof viol 0
F2F not allowed 0 hl - TCP viol 0
corrupted packet 0 hl - new conn 0
clr pkt on vpn 0 partial conn 0
encrypt failed 0 drop template 0
decrypt failed 0 outb - no conn 0
interface down 0 cluster error 0
XMT error 0 template quota 0
anti spoofing 0 Attack mitigation 0
local spoofing 0 sanity error 0
monitored spoofed 0 QXL decision 0
cat /proc/ppk/viol_statistics
Background:
Displays violations statistics
Diagnostics:
Collect this output to see the current status
Example:
Violation Packets Violation Packets
-------------------- --------------- -------------------- ---------------
pkt is a fragment 0 pkt has IP options 406
ICMP miss conn 932 TCP-SYN miss conn 169
TCP-other miss conn 17 UDP miss conn 10305933
other miss conn 0 VPN returned F2F 0
ICMP conn is F2Fed 7 TCP conn is F2Fed 23665
UDP conn is F2Fed 6275273 other conn is F2Fed 0
uni-directional viol 0 possible spoof viol 0
TCP state viol 0 out if not def/accl 0
bridge, src=dst 0 routing decision err 0
sanity checks failed 0 temp conn expired 0
fwd to non-pivot 0 broadcast/multicast 0
cluster message 0 partial conn 0
PXL returned F2F 0 cluster forward 0
chain forwarding 0 general reason 0
cat /proc/ppk/mcast_statistics
Background:
Displays SecureXL multicast statistics
Diagnostics:
Collect this output to see the current status
Example:
Name Value Name Value
-------------------- --------------- -------------------- ---------------
in packets 406 out packets 0
if restricted 0 conns with down if 0
f2f packets 0 f2f bytes 0
dropped packets 0 dropped bytes 0
accel packets 0 accel bytes 0
mcast conns 1
Prints the general help message with available parameters
fwaccel ehelp
Prints the extended help message with available parameters
fwaccel <parameter> -h
Prints the specific help message for given <parameter>
fwaccel on [-a] [-q]
Starts acceleration on-the-fly (if Performance Pack is installed)
"-q" flag suppresses the output
"-a" flag means to start acceleration on all Virtual Systems
Returned strings:
SecureXL device is enabled.
Failed to start SecureXL.
No license for SecureXL.
SecureXL is disabled by the firewall. Please try again later.
The installed SecureXL device is not compatible with the installed firewall (version mismatch).
The SecureXL device is in the process of being stopped. Please try again later.
SecureXL cannot be started while "flows" are active.
SecureXL is already started.
SecureXL will be started after a policy is loaded.
fwaccel: Failed to check FloodGate-1 status. Acceleration will not be started.
FW-1: SecureXL acceleration cannot be started while QoS is running in express mode. Please disable FloodGate-1 express mode or SecureXL.
FW-1: SecureXL acceleration cannot be started while QoS is running with citrix printing rule. Please remove the citrix printing rule to enable SecureXL.
FW-1: SecureXL acceleration cannot be started while QoS is running with UAS rule. Please remove the UAS rule to enable SecureXL.
FW-1: SecureXL acceleration cannot be started while QoS is running. Please remove the QoS blade to enable SecureXL.
Failed to enable SecureXL device
fwaccel_on: failed to set process context <VSID>
fwaccel off [-a] [-q]
Stops acceleration on-the-fly
"-q" flag suppresses the output
"-a" flag means to stop acceleration on all Virtual Systems
Returned strings:
SecureXL device disabled
SecureXL device is not active
Failed to disable SecureXL device
fwaccel_on: failed to set process context <VSID>
fwaccel ver
Shows SecureXL FWAccel and FireWall version
Example:
Firewall version: R77.10 - Build 243
Acceleration Device: Performance Pack
Accelerator Version 2.1
Firewall API version: 2.90NG (13/6/2013)
Accelerator API version: 2.90NG (13/6/2013)
fwaccel stat [-a]
Shows SecureXL status
"-a" flag means to show SecureXL status for all Virtual Systems
Example:
Accelerator Status : on
Accept Templates : enabled
Drop Templates : disabled
NAT Templates : disabled by user
Returned strings:
Accelerator Status:
Accelerator Status : on
Accelerator Status : off
Accelerator Status : disabled by Firewall
Accelerator Status : waiting for policy load
Accelerator Status : no license for SecureXL
Accept Templates:
Accept Templates : enabled
Accept Templates : not supported by the SecureXL device
Interface to DOS mitigation techniques in SecureXL.
Commands:
blacklist - Interface to the IP blacklist in SecureXL.
config - Interface to DOS mitigation configuration in SecureXL.
pbox - Interface to the penalty box policy in SecureXL.
rate - Interface to the rate limiting policy in SecureXL.
stats - Interface to DOS real-time statistics.
tab - Manage DOS tables.
whitelist - Interface to the IP whitelist in SecureXL.
fwaccel dos blacklist
Interface to the IP blacklist in SecureXL..
Commands:
add - Add IPs to the blacklist.
clear - Remove all IPs from the blacklist.
get - Print the blacklist.
remove - Remove IPs from the blacklist.
fwaccel dos config
Interface to DOS mitigation configuration in SecureXL.
Commands:
get - View configuration parameters.
set - Modify configuration parameters.
fwaccel dos pbox
Interface to the penalty box policy in SecureXL.
Commands:
whitelist -Interface to the penalty box source IP whitelist in SecureXL.
fwaccel dos rate
Interface to the rate limiting policy in SecureXL.
Commands:
get - View rule information in the rate limiting policy.
install - Manually install a rate limiting policy via stdin. E.g.:
fw samp get -l -k req_type -t in -v quota | fwaccel dos rate install
Note: Installing a new rate limiting policy with more than one rule will automatically enable the rate limiting feature. To manually disable the feature after this command, run:
fwaccel dos config set --disable-rate-limit
In order to delete the current policy, simply install a new policy with zero rules.
fwaccel dos stats
Interface to DOS real-time statistics.
Commands:
clear - Clear real-time statistics.
get - View real-time statistics.
fwaccel dos tab
Manage DOS tables.
Commands:
get - Print contents of the given DOS table.
fwaccel dos whitelist
Interface to the IP whitelist in SecureXL.
Flags:
-a addr/mask - Add entry to whitelist (/mask is optional)
-d addr/mask - Delete entry from whitelist (/mask is optional)
-l filename - Load whitelist from filename
-L - Load whitelist from $FWDIR/conf/dos-whitelist-v4.conf
-s - Show current whitelist entries
-F - Flush all whitelist entries
This whitelist overrides entries in the blacklist. Before using 3rd-party or automatic blacklists, add trusted networks and hosts to the whitelist to avoid outages.
This whitelist also unblocks ip options and fragments from trusted sources when the --enable-drop-opts or --enable-drop-frags features are active.
For whitelisting of rate limiting policy, refer to the bypass action for the fw samp_policy command. I.e.: "fw samp -a b ..."
To replace the current whitelist with the contents of a new file, use both the -F and -l (or -L) options on the same command line.
The whitelist file should contain one entry per line in the format of the -a option. Lines beginning with '#' and blank lines are ignored.
fwaccel dos whitelist -L is automatically run at boot time.
The file that the -L option loads, does not exist by default.
See also:
fwaccel dos pbox whitelist fwaccel synatk whitelist fw samp -a b
Prints the general help message with available parameters
sim ver [-k]
Shows SecureXL SIM version
sim if
Prints the list of interfaces used and seen by the SecureXL implementation (Performance Pack)
Configuration flags (sum of the following values in the "F" column):
0x0001 - If set, the packet should be dropped at the end of the inbound processing, if the packet is "cut-through" packet. In outbound, all the packets should ne forwarded to the network.
0x0002 - If set, and the SIM "tcp" feature is enabled (on), then an appropriate notification should be sent whenever a TCP state change occurs (connection established/teared down).
0x0004 - If set, then when encapsulating an encrypted packet (UDP encapsulation), the UDP header's checksum field should be set correctly. If flag is not set, then the UDP header's checksum field should be set to zero. It is safe to ignore this flag if it is set to 0 (i.e., still calculate the checksum).
0x0008 - If set, then when the Connections Table has reached the specified limit, new connections that match a template should not be created and the packet matching the template should be dropped. If flag is not set, the packet should be forwarded to the firewall.
0x0010 - If set, then fragments should be forwarded to the firewall.
0x0020 - If set, then connection creation from templates should be disabled. Connection can still be offloaded to the device. This flag disables only the creation of TCP templates.
0x0040 - Accelerated connections should periodically be refreshed in the firewall through the notification. The refreshing should be done only if this global flag is set.
0x0080 - If set, then connection creation from templates should be disabled. Connection can still be offloaded to the device. This flag disables only the creation of non-TCP templates.
0x0100 - If set, then sequence verification violations should be allowed for connections that did not complete the 3-way handshake process (instead of F2F-ing violating packets).
00x200 - If set, then sequence verification violations should be allowed for connections that completed the 3-way handshake process (instead of F2F-ing violating packets).
0x0400 - If set, then RST TCP packets should be forwarded to firewall.
0x0800 - If set, then Path MTU Discovery should not be enforced for IP multicast packets.
0x1000 - If set, then SIM "drop_templates" feature should be disabled.
0x2000 - Indicates whether Link Selection Load Sharing feature was enabled by the user.
0x4000 - If set, then SecureXL asynchronic notification feature should be disabled.
0x8000 - Indicates that Firewall Connections Table capacity is unlimited.
Examples:
F=039 means the sum of the following flags:
0x0001
0x0008
0x0010
0x0020
F=0x00009a16 means the sum of the following flags:
0x0002
0x0004
0x0010
00x200
0x0800
0x1000
0x8000
sim vpn <on | off>
Enables/Disables acceleration of VPN traffic:
sim vpn on = enable acceleration of VPN traffic
sim vpn off = disable acceleration of VPN traffic
sim ranges
Prints the loaded ranges
sim ranges -l
Prints the list of loaded ranges
sim ranges -a
Prints all loaded ranges
sim ranges range_id
Prints specified loaded range
sim ranges -s range_id
Prints summary for specified loaded range
sim affinity
Controls network interfaces' affinity settings Note: Command is available only on Linux-based OS
sim affinity -h
Prints the help message with available options for 'affinity' parameter
sim affinity -l
Prints the current network interfaces' affinity
sim affinity -a
Sets the network interfaces' affinity in 'Automatic' mode
sim affinity -s
Sets the network interfaces' affinity in 'Static' ('Manual') mode
sim tmplquota
Controls SIM module's template quota Note: Command is available only on Linux-based OS
sim tmplquota -h
Prints the help message with available options for 'tmplquota' parameter
sim tmplquota -e <1 | 0>
Enables/Disables template quota feature:
sim tmplquota -e 1 = enable the template quota feature
sim tmplquota -e 0 = disable the template quota feature
sim tmplquota -q <quota>
Sets maximum of connections per second per template (quota is allowed)
sim tmplquota -d <drop_duration>
Sets drop duration (in seconds) for drop state
sim tmplquota -m <0 | 1>
Controls monitor only mode:
sim tmplquota -m 0 = disable monitor only mode (default)
sim tmplquota -m 1 = enable monitor only mode
sim tmplquota -r
Resets SIM module template quota values to their defaults
sim tmplquota -d <file_name>
Loads exclusion list of Source IP addresses / Source Subnets from the file
sim tab -d templates
Prints only templates in drop state (output is printed into /var/log/messages files and into Linux kernel ring buffer (output of 'dmesg' command))
sim affinityload
Applies SIM Affinity in 'Automatic' mode Note: Command is available only on Linux-based OS
sim dropcfg
Configures drop parameters (run 'sim dropcfg') Notes:
Since R80.20 "sim dropcfg" is deprecated, replaced by "fw samp" (see sk112454)
Prints the help message with available options for 'dropcfg' parameter
sim dropcfg -l
Prints current drop configuration
sim dropcfg -f </path_to/file_name>
Sets drop configuration file Note: The drop rules configuration does not survive the reboot. Therefore, in order to apply the configured drop rules after the reboot, use a startup script (e.g., /etc/rc.d/rc.local) to run the 'sim dropcfg -f </path_to/file_name>' command automatically during each boot)
sim dropcfg -e
Enforces drop configuration on the external interface only
sim dropcfg -y
Avoids confirmation
sim dropcfg -r
Resets drop rules
sim dbg
Note: From R80.20 "sim dbg" is deprecated, replaced by "fwaccel dbg".
Controls SecureXL SIM debugging (run 'sim dbg -h') By default, debug messages will be printed to /var/log/messages file, therefore you must set the usual kernel debugging options with: [Expert@HostName]# fw ctl debug 0 [Expert@HostName]# sim dbg resetall [Expert@HostName]# fwaccel dbg resetall [Expert@HostName]# fw ctl debug -buf 32000 [Expert@HostName]# sim dbg -m MODULE + FLAG1 FLAG2 ... FLAGn [Expert@HostName]# sim dbg list [Expert@HostName]# fwaccel dbg list [Expert@HostName]# fw ctl kdebug -T -f > /var/log/debug.txt
sim dbg -h
Prints the help message with available options, list of debug modules and their flags
sim dbg list
Prints all currently enabled debug flags for all modules
sim dbg resetall
Resets all debug flags for all modules to their default (none)
sim dbg -m MODULE reset
Resets all debug flags for specified module to their default (none)
sim dbg -m MODULE all
Enables all supported debug flags for specified module
sim dbg -m MODULE + FLAG1 FLAG2 ... FLAGn
Enables specified debug flags for specified module
sim dbg -m MODULE - FLAG1 FLAG2 ... FLAGn
Disables specified debug flags for specified module
Prints the general help message with available parameters
fw ctl multik stat
Prints the summary table for CPU cores and CoreXL FW instances
fw ctl multik start
Starts CoreXL
fw -i Instance_ID ctl multik start
Starts specific CoreXL FW instance
fw ctl multik stop
Stops CoreXL
fw -i Instance_ID ctl multik stop
Stops specific CoreXL FW instance
fw ctl affinity <options>
Controls CoreXL affinities of interfaces/processes/CoreXL FW instances to CPU cores
fw ctl affinity
Prints the help message with available options
fw -d ctl affinity -corelicnum
Prints the number of system CPU cores allowed by CoreXL license
fw ctl affinity -l
Prints the current CoreXL affinities - output shows affinities of interfaces/processes/CoreXL FW instances to CPU cores
fw ctl affinity -l -r
Prints the current CoreXL affinities in reverse order - output shows CPU cores and which interface/process/CoreXL FW instance is affined to each CPU core
fw ctl affinity -l -a
Prints all current CoreXL affinities - output shows affinities of interfaces/processes/CoreXL FW instances to CPU cores, and also shows targets without specific affinity
fw ctl affinity -l -v
Prints the current CoreXL affinities - verbose output shows affinities of interfaces/processes/CoreXL FW instances to CPU cores (targets are shown as 'Interface' (with IRQ), 'Kernel', 'Process'
fw ctl affinity -l -q
Prints the current CoreXL affinities - output shows affinities of interfaces/processes/CoreXL FW instances to CPU cores, and suppresses errors
fw ctl affinity -l -r -a -v
Prints the current CoreXL affinities - verbose output that combines all possible outputs (shows all targets in reverse order)
fw ctl affinity -l -p PID [-r] [-a] [-v]
Prints the current CoreXL affinity of the specified process (by PID) to CPU cores
fw ctl affinity -l -n Daemon_Name [-r] [-a] [-v]
Prints the current CoreXL affinity of the specified process (by name [maximal length = 255 characters]) to CPU cores
fw ctl affinity -l -k Instance_ID [-r] [-a] [-v]
Prints the current CoreXL affinity of the specified CoreXL FW instance to CPU cores
Prints the current CoreXL affinities - extended output Notes:
If "-vsid" flag is omitted, the current context will be used (in which the command was issued)
The "-vsid" flag accepts either a single VSID (e.g., '-vsid 7'), or a range of VSID numbers (e.g., '-vsid 0-5'), or a combination (e.g., '-vsid 0-2 4')
The "-cpu" flag accepts either a single CPU ID (e.g., '-cpu 7'), or a range of CPU ID numbers (e.g., '-cpu 0-5'), or a combination (e.g., '-cpu 0-2 4')
The "-flags" requires at least one of the following arguments (multiple arguments must be specified together):
e - do not print exception processes
k - do not print kernel threads
t - print all process threads
n - print process name instead of /proc/<PID>/cmdline
h - print CPU mask in Hex format
o - print output into the '/tmp/affinity_list_output' file (VSX R77.10 and above)
fw ctl affinity -s
Sets CoreXL Affinity
fw ctl affinity -s -i Interface_NameCPU_IDs | all
Sets affinities of the specified interface to CPU cores
fw ctl affinity -s -p PIDCPU_IDs | all
Sets affinities of the specified process (by PID) to CPU cores
Sets affinity of the specified process (by name) to CPU cores Notes:
If "-vsid" flag is omitted, the current context will be used (in which the command was issued)
The "-vsid" flag accepts either a single VSID (e.g., '-vsid 7'), or a range of VSID numbers (e.g., '-vsid 0-5'), or a combination (e.g., '-vsid 0-2 4')
The "-cpu" flag accepts either a single CPU ID (e.g., '-cpu 7'), or a range of CPU ID numbers (e.g., '-cpu 0-5'), or a combination (e.g., '-cpu 0-2 4')
Sets affinities of Virtual Devices (VS, VR, VSW) to CPU cores Notes:
If "-vsid" flag is omitted, the current context will be used (in which the command was issued)
The "-vsid" flag accepts either a single VSID (e.g., '-vsid 7'), or a range of VSID numbers (e.g., '-vsid 0-5'), or a combination (e.g., '-vsid 0-2 4')
The "-cpu" flag accepts either a single CPU ID (e.g., '-cpu 7'), or a range of CPU ID numbers (e.g., '-cpu 0-5'), or a combination (e.g., '-cpu 0-2 4')
Sets affinities of the specified FWK instances to CPU cores Notes:
The "-inst" flag accepts either a single FWK_ID (e.g., '-inst 7'), or a list of FWK_ID numbers (e.g., '-inst 0 2 4')
The "-cpu" flag accepts either a single CPU ID (e.g., '-cpu 7'), or a range of CPU ID numbers (e.g., '-cpu 0-5'), or a combination (e.g., '-cpu 0-2 4')
fw ctl affinity -s -d -fwkall Number_of_CPUs
Sets affinities of all FWK instances to CPU cores (where Number_of_CPUs is an integer number)
fw ctl affinity -vsx_factory_defaults
Resets all VSX affinity settings (prompts the user) (VSX R77 and above)
fw ctl affinity -vsx_factory_defaults_no_prompt
Resets all VSX affinity settings (does not prompt the user) (VSX R77 and above)
Shows Multi-Queue status of active supported interfaces
cpmq get -a
Shows Multi-Queue status of all supported interfaces (those with enabled Multi-Queue and those with disabled Multi-Queue)
cpmq get -v
Shows Multi-Queue status of active supported interfaces with IRQ affinity information
cpmq get rx_num igb
Shows the number of active RX queues for interfaces that use igb driver
cpmq get rx_num ixgbe
Shows the number of active RX queues for interfaces that use ixgbe driver
cpmq set [-f]
Enables/Disables Multi-Queue per interface (the "-f" flag forces the operation)
cpmq set rx_num all default [-f]
Sets the number of active RX queues to the number of CPU cores that run as CoreXL SND (CPU cores that are not used by CoreXL FW instances) - for all interfaces (those that use igb driver and those that use ixgbe driver) Note: This is recommended configuration
cpmq set rx_num igb default [-f]
Sets the number of active RX queues to the number of CPU cores, which are not used by CoreXL FW instances (recommended) - for interfaces that use igb driver
cpmq set rx_num ixgbe default [-f]
Sets the number of active RX queues to the number of CPU cores, which are not used by CoreXL FW instances (recommended) - for interfaces that use ixgbe driver
cpmq set rx_num all <number> [-f]
Sets the number of active RX queues to the number between 2 and the number of CPU cores - for all interfaces (those that use igb driver and those that use ixgbe driver)
cpmq set rx_num igb <number> [-f]
Sets the number of active RX queues to the number between 2 and the number of CPU cores - for interfaces that use igb driver
cpmq set rx_num ixgbe <number> [-f]
Sets the number of active RX queues to the number between 2 and the number of CPU cores - for interfaces that use ixgbe driver
cpmq set affinity
Sets the IRQ affinity for Multi-Queue interfaces after the following occurs (in the given order):
Multi-Queue is enabled on an interface
The interface status is changed to 'down'
The machine is rebooted
The interface status is changed back to 'up'
Run this command after the interface status is changed back to 'up' to reset the IRQ affinity for this interface.
cpmq reconfigure [-q]
Reconfigures the Multi-Queue (the "-q" flag suppresses the output):
After changing the number of CoreXL FW instances
After changing the physical interfaces on the machine
After changing the number of CPU cores, to which the fwk processes on VSX are assigned
Output of 'fwaccel stat' command shows that 'Accept Templates' are disabled after specific rule:
Accept Templates : disabled by Firewall
disabled from rule #251
Thorough analysis of Firewall's Connections Table (using 'connstat' utility from sk85780) during the issue (~99800 concurrent connections) showed that most matched rules are lower than Rule #251 (after which SecureXL Accept Templates are disabled):
Relevant outputs were collected during a period of time from Security Gateway with the help of the shell script that runs relevant commands every 1 second. All outputs were analyzed and correlated to each other.
Output of 'vmstat' command showed that CPU time is consumed by:
IOWait : min 0% , average 36.6% , max 82%
Kernel Space : min 11% , average 16.2% , max 45%
User Space : min 0% , average 1.8% , max 36%
This leaves IDLE at min 0% , average 45.3% , max 89%
Example:
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
......
1 7 334816 708020 820 37388 2152 0 3280 4 18012 17602 0 13 7 80 0
......
0 9 334348 652800 988 29420 1664 0 3944 4 21575 19725 1 13 5 80 0
......
1 12 291604 723400 884 29696 196 0 4272 12 20799 18273 5 18 0 77 0
Output of 'cat /proc/stat' command showed that CPU time is consumed by:
User Space : min 0% , average 1% , max 7%
Kernel Space : min 0% , average 1% , max 3%
Idle : min 16% , average 34% , max 54%
IOWait : min 32% , average 48% , max 69%
IRQ : min 0% , average 1% , max 1%
SoftIRQ : min 11% , average 14% , max 19
Output of 'top' command showed:
fw_worker_0 / fw_worker_1 / fw_worker_2 - constantly consume CPU at ~15-20%
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6283 root 19 0 54688 16m 12m D 87 0.8 0:00.56 fw
2283 root 15 0 0 0 0 R 30 0.0 4844:09 fw_worker_2
2209 root 15 0 0 0 0 R 16 0.0 5254:57 fw_worker_1
2144 root 15 0 0 0 0 S 14 0.0 5227:59 fw_worker_0
Output of 'ps auxwf' command correlated to the output of 'top' command showed that during the spikes of CPU load by 'fw', the CPD daemon calls the 'sim affinity -c' command, which calls a series of Check Point shell script that calculate the current load by running various 'fw ctl' commands. These commands process high amount of data - and this causes the spike in CPU load. The CPD daemon calling the 'sim affinity -c' command shows that SIM Affinity is configured in Automatic mode (refer to sk63330 - Explanation about 'sim affinity -c' , 'fwaffinity_used_cpus' , 'fw ctl affinity -l -v'):
