Table of Contents:
-
Introduction
-
Firewall Priority Queues in R77.30 / R80.x / R81.x
-
Configuration on Security Gateway
-
Manual Configuration
-
Monitoring
-
Limitations
-
Related solutions
-
Installation/Disablement of Priority Queues and Evaluation of Heavy Connections Mechanism
-
Evaluation of Heavy Connections
-
Elephant Connection Detection Mechanism
(I) Introduction
Packets could be dropped by Firewall when CPU cores, on which Firewall runs, are fully utilized. Such packet loss might occur regardless of the connection's type (for example, local SSH or connection to Security Management Server server).
To help mitigate the above issue, Firewall Priority Queues feature was introduced in R77.30 Security Gateway.
(II) Firewall Priority Queues in R77.30 / R80.x / R81.x
Note: starting from R80.40, the Firewall Priority Queues are enabled by default. In R80.30 and lower, the Firewall Priority Queues are disabled by default.
The Priority Queues (PrioQ) mechanism is intended to prioritize part of the traffic, when we need to drop packets because the Security Gateway is stressed (CPU is fully utilized).
In Security Gateway R77.20 and lower versions, when the CPU became fully utilized, part of the traffic was dropped regardless of the traffic type. As a result, control connections (described below) were dropped. This had serious negative impact (e.g., no SSH connectivity). In addition, several "heavy" connections could cause high CPU load on the Security Gateway and cause issues for all other connections.
Security Gateway R77.30 / R80.x / R81.x handle both aforementioned cases in the following ways:
- Prioritizing control connections over data connections.
- Each connection of the same priority will get an equal share of CPU resources.
Security Gateway R77.30 / R80.x / R81.x "protects" the CPU cores, on which the Firewall is running.
It is important to notice the differences between Firewall Priority Queues (PrioQ) and the QoS Software Blade.
Both prioritize traffic. Note the following differences:
| |
Firewall Priority Queues (PrioQ)
in R77.30 / R80.x / R81.x |
QoS Software Blade |
| What is mitigated? |
CPU load |
Network downstream load |
When it is active
on Security Gateway? |
Only in extreme
conditions (CPU is overloaded) |
Always (from the moment the blade
is enabled and policy installed) |
| Is the configuration predefined or does it depend on policy? |
Predefined |
Policy driven |
-
Explanation about Control Connections
Firewall R77.30 / R80.10 and higher assign higher priority to control connections than to other connections.
By default, the following services are considered by Firewall R77.30 / R80.10 and higher as control connections:
- Check Point CPMI
- Policy installation/fetch (CPD daemon)
- Check Point Remote Installation (CPRID daemon)
- SSH
- DHCP
- OSPF
- BGP
- VRRP
Not all control services get the same priority. Firewall R77.30 / R80.10 and higher prioritize some control services over the other control services.
-
Explanation about Priority Queues
By default, there are 8 priority queues (up to 16 by manual configuration).
Queue
Priority
(0=highest) |
Queue
Name |
Type of connections |
Can
connections
migrate? |
Limited
Queue
Length
(default=512) |
| 0 |
Routing |
DHCP, VRRP, OSPF, BGP, IGMP, PIM, ... |
No |
Yes |
| 1 |
Control |
GUI / SSH / ports 18xxx / Mgmt services (see sk52421) |
Yes |
Yes |
| 2 |
Cluster sync |
Local / Full sync |
No |
Yes |
| 3 |
High priority |
User defined |
Yes |
Yes |
| 4 |
Light Data Queue |
Light connections |
Yes |
Yes |
| 5 |
Default Data Queue |
Medium weight connection, or New connection |
Yes |
Yes |
| 6 |
Log notification |
Log and Drop notifications |
No |
Yes |
| 7 |
Heavy Data Queue |
Heavy connections |
Yes |
Yes |
The priority queues are active when the Firewall consumes 100% of CPU. When a packet of a new connection arrives, the connection is classified in order to know to which queue it is supposed to go. For example, packets of a new data connection will go to the default data queue.
The Firewall priority Queues are enabled in the active priority mode: Initially, each connection is assigned to a queue by the connection type: Routing, Control, Data, etc. Nevertheless, for some connection types, the connection might migrate to a different queue. We know what is the load of each connection all the time, so the connections can migrate between the queues, and by doing so get lower/higher priority with respect to other connections.
For example, a data connection begins its life in the Default Data Queue (Queue #5) and if this connection becomes heavier, it will migrate to a lower priority queue (Heavy Data Queue - Queue #7). Same is true vice versa. If a connection becomes lighter, it will migrate to a higher priority queue (e.g., Light Data Queue - Queue #4).
To decide whether a connection should migrate or not, the Firewall keeps the load of the average connection all the time. For each connection (by its type), the Firewall knows to how many queues the connection was assigned and what is the CPU load caused by this connection. Based on that information, the Firewall matches weight to a queue, so that the average connection will belong to the median queue (out of all possible queues for the connection type) and each queue has an equal share of weights.
In R80.20, in VSX mode, the Firewall priority Queues are enabled in the static priority mode. This means that after a connection was classified to a specific queue, it cannot migrate to a different queue (classification is done only according to the connection type and not according to the load of the connection).
(III) Configuration on Security Gateway
For instructions on how to install and disable the Firewall Priority Queues, refer to Installation/Disablement of Priority Queues and Evaluation of Heavy Connections Mechanism.
(IV) Manual Configuration
To perform manual configuration, use $FWDIR/conf/prioq.conf
prioq.conf is a customized classification file.
Entry format: {Type, Interface, IP, src_port_num, dst_port_num, protocol}
| Field |
Options |
Explanation |
| Type |
CONTROL / ROUTE |
To which queue the rule refers. |
| interface |
any / external / internal / <interface_name> |
On which interface the rule will be. |
| IP |
any / local_src / local_dst |
Which kind of connections. |
| Source Port Number |
0 (any) / port_number |
Source Port Number. |
| Destination Port Number |
0 (any) / port_number |
Destination Port Number. |
| Protocol Number |
protocol_num |
Protocol Number. |
Example of default prioq file:
You can also add BFD:
#BFD (singlehop)
{ROUTE,any,any,0,3784,17}
#BFD (multihop)
{ROUTE,any,any,0,4784,17}
(V) Monitoring
Administrator can monitor the Firewall Priority Queues in the following ways.
Note: Firewall Priority Queues on Security Gateway must already be enabled. Refer to Installation/Disablement of Priority Queues and Evaluation of Heavy Connections Mechanism.
-
Use the sk101878 - CPView Utility.
-
Go to Advanced -> PrioQ -> Overview
Provides general PrioQ information (cross instances)
Example:
| Security Gateway R77.30 |
Security Gateway R80.10 |
 |
 |
-
Go to Advanced -> PrioQ -> Instances
Provides general information about the priority queues and information per instance including the priority queues of the specific instance.
Note: When CPU cores are not fully utilized, all values on this screen will be 0 (zero).
Example:
| Security Gateway R77.30 |
Security Gateway R80.10 |
 |
 |
-
Use SmartLog (supported only for Dynamic PrioQ).
A log is issued for every Heavy Connection (that consumes more than 10% of the CPU resources) that such connection was detected and its priority is decreased.
Example:
(VI) Limitations
Refer to the Configuration Limitations section.
(VIII) Installation/Disablement of Priority Queues and Evaluation of Heavy Connections Mechanism
Configuration on Security Gateway R77.30
Note: Firewall Priority Queues and Evaluation of Heavy Connections mechanism are disabled by default.
Important Notes:
-
Before enabling these features, refer to "Limitations" section.
-
In R77.30, these two modes are the only officially supported modes (other modes are not supported, and therefore are not mentioned):
| Mode Number |
Mode Name |
Explanation |
| 0 |
Off |
Default.
Firewall Priority Queues and Evaluation of Heavy Connections are completely disabled. |
| 9 |
On |
Firewall Priority Queues feature and connection statistics feature (in cpview) is fully enabled.
Note: This mode also fully enables the CoreXL Dynamic Dispatcher (refer to sk105261). |
Instructions:
-
To check the current mode on Security Gateway:
[Expert@HostName]# fw ctl multik get_mode
-
To fully enable the Firewall Priority Queues on Security Gateway:
Note: In cluster environment, this procedure must be performed on all members of the cluster.
-
Run in Expert mode:
[Expert@HostName]# fw ctl multik set_mode 9
-
Reboot (in cluster, this might cause fail-over).
-
To completely disable the Firewall Priority Queues on Security Gateway:
Note: In cluster environment, this procedure must be performed on all members of the cluster.
-
Run in Expert mode:
[Expert@HostName]# fw ctl multik set_mode 0
-
Reboot (in cluster, this might cause fail-over).
Configuration on Security Gateway R80.10 and higher
Important Notes:
-
In R80.10, Firewall Priority Queues are disabled by default.
-
In R80.20, Firewall Priority Queues are disabled by default, and the Evaluation of Heavy Connections mechanism is enabled by default (Evaluator-only mode)
-
Starting in R80.10, configuration of Firewall Priority Queues and CoreXL Dynamic Dispatcher were separated and are no longer related to each other.
-
Before enabling these features, refer to "Limitations" section.
-
In R80.10 and higher, these three modes are the only officially supported modes (other modes are not supported, and therefore are not mentioned):
| Mode Number |
Mode Name |
Explanation |
| 0 |
Off |
Default.
Firewall Priority Queues and Evaluation of Heavy Connections are completely disabled. |
| 1 |
Evaluator-only
(Connections
Statistics) |
Firewall Priority Queues feature is disabled, but monitoring of Heavy Connections
(that consume the most CPU resources) is enabled in CPView Utility. |
| 2 |
On |
Firewall Priority Queues feature is fully enabled. |
Instructions:
-
To check the current mode on Security Gateway:
[Expert@HostName]# fw ctl multik prioq
Example output:
[Expert@R80_10_SA:0]# fw ctl multik prioq
Current mode is Off
Available modes:
0. Off
1. Evaluator-only
2. On
Choose the desired mode number: (or 3 to Quit)
-
To fully enable the Firewall Priority Queues on Security Gateway:
Note: In cluster environment, this procedure must be performed on all members of the cluster.
-
Run in Expert mode:
[Expert@HostName]# fw ctl multik prioq
Example output:
[Expert@R80_10_SA:0]# fw ctl multik prioq
Current mode is Off
Available modes:
0. Off
1. Evaluator-only
2. On
Choose the desired mode number: (or 3 to Quit)
2
New mode is: On
Please reboot the system
[Expert@R80_10_SA:0]#
-
Choose the mode number 2 "On".
-
Reboot (in cluster, this might cause fail-over).
-
To enable the monitoring of Heavy Connections (that consume more than 10% of the CPU resources) in CPView Utility on Security Gateway:
Notes: This mode is not supported when Security Gateway is configured in VSX mode. In cluster environment, this procedure does not have to be performed on all members of the cluster because it enables monitoring only.
-
Run in Expert mode:
[Expert@HostName]# fw ctl multik prioq
-
Choose the mode number 1 "Evaluator-only".
-
Reboot (in cluster, this might cause fail-over).
-
To completely disable the Firewall Priority Queues on Security Gateway:
Note: In cluster environment, this procedure must be performed on all members of the cluster.
-
Run in Expert mode:
[Expert@HostName]# fw ctl multik prioq
-
Choose the mode number 0 "Off".
-
Reboot (in cluster, this might cause fail-over).
Configuration Limitations
-
The Firewall Priority Queues cannot be enabled in the following scenarios:
- In R77.30 and R80.10 VSX mode - Firewall Priority Queues are not supported at all (on any of the VSs, including VS0)
- In R80.20 VSX mode:
- "Evaluator-only" mode is not supported. (VSX and USFW mode are, however, supported since Jumbo Hotfix Accumulator for R80.40 Take 78 (PRJ-13177))
- The Evaluator of Heavy Connections mechanism is not supported.
- Firewall Priority Queues is supported in static priority mode only (see "Explanation about Priority Queues" section).
- SAM acceleration card is installed on the appliance
- Carrier Grade NAT (CGN) is configured
- Security Gateway is configured in Monitor Mode (per sk101670)
- 6in4 tunnel (SIT interface) is configured
- Some lines in the $FWDIR/boot/modules/fwkern.conf file are commented out (refer to sk106309).
-
When SecureXL and a CoreXL FW instance are running on the same CPU core, Priority Queues and Top Connections may not function as expected.
Such a configuration is the default on the 2200 appliance and on the 4600 appliance.
To resolve this, disable the synchronous dequeue feature in CoreXL by permanently setting the value of kernel parameter fwmultik_sync_processing_enabled to 0 (zero).
(IX) Evaluation of Heavy Connections
Introduction
This feature allows the administrator to monitor the Heavy Connections (that consume the most CPU resources) without interrupting the normal operation of the Firewall.
After enabling this feature (refer to Installation/Disablement of Priority Queues and Evaluation of Heavy Connections Mechanism), the relevant information is available in CPView Utility.
For detecting Elephant connection, refer to the Elephant Connection Detection Mechanism.
Note: This feature is not supported in VSX mode for R77.30/R80.10/R80.20. However, VSX and USFW mode are supported since Jumbo Hotfix Accumulator for R80.40 Take 78 (PRJ-13177).
Configuration on Security Gateway
For installation and disablement of the Evaluation of Heavy Connections mechanism, refer to Installation/Disablement of Priority Queues and Evaluation of Heavy Connections Mechanism.
Monitoring
Administrator can monitor the Evaluation of Heavy Connections in the following ways:
Monitoring Heavy Connections
-
Set the mode of the Firewall Priority Queues on Security Gateway to enabled:
Version of
Security Gateway |
Command |
| R80.10 |
[Expert@HostName]# fw ctl multik prioq
Select mode 1 "Evaluator-only". |
| R77.30 |
[Expert@HostName]# fw ctl multik set_mode 9 |
-
Reboot (in cluster, this might cause fail-over).
-
Run the sk101878 - CPView Utility:
[Expert@HostName]# cpview
-
Go to CPU -> Top-Connections.
This screen displays top 10-30 (configurable) connections that consume the most CPU resources.
Example:
| Security Gateway R77.30 |
Security Gateway R80.10 |
 |
 |
Use SmartLog
Note: This log is supported only for Dynamic PrioQ.
A log is issued for every Heavy Connection (that consumes more than 10% of the CPU resources) that was detected and its priority is decreased.
Example:
Limitations
Refer to Configuration Limitations section.
(X) Elephant Connection Detection Mechanism
Starting Jumbo Hotfix Accumulator for R80.20 Take 47 and newer versions, a heavy connection detection mechanism was added for Kernel mode FW only.
Every connection which fulfills the following 3 conditions will be considered as a heavy connection flow and will be reported.
Heavy connection flow system definition:
- Specific instance CPU is over 60%
- Suspected connection lasts more than 10s
- Suspected connection utilizes more than 50% of the total work the instance does
- In other words, connection CPU utilization must be > 30%
Several notes:
- Identifying elephant depends only on the specific instance related to it.
- Allows us to tackle cases where several elephant connections are running on the system
- The 10s threshold, prevent us from falsely identifying short connections with a packet burst
- Connection overall utilization is an average calculated throughout the connection life cycle
- Average is collected according to the formula - new_average = α*old_average + (1-α)*new_value
The system saves heavy connection data for the last 24 hours and CPDiag has a matching collector which uploads this data for diagnosis purposes.
On the system itself, heavy connection data is accessible using the command: "fw ctl multik print_heavy_conn"
Its output is as follows:
- Source - source IP
- SPort – source port
- Dest - Destination IP
- DPort – Destination port
- IPP – ipp
- Instance – instance number
- Instance load – instance CPU load while the connection was captured
- Connection instance load – The load percentage of the connection on the instance
- (100 * Connection CPU load/Instance CPU load)
