Support Center > Search Results > SecureKnowledge Details
Dynamic Balancing for CoreXL Technical Level
Solution

Table of Contents

  • Introduction
  • Key Features
  • System requirements
  • Usage
  • Monitoring
  • SmartConsole Extension
  • Super Instance Feature
  • HyperFlow Resource Allocation
  • Advanced Configuration
  • FAQs
  • Known Limitations

Introduction

Dynamic Balancing (Dynamic Split) is a performance-enhancing daemon that balances the load between CoreXL SNDs and CoreXL Firewalls. It dynamically changes the split between CoreXL SNDs and CoreXL Firewalls and does not require a reboot or cause an outage.

Each Check Point Security Gateway's CPU belongs to one of two groups, each of which performs a different task: 

  • Firewall Instance
  • SND (with the exception of a single CPU running FWD in large User-Space appliances). 

The distribution of jobs across a Security Gateway's CPUs is referred to as the Security Gateway's split. As the distribution of work across these groups depends on your security policy and traffic, we highly recommend that you configure your split to fit your specific needs.

CoreXL's Out-of-the-Box Dynamic Balancing performs a dynamic change of the split. It monitors your system and makes changes as needed.

Starting in R81, this feature is on by default. 

Note: If you manually changed CoreXL or Multi-Queue affinity settings on the old version, the feature is not on by default on the upgraded version. 

Key Features

  • Out-of-the-box optimization
  • A flexible split to suit your profile

System requirements

  • Supported Platforms: Dynamic Balancing is supported on Check Point Appliances. 

  • Supported Models: Dynamic Balancing is supported on these models:
    • All models in these series: 7000, 15000, 16000, 23000, 26000 and 28000
    • 5000 series: 5600, 5800 and 5900
    • 6000 series: 6200T, 6400, 6500, 6600, 6700, 6800 and 6900
    • 3000 series: 3100, 3200, 3600 and 3800

    Notes:
    • On models with fewer than eight cores, you must enable a GNAT port allocation feature. Refer to sk165153. If Dynamic Balancing was enabled before or is on by default, post-enabling GNAT and the required reboot, Dynamic Balancing automatically starts running. 
    • R80.40 is the last supported version for 2200 / 4000 / 12000 / 13000 / 21000 appliances.  
  • Supported Versions: Check Point R80.40 with Jumbo Hotfix Take 25 and higher

  • Supported Configurations: Security Gateway (Kernel or USFW), Stand-Alone, VSX, and Maestro. 
    • Maestro support is for R81.20 and higher versions. Dynamic Balancing is on by default in Maestro R81.20 and higher versions (including Maestro VSX and Maestro mixed appliances).  
    • VSX support is for:
      • R80.40 Jumbo Hotfix Take 126
      • R81 with Jumbo Hotfix Take 58 
      • R81.10 and higher versions 
      • Note: Starting in R81.20, VSX support is on by default.

  • Supported features: IPv6, Management Data Plane Separation (MDPS), Bridge mode 

Usage

Show / Hide this section

To enable Dynamic Balancing

1. In Expert mode, run dynamic_balancing -o enable
  • In Maestro Expert mode, run g_dynamic_balancing -o enable
  • In Clish/gClish, run set dynamic-balancing state enable
2. You can receive a prompt if one of these requirements is not met. 
a. On non-VSX environments, if your current CoreXL split is not the default, you are asked to change the number of instances (not applicable to Maestro mixed appliances).
    • In Expert mode, Enter y
    • In Clish (non-interactive), run the command set dynamic-balancing state enable set_default_fw_instances
b. On VSX environments, if your current FWK affinity is not the same on all VSs, you are asked to set it to default. Refer to the Performance Tuning R81 Administration Guide and refer to this section: CoreXL > CoreXL Commands > fw ctl affinity > Running the fw ctl affinity -s command in VSX Mode.  
3. Reboot your appliance. In Maestro, reboot each Security Gateway Member (SGM) gradually.

Important note for cluster environments
: You must configure all cluster members in the same way. In High Availability mode, start with the Standby member. For more information, refer to the ClusterXL R81.10 Administration Guide (section: ClusterXL Requirements and Compatibility > Software Requirements for Cluster Members).

Syntax
Expert: dynamic_balancing -o { enable | disable | stop | start }
In Maestro Expert mode: use g_dynamic_balancing to run the command on all SGMs.
Clish/gClish: set dynamic-balancing state { enable | disable | stop | start }
Note: You can use start/stop commands on specific SGMs with this gClish command: set dynamic-balancing state {start | stop} member_ids <members IDs>
Action Description
enable Starts Dynamic Balancing from the Security Gateway’s default split.
disable Returns the Security Gateway to its default configuration.
stop Stops Dynamic Balancing from making split changes. Survives reboot. The CoreXL split is restored after the reboot to the same configuration when the command was executed.
start Starts Dynamic Balancing after it is stopped by a user.

Monitoring

Show / Hide this section
  • Verify the Dynamic Split Status in CPView under the SysInfo tab:

  • Monitor Dynamic Balancing through CPView under the CPU tab:

  • Check the Dynamic Balancing status:

    Expert: dynamic_balancing –p
    Maestro Expert mode: g_dynamic_balancing -p
    Clish/gClish:  show dynamic-balancing state
    Output: Dynamic Balancing is currently On/Off


SmartConsole Extension

Show / Hide this section

The SmartConsole Extension provides monitoring and control over Dynamic Balancing.

To enable this feature, use the Import SmartConsole Extension... option in SmartConsole with this URL: https://dannyjung.de/ds.json 

For more information, refer to this CheckMates discussion.


Super Instance Feature

Show / Hide this section
Super Instance was introduced in these versions
  • R80.40 Jumbo Hotfix Take 180
  • R81 Jumbo Hotfix Take 77
  • R81.10 Jumbo Hotfix Take 79
Note: It is not available for VSX.

This feature introduces improvements in Elephant Flow resource utilization (refer to sk164215 - How to Detect and Handle Heavy Connections). Dynamic Balancing detects Elephant Flows and dedicates available resources to the Firewall instance that handles the connection. This feature is on by default.

How it works:
  • Upon detecting an Elephant Flow, Dynamic Balancing stops the Firewall instance that handles the connection, and its logical sibling instance (on SMT HyperThreading environments).
    • New connections are not dispatched to either one of the Firewall instances,
    • The Elephant Flow gradually benefits from the free CPU processing power of the entire physical core.
  • Upon the completion of the Elephant Flow or high overall system utilization, Dynamic Balancing removes the Super Instance by starting the two instances back.
  • The CPU tab in CPView shows the logical sibling core as CoreXL_FW_RESERVED.


HyperFlow Resource Allocation

Show / Hide this section

In R81.20 and higher, alongside the regular CoreXL split decisions, Dynamic Balancing is responsible for HyperFlow cores activation (refer to sk178070 - HyperFlow in R81.20 and higher).

How it works:
  • When it detects an Elephant Flow, Dynamic Balancing allocates cores for the HyperFlow’s Parallel Processing Engine (PPE) threads at the expense of Firewall or SND cores (whichever yields a more balanced state).
  • Dynamic Balancing decides to allocate more PPE threads depending on the Elephant Flow throughput.
  • When the Elephant Flow terminates, or high overall system utilization occurs, Dynamic Balancing deactivates the PPE threads and returns the removed Firewalls/SNDs to their initial state.
  • Dynamic Balancing prevents an activation that interferes with preserving the balanced state of the system. It always prioritizes total throughput (Firewalls and SNDs) over a single connection (PPEs)
  • The Super Instance feature works seamlessly on top of HyperFlow, and provides an additional boost to the Elephant Flow


Advanced Configuration

Show / Hide this section
Dynamic Balancing has various configuration parameters which can be changed during run-time. The ability to change them during run time was introduced in these versions:
  • R80.40 Jumbo Hotfix Take 180
  • R81 Jumbo Hotfix Take 77
  • R81.10 Jumbo Hotfix Take 79
Clish:  set dynamic-balancing config <config> <value>
Expert: dynamic_balancing –v <config> <value>
Note: Use “default” as the <value> to change <config> to default

Name Purpose Unit Default Min/Max
ALPHA Balances decision sensitivity, the minimal difference between Firewall CPUs and SNDs CPUs to perform a change. N/A 10 0/100
EMERGENCY_CPU_HANDLING_THRESHOLD When the FWs average utilization is above this, SNDs must work harder, by a factor, for a change to take place. % 50 0/100
EMERGENCY_CPU_HANDLING_FACTOR The factor by which the SNDs must work harder in the above parameter. For example, 70 translates to a factor of 1.7 N/A 70 0/100
VERBOSE_DEBUG Enable verbose debugs BOOLEAN 0 0/1
VERIFY_INTERVAL State verification interval occurs one time every 300 main loop cycles, which usually translates to every 5 minutes.
During this verification, Dynamic Balancing makes sure its internal state is in sync with the system’s state.
Use 0 to disable state verification checks
N/A 300 0/1000
RECOVERY_ATTEMPTS Limit the number of attempts to recover after a state verification failure, disabled by default. N/A 0 0/100
SUPER_INST_ENABLED Enable Super Instance feature BOOLEAN 1 0/1
SUPER_INST_ADD_FW_LOAD Super instances can be added if FWs average utilization is below this. % 60 0/100
SUPER_INST_REMOVE_FW_LOAD Super instances are removed if the FWs average utilization is above this. % 70 0/100
DMD_WAKEUP_FW_LOAD HyperFlow PPE threads "wake up" if FWs average utilization is below this, and an Elephant Flow was detected. % 60 0/100
DMD_SLEEP_FW_LOAD HyperFlow PPE threads "go to sleep" if FWs average utilization is above this. % 70 0/100


FAQs

Click Here to Show the Entire Article
  • Who should use Dynamic Balancing?
    Dynamic Balancing is especially beneficial for:
    • Customers who use non-default splits
    • Customers with env. bottle-necks by Secure Network Distributors (SNDs)
    • Customers with environments that may change usage over time - you can "auto tune" your device with no outage, no reboots, no need for any skills!
  • Why don't I get more Secure Network Distributors (SNDs)?
    There are few possible reasons:
    • Average utilization of SNDs vs FireWall is relatively close (we require 10% difference by default)
    • FireWalls are utilized more than 50% (EMERGENCY_CPU_HANDLING_THRESHOLD), and SNDs are not working harder, by a factor of 1.7 (EMERGENCY_CPU_HANDLING_FACTOR). This is done to prevent a situation where more than one heavily utilized FireWall instance works on a single CPU.
    • All eligible SNDs are already used as SNDs
  • Why don't I get more Firewalls?
    There are few possible reasons:
    • Average utilization of SNDs vs Firewalls is relatively close (we require 10% difference by default)
    • All loaded firewall instances are already active
  • How does it work with Cluster?
    In Cluster HA mode, changes the active member makes are synced to the passive member. This does not apply to Virtual Routing Redundancy Protocol (VRRP).
  • Feature enablement on one member is only in cluster configuration - any sync related issues?
    No, as at any given time, both members will have same number of Firewall instances. Dynamic Workloads will merely “stop” the instance, meaning new connection are not to be dispatched to it.
  • How does it work with VSX?
    Dynamic Balancing in VSX is similar to Security Gateways in that it aims to balance the SNDs and FWs cores. As opposed to Security Gateways, FW instances do not have static core affinity, and their amount does not determine the amount of SNDs. As a result, Dynamic Balancing does not require a certain amount of FW instances to be configured, and only adjusts their core affinity.

    When adding an SND, the feature sets the affinity of FWK processes in all VSs to the list of new cores (rather than move a FW instance from one core to a different core, as done in Security Gateways). The maximum quantity of SND cores will be according to the NIC driver, with the highest number of queues in all VSs.

    When you add a VS, the feature sets the new VS’s FWK to the current FWKs cores affinity.
  • Why requiring a reboot upon 1st enablement of the feature?
    Dynamic Balancing requires several configuration changes that can only be set upon boot, to allow best performance in all splits (mostly relevant when more SNDs are needed vs default split).
  • What are the scenarios in which the feature is to automatically turn itself off?
    Dynamic Balancing monitors the system periodically for manual changes that might conflict with its own actions (such as: Firewall instances state or affinity changes, interfaces affinity changes etc.), and will stop itself if such action is detected. Refer to sk163815 for more information.  
  • Can excessive split changes impact production environment?
    Dynamic Balancing has mechanisms within its logic aimed to prevent un-wanted changes (uses threshold to not be over sensitive, detects frequent conflicting changes and more). Essentially, it uses existing, established Firewall/Linux commands, that were used in Check Point devices for years, while automating their use to be as effective as possible.
  • Where do I monitor feature is enabled? 
  • Do I need to perform reboot once I want to disable/stop Dynamic Balancing?
    Stopping the Dynamic Balancing does not require a reboot. Disabling Dynamic Balancing requires a reboot, since part of the disable process is to revert the configuration back to default, which includes several settings that can only be set on boot (mainly due to memory allocations performed at system init).
  • If Dynamic Balancing is enabled, is the CoreXL setting in cpconfig disabled? Will I not be able to change CoreXL numbers?
    No, but changes made in the CoreXL cpconfig only take effect after reboot. Note that on non-VSX environments, rebooting with a non-default instance number (that is, manual changes done by the user) prevents Dynamic Balancing from starting in order not to overwrite the users' actions.
    An alert is sent to the user in such scenarios:
  • Why do I see more than one Firewall instance on a single CPU, in non-VSX environments?
    When Dynamic Balancing adds additional SNDs:
    1. It stops the Firewall instance(s) of the physical CPU that is going to become an SND. This means that new connections are no longer dispatched to this instance, but it will continue to handle its existing connections.

      The "fw ctl multik stat" command shows these Firewall instances as inactive:



    2. It sets the affinity of the stopped Firewall instance(s) to the affinity of the active Firewall instances belonging to the same NUMA node.

      • The "fw ctl affinity -l -r" command shows these Firewall instances on all (active) Firewall instances' CPUs:
      • Note that CPU 20 has different inactive Firewall instances on it because it is on a different NUMA node than CPUs 7-17.


    3. It turns the free CPU to start working as SND.
  • Why in some scenarios, I can still see the connections being opened on the stopped instances?
    Although the Dynamic Dispatcher excludes the stopped instance from its dispatching calculations, in cases where a VPN tunnel was previously created on this instance, some connections, such as NAT-T or ones belong to a non-accelerated tunnel, may be opened on this instance to allow a certain performance optimization.
  • What does “BOTH” or “OTHER” mean in the CPVIEW CPU tab?
    There are two primary types of CPUs: CoreXL_FW and CoreXL_SND.
    "BOTH" means the CPU currently operates as a FW and an SND.
    "OTHER" means the CPU does not currently operate as a FW or SND.
    In most scenarios, this is an expected, temporary state, caused by CPU role change.
    If this is not a temporary state, it could imply an affinity misconfiguration.


Notes

Dynamic Balancing manages network card ports that have Multi-Queue enabled. The "mq_mng --show" command shows such ports as "Dynamic".
While Dynamic Balancing is active, it assumes control over several resources (listed below). Manual changes may not work, or cause Dynamic Balancing to stop its work (refer to sk163815 for more details):

  • Changes in affinity of CoreXL Firewall instances, starting or stopping CoreXL Firewall instances, and changing the number of CoreXL Firewall instances.
  • Changes in Multi-Queue affinity/mode, or changes in the number of RxTx queue weights.


Known Limitations

Issue ID Description Comments
PRJ-15874 When you downgrade to Jumbo Hotfix Take where the Dynamic Balancing is not supported, it remains enabled. In this case, the affinity of the Security Gateway will be configured incorrectly.  Disable the Dynamic Balancing before you uninstall the Jumbo Hotfix 

Give us Feedback
Please rate this document
[1=Worst,5=Best]
Comment