Support Center > Search Results > SecureKnowledge Details
CoreXL Dynamic Dispatcher in R77.30 / R80.10 and above
Solution

Table of Contents:

  1. Introduction
  2. CoreXL Dynamic Dispatcher
  3. Configuration on Security Gateway R77.30
  4. Configuration on Security Gateway R80.10 and above
  5. Monitoring CoreXL load distribution
  6. Limitations
  7. FAQ
  8. Related documentation
  9. Related solutions

 

(1) Introduction

CoreXL is a performance-enhancing technology for Security Gateways on platforms with multiple CPU cores. CoreXL enhances Security Gateway performance by enabling the processing CPU cores to concurrently perform multiple tasks.

On a Security Gateway with CoreXL enabled, the Firewall kernel is replicated multiple times. Each replicated copy, or Firewall instance, runs on one processing CPU core. These Firewall instances handle traffic concurrently, and each Firewall instance is a complete and independent Firewall inspection kernel. When CoreXL is enabled, all the Firewall kernel instances in the Security Gateway process traffic through the same interfaces and apply the same security policy.

The CoreXL software architecture includes the Secure Network Distributor (SND). The SND is responsible for:

  • Processing incoming traffic from the network interfaces
  • Securely accelerating authorized packets (if SecureXL is running)
  • Distributing non-accelerated packets or Medium Path packets among CoreXL FW kernel instances - this functionality is also referred to as dispatcher

Traffic received on network interface cards (NICs) is directed to a processing core running the SND.

The dispatcher is executed when a packet should be forwarded to a CoreXL FW instance (in Slow path and Medium path - see sk98737 for details) and is in charge of selecting the CoreXL FW instance that will inspects the packet.

In R77.20 and lower versions, traffic distribution between CoreXL FW instances is statically based on Source IP addresses, Destination IP addresses, and the IP 'Protocol' type. Therefore, there are possible scenarios where one or more CoreXL FW instances would handle more connections, or perform more processing on the packets forwarded to them, than the other CoreXL FW instances.

This may lead to a situation, where the load is not balanced across the CPU cores, on which the CoreXL FW instances are running.

To help mitigate the above issue, CoreXL Dynamic Dispatcher feature was introduced in R77.30 Security Gateway.

 

(2) CoreXL Dynamic Dispatcher

Rather than statically assigning new connections to a CoreXL FW instance based on packet's IP addresses and IP protocol (static hash function), the new dynamic assignment mechanism is based on the utilization of CPU cores, on which the CoreXL FW instances are running.

The dynamic decision is made for first packets of connections, by assigning each of the CoreXL FW instances a rank, and selecting the CoreXL FW instance with the lowest rank.

The rank for each CoreXL FW instance is calculated according to its CPU utilization.

The higher the CPU utilization, the higher the CoreXL FW instance's rank is, hence this CoreXL FW instance is less likely to be selected by the CoreXL SND.

The CoreXL Dynamic Dispatcher allows for better load distribution and helps mitigate connectivity issues during traffic "peaks", as connections opened at a high rate that would have been assigned to the same CoreXL FW instance by a static decision, will now be distributed to several CoreXL FW instances.

When CoreXL Dynamic Dispatcher is enabled, the dynamic decision is always made (even when there is no significant load).

 

(3) Configuration on Security Gateway R77.30

CoreXL Dynamic Dispatcher is disabled by default. Meaning, CoreXL statically assigns new connections to a CoreXL FW instance based on packet's IP addresses and IP 'Protocol' type (static hash function).

Important Notes:

  • Before enabling the CoreXL Dynamic Dispatcher, refer to "Limitations" section and to "FAQ" section.

  • In R77.30, when working with CoreXL Dynamic Dispatcher, the only officially supported modes are:

    Mode Number Mode Name Explanation
    0 Off Default.
    CoreXL Dynamic Dispatcher is completely disabled. CoreXL statically assigns new connections to a CoreXL FW instance based on packet's IP addresses and IP 'Protocol' type.
    4 Full-DD Important Note: Supported only after installing the hotfix from sk109772 - R77.30 NGTP, NGTX and HTTPS Inspection performance and memory consumption optimization.
    • CoreXL Dynamic Dispatcher is fully enabled.
    • Firewall Priority Queue feature in each CoreXL FW instance (refer to sk105762) is fully disabled.
    9 On CoreXL Dynamic Dispatcher is fully enabled.
    Note: This mode also fully enables the Firewall Priority Queues feature in each CoreXL FW instance (refer to sk105762).

Instructions:

  • To check the current mode on Security Gateway:

    [Expert@HostName]# fw ctl multik get_mode

    Example output:

    [Expert@R77.30:0]# fw ctl multik get_mode
    Current mode is Off
    [Expert@R77.30:0]#
    
  • To fully enable the CoreXL Dynamic Dispatcher on Security Gateway:

    Note: In cluster environment, this procedure must be performed on all members of the cluster. Since a reboot is required, it is recommended to follow the Gaia Installation and Upgrade Guide - either "Minimal Effort" procedure, or "Zero Downtime" procedure.

    1. Run in Expert mode:

      [Expert@HostName]# fw ctl multik set_mode 9

      Example output:

      [Expert@R77.30:0]# fw ctl multik set_mode 9
      Please reboot the system
      [Expert@R77.30:0]#
      
    2. Reboot (in cluster, this might cause fail-over).

  • To completely disable the CoreXL Dynamic Dispatcher on Security Gateway:

    Note: In cluster environment, this procedure must be performed on all members of the cluster. Since a reboot is required, it is recommended to follow the Installation and Upgrade Guide for Gaia Platforms - Chapter 'Upgrading ClusterXL Deployments' - either "Minimal Effort" procedure, or "Zero Downtime" procedure.

    1. Run in Expert mode:

      [Expert@HostName]# fw ctl multik set_mode 0

      Example output:

      [Expert@R77.30:0]# fw ctl multik set_mode 0
      Please reboot the system
      [Expert@R77.30:0]#
      
    2. Reboot (in cluster, this might cause fail-over).

 

(4) Configuration on Security Gateway R80.10 and above

CoreXL Dynamic Dispatcher is enabled by default. Meaning, CoreXL dynamically assigns new connections to a CoreXL FW instance based on the utilization of CPU cores.

Important Notes:

  • Refer to "Limitations" section.
  • Refer to "FAQ" section.

Instructions:

  • To check the current mode on Security Gateway:

    [Expert@HostName]# fw ctl multik dynamic_dispatching get_mode

    Example output:

    [Expert@R80.10:0]# fw ctl multik dynamic_dispatching get_mode
    Current mode is On
    [Expert@R80.10:0]#
    
  • To enable the CoreXL Dynamic Dispatcher on Security Gateway:

    Note: In cluster environment, this procedure must be performed on all members of the cluster. Since a reboot is required, it is recommended to follow the Installation and Upgrade Guide - Chapter 'Upgrading ClusterXL Deployments' - either "Minimal Effort Upgrade" procedure, or "Zero Downtime Upgrade" procedure.

    1. Run in Expert mode:

      [Expert@HostName]# fw ctl multik dynamic_dispatching on

      Example output:

      [Expert@R80.10:0]# fw ctl multik dynamic_dispatching on
      New mode is: On
      Please reboot the system
      [Expert@R80.10:0]#
      
    2. Reboot (in cluster, this might cause fail-over).

  • To disable the CoreXL Dynamic Dispatcher on Security Gateway:

    Note: In cluster environment, this procedure must be performed on all members of the cluster. Since a reboot is required, it is recommended to follow the Installation and Upgrade Guide - Chapter 'Upgrading ClusterXL Deployments' - either "Minimal Effort Upgrade" procedure, or "Zero Downtime Upgrade" procedure.

    1. Run in Expert mode:

      [Expert@HostName]# fw ctl multik dynamic_dispatching off

      Example output:

      [Expert@R80.10:0]# fw ctl multik dynamic_dispatching off
      New mode is: Off
      Please reboot the system
      [Expert@R80.10:0]#
      
    2. Reboot (in cluster, this might cause fail-over).

 

(5) Monitoring CoreXL load distribution

This section describes how administrator can monitor either the current CoreXL load distribution to decide whether to enable CoreXL Dynamic Dispatcher, or the CoreXL Dynamic Dispatcher in action (after it was enabled):

Note: To monitor the CoreXL Dynamic Dispatcher in action, it must already be enabled with fw ctl multik set_mode 9 command and Security Gateway must already be rebooted.

The examples provided below were taken from the machine without VPN blade or VoIP traffic and with disabled CoreXL Dynamic Dispatcher. These examples show an excellent reason to enable CoreXL Dynamic Dispatcher to mitigate the load imbalance between CoreXL FW instances:

  1. Check on which CPU cores the CoreXL FW instances are running with the fw ctl affinity -l -r command.

    Example (CoreXL FW instances are running on CPU 1, CPU 2, and CPU 3):

    [Expert@HostName]# fw ctl affinity -l -r
    CPU 0:  eth0 eth1 eth2
    CPU 1:  fw_2
    CPU 2:  fw_1
    CPU 3:  fw_0
    All:    cpca status_proxy in.geod fwm cpstat_monitor fwd mpdaemon cpsead cpd cprid
    [Expert@HostName]#
    
  2. Check the current CPU utilization by each CoreXL FW instance with the top command.

    Note: If the output does not show all CPU cores (if 3rd line shows "Cpu(s):"), then press "1" and then "Shift+W".

    Example (Load on CPU 3 is 24% by SoftIRQ; CoreXL FW instance 0 is consuming 18%; other CoreXL FW instances (fw_worker threads) are idle):

    Tasks: 118 total,   3 running, 115 sleeping,   0 stopped,   0 zombie
    Cpu0  :  0.0%us,  0.0%sy,  0.0%ni, 94.0%id,  0.0%wa,  1.2%hi,  4.8%si,  0.0%st
    Cpu1  :  0.0%us,  1.2%sy,  0.0%ni, 98.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
    Cpu2  : 40.2%us,  9.2%sy,  0.0%ni, 49.4%id,  0.0%wa,  0.0%hi,  1.1%si,  0.0%st
    Cpu3  :  1.3%us,  1.3%sy,  0.0%ni, 73.3%id,  0.0%wa,  0.0%hi, 24.0%si,  0.0%st
    Mem:   4078484k total,  4021144k used,    57340k free,   241380k buffers
    Swap:  3140696k total,       64k used,  3140632k free,   414744k cached
    
      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
    15964 admin     18   0  1644  436  368 R   41  0.0   0:00.42 cp_logrotate
     4129 admin     15   0     0    0    0 R   18  0.0   8:23.67 fw_worker_0
    15954 admin     15   0  2176 1112  840 R    2  0.0   0:00.09 top
       14 admin     10  -5     0    0    0 S    0  0.0  47:01.09 events/0
     4995 admin     15   0  207m  66m  24m S    1  1.7  79:04.08 cpd
        1 admin     15   0  2044  724  624 S    0  0.0   0:00.43 init
        2 admin     RT  -5     0    0    0 S    0  0.0   3:08.83 migration/0
        3 admin     15   0     0    0    0 S    0  0.0   0:02.15 ksoftirqd/0
        4 admin     RT  -5     0    0    0 S    0  0.0   0:00.05 watchdog/0
        5 admin     RT  -5     0    0    0 S    0  0.0   2:46.42 migration/1
        6 admin     15   0     0    0    0 S    0  0.0   0:00.01 ksoftirqd/1
        7 admin     RT  -5     0    0    0 S    0  0.0   0:00.36 watchdog/1
        8 admin     RT  -5     0    0    0 S    0  0.0   2:36.56 migration/2
        9 admin     17   0     0    0    0 S    0  0.0   0:00.24 ksoftirqd/2
    
  3. Check the distribution of connections across all CoreXL FW instances with the fw ctl multik stat command.

    Example (refer to the "Peak" column - CoreXL FW instance 0 processed almost all connections, while other CoreXL FW instances were idle):

    [Expert@HostName]# fw ctl multik stat
    ID | Active  | CPU    | Connections | Peak    
    ----------------------------------------------
     0 | Yes     | 3      |           1 |    21623
     1 | Yes     | 2      |           0 |        6
     2 | Yes     | 1      |           1 |       13
    [Expert@HostName]#
    

 

(6) Limitations

The CoreXL Dynamic Dispatcher is not supported in the following scenarios:

  • Security Gateway is configured in VSX mode (not supported on any of the VSs, including VS0)
  • SAM acceleration card is installed on the appliance
  • Carrier Grade NAT (CGN) is configured
  • Security Gateway is configured in Monitor Mode (per sk101670)
  • 6in4 tunnel (SIT interface) is configured
  • Some lines in the $FWDIR/boot/modules/fwkern.conf file are commented out (refer to sk106309).

The following types of traffic are not load-balanced by the CoreXL Dynamic Dispatcher (this traffic will always be handled by the same CoreXL FW instance):

  • VoIP
  • VPN encrypted packets

Additional known limitations:

 

(7) FAQ

  • When should I enable CoreXL Dynamic Dispatcher?

    Note: The thresholds given in this answer are approximate and should be suitable for majority of environments.

    Administrator should enable CoreXL Dynamic Dispatcher if:
    1. A CoreXL FW instance consumes its CPU core at 85% (and above) even for 1 second.
    2. Other CoreXL FW instances consume their CPU cores at 75% (and below) - i.e., the difference in CPU consumption between overloaded and normally loaded CoreXL FW instances is 10% and more.
    In addition, refer to "Limitations" section.
  • When will Security Gateway benefit from enabling CoreXL Dynamic Dispatcher?
    • When CPU load is not properly balanced, Dynamic Dispatcher will distribute the load equally between the CPU cores.
    • Dynamic Dispatcher will mostly benefit on machines with large number of CPU cores (multi core machines). On machine with small number of CPU cores, the static hash distribution of traffic is good enough, and the benefit of enabling Dynamic Dispatcher is low.

    Additional common scenarios, in which enabling CoreXL Dynamic Dispatcher will improve Security Gateway performance, are cases when traffic is dropped/delayed due CPU load on Security Gateway.
    For example, during traffic peaks, during policy installation.
    Refer to "Monitoring CoreXL load distribution" section.
  • Can CoreXL assign some connections statically and some connections dynamically?
    By default, CoreXL assigns all connections statically.
    If you fully enable the CoreXL Dynamic Dispatcher, then CoreXL assigns all connections dynamically.
    In addition, refer to "Limitations" section.
  • Does affinity of interfaces change when enabling CoreXL Dynamic Dispatcher?
    CoreXL Dynamic Dispatcher does not affect affinity of interfaces.
    For information about affinity of interfaces when CoreXL is enabled, refer to sk98737 - ATRG: CoreXL - section "(4) Architecture".

 

  • Performance Tuning Administration Guide (R77.X, R80.10)

 

Give us Feedback
Please rate this document
[1=Worst,5=Best]
Comment