The information you are about to copy is INTERNAL!
DO NOT share it with anyone outside Check Point.
Various traffic issues on cluster due to FWD daemon taking all slots on cluster subscriber list
Technical Level
Solution ID
sk109596
Technical Level
Product
ClusterXL, Cluster - 3rd party
Version
R77, R77.10, R77.20, R77.30, R80.10
OS
Gaia, SecurePlatform 2.6, Crossbeam XOS, IPSO 6.2
Platform / Model
All
Date Created
12-Jan-2016
Last Modified
12-Nov-2019
Symptoms
Various issues with traffic through a cluster:
Random traffic outages
Dynamic / Static routing stops working
Users are not recognized by Identity Awareness
All Identity Awareness commands on cluster members show: "daemon did not respond or not running!"
/var/log/messages file on cluster members repeatedly shows: FW-1: cxl_status_update_register: registering Name_of_Process not successfull because too many processes are registered.
Examples:
FW-1: cxl_status_update_register: registering pdpd not successful because too many processes are registered.
FW-1: cxl_status_update_register: registering pepd not successful because too many processes are registered.
FW-1: cxl_status_update_register: registering routed not successful because too many processes are registered.
FW-1: cxl_status_update_register: registering cprd PID 22279 not successful because too many processes are
FW-1: cxl_status_update_register: registering cvpnd not successful because too many processes are registered.
FW-1: cxl_status_update_register: registering ted not successful because too many processes are registered.
FW-1: cxl_status_update_register: registering gated not successful because too many processes are registered.
registered.
After enabling generation of core dump files per sk92764 / sk53363, core dump files for various processes might be generated in the /var/log/dump/usermode/ directory:
fwd
routed
pdpd
pepd
cvpnd
gated
Output of "cphaconf debug_data" command (under cluster debug 'fw ctl debug -m cluster + conf') on cluster members shows that cluster subscriber list is filled only with instances of FWD daemon (refer to sk93306 - Chapter 'ClusterXL Monitoring and Troubleshooting' - section "'cphaconf' command" - "cphaconf debug_data").
Example:
;---- Status subscribers ----
;
;Subscriber: 0 pid 5823 sig 18 desc fwd PID 5823;
;Subscriber: 1 pid 760 sig 1 desc fibmgrd;
;Subscriber: 2 pid 7268 sig 18 desc fwd PID 7268;
;Subscriber: 3 pid 5831 sig 18 desc fwd PID 5831;
;Subscriber: 4 pid 15189 sig 18 desc fwd PID 15189;
;Subscriber: 5 pid 10097 sig 18 desc fwd PID 10097;
;Subscriber: 6 pid 15839 sig 18 desc fwd PID 15839;
;Subscriber: 7 pid 23038 sig 18 desc fwd PID 23038;
;Subscriber: 8 pid 4106 sig 18 desc fwd PID 4106;
;Subscriber: 9 pid 29513 sig 18 desc fwd PID 29513;
;Subscriber: 10 pid 17118 sig 18 desc fwd PID 17118;
;Subscriber: 11 pid 27227 sig 18 desc fwd PID 27227;
;Subscriber: 12 pid 3092 sig 18 desc fwd PID 3092;
;Subscriber: 13 pid 23341 sig 18 desc fwd PID 23341;
;Subscriber: 14 pid 15326 sig 18 desc fwd PID 15326;
;Subscriber: 15 pid 13008 sig 18 desc fwd PID 13008;
Cause
FWD daemon registers itself too many times to cluster status updates. As a result, it takes all the available 16 registration slots, preventing other daemon from registering.