Support Center > Search Results > SecureKnowledge Details
"BUG: soft lockup - CPU#x stuck for 10s! [fw_worker_Z:...]" appears repeatedly in /var/log/messages file Technical Level
Symptoms
  • /var/log/messages file repeatedly shows soft lockups for "fw_worker" processes similar to these:

    kernel: BUG: soft lockup - CPU#x stuck for 10s! [fw_worker_Z:...]
    ... ...
    kernel: Call Trace:
    kernel: [...] :fw_0:fwmultik_enqueue_data+...
    kernel: [...] :fw_Z:fwmultik_msg_send_do+...
    kernel: [...] :fw_Z:fwmultik_msgblock_get_write_ptr+...
    kernel: [...] :fw_Z:fwmultik_msg_send_to_all+...
    kernel: [...] :fw_Z:fwlddist_flush_msg_block+...
    kernel: [...] :fw_Z:ld_getid+...
    kernel: [...] :fw_Z:fwlddist_put+...
    ... ...
    
  • /var/log/messages file repeatedly shows soft lockups for "fw_worker" processes similar to these:

    kernel: BUG: soft lockup - CPU#x stuck for 10s! [fw_worker_Z:...]
    ... ...
    kernel: Call Trace:
    kernel: [...] :fw_0:fwmultik_pending_queue_try_flush_kernel+...
    kernel: [...] :fw_0:fwmultik_enqueue_data_kernel+...
    kernel: [...] :fw_0:fwmultik_enqueue_data+...
    kernel: [...] :fw_Z:fwmultik_msg_send_do+...
    kernel: [...] :fw_Z:fwmultik_msgblock_get_write_ptr+...
    kernel: [...] :fw_Z:fwmultik_msg_send_to_all+...
    kernel: [...] :fw_Z:fwlddist_flush_msg_block+...
    kernel: [...] :fw_Z:ld_getid+...
    kernel: [...] :fw_Z:fwlddist_put+...
    ... ...
    
  • /var/log/messages file repeatedly shows soft lockups for "fw_worker" processes similar to these:

    kernel: BUG: soft lockup - CPU#x stuck for 10s! [fw_worker_Z:...]
    ... ...
    kernel: Call Trace:
    kernel: [...] :fw_Z:fwmultik_queue_async_dequeue_cb+...
    kernel: [...] :fw_Z:kernel_thread_run+...
    kernel: [...] autoremove_wake_function+...
    kernel: [...] :fw_Z:kiss_kthread_run+...
    kernel: [...] :fw_Z:plat_run_thread+...
    kernel: [...] child_rip+...
    kernel: [...] :fw_Z:plat_run_thread+...
    kernel: [...] child_rip+...
    ... ...
    
  • When VLAN interfaces are configured, the /var/log/messages file repeatedly shows:

    ;FW-1: _fwhamultik_set_mem: changing IF_UNIQUE(i) from X to Y(changed by [fwhaif.c:N]);
    ;FW-1: _fwhamultik_set_mem: changing IF_UNIQUE(ifn) from Y to X(changed by [fwhaif.c:M]);
    
Cause

A soft lockup isn't necessarily anything 'crashing', it is the symptom of a task or kernel thread using and not releasing a CPU for a longer period of time than allowed; in Check Point the default fault is 10 seconds. This is adjustable by editing the value within /proc/sys/kernel/softlockup_thresh and requires a reboot when change. Changing this value will only extend the threshold for when a softlock trace is triggered. The technical reasons behind a softlock involves CPU interrupts and nmi-watchdog while the CPU is under heavy load. For each online CPU on the system, a watchdog process gets created which will send a non maskable interrupt (NMI) to all CPUs in the system who in turn will (non-intrusively) print the stacktraces of their currently running tasks. Under normal circumstances those messages may go away if the load on the CPU is decreased. The stack traces of those tasks are supposed to give us a first idea what the tasks were doing at the time of the issue. However, to be able to examine the root cause behind the messages, a kernel dump would be needed.


Solution
Note: To view this solution you need to Sign In .