Table of Contents:
Background
When a User Space process performs an illegal operation (e.g., trying to access a memory cell that does not belong to the allocated range of memory cells), then the Linux kernel closes the process, and the process crashes from the user's point of view.
Obviously, in such an unexpected state, it is not possible to create any logs about the exact events that led to the crash.
To identify the root cause of such crashes, developers invented a simple mechanism that allows the user to dump (write a file to the disk) the whole content of the memory that was used by the User Space process.
Such files are called Core Dump files.
Introduction
Generation of User Mode core dump files on Linux OS has to be enabled each time the machine boots using the 'ulimit -c unlimited' command (for more information about the shell built-in command ulimit, refer to the shell manual page).
To make this process automatic, the following mechanism was implemented in Check Point's SecurePlatform OS ( the predecessor of Gaia OS):
um_core enable command generates the /etc/sysconfig/enable_cores file.
um_core disable command deletes the /etc/sysconfig/enable_cores file.
/etc/initscript start-up script checks whether the /etc/sysconfig/enable_cores file exists. If this file exists, then it runs the ulimit -c unlimited command.
Note: For more information about the um_core command, refer to sk18307: Enabling the generation of User Mode core dump files on SecurePlatform OS.
In Gaia OS, the /etc/sysconfig/enable_cores file always exists on the system.
How it works in Gaia OS
Starting in R75.47 and R76, the mechanism that configures the Linux kernel to generate and process the User Mode core dump files was improved.
Starting in R75.47 and R76, the Core Dump Manager (CDM) utility was added to manage User Mode core dump files generated in Gaia OS. Currently, these two policies are enforced by CDM:
- Limit the total amount of disk space used by core dump files.
- Limit the number of core dump files per process.
Both these limits can be configured by the user in Clish or in Gaia Portal.
When the User Mode core dump file should be generated, the Linux kernel checks the /proc/sys/kernel/core_pattern file, which defines how to dump the core file. The default content of this file is:
|/etc/coredump/compress.sh /var/log/dump/usermode/%e.%p.core
This tells the Linux kernel to dump the core file into /var/log/dump/usermode/ProcessName.PID.core, and pipe this core file to the /etc/coredump/compress.sh shell script (the process's executable name and its PID are provided as arguments).
The /etc/coredump/compress.sh shell script performs the following:
- Checks whether CDM is enabled.
- If CDM is disabled, then this script exits. The core dump file is not processed in any way. Core dump files should be located in
/var/log/dump/usermode/ProcessName.PID.core
- If CDM is enabled, then this script continues to the next step.
- If the
/etc/coredump/proclist.conf file exists on the machine, then the process, for which the core file was dumped, is checked against this list.
- For each process listed in the
/etc/coredump/proclist.conf file, the core dump file will be GZIPed (compressed) in /var/log/dump/usermode/.
- Core dump files from processes that are not listed in the file are not compressed.
- Runs the CDM (
/bin/cdm) that performs the necessary operations based on its configuration file - /etc/coredump/cdm.conf.
Important Note: Users should not edit the /etc/coredump/cdm.conf file manually. This file must hold the same limits that were configured in the Gaia OS database /config/db/initial.
Controlling Core Dump Manager (CDM) in Gaia Portal
- Open the Gaia Portal in your web browser.
Note: Refer to sk92668 (Browsers supported to work with Gaia Portal).
- Obtain the lock over the configuration database (click on the padlock icon at the top).
- Go to the System Maintenance pane - go to the Core Dump section.
- Check the box Enable Core Dumps.
- Set the desired limits.
- Click on the Apply button.
- There is no need to reboot. These settings also apply to processes that are already running.
Example:
Controlling Core Dump Manager (CDM) in the Command Line
Important Note:
After any changes, the Gaia OS configuration must be saved:
HostName> save config
- Check status of core dumping:
HostName> show core-dump status
Example:
Core dumps enabled
- Check the current total disk space limit:
HostName> show core-dump total
Example:
Total core dumps space limit is 1000 Megabytes
- Check the current amount of core dump files per process:
HostName> show core-dump per_process
Example:
Each executable is allowed to have 2 dump files
- Configure the total disk space limit:
HostName> set core-dump total Total_Disk_Space_Limit_in_MB
HostName> save config
- Configure the amount of core dump files per process:
HostName> set core-dump per_process Amount_of_Core_Dump_Files_per_Process
HostName> save config
- Enable dumping of core files:
HostName> set core-dump enable
HostName> save config
Note: There is no need to reboot. These settings also apply to processes that are already running.
- Disable core dumping:
HostName> set core-dump disable
HostName> save config
Note: There is no need to reboot. These settings also apply to processes that are already running.
- Check core dumping settings (must match):
[Expert@HostName]# grep cdm /config/db/initial
Example:
cdm:per_exec 2
cdm:total 1000
cdm:enable 1
[Expert@HostName]# cat /etc/coredump/cdm.conf
Example:
enable 1
total 1000
per_exec 2
Notes
- The CDM binary file is located in
/bin/cdm.
- Executing either of these two commands triggers the execution of CDM to ensure system consistency:
HostName> set core-dump total VALUE_in_MBHostName> set core-dump per_process VALUE
- CDM has a strict rule (hard coded) to erase core dump files if free disk space is below 200 MB.
- If the binary file that has crashed is the CDM executable file
/bin/cdm, then the CDM will abort its execution to prevent a loop of crashes and generation of core dump files.
