Support Center > Search Results > SecureKnowledge Details
SmartLog indexes fill up the disk space
Symptoms
  • Old SmartLog indexes and SmartView Tracker logs are deleted.

  • SmartLog indexes are deleted on some Domains, while not deleted on other Domains.

  • "There is not enough disk space available on the correlation unit. Please free disk space" a correlation unit error in SmartView Monitor.
Cause

SmartLog periodically checks the amount of disk space left. When the limit is reached, SmartLog deletes its old indexes.

The size is tested on the $SMARTLOGDIR/data/ folder. SmartLog consumption of disk space may also influences Log deletion policy.


In R77 and R75.47 the problem does not exist since index files are automatically deleted when log files are removed. There is an option in SmartDashboard R77 and above, in the network object to limit index size by days or disk size.


Solution

Background

SmartLog parameters that are relevant to this issue can be configured in the following file on Security Management Server:

  • in pre-R76:

    $SMARTLOGDIR/conf/smartlog_settings.txt

  • in R76:

    $SMARTLOGDIR/smartlog_settings.txt

    Note: In R76, SmartLog parameters are located in two files:

    • $SMARTLOGDIR/conf/smartlog_settings.conf (replaced during upgrades)
    • $SMARTLOGDIR/smartlog_settings.txt

 

Configuring the minimal free disk space before deleting old indexes

Follow these steps on Security Management Server:

  1. Backup the current file:

    • In pre-R76:

      [Expert@HostName]# cp $SMARTLOGDIR/conf/smartlog_settings.txt $SMARTLOGDIR/conf/smartlog_settings.txt_ORIGINAL

    • In R76:

      [Expert@HostName]# cp $SMARTLOGDIR/smartlog_settings.txt $SMARTLOGDIR/smartlog_settings.txt_ORIGINAL


  2. Edit the current file:

    • In pre-R76:

      [Expert@HostName]# vi $SMARTLOGDIR/conf/smartlog_settings.txt

    • In R76:

      [Expert@HostName]# vi $SMARTLOGDIR/smartlog_settings.txt


  3. Change:

    from

    min_disk_space (10240)

    to

    min_disk_space (DESIRED_MIN_DISK_SPACE_in_MB)

    Notes:

    • The DESIRED_MIN_DISK_SPACE_in_MB can be any number between 2048 and some value lower than the total disk size.

    • On R76, if this parameter does not exist in $SMARTLOGDIR/smartlog_settings.txt, then add it manually, so it will override the value in $SMARTLOGDIR/conf/smartlog_settings.conf.

  4. Restart SmartLog services:

    [Expert@HostName]# smartlogstop
    [Expert@HostName]# smartlogstart


Log files Storage Policy

  • Since log files and SmartLog indexes share the same disk partition, their limits refer to the same storage.

  • By default, SmartLog limit is reached first. SmartLog limit is 10G while log files limit is 20M.

  • If defaults were changed so Log files limit is higher, than log files will be deleted before the indexes do.

  • There is also an option to control max size of indexes using the max_disk_space_usage parameter that limits the maximum disk size you allow indexes to use (in MegaBytes).
    This parameter does not exist by default and should be added to $SMARTLOGDIR/smartlog_settings.txt (on R76 and above) or $SMARTLOGDIR/conf/smartlog_settings.txt


Notes for Multi-Domain Management (MDM) Server:

  • The same configuration steps should be implemented in the context of each Domain Management Server.

  • Manually navigate to each Domain's Smartlog directory, as $SMARTLOGDIR does not currently work in MDM environment.

  • Important: Since all Domains are probing the same folder in order to verify that they are not over the free disk space limit, there is a race condition - which Domain will delete its index first.

    To avoid the race condition, it is possible to configure, per Domain, the maximum disk size you allow it to use (in MegaBytes). The configuration is done using the max_disk_space_usage parameter in the smartlog_setting.txt file.

    This is different from setting the min_disk_space parameter, which probes the free disk size left on the machine, which is actually shared between all Domains.

    Example:
    Setting max_disk_space_usage (1000) will force the Domain to use only 1 GB of disk space, and then the Domain will start deleting its old indexes.

  • max_disk_space_usage parameter can be used regardless of MDM.

  • max_index_size is an internal parameter to set the size of a core within an index. Increasing this parameter would improve search speed for rare logs but decrease speed with queries that return many results.
    To limit the size of the index, use the max_disk_space_usage parameter.

 

Related solutions

This solution is about products that are no longer supported and it will not be updated
Applies To:
  • This SK replaces sk93164

Give us Feedback
Please rate this document
[1=Worst,5=Best]
Comment