Support Center > Search Results > SecureKnowledge Details
'kernel: neighbour table overflow' appears repeatedly in /var/log/messages files Technical Level
Symptoms
  • 'kernel: neighbour table overflow' message appears repeatedly in /var/log/messages files.
Cause

ARP cache is overflowing. 
Most likely reason - too much traffic on the network (generated by some application, by some hosts, or by related factors).


Solution

Table of Contents:

  1. Background
  2. Explanation
  3. Action plan
  4. Instructions for Gaia OS
  5. Instructions for SecurePlatform OS
  6. Related documentation
  7. Related solutions

 

(1) Background

The problem is not related to Check Point products, but to Linux OS.

The solution to the problem is to increase the threshold levels for ARP cache.

Note: If you have additional questions about ARP cache mechanism and its thresholds, then refer to articles and books available on the Internet.

 

(2) Explanation

This section provides a basic explanation about the relevant Linux Kernel parameters.

  • These 3 parameters are defined in the Linux Kernel Code in the header file /include/net/neighbour.h as integer.

  • As of R80.30, Gaia Portal accepts maximal value of 131072. In prior versions that value was 16384.

  • gc_thresh1

    The minimum number of entries to keep in the ARP cache.
    The garbage collector will not run if there are fewer than this number of entries in the cache. Note: On kernel 2.6.18, this value was ignored, meaning the garbage collector would activate regardless of the number of entries in the cache.

  • gc_thresh2

    The soft maximum number of entries to keep in the ARP cache.
    The garbage collector will allow the number of entries to exceed this for 5 seconds before collection will be performed.

  • gc_thresh3

    The hard maximum number of entries to keep in the ARP cache.
    The garbage collector will always run if there are more than this number of entries in the cache.

    In order for the garbage collector to work properly, and not to overload the machine with garbage collections, when changing the gc_thresh3 parameter, user should (note: does not have to) change the gc_thresh2 and gc_thresh1 parameters accordingly.

 

(3) Action plan

Follow these guidelines (for both SecurePlatform OS and Gaia OS):

  1. Find the threshold levels, at which the error messages about "neighbour table overflow" does not appear anymore.

  2. Set the new threshold levels permanently.

  3. Addition note: Check (with "arp -an" command) if ARP table on Security Gateway / Cluster member contains multiple entries with MAC address of your ISP device. In such case, consider changing the default route from "route through interface (external, leading to ISP)" to "route through next hop IP address (of ISP device)" (refer to "related documentation" section).

 

(4) Instructions for Gaia OS

Gaia Portal / Gaia Clish will override any settings placed in the /etc/sysctl.conf file. Any changes made to this file do not take effect after a reboot.

To configure threshold level for ARP cache on Gaia OS:

Note: As of R80.30 2.6.18, Gaia OS accepts maximal value of 131072. In prior versions that value was 16384.

  • In Gaia Portal:

    Go to Network Management section - click on ARP page - go to section ARP Table Settings section - enter the desired value in Maximum Entries field.

  • In Gaia Clish:

    HostName> set arp table cache-size <Number_of_Entries>
    HostName> save config

To check the current threshold level for ARP cache on Gaia OS::

  • In Gaia Clish:

    HostName> show arp table cache-size
  • In Expert mode:

    [Expert@HostName:0]# dbget ip:arp:cache_size

Notes:

  • Once set in the Gaia Portal / Gaia Clish, the settings will survive a reboot.
  • Settings are applies immediately (reboot, restart of any services, policy installation are not required)

The Maximum Entries value in the Gaia Portal corresponds to gc_thresh3 parameter in Linux kernel.

Note: In Gaia OS, the value of gc_thresh1 and the value of gc_thresh2 are automatically determined by the value of gc_thresh3 value:

  • value of gc_thresh1 is 1/8 the value of gc_thresh3
  • value of gc_thresh2 is 1/2 the value of gc_thresh3

 

(5) Instructions for SecurePlatform OS

  1. Check the current threshold levels:

    [Expert@HostName]# cat /proc/sys/net/ipv4/neigh/default/gc_thresh1
    [Expert@HostName]# cat /proc/sys/net/ipv4/neigh/default/gc_thresh2
    [Expert@HostName]# cat /proc/sys/net/ipv4/neigh/default/gc_thresh3

    By default, it should return such values as 128 or 256 or 512 or 1024.

  2. Increase the current levels to the next value - multiply the current value by 2:

    • if gc_thresh1=128, then set gc_thresh1=256
    • if gc_thresh2=256, then set gc_thresh2=512
    • if gc_thresh3=512, then set gc_thresh3=1024

    To increase the threshold levels for ARP cache on-the-fly, run:

    [Expert@HostName]# echo 256 > /proc/sys/net/ipv4/neigh/default/gc_thresh1
    [Expert@HostName]# echo 512 > /proc/sys/net/ipv4/neigh/default/gc_thresh2
    [Expert@HostName]# echo 1024 > /proc/sys/net/ipv4/neigh/default/gc_thresh3

    or

    [Expert@HostName]# sysctl -w net.ipv4.neigh.default.gc_thresh1=256
    [Expert@HostName]# sysctl -w net.ipv4.neigh.default.gc_thresh2=512
    [Expert@HostName]# sysctl -w net.ipv4.neigh.default.gc_thresh3=1024

    Important Note: Keep increasing the values (multiply by 2), until the error messages about "neighbour table overflow" disappear.

  3. To make these changes permanent:

    1. Modify the /etc/sysctl.conf file to include the following lines (pay attention to spaces around the "=" sign):

      net.ipv4.neigh.default.gc_thresh1 = <NEW_VALUE_FOR_GC_THRESH1>
      net.ipv4.neigh.default.gc_thresh2 = <NEW_VALUE_FOR_GC_THRESH2>
      net.ipv4.neigh.default.gc_thresh3 = <NEW_VALUE_FOR_GC_THRESH3>
    2. Reboot the machine:

      [Expert@HostName]# reboot

      Important Note: Do NOT run the "sysctl -p /etc/sysctl.conf" command to check that there are no mistakes in the /etc/sysctl.conf file. This command will load all kernel parameters from the /etc/sysctl.conf file - including the IPv4 Forwarding net.ipv4.ip_forward = 0. Meaning that the moment this command is issued, no traffic will be passed through this machine.

    3. After reboot, verify whether the new values were preserved:

      [Expert@HostName]# cat /proc/sys/net/ipv4/neigh/default/gc_thresh1
      [Expert@HostName]# cat /proc/sys/net/ipv4/neigh/default/gc_thresh2
      [Expert@HostName]# cat /proc/sys/net/ipv4/neigh/default/gc_thresh3

    4. If there was a mistake in the /etc/sysctl.conf file, edit the file, and reboot the machine again.

 

(6) Related documentation

 

Give us Feedback
Please rate this document
[1=Worst,5=Best]
Comment