Support Center > Search Results > SecureKnowledge Details
"NAT Hide failure - there are currently no available ports for hide operation. Please refer to sk156852." error in SmartConsole / SmartLog / SmartView Tracker Technical Level
Symptoms
  • 'Message information' label in the log card shows "NAT Hide failure - there are currently no available ports for hide operation. Please refer to sk156852."
Solution

NAT Exhausted Pool

This feature is implemented starting from:

 


For R80.10, R80.20, and R80.30:

What is a "pool"?

  • When a Security Gateway allocates a source port for a Hide NAT operation, it can allocate the same port for different connections, as long as certain properties of the connections are different.

    These properties are: IP protocol, Hide Source IP address, Destination IP address.

    This 3-tuple is called a NAT pool.

  • Two connections can get the same port, if their NAT pools are different (at least one of the values is different). For example, the 2 NAT pools <6, 1.1.1.1, 2.2.2.2> and <6, 1.1.1.1, 3.3.3.3> are different because their destination IP addresses are different.

What is the format of the "NAT Exhausted Pool" label?

<IP protocol>, <Hide Source IP address>, <Destination IP address>

Example log card of "NAT exhausted pool"

  • IP protocol: 6 (TCP)
  • Hide Source IP address: 110.16.4.80
  • Destination IP address: 110.16.4.84

To solve this issue, configure "Hide behind range" as described in sk140432.


For R80.40 and higher:

What is a "pool"?

  • When a Security Gateway allocates a source port for a Hide NAT operation, it can allocate the same port for different connections, as long as certain properties of the connections are different.

    These properties are: IP protocol, Hide Source IP address, Destination IP address, and Destination Port (Destination Port is not always used, as explained below).

    This 4-tuple is called a NAT pool.

  • Two connections can get the same port if their pools are different (at least one of the values is different). For example, the 2 NAT pools <6, 1.1.1.1, 2.2.2.2, 443> and <6, 1.1.1.1, 3.3.3.3, 443> are different because their destination IP addresses are different.

What is the format of the "NAT Exhausted Pool" label?

The label has 2 format options:

  • If we use GNAT, we use the 4-tuple:

    <IP Protocol, Hide Source IP address, Destination IP address, Destination Port>

  • If we use static port allocation, we use the 3-tuple:

    <IP Protocol, Hide Source IP address, Destination IP address>

Example log for "NAT exhausted pool"

  • IP protocol: 6 (TCP)
  • Hide Source IP address: 110.16.4.80
  • Destination IP address: 110.16.4.84
  • Destination port: 888

The use of a destination port in the NAT pool

  • If a Security Gateway uses static NAT port allocation, the destination port ('dport') is not part of the pool.

  • If a Security Gateway uses GNAT, the destination port ('dport') is part of the pool, but it can still be 0, if the specific destination port is not present in the xlate_use_dport_services kernel table.

The kernel table 'xlate_use_dport_services'

  • The properties of the kernel table 'xlate_use_dport_services' are configured in the file table.def - including the destination ports which are used in the NAT pool (for all other destination ports, the value 0 is used).

  • For general information about the table.def file and its location, see sk98339.

  • By default, the kernel table 'xlate_use_dport_services' contains common ports and protocols, for which a Security Gateway might exhaust its available NAT ports.

  • Generally speaking, it is not recommended to modify this kernel table, except in this case:

    If you have NAT port exhaustion for connections whose NAT pools differ only in the destination port, and those destination ports are not present in the kernel table, then adding them to the kernel table might solve the NAT port exhaustion.

    For example, a server that handles two services, one on dport 555 and one on dport 556. Both ports are UDP. In this case you should add these values in the configuration of the kernel table 'xlate_use_dport_services' in the applicable table.def file: <555, 17>, <556, 17>

    Therefore, in this case, instead of having one NAT pool for this server, you get two NAT pools, which contain twice the number of NAT ports than would one pool.

When a NAT pool is exhausted at 85% or above, the Security Gateway sends an alert log to notify that the NAT pool is about to be exhausted.

Example log:

To resolve this issue:

  1. Configure "Hide behind range" as described in sk140432.
  2. If necessary, add the dport to xlate_use_dport_services, as described above.
  3. Configure the applicable value of nat_limit as described in sk36708.

Important Notes:

  • Never add ports that might be used for data connections (for example, FTP).

  • Modifying the kernel table without rebooting the Security Gateway might cause temporary collisions in connection links, until the old connections are deleted.


How to review the statistics of the NAT pools in CPView

Run the cpview command on the Security Gateway -> click Advanced -> NAT.

Note - CPview refreshes the statistics every one minute and every two seconds when you stay on the tab.

This page opens:

Columns in the row marked in red:

  • Instance: The CoreXL Firewall instance, on which this NAT pool is used. In this example, all CoreXL Firewall instances are used.
  • Hide IP: The translated Hide IP address after the NAT translation.
  • Dst IP: The destination IP address.
  • Dport: The destination port. In this example, it is 0.
  • Proto: The IP protocol of the connection. In this case, 6 represents the TCP connection.
  • Port Usage: The number of ports used for this specific pool.
  • Capacity: The total number of NAT ports that can be used for each NAT pool.
  • Used: The percentage of NAT ports used out of the total capacity.

The different type of ports:

  • High: Ports for general use, from 10,000 to 60,000.
  • Low: Reserved ports for services that require ports from 600 to 1024.
  • Extra: Reserved ports for VoIP connections, from 60,000 and above.

Note - CPview shows only the 2 busiest NAT pools.

 

 

How to review the statistics of NAT pools with SNMP

Note - SNMP statistics for NAT pools are supported only in R80.40 and higher.

  1. Enable SNMP on the Security Gateway / each Cluster Member:

    For more information, see the Gaia Administration Guide for your version.

    1. From the Expert mode, go to Gaia Clish:

      clish

    2. Enable the SNMP v1/v2:

      set snmp agent-version any

    3. Configure the SNMP community as read-only:

      set snmp community public read-only

    4. Start the SNMP Agent:

      set snmp agent on

    5. Save the changes:

      save config

  2. Make sure CPView service is running:

    cpview -s stat

    If CPView service is not running, then run this command:

    cpview -s on

  3. For NAT statistics queries, you can get all the information that you see in CPview from SNMP.

    IPv4 Hide NAT pools:

    • Concurrent Connections:

      snmpwalk -Os -c public -v 2c localhost 1.3.6.1.4.1.2620.1.56.1301.1

    • Connection Session Rate:

      snmpwalk -Os -c public -v 2c localhost 1.3.6.1.4.1.2620.1.56.1301.2

      Shows how much "NATed" connections opened in the last update interval.

    • High Ports:

      snmpwalk -Os -c public -v 2c localhost 1.3.6.1.4.1.2620.1.56.1301.3

    • Low Ports:

      snmpwalk -Os -c public -v 2c localhost 1.3.6.1.4.1.2620.1.56.1301.4

    • Extra Ports:

      snmpwalk -Os -c public -v 2c localhost 1.3.6.1.4.1.2620.1.56.1301.5

    IPv6 Hide NAT Pools:

    • Concurrent Connections:

      snmpwalk -Os -c public -v 2c localhost 1.3.6.1.4.1.2620.1.56.1301.6

    • Connection Session Rate:

      snmpwalk -Os -c public -v 2c localhost 1.3.6.1.4.1.2620.1.56.1301.7

      Shows how much "NATed" connections opened in the last update interval.

    • High Ports:

      snmpwalk -Os -c public -v 2c localhost 1.3.6.1.4.1.2620.1.56.1301.8

    • Low Ports:

      snmpwalk -Os -c public -v 2c localhost 1.3.6.1.4.1.2620.1.56.1301.9

    • Extra Ports:

      snmpwalk -Os -c public -v 2c localhost 1.3.6.1.4.1.2620.1.56.1301.10

    For example:

    CPview shows two high port pools:

    You can query all SNMP counters at the same time:

    [Expert@Sec_GW:0]# snmpwalk -Os -c public -v 2c localhost 1.3.6.1.4.1.2620.1.56.1301.1
    enterprises.2620.1.56.1301.1.0 = Gauge32: 127      Explanation: concurrent connections
    [Expert@Sec_GW:0]#
    [Expert@Sec_GW:0]# snmpwalk -Os -c public -v 2c localhost 1.3.6.1.4.1.2620.1.56.1301.2
    enterprises.2620.1.56.1301.2.0 = Gauge32: 1        Explanation: connection session rate
    [Expert@Sec_GW:0]#
    [Expert@Sec_GW:0]# snmpwalk -Os -c public -v 2c localhost 1.3.6.1.4.1.2620.1.56.1301.3
    enterprises.2620.1.56.1301.3.1.1.1.0 = Gauge32: 1                     Explanation: Index of pool #1 
    enterprises.2620.1.56.1301.3.1.1.2.0 = Gauge32: 2                     Explanation: Index of pool # 2  
    enterprises.2620.1.56.1301.3.1.2.1.0 = STRING: "0"                    Explanation: CoreXL FW instance of pool # 1
    enterprises.2620.1.56.1301.3.1.2.2.0 = STRING: "0"                    Explanation: CoreXL FW instance of pool # 2
    enterprises.2620.1.56.1301.3.1.3.1.0 = STRING: "4.4.4.4"              Explanation: Hide IP address of pool # 1
    enterprises.2620.1.56.1301.3.1.3.2.0 = STRING: "172.16.4.166"         Explanation: Hide IP address of pool # 2
    enterprises.2620.1.56.1301.3.1.4.1.0 = STRING: "200.1.1.2"            Explanation: Destination IP address of pool # 1
    enterprises.2620.1.56.1301.3.1.4.2.0 = STRING: "184.30.25.223"        Explanation: Destination IP address of pool # 2
    enterprises.2620.1.56.1301.3.1.5.1.0 = Gauge32: 0                     Explanation: Destination port of pool # 1
    enterprises.2620.1.56.1301.3.1.5.2.0 = Gauge32: 0                     Explanation: Destination port of pool # 2
    enterprises.2620.1.56.1301.3.1.6.1.0 = Gauge32: 17                    Explanation: Protocol of pool # 1
    enterprises.2620.1.56.1301.3.1.6.2.0 = Gauge32: 6                     Explanation: Protocol of pool # 2
    enterprises.2620.1.56.1301.3.1.7.1.0 = Gauge32: 42                    Explanation: Port usage of pool # 1
    enterprises.2620.1.56.1301.3.1.7.2.0 = Gauge32: 1                     Explanation: Port usage of pool # 2
    enterprises.2620.1.56.1301.3.1.8.1.0 = Gauge32: 16667                 Explanation: Capacity of pool # 1
    enterprises.2620.1.56.1301.3.1.8.2.0 = Gauge32: 16667                 Explanation: Capacity of pool # 2
    enterprises.2620.1.56.1301.3.1.9.1.0 = Counter64: 0                   Explanation: Used of pool # 1
    enterprises.2620.1.56.1301.3.1.9.2.0 = Counter64: 0                   Explanation: Used of pool # 2
    

    You can query each SNMP counter separately:

    [Expert@Sec_GW:0]# snmpwalk -Os -c public -v 2c localhost 1.3.6.1.4.1.2620.1.56.1301.3.1.1
    enterprises.2620.1.56.1301.3.1.1.1.0 = Gauge32: 1
    enterprises.2620.1.56.1301.3.1.1.2.0 = Gauge32: 2
    [Expert@Sec_GW:0]#
    [Expert@Sec_GW:0]# snmpwalk -Os -c public -v 2c localhost 1.3.6.1.4.1.2620.1.56.1301.3.1.2
    enterprises.2620.1.56.1301.3.1.2.1.0 = STRING: "0"
    enterprises.2620.1.56.1301.3.1.2.2.0 = STRING: "0"
    [Expert@Sec_GW:0]#
    [Expert@Sec_GW:0]# snmpwalk -Os -c public -v 2c localhost 1.3.6.1.4.1.2620.1.56.1301.3.1.3
    enterprises.2620.1.56.1301.3.1.3.1.0 = STRING: "4.4.4.4"
    enterprises.2620.1.56.1301.3.1.3.2.0 = STRING: "172.16.4.166"
    [Expert@Sec_GW:0]#
    [Expert@Sec_GW:0]# snmpwalk -Os -c public -v 2c localhost 1.3.6.1.4.1.2620.1.56.1301.3.1.4
    enterprises.2620.1.56.1301.3.1.4.1.0 = STRING: "200.1.1.2"
    enterprises.2620.1.56.1301.3.1.4.2.0 = STRING: "184.30.25.223"
    [Expert@Sec_GW:0]#
    [Expert@Sec_GW:0]# snmpwalk -Os -c public -v 2c localhost 1.3.6.1.4.1.2620.1.56.1301.3.1.5
    enterprises.2620.1.56.1301.3.1.5.1.0 = Gauge32: 0
    enterprises.2620.1.56.1301.3.1.5.2.0 = Gauge32: 0
    [Expert@Sec_GW:0]#
    [Expert@Sec_GW:0]# snmpwalk -Os -c public -v 2c localhost 1.3.6.1.4.1.2620.1.56.1301.3.1.6
    enterprises.2620.1.56.1301.3.1.6.1.0 = Gauge32: 17
    enterprises.2620.1.56.1301.3.1.6.2.0 = Gauge32: 6
    [Expert@Sec_GW:0]#
    [Expert@Sec_GW:0]# snmpwalk -Os -c public -v 2c localhost 1.3.6.1.4.1.2620.1.56.1301.3.1.7
    enterprises.2620.1.56.1301.3.1.7.1.0 = Gauge32: 42
    enterprises.2620.1.56.1301.3.1.7.2.0 = Gauge32: 1
    [Expert@Sec_GW:0]#
    [Expert@Sec_GW:0]# snmpwalk -Os -c public -v 2c localhost 1.3.6.1.4.1.2620.1.56.1301.3.1.8
    enterprises.2620.1.56.1301.3.1.8.1.0 = Gauge32: 16667
    enterprises.2620.1.56.1301.3.1.8.2.0 = Gauge32: 16667
    [Expert@Sec_GW:0]#
    [Expert@Sec_GW:0]# snmpwalk -Os -c public -v 2c localhost 1.3.6.1.4.1.2620.1.56.1301.3.1.9
    enterprises.2620.1.56.1301.3.1.9.1.0 = Counter64: 0
    enterprises.2620.1.56.1301.3.1.9.2.0 = Counter64: 0
    
This solution has been verified for the specific scenario, described by the combination of Product, Version and Symptoms. It may not work in other scenarios.

Give us Feedback
Please rate this document
[1=Worst,5=Best]
Comment