Connectivity Issues encountered after Policy Install

After Policy Install, you may encounter a variety of issues. This article discusses a number of the more common issues that we have encountered and dealt with.

Table of Contents

• Scenario 1: Connection Continuity after Policy Installation
• Scenario 2: UDP traffic was dropped after policy installation with 'Server to client packet of an old UDP session' log in SmartView Tracker
• Scenario 3: fw_handle_old_conn_recovery Reason: Other protocol packet that belongs to an old connection
• Scenario 4: Security Gateway might crash upon policy installation after deleting some rules from the rulebase

Scenario 1

Title: Connection Continuity after Policy Installation

Product: Security Gateway

Solution:

Background

The connections table is created for the first time when the first policy is installed on the Security Gateway appliance after initial start-up. The table is not destroyed when a new policy is installed so that the appliance keeps the information it has accumulated for each currently active connection. (When a policy is uninstalled, the connections table is deleted.)

In the process of installing the new policy, some of the old connection entries are not removed from the connection table:

• If the "Keep all connections" setting has been enabled for a Check Point Security Gateway, all connections will be kept open until they have ended, and the newly installed policy will be enforced only for new connections.
  
  
• Connection entries that belong to a service for which the "Keep connections open after Policy has been installed" setting has been enabled are not overwritten during policy installation and remain active after policy reinstallation.
  
  
• Data connections cannot be rematched against the rule base. They are kept open if their associated control connection is explicitly kept, as described above. In addition, if the "Keep Data Connections" setting has been enabled, data connections will be retained until they time out, regardless of whether the control connection has been successfully rematched.
  
  
• Security server server-side connections are retained.
  
  
• Connections using NAT's IP pool feature will be removed from the table, if the IP pool has been modified.
  
  
• If "Rematch Connections" has been specified on the Security Gateway's Connection Persistence page, Data connections will be removed from the table.
  
  
• Remaining connections are flagged as "old". When a new packet successfully matches a connection that is marked as "old", it is reverified against the new rulebase. If the packet is accepted, the Security Gateway will change the connection's status back to that of a regular connection. If the new packet does not pass the rulebase, it will be dropped and the "old" connection will time out.

Considerations and Restrictions

• First packet direction

  Each connection has 2 peers; the client and the server. The client initiates the connection to the server. When a new connection is established, the first packet is from the client to server and this is what is matched against the Policy. If the first packet arrives from the server side, and it does not belong to an established TCP connection, the receiving host will silently discard this packet. UDP, ICMP and other (not TCP) packets are dropped since there is no way to guarantee these are valid packets that belong to the connection.
  
  

• Data connections

  Data connections are connections that are dynamically created within an existing control connection, for example FTP. The initial control connection is used only for sending commands; actual file transfers are done by new connections. These auxiliary connections will be accepted and connectivity will not be affected. Data connections cannot usually be inferred from the Policy, as they are created according to the flow of the control protocol. By default, when loading a new Policy, the Security Gateway deletes all the data connections entries from the table as they are likely to get the wrong results if a Policy match for a data connection packet is attempted.
  
  

• IPSO Flows / SecureXL connections table

  During Policy installation, IPSO Flows / SecureXL connections table will be cleared and re-created, irrespective of connection persistence settings. This clearing and re-creating are very expensive depending on the active connections in the table at that point. Also, all the packets will be F2F (Forwarded to FireWall in slowpath) until IPSO flows are created again.
  
  Notes:
  
  • Since R80.20, the SecureXL connections table is not cleared during policy installation.
  • In addition, Check Point does not support IPSO since R80.10.

• Impact on CPU usage

  Most of the Check Point processes will consume high CPU and idle CPU can be as low as zero for high traffic condition.

Recommendations

Policy installation operation is very CPU intensive and can cause resource contention on busy system, as a result there will be some packet loss during install operation. To minimize disruption, the following configuration changes are recommended:

• Enable "Keep all connections" so that connection does not have to rematch new Policy.
  
  
• Increase the Log Buffer size (refer to sk52100 - /var/log/messages files show - 'Log buffer is full').
  
  
• Redirect all Check Point kernel debug messages from console to a file (refer to sk41400 - How to redirect all Check Point kernel debug messages from console to file).
  
  
• Install policy during non-peak hours only.

---

Scenario 2

Title: UDP traffic was dropped after policy installation with 'Server to client packet of an old UDP session' log in SmartView Tracker

Product: Security Gateway, ClusterXL, Cluster - 3rd party, VSX

OS: Gaia

Symptoms:

• UDP traffic was dropped after policy installation with 'Server to client packet of an old UDP session' log in SmartView Tracker.

Cause: 

This log can be generated in the following cases:

1. When 'Drop out of state UDP packets' global property and 'Log on drop' global property for out of state UDP packets are enabled per sk103084 (How to configure the Security Gateway to drop Out of State UDP packets).
   
   
2. A UDP packet was detected that belongs to an old connection (after policy installation). (When run zdebug+ drop see: "fw_handle_old_conn_recovery Reason: old packet rule base drop")

Note: Check Point Security Gateway never re-matches Server-to-Client connections (unless it is VPN RDP packet):

• If a Server-to-Client packet in question is a TCP, then Security Gateway mangles the packet.
• If a Server-to-Client packet in question is a UDP, then Security Gateway simply drops it.

Solution:

Clarification: "Keep connections open after Policy has been installed" keeps all control and data connections open until the connections have ended, even if they are not allowed under the new policy. This overrides the settings in the Connection Persistence page. If you change this property, the change will not affect open connections, but only future connections

One possible solution is:

1. In SmartDashboard, customize the Service by selecting "Keep connections open after Policy has been installed".
   
   
   
   
2. Install policy on the involved Security Gateway.

---

Scenario 3

Title: fw_handle_old_conn_recovery Reason: Other protocol packet that belongs to an old connection

Product: IPSec VPN, Security Gateway

OS: SecurePlatform, IPSO 4.x, IPSO 6.2, SecurePlatform 2.6, Gaia

Symptoms:

• VPN traffic passing through a firewall (not an endpoint) is dropped.
• fw zdebug + drop shows: fw_handle_old_conn_recovery Reason: Other protocol packet that belongs to an old connection

Cause:

When the connection is removed from the connection table, arriving packets will not be matched by the connection table.

Solution:

There are two options to resolve this issue permanently:

1) Select only this type of traffic to persist through a policy push. Open the TCP Service for the packet that is being dropped, and select "Keep connections open after policy has been installed".

OR

2) Allow all traffic to persist past a policy push. Open the firewall (cluster) objects properties, expand Advanced and select Connection Persistence. Select "Keep all connections".

After performing either of these, you will need to push a policy to the gateways and clear the connections table on the gateways.

1. To clear the connection table, run the following command on the gateway: #fw tab -t connections -x
2. When you run this command on the primary, it will delete all connections from all cluster members simultaneously.
3. The system will ask if you are sure, input "yes".

Notes:

• This procedure will not remove entries from translation tables.
• As #fw tab -t connections -x removes all connections, it is possible to simply stop the traffic in question for a few minutes. This will time out the session and resolve the issue without removing all other connections.

---

Scenario 4

Title: Security Gateway might crash upon policy installation after deleting some rules from the rulebase

Product: Security Gateway

Version: R70, R71, R75

OS: 

Symptoms:

• Security Gateway might crash upon policy installation after deleting some rules from the rulebase if 'Connection Persistence' is set to 'Keep all connections' in the Security Gateway object.
• The stack printed on the console contains these lines:
  
• EIP is at fw_set_new_rule_uuid
• Process fw (pid: XXX, ti=XXX task=XXX task.ti=XXX)

Solution:

This problem was fixed. The fix is included in:

• Check Point R76
• Check Point R75.45
• Check Point R75.052 on 61000 appliance

Check Point recommends to always upgrade to the most recent version (upgrade Security Gateway / upgrade VSX / upgrade Security Management Server / upgrade Multi-Domain Security Management Server).

For lower versions, Check Point can supply a Hotfix. Contact Check Point Support to get a Hotfix for this issue.
A Support Engineer will make sure the Hotfix is compatible with your environment before providing the Hotfix.
For faster resolution and verification, please collect CPinfo files from the Security Management Server and Security Gateways involved in the case.

Hotfix installation instructions:

1. Hotfix has to be installed on Security Gateway.

   Note: In cluster environment, this procedure must be performed on all members of the cluster.
   
   
2. Transfer the hotfix package to the machine (into some directory, e.g., /some_path_to_fix/).
   
   

3. Unpack the hotfix package:

   [Expert@HostName]# cd /some_path_to_fix/
   [Expert@HostName]# tar zxvf fw1_wrapper_HOTFIX_NAME.tgz
   

4. Install the hotfix:

   [Expert@HostName]# ./fw1_wrapper_HOTFIX_NAME

   Note: The script will stop all of Check Point services (cpstop) - read the output on the screen.
   
   
5. Reboot the machine.

Applies To: 00853274 , 00853798 , 00853799 , 00853800 , 00857677 , 00859676 , 00885390 , 00903026 , 00936263 , 01056437 , 01056440 , 01057695 , 01060611 , 01111724 , 01118586 , 01119306 , 01119814 , 01121955 , 01129793

---

Related Solution: sk142355 - VPN tunnel goes down after policy push, and it needs to reset the tunnel to bring it up.

---

Scenario 5

Title: new policy installed is not enforced as intended.

Product: Security Gateway

Version: All

Symptoms:

1. Post policy installation the changed rule or disabled setting configuration is not enforced by the GW.
2. The traffic from before the policy installation is still allowed through the GW as it did before.
3. Changes to NAT rules for certain connections are not enforced properly.

Cause:

• The connection/NAT table/s (or other) is still holding the past entry in its cache and it is being forwarded by SecureXL.
• also related to "keep data connections" and "keep all connections" setting.

 
Solution:
In the short term: 

1. Understand the changes made to configuration after policy installation and its' affect to the relevant table.

• Look through the relevant table on the GW using the following command:
  • #fw tab -t <table name> 
    • The command will show you the entries in hexadecimal format.
    • You can use the following site to convert Hex to IPv4: link here
    • You can use the following site to convert Hex to port number: link here
• Erase the old entry in the relevant table using the following command:
  • #fw tab -t <table name> -x -e <the relevant tuple in hex>

     2. In the long term:

• Understand the necessary "connection persistence" setting for your environment post policy installation.

Important note:

• Be aware that erasing the wrong entry during production could result in stateful inspection drops and an impact to connectivity if you erase the wrong entry, please make sure and check twice before doing so.