Connectivity Issues encountered after Policy Install
After Policy Install, you may encounter a variety of issues. This article discusses a number of the more common issues that we have encountered and dealt with.
Table of Contents
Scenario 1: Connection Continuity after Policy Installation
Scenario 2: UDP traffic was dropped after policy installation with 'Server to client packet of an old UDP session' log in SmartView Tracker
Scenario 3: fw_handle_old_conn_recovery Reason: Other protocol packet that belongs to an old connection
Scenario 4: Security Gateway might crash upon policy installation after deleting some rules from the rulebase
Title: Connection Continuity after Policy Installation
Product: Security Gateway
The connections table is created for the first time when the first policy is installed on the Security Gateway appliance after initial start-up. The table is not destroyed when a new policy is installed so that the appliance keeps the information it has accumulated for each currently active connection. (When a policy is uninstalled, the connections table is deleted.)
In the process of installing the new policy, some of the old connection entries are not removed from the connection table:
- If the "
Keep all connections" setting has been enabled for a Check Point Security Gateway, all connections will be kept open until they have ended, and the newly installed policy will be enforced only for new connections.
- Connection entries that belong to a service for which the "
Keep connections open after Policy has been installed" setting has been enabled are not overwritten during policy installation and remain active after policy reinstallation.
- Data connections cannot be rematched against the rule base. They are kept open if their associated control connection is explicitly kept, as described above. In addition, if the "
Keep Data Connections" setting has been enabled, data connections will be retained until they time out, regardless of whether the control connection has been successfully rematched.
- Security server server-side connections are retained.
- Connections using NAT's IP pool feature will be removed from the table, if the IP pool has been modified.
- If "
Rematch Connections" has been specified on the Security Gateway's Connection Persistence page, Data connections will be removed from the table.
- Remaining connections are flagged as "old". When a new packet successfully matches a connection that is marked as "old", it is reverified against the new rulebase. If the packet is accepted, the Security Gateway will change the connection's status back to that of a regular connection. If the new packet does not pass the rulebase, it will be dropped and the "old" connection will time out.
Considerations and Restrictions
Policy installation operation is very CPU intensive and can cause resource contention on busy system, as a result there will be some packet loss during install operation. To minimize disruption, the following configuration changes are recommended:
Title: UDP traffic was dropped after policy installation with 'Server to client packet of an old UDP session' log in SmartView Tracker
Product: Security Gateway, ClusterXL, Cluster - 3rd party, VSX
- UDP traffic was dropped after policy installation with '
Server to client packet of an old UDP session' log in SmartView Tracker.
This log can be generated in the following cases:
- When '
Drop out of state UDP packets' global property and '
Log on drop' global property for out of state UDP packets are enabled per sk103084 (How to configure the Security Gateway to drop Out of State UDP packets).
- A UDP packet was detected that belongs to an old connection (after policy installation). (When run zdebug+ drop see: "fw_handle_old_conn_recovery Reason: old packet rule base drop")
Note: Check Point Security Gateway never re-matches Server-to-Client connections (unless it is VPN RDP packet):
- If a Server-to-Client packet in question is a TCP, then Security Gateway mangles the packet.
- If a Server-to-Client packet in question is a UDP, then Security Gateway simply drops it.
Clarification: "Keep connections open after Policy has been installed" keeps all control and data connections open until the connections have ended, even if they are not allowed under the new policy. This overrides the settings in the Connection Persistence page. If you change this property, the change will not affect open connections, but only future connections
One possible solution is:
- In SmartDashboard, customize the Service by selecting "Keep connections open after Policy has been installed".
- Install policy on the involved Security Gateway.
Title: fw_handle_old_conn_recovery Reason: Other protocol packet that belongs to an old connection
Product: IPSec VPN, Security Gateway
OS: SecurePlatform, IPSO 4.x, IPSO 6.2, SecurePlatform 2.6, Gaia
- VPN traffic passing through a firewall (not an endpoint) is dropped.
- fw zdebug + drop shows: fw_handle_old_conn_recovery Reason: Other protocol packet that belongs to an old connection
When the connection is removed from the connection table, arriving packets will not be matched by the connection table.
There are two options to resolve this issue permanently:
1) Select only this type of traffic to persist through a policy push. Open the TCP Service for the packet that is being dropped, and select "Keep connections open after policy has been installed".
2) Allow all traffic to persist past a policy push. Open the firewall (cluster) objects properties, expand Advanced and select Connection Persistence. Select "Keep all connections".
After performing either of these, you will need to push a policy to the gateways and clear the connections table on the gateways.
- To clear the connection table, run the following command on the gateway: #fw tab -t connections -x
- When you run this command on the primary, it will delete all connections from all cluster members simultaneously.
- The system will ask if you are sure, input "yes".
- This procedure will not remove entries from translation tables.
- As #fw tab -t connections -x removes all connections, it is possible to simply stop the traffic in question for a few minutes. This will time out the session and resolve the issue without removing all other connections.
Title: Security Gateway might crash upon policy installation after deleting some rules from the rulebase
Product: Security Gateway
Version: R70, R71, R75
- Security Gateway might crash upon policy installation after deleting some rules from the rulebase if '
Connection Persistence' is set to '
Keep all connections' in the Security Gateway object.
- The stack printed on the console contains these lines:
- EIP is at fw_set_new_rule_uuid
- Process fw (pid: XXX, ti=XXX task=XXX task.ti=XXX)
This problem was fixed. The fix is included in:
Check Point recommends to always upgrade to the most recent version (upgrade Security Gateway / upgrade VSX / upgrade Security Management Server / upgrade Multi-Domain Security Management Server).
For lower versions, Check Point can supply a Hotfix. Contact Check Point Support to get a Hotfix for this issue.
A Support Engineer will make sure the Hotfix is compatible with your environment before providing the Hotfix.
For faster resolution and verification, please collect CPinfo files from the Security Management Server and Security Gateways involved in the case.
Hotfix installation instructions:
Hotfix has to be installed on Security Gateway.
Note: In cluster environment, this procedure must be performed on all members of the cluster.
- Transfer the hotfix package to the machine (into some directory, e.g.,
Unpack the hotfix package:
[Expert@HostName]# cd /some_path_to_fix/
[Expert@HostName]# tar zxvf fw1_wrapper_HOTFIX_NAME.tgz
Install the hotfix:
Note: The script will stop all of Check Point services (
cpstop) - read the output on the screen.
- Reboot the machine.
Applies To: 00853274 , 00853798 , 00853799 , 00853800 , 00857677 , 00859676 , 00885390 , 00903026 , 00936263 , 01056437 , 01056440 , 01057695 , 01060611 , 01111724 , 01118586 , 01119306 , 01119814 , 01121955 , 01129793
Related Solution: sk142355 - VPN tunnel goes down after policy push, and it needs to reset the tunnel to bring it up.
Title: new policy installed is not enforced as intended.
- Post policy installation the changed rule or disabled setting configuration is not enforced by the GW.
- The traffic from before the policy installation is still allowed through the GW as it did before.
- Changes to NAT rules for certain connections are not enforced properly.
Solution:In the short term:
- The connection/NAT table/s (or other) is still holding the past entry in its cache and it is being forwarded by SecureXL.
- also related to "keep data connections" and "keep all connections" setting.
- Understand the changes made to configuration after policy installation and its' affect to the relevant table.
- Look through the relevant table on the GW using the following command:
- #fw tab -t <table name>
- The command will show you the entries in hexadecimal format.
- You can use the following site to convert Hex to IPv4: link here
- You can use the following site to convert Hex to port number: link here
- Erase the old entry in the relevant table using the following command:
- #fw tab -t <table name> -x -e <the relevant tuple in hex>
2. In the long term:
- Understand the necessary "connection persistence" setting for your environment post policy installation.
- Be aware that erasing the wrong entry during production could result in stateful inspection drops and an impact to connectivity if you erase the wrong entry, please make sure and check twice before doing so.