When troubleshooting logging related issues in a distributed environment, proceed as follows:

 

(Before going through these steps, make sure the log directory in $FWDIR/log on SmartCenter exist. If this is Multi-Domain, make sure that both CMA/CLM and log directory exist in /var/log/mds_logs/CMA_name/log.)

 

1. In SmartConsole, go to 'Policy' menu - click on 'Install Database...' - select the Security Management Server and Log Servers - click 'OK'.

2. Ensure that you have not run out of disk space on the Security Management Server / Log Servers, to which the logs are being sent:

   • On Gaia / SecurePlatform / Linux / IPSO OS:

     Run df -kh command - check the "Use%" column

   • On Windows OS:

     On Desktop, open 'My Computer' - right-click on the relevant hard disk - click on 'Properties' - check the "Free space" line.
   
   If needed, delete or move the unneeded logs / files to an external storage device.

3. Is the Security Gateway configured to send logs to Security Management Server / Log Server?

   In SmartConsole, open the Security Gateway object - check each setting in the "Logs" section.
   If any change was made, install policy.

4. Is the Security Management Server able to communicate over SIC with Security Gateway ?

   In SmartConsole, open the Security Gateway object - on 'General Properties' pane, in "Secure Internal Communication" section - click on "Test SIC Status...".

   If this fails, then it might be due to connectivity / routing issues between the Security Gateway and the Security Management Server.

5. Is the Security Gateway able to communicate (other than SIC) with the Security Management Server?

   Test by sending pings from the Security Gateway to the Security Management Server.
   Test by sending pings from the Security Management Server to the Security Gateway.

   Note: Security policy must allow ICMP between the Security Gateway and the Security Management Server.

   If this fails, and security policy will allows ICMP, then it is most likely a routing issue on Security Gateway. Check the routing table on Security Gateway - there has to be a route to Security Management Server's network / Security Management Server's IP address:

   netstat -rn

6. Is the Security Management Server listening on TCP port 257?

   • On Gaia / SecurePlatform / Linux / IPSO OS:

     # netstat -anp | grep ":257"

   • On Windows OS:

     netstat -abno | findstr ":257"

7. Check the Log Policy settings in log_policy.C file on the Security Management Server:

   Note: Settings in this file have to match the settings in SmartDashboard in the Security Management Server object.

   • On Gaia / SecurePlatform / Linux / IPSO OS:

     $FWDIR/conf/log_policy.C

   • On Windows OS:

     %FWDIR%\conf\log_policy.C

   Example:

   (
           :stop_logging_on_free_disk_space (true)
           :min_free_disk_space (15)
           :stop_free_disk_space_metrics (mbytes)
           :reject_connections (false)
           :alert_on_disk_space (true)
           :alert_free_disk_space (396)
           :alert_free_disk_space_metrics (mbytes)
           :alert_type (alert)
           :log_switch_on_file_size (false)
           :scheduled_switch (false)
           :forward_logs (false)
           :log_delete_on_below (true)
           :log_delete_below_metrics (mbytes)
           :log_delete_below_value (495)
           :log_keep_on_days (false)
           :log_delete_on_run_script (false)
           :dlp_blob_delete_on_run_script (false)
           :dlp_blob_delete_above_value_percentage (20)
           :dlp_blob_delete_on_above (true)
           :packets_capture_reserved_disk_metrics (mbytes)
           :packets_capture_reserved_disk_size_MB (500)
           :dlp_blob_fetch_bulk_size (200)
           :dlp_blob_fetch_interval (5)
           :dlp_blob_retry_interval (180)
   )
   

8. Are any logs coming from the Security Gateway to Security Management Server?

   • On Gaia / SecurePlatform / Linux / IPSO OS:

     # tcpdump -n -i INTERFACE_NAME host IP_ADDRESS_of_GW and tcp port 257

   • On Windows OS:

     Use WireShark and filter for TCP port 257 and IP Address of Security Gateway.
   
   Note: When NAT is configured on the Security Management object, make sure to check the 'Apply for Security Gateway Control Connections' checkbox.
   
   
   
   

9. Verify that topology has not changed, in case customer has reinstalled firewall or added an interface to it.
   Topology in the gateway object has to match interfaces and ip assigned to them in CLI of the gateway.
   In some cases Anti-spoofing may block logging if topology has changed. If this is the case - to resolve this issue go to the object of the problematic gateway or cluster and do get interfaces with topology.
   Note. Make sure you know what interface configuration looks like before you do this step, as you may have to re-assign Anti-spoofing groups if you had any configured. Take a database revision control or a backup of the management, or at least cpinfo, or screen shots of the interfaces configuration before doing fetch interfaces with topology.
   Save the changes.
   Install policy to the gateway.

10. Is the active firewall log file fw.log growing on the Security Gateway?

    • On Gaia / SecurePlatform / Linux :

      # watch -d -n 2 "ls -l $FWDIR/log/fw.log"

    • On Windows OS:

      • Either use Windows Explorer to monitor the size of the %FWDIR%\log\fw.log file
      • Or use Windows Command Prompt and repeatedly run the command
        dir %FWDIR%\log\fw.log

    If the active firewall log file is growing, then the Security Gateway is logging locally instead of forwarding the logs to the Security Management Server.

    This could be a connectivity / routing issue, or it could be the way the logging was configured on this Security Gateway. Check the Security Gateway object to ensure it is configured to send logs to the Management Server.

11. Check the masters file on the Security Gateway.

    • On Gaia / SecurePlatform / Linux / IPSO OS:

      # cat $FWDIR/conf/masters

    • On Windows OS:

      open %FWDIR%\conf\masters

    The Hostname or IP Address of the Security Management Server must be listed in this file.

    Example:

    [Policy]
    Hostname_of_Management_Server
    
    [Log]
    Hostname_of_Management_Server
    
    [Alert]
    Hostname_of_Management_Server
    

12. The active firewall log file fw.log might be corrupted on the Security Gateway.

    1. Create a temporary folder anywhere outside $FWDIR/log/ (on Windows OS: %FWDIR%\log\)
       
       
    2. Stop all Check Point services with cpstop command
       Note: This will stop all traffic. In cluster, this will cause fail-over.
       
       
    3. Move all fw.log* files from the $FWDIR/log/ (on Windows OS: %FWDIR%\log\) folder to a new temporary folder
       Note: Do not move the folder "log" itself
       
       
    4. Start all Check Point services with cpstart command
       
       
    5. Check if $FWDIR/log/fw.log (on Windows OS: %FWDIR%\log\fw.log) file was created and if it is growing

13. The active firewall log file fw.log might be corrupted on the Security Management Server.

    Switch the active firewall log on the Security Management Server:

    • Either from SmartView Tracker : go to "Network & Endpoint" tab - go to 'File' menu - click on 'Switch Active File...'
    • Or from command line:
      fw logswitch

    If switching the active log does not work / fails, perform the following on the Security Management Server:

    1. Create a temporary folder anywhere outside $FWDIR/log/ (on Windows OS: %FWDIR%\log\)
       
       
    2. Stop all Check Point services with cpstop command
       
       
    3. Move all fw.log* files from the $FWDIR/log/ (on Windows OS: %FWDIR%\log\) folder to a new temporary folder
       Note: Do not move the folder "log" itself
       
       
    4. Start all Check Point services with cpstart command
       
       
    5. Check if $FWDIR/log/fw.log (on Windows OS: %FWDIR%\log\fw.log) file was created and if it is growing

14. Debug FWD on the Gateway to show why it is logging locally or logging to the wrong log server.
    
    # fw debug fwd on TDERROR_ALL_FWLOG_DISPATCH=5
    
    Replicate the logging issue.
    
    # fw debug fwd off TDERROR_ALL_FWLOG_DISPATCH=0

 

 

Related Solution for Gaia Embedded Appliances: sk112858 - ATRG: Gaia Embedded Appliances.

Related Solution: sk98317 - Connectivity problems between the Security Gateway and the Log Server

Imported from Nokia support database