Support Center > Search Results > SecureKnowledge Details
"Internal System Error occurred" log in SmartView Tracker while trying to categorize resources
Symptoms
  • SmartView Tracker shows the following alert logs from Application Control & URL Filtering blade for relevant HTTPS sites (depending on the Fail Mode configuration):

    • "Internal System Error, allowing request (as configured in engine settings) See sk64162 for more information."
    • "Internal System Error, blocking request (as configured in engine settings) See sk64162 for more information."

  • Connections are blocked, or categorization is bypassed according to 'Fail Mode' configuration:
    Application & URL Filtering Engine Settings in 'Application & URL Filtering' tab -> 'Advanced' -> 'Engine Settings' are:

    • In the 'Fail Mode' section - either 'Block all requests (Fail-close)' or 'Fail-open' option is selected
    • In the 'URL Filtering' section - selected the 'Categorize HTTPS sites' option

Cause

An error caused by internal state prevents categorization of the resource referenced in the connection. Decision regarding enforcement of this connection depends on the 'Fail Mode' configuration.


Solution

This article describes different scenarios when URL Filtering blocks HTTPS traffic with the "Internal System Error occurred, blocking request (as configured in engine settings). See SK64162 for more information" message in SmartView Tracker.

Each Scenario has additional details, mostly received from debug, its own cause and solution.
See the Table of Contents below.

Table of Contents

  • Issues fixed in R80.10
  • Scenarios with available Hotfixes 
    • Scenario 1 - HTTP/HTTPS Traffic passing through 3rd party proxy is dropped with "Internal System Error"
    • Scenario 2 - Random issues with HTTP web browsing 
    • Scenario 3 - "Internal System Error occurred" log for IDN HTTPS websites with HTTPS Categorization enabled
    • Scenario 4 - HTTPS Inspection fails with "Internal System Error occurred" log from URL Filtering blade
  • Configuration Issues
    • Scenario 5 - URL Filtering blade does not work as expected on VSX Gateway
    • Scenario 6 - Custom application traffic over HTTPS is dropped by Application Control blade
    • Scenario 7 - Application Control policy fails to match rule with Access Role with "Internal system error

 

The following issues were fixed in R80.10 or earlier:

# Known
Limitation
Additional Symptoms and Cause Also fixed in
1 01175524,
01356085
Some web sites are not shown after adding 'Encrypts communications' category to Application Control and URL Filtering policy.

Show / Hide the Cause
Cause: 'Encrypts communications' category belongs to applications that encrypt or might encrypt their traffic (using SSL, SSH, or any other encryption protocol) like uTorrent, OpenSSH, Facebook, anonymizer websites, and more.
'Internal system error occurred' log indicates an issue handling connections in specific scenario when a Proxy is used.
 
2 01243786, 01244345, 01246244, 01246245, 01246246, 01269447, 01271870, 01271872, 01271874, 01295795, 01321333, 01351357, 01353017, 01382415, 01392050  Output of fw tab -t appi_connections -s command on Security Gateway shows that the appi_connections kernel table has reached its limit.

Note: This size is derived from the general fw connection table size.



Cause: When a connection is created, it is written both in the connections kernel table (id 8158) and the appi_connections kernel table (id 55). In some cases, when only URL Filtering is enabled, the connection may not be cleared from the appi_connection kernel table causing the table to reach the limit. When the limit is reached, new connections will not be marked to be inspected and an engine failure will be declared.
Check Point R77.10 
3 01835506, 01849370, 01884821, 01886146, 01844696 
With Anti-Virus, Application Control and URL Filtering blades enabled and APPI rule base configured to block "Malware / Malicious sites" with UserCheck message, when downloading Eicar test file from eicar.org over HTTPS, the UserCheck page is not displayed.
Cause: Due to code limitation, we try to set error page twice on the same session.
 
4 01191701, 02385877, 01200219, 01200218, 02539513, 01336383, 01284965, 02343378, 02346791
RAD debug (per sk92743 - 'rad_admin rad debug on all') shows:
... rad_kernel_urlf_request_set_url: url is not valid - no dot character in it && not valid ipv6!;
...
... [ERROR]: rad_kernel_url_validator: url is not valid - no dot character in it && not valid ipv6!;
... [ERROR]: rad_kernel_url_normalizer_normalize: url is not valid, return;;
... [ERROR]: rad_kernel_malware_request_prepare: rad_kernel_url_normalizer_normalize failed;

Cause: URL Filtering does not detect sites that do not contain the dot character ('.') in URL because validation of IPv4 URL expects a dot in URL.
Check Point R77.10

Jumbo HFA for R77 since Take_1

Jumbo HFA for R76SP.50 since Take_16
5 01466938, 01495623, 01471486, 01495401, 01476226, 01661973, 01611419, 01474787, 01554277, 01462230, 01859469, 01474415

Kernel debug ('fw ctl debug -m WS all' and 'fw ctl debug -m kiss all') shows:

[SID: ...] {connection} appi_transaction_http_process_rb_match_result: MATCH on rule NNN 
(application: ..., action: ...); [SID: ...] {connection} appi_transaction_http_prepare_uc_match_data: uuid: ...; [SID: ...] ws_http_session_make_abs_url_ex: a non-default scheme was not found, URL is considered relative; [SID: ...] ws_http_session_make_abs_url_ex: found 'host:' header: 'PROBLEMATIC_URL'; [SID: ...] ws_http_session_make_abs_url_ex: url is not ipv6!; [SID: ...] ws_http_session_ext_log_header_get_ex: getting header X-Forwarded-For; [SID: ...] ws_http_session_ext_log_header_get_ex: getting header user-agent; [SID: ...] ws_global_policy_headers_htab_get: [ERROR]: kiss_htab_get() failed; [SID: ...] ws_global_policy_get_header_string: [ERROR]: ws_global_policy_headers_htab_get() failed; [SID: ...] ws_http_session_ext_log_header_get_ex: [ERROR]: ws_global_policy_get_header_string() failed; [SID: ...] ws_module_get_session_attribs: [ERROR]: failed to get 'User-Agent' header; [SID: ...] ws_module_get_attribs: [ERROR]: ws_module_get_session_attribs() failed;
Cause: Entry was not found in relevant kernel table.
Check Point R77.30
6 01430167, 01434195, 01434385, 01471225, 01480934, 01502668

Application & URL Filtering Engine settings are:

  • In the 'Fail Mode' section - selected the 'Block all requests (Fail-close)' option
  • In the 'URL Filtering' section - selected the 'Categorize HTTPS sites' option
Show / Hide the Cause
Cause: If a server in its response adds some additional data before the handshake, Security Gateway might fail to match it as Server' "Hello" when "Categorize HTTPS sites" and "Fail-close" are enabled.
Check Point R77.30
7 01431893, 01433702, 01433786, 01434216, 01471298, 01480952, 01535820, 01535835, 01550599, 01554459, PRHF-125,
PRHF-206,
PRHF-535,
PMTR-12335

Application Control debug ('fw ctl debug -m APPI all') shows that Server's response is rejected:

{connection} appi_conn_container_htab_get_connection: [Source_IP:Source_Port - > Dest_IP:Dest_Port] ;
{connection} appi_conn_notify_event_ex: handling parser context HTTP_RESPONSE_DATA ;
{connection} appi_conn_handle_parser_event: found aspii context 184 for connection ... ;
{connection} appi_conn_get_conn_transaction: getting connection transaction;
{connection} appi_conn_get_conn_transaction: connection transaction found;
{connection} appi_conn_notify_event_ex: EVENT END: < parser >;
{policy} appi_cmi_handler_match_cb: conn returned: action [Accept];
...
{policy} appi_rad_uf_cmi_handler_match_cb: match on context or signature 
that is not the registered - won't be sent to RAD; {policy} appi_rad_uf_cmi_handler_match_cb: skip notify for context_id 184; {policy} appi_rad_uf_cmi_handler_match_cb: conn returned: action [Accept]; {urlf_ssl} appi_rad_uf_cmi_handler_server_response: buf len ...; {connection} appi_rad_uf_cmi_handler_get_cmi_opaque: app_conn_opq ...; {policy} appi_rad_uf_cmi_handler_get_cmi_opaque: opaque found ...; {urlf_ssl} [ERROR]: appi_rad_uf_cmi_handler_server_response: no hello done, failed;

Show / Hide the Cause

Cause: When "Categorize HTTPS sites" feature is enabled (go to 'Application & URL Filtering' tab - open 'Advanced' - click on 'Engine Settings' - check the box 'Categorize HTTPS sites'), the non-HTTPS traffic with SSL handshake (e.g., SFTP) causes "Internal System Error" because Pattern Matcher feature does not recognize the Server's "Hello" response.
Check Point R77.30

Jumbo HFA for R77.20 since Take_16

Jumbo HFA for R76SP.10 since Take_37

Jumbo HFA for R80.10 since Take_142

Check Point R80.20
8 02370555,
02391624
With URL Filtering blade enabled, when accessing a web server with short domain name (for example, FQDN less than 4 characters as in URL address http://vb/), user gets the "Internal System Error occurred" error.  
01872944,
01874683, 01977434, 02188954, 01875566, 02182699, 02188956, 02021465, 02022477, 01881544, 01894400;
01907475, 01944831, 02386041, 02357623, 02467694, 02189862,
01919277,
02353010, 02189958, 02021467, 01912286,
02467666,
01919574,
01977453,
01912368,
02388533,
02310566 
  • Users are unable to access HTTPS sites.

  • "cmi_execute_ex: Failed to execute the pattern matcher!;" in /var/log/messages file on Security gateway.

  • Kernel debug (fw ctl debug -m APPI all' + 'fw ctl debug -m NRB all) during the issue shows that Security Gateway fails to execute the pattern matcher on the HTTPS site:

    {connection} appi_conn_container_set_connection: [Source_IP:Source_Port -> Dest_IP:443] ;
    {connection} appi_conn_notify_event_ex: EVENT END: <app_found>;
    {policy} appi_cmi_handler_match_cb: conn returned: action [Accept];
    ... ...
    {policy} appi_rad_uf_cmi_handler_match_cb: pm_match, ...;
    ... ...
    {policy} appi_rad_uf_cmi_handler_match_signature: SSL client hello;
    {urlf_ssl} appi_rad_uf_cmi_handler_match_ssl_client_hello: conn: dst Dest_IP, dport 1bb ;
    ... ...
    {urlf_ssl} appi_rad_uf_by_ssl_cn_cache_match: dest ip Source_IP_in_Hex dport 1bb;
    ... ...
    {policy,urlf_ssl} appi_rad_uf_cmi_handler_match_cb: call appi_user_cmi_handler_handle_url()
    with cn='My.Site.com' (16);
    {urlf_ssl} appi_user_cmi_handler_handle_url: url_https_normalized = 'https://My.Site.com' (24);
    ;cmi_execute_ex: Failed to execute the pattern matcher!;
    {urlf_ssl} [ERROR]: appi_user_cmi_handler_handle_url: appi_cmi_exec_from_app failed (via hook);
    {urlf_ssl} [ERROR]: appi_rad_uf_cmi_handler_match_cb: appi_user_cmi_handler_handle_url() failed;

Show / Hide the Cause
Cause: New application was added to Application & URL Filtering database (in July 2015), which is not processed correctly by the Pattern Matcher when "Categorize HTTPS sites" option is enabled ("Application & URL Filtering" tab - expand "Advanced" - click on "Engine Settings" - go to section "URL Filtering").
Jumbo HFA for R77.30 since Take_98

Jumbo HFA for R77.20 since Take_180

Jumbo HFA for R76SP.30 since Take_58
10 01518127,
01556228
  • Access to web sites fails:
    • a blank page in user's web browser
    • multiple "Internal System Error" logs from Application Control / URL Filtering in SmartView Tracker or SmartLog
  • Kernel debug of NRB and APPI modules shows:

    {session} nrb_column_ip_match: matching column Source IP (handle IP_ADDRESS_in_HEX); 
    {session} nrb_column_ip_match: is_scope_src=0; 
    {session} nrb_column_ip_get_is_identity_required: rule: (1) has role defined, need NAC, match_index: (1) ; 
    {session} nrb_column_ip_match: data is not ready yet (0x38c); 
    {session} nrb_rulebase_default_match: rulebase return match status 'POSSIBLE' ; 
    {connection} appi_transaction_exe_rulebase: rulebase match returned: POSSIBLE; 
    {connection} [ERROR]: appi_transaction_http_process_rb_match_result: invalid match status in this stage; 
    {connection} [ERROR]: appi_transaction_exe_rulebase: transaction_process_rb_match_result_func failed; 
    {session} nrb_handle_abs_destroy: called for IP_ADDRESS_in_HEX; 
    {connection} [ERROR]: appi_transaction_http_exe_rulebase_event: appi_transaction_exe_rulebase failed; 
    {connection} appi_transaction_http_exe_rulebase_event: END: new state: <can_exe_rulebase>, 
    result: [Accept], returned: FALSE; {connection} [ERROR]: appi_conn_handle_parser_event: transaction_execute_rulebase_func failed; {connection} [ERROR]: appi_conn_notify_event_ex: appi_conn_handle_parser_event failed; {connection} appi_conn_notify_event_ex: EVENT END: <parser>;
Show / Hide the Cause
Cause: A misconfiguration between the Proxy and the Security Gateway:
  • On the Security Gateway, the box "Detect users behind http proxy using X Forward-For header" was checked (Identity Awareness pane)
  • On the Proxy server - the "X-Forwarded-For" was disabled.

When 'X-Forwarded-For' (XFF) is disabled on the Proxy, "unknown" is added to the 'X-Forward-For' HTTP Header.

Example from Squid proxy:
Via: 1.1 localhost (squid/3.1.19) 
X-Forwarded-For: unknown 

Example scenario:

  1. Enable Identity Awareness blade and Application Control blade on Security Gateway.
  2. Open Security Gateway properties - go to 'Identity Awareness' pane - check the box "Detect users behind http proxy using X Forward-For header".
  3. In Application Control policy, add a rule with access role in 'Source' column.
  4. Install policy.
  5. On Proxy server - disable 'X-Forward-For' HTTP Header (e.g., on Squid, use directive 'forwarded_for off').
  6. On the Client, browse to a web site that should hit the Application Control rule with access rule.
  7. SmartView Tracker / SmartLog will show multiple "Internal System Error" logs from Application Control / URL Filtering.
Check Point R77.30

Check Point R77.20 for 600 / 1100 / 1200R Appliance 


Important Note: The configuration of "X-Forwarded-For" (XFF) must be identical on both the Security Gateway and the Proxy Server.
11 01842368, 02038367, 02049297, 01917498, 01875817, 02385802, 02538345

01412858, 01624690, 01517886, 01805004, 01560621, 01853655, 01610627, 01511792,
01769543, 01525328, 01639039, 01536626,
02411122, 02104025, 02038356
  • Kernel debug (of APPI and RAD_KERNEL modules) shows:

    {policy,urlf_ssl} appi_rad_uf_cmi_handler_match_cb: call appi_user_cmi_handler_handle_url() with cn='' (0);
    ... ... ...
    {global} [ERROR]: rad_kernel_urlf_request_set_url: cp_lsubstring_search for path slash failed;
    {policy} [ERROR]: appi_rad_uf_cmi_handler_prepare_rad_request: rad_kernel_urlf_request_set_url() failed;
    {policy} [ERROR]: appi_rad_uf_cmi_handler_match_cb_handle_url: 
    appi_rad_uf_cmi_handler_prepare_rad_request() failed; {global} rad_kernel_urlf_request_dtor: going to free original url buffer; {policy} [ERROR]: appi_rad_uf_cmi_handler_match_cb: appi_rad_uf_cmi_handler_match_cb_handle_url() failed; {global} appi_global_policy_get_fail_action: fail action is ACCEPT;
  • HTTPS Inspection is disabled, but the "Categorize HTTPS sites" feature is enabled.


Cause: The "Categorize HTTPS sites" feature examines the CN of the public certificate and makes categorization decisions accordingly. If the CN field in the HTTPS site's certificate is empty, that site can not be categorized correctly because "Categorize HTTPS sites" feature is CN-based.

Show / Hide the Code Improvement

Code was improved in the following way:

Status of
Categorization
by IP addresses
Value of relevant
kernel parameter
Behavior of Security Gateway
if the CN field in the HTTPS site's certificate is empty
Enabled appi_urlf_ssl_cn_allow_ip_categorization=1 Default.
If the CN field in the HTTPS site's certificate is empty, then Security Gateway ignores it and replaces it with site's IP address.
Connection is allowed.
Disabled appi_urlf_ssl_cn_allow_ip_categorization=0 If the CN field in the HTTPS site's certificate is empty, then Security Gateway is not able to categorize that site correctly.
Connection is dropped, and the above log is generated.

Note: By default, Categorization by IP addresses is integrated and enabled (appi_urlf_ssl_cn_allow_ip_categorization=1) since:

  • R77.30
  • Take_5 of R77.20 Jumbo Hotfix Accumulator
  • R76SP.30 for 61000 / 41000
  • Take_67 of R76SP.10 Jumbo Hotfix Accumulator
Jumbo HFA for R77.30 since Take_98

Jumbo HFA for R76SP.50 since Take_16

Show / Hide the Workaround


Available Workaround
:

Disable HTTPS Categorization and enable HTTPS Inspection:

  1. In SmartDashboard, go to 'Application & URL Filtering' tab.

  2. Open 'Advanced'.

  3. Click on 'Engine Settings'.

  4. Clear the 'Categorize HTTPS' box.

  5. Open 'HTTPS Inspection'.

  6. Configure the relevant policy, enable HTTPS Inspection on the involved Security Gateway, etc.

  7. Install the policy.

12 01448602, 01715608, 01638657, 01926085, 01450281, 01449446, 01813264, 01712305, 01715606, 01969817, 01554272
Issue happens only in R77.20 with HTTPS Inspection disabled.

Cause: Conflict between 'Fail-close' mode and 'Categorize HTTPS sites' feature.

Show / Hide the Code Improvement

Code was improved in the following way:

A new kernel parameter was added to allow Categorization by IP addresses and not only by Fail Mode settings (refer to the Configuration instructions below):

Kernel Parameter Value Description
appi_urlf_ssl_cn_allow_ip_categorization=1 Security Gateway allows categorization by IP addresses. This is the default.
appi_urlf_ssl_cn_allow_ip_categorization=0 Security Gateway uses only Fail Mode settings.

Check Point R77.30

Check Point R76SP.30 for 61000 / 41000

Jumbo HFA for R77.20 since Take_5

Jumbo HFA for R76SP.10 since Take_67

13 01175524,
01356085

"Internal system error occurred" log in SmartView Tracker/SmartLog and some web sites are not shown after adding 'Encrypts communications' category to Application Control & URL Filtering policy

Show / Hide the Cause
Cause: 'Encrypts communications' category belongs to applications that encrypt or might encrypt their traffic (using SSL, SSH, or any other encryption protocol) like uTorrent, OpenSSH, Facebook, anonymizer websites, and more.
'Internal system error occurred' log indeed indicates an issue handling connections in specific scenario in case a Proxy is used.
Check Point R80

Show / Hide the Workaround


Available Workaround
:

Remove the 'Encrypts communications' category from the policy:

  1. Open SmartDashboard
  2. Go to Application & URL Filtering tab -> Policy
  3. In the rule, right-click in the 'Applications/Sites' column
  4. Click 'Add category'
  5. Clear the 'Encrypts communications' checkbox
  6. Click OK
  7. Save and install the Security policy
14 01362385,
01366990,
01377452,
01379645,
01396795,
01402500,
01404287,
01405849,
01414498

Kernel debug ('fw ctl debug -m APPI + error connection global policy') shows:

{connection} [ERROR]: appi_transaction_conn_process_rb_match_result: invalid match status in this stage; 
{connection} [ERROR]: appi_transaction_exe_rulebase: transaction_process_rb_match_result_func failed; 
{session} nrb_handle_abs_destroy: called for ...; 
{connection} [ERROR]: appi_conn_check_can_exe_rb_flag: appi_transaction_exe_rulebase failed; 
{connection} [ERROR]: appi_conn_handle_parser_event: appi_conn_check_can_exe_rb_flag failed; 
{connection} [ERROR]: appi_conn_notify_event_ex: appi_conn_handle_parser_event failed; 
{policy} [ERROR]: appi_cmi_handler_match_cb: failed to handle 'PARSER_EVENT'; 
{global} appi_global_policy_get_fail_action: fail action is REJECT;
Check Point R77.20

 

Scenarios with available Hotfixes

Contact Check Point Support to get a Hotfix for the issues, listed below. 
A Support Engineer will make sure the Hotfix is compatible with your environment before providing the Hotfix.
For faster resolution and verification please collect CPinfo files from the Security Management and Security Gateways involved in the case.

Show / Hide Hotfix installation instructions
  1. Hotfix has to be installed on Security Gateway.

    Note: In cluster environment, this procedure must be performed on all members of the cluster.

  2. Transfer the hotfix package to the machine (into some directory, e.g., /some_path_to_fix/).

  3. Unpack the hotfix package:

    [Expert@HostName]# cd /some_path_to_fix/
    [Expert@HostName]# tar -zxvf fw1_wrapper_HOTFIX_NAME.tgz
    
  4. Install the hotfix:

    [Expert@HostName]# ./fw1_wrapper_HOTFIX_NAME
    
    Note: The script will stop all of Check Point services (cpstop) - read the output on the screen.

  5. Reboot the machine.

 

Scenario 1: HTTP/HTTPS Traffic passing through 3rd party proxy is dropped with "Internal System Error"

Symptoms:

  • HTTP/HTTPS Traffic passing through 3rd party proxy (before the firewall) is dropped with "Internal System Error".

  • WS module debug shows:
    {session} ... ws_http_session_get_header_value: got header:X-Forwarded-For len:7 value:unknown;
    {session} ... ws_http_session_get_x_forwarded_ip_from_string: x-forwarded-for header value:  (len=7): unknown;
    {session} ... ws_http_session_get_x_forwarded_ip_from_string: [ERROR]: x-forwarded-for header with 'unknown' value found, cannot convert to IP.;
    {session} ... ws_http_session_ext_log_header_save: [ERROR]: ws_http_session_get_x_forwarded_ip_from_string() failed;
    {session} ... ws_http_session_ext_log_header_get_ex: [ERROR]: ws_http_session_ext_log_header_save() failed;
    {module} ... ws_module_get_session_attribs: [ERROR]: failed to get forwarded ip;
    {module} ... ws_module_get_attribs: [ERROR]: ws_module_get_session_attribs() failed        
             

Known Limitation: 02335915

Cause: Identity Awareness and Application control blades are enabled on the Security Gateway and the option Detect users located behind http proxy using X Forwarded-For header is enabled. The X-Forwarded-For (XFF) header in the HTTP flow contains the value 'unknown'. As the Security Gateway is unable to parse the 'unknown' value as IP, the traffic is dropped.


Scenario 2: Random issues with HTTP web browsing - traffic latency increases, and at some point web browsing stops working

Symptoms:

  • SmartView Tracker during the issue shows multiple logs "Internal System Error occurred" for all HTTP traffic.
  • Restarting RAD daemon (with 'rad_admin stop ; rad_admin start' commands) resolves the issue until it occurs again.

  • Debug of RAD daemon ('rad_admin rad debug on all all') during the issue shows (in $FWDIR/log/rad.elg*):

    • [rad_repository_container_data.h:XXX] CRadRepositoryContaineData::get: [INFO] enter to ...
      [rad_repository_container_data.h:XXX] CRadRepositoryContaineData::get: [MISC] list: <CRadValueUInt> free objects = 0, used XXX of XXX
      [rad_repository_container_data.h:XXX] CRadRepositoryContaineData::get: [ERROR] container <CRadValueUInt> is on its maximum capacity
      [rad_repository_container_data.h:XXX] CRadRepositoryContaineData::get: [INFO] exit from ..
      
    • [rad_http_request.cpp:XXX] CRadHttpRequest::getBuilder: [ERROR] unable to find builder 'malware+0'
      [rad_http_request.cpp:XXX] CRadHttpRequest::build: [ERROR] unable to find builder for 'malware+0'
      [rad_query.cpp:XXX] CRadQuery::build: [ERROR] failed to build http request from values map
      [rad_fetcher.cpp:XXX] CRadFetcher::fetch: [ERROR] error while build message for service 'urlf
      
    • [rad_http_request.cpp:XXX] CRadHttpRequest::getBuilder: [ERROR] unable to find builder 'categorization+0'
      [rad_http_request.cpp:XXX] CRadHttpRequest::build: [ERROR] unable to find builder for 'categorization+0'
      [rad_query.cpp:XXX] CRadQuery::build: [ERROR] failed to build http request from values map
      [rad_fetcher.cpp:XXX] CRadFetcher::fetch: [ERROR] error while build message for service 'appi
      

Known Limitation: 01612968 , 01640299 , 01652963 , 01662822 , 01670027 , 01674403 , 01675502


Scenario 3: "Internal System Error occurred" log for IDN HTTPS websites with HTTPS Categorization enabled

Symptoms:

  • "Internal System Error occurred" log in SmartDashboard while the client is trying to reach HTTPS websites with internationalized domain name (name contain non-English characters)
  • Categorize HTTPS sites feature is enabled (SmartDashboard > Application Control & URL Filtering > Advanced > Engine Settings)

  • HTTPS Inspection feature is disabled (Security Gateway > Properties > HTTPS Inspection)

Known Limitation: 02525513

Cause: Certificate Name (CN) of the HTTPS site's certificate contains non-English characters. When the client is trying to reach the website with internationalized domain name (IDN), browsers converts the URL to a Punycode with prefix of "xn--",

(e.g. IDN: жить.рф = Punycode: xn--f1ae4a2b.xn--p1ai).

Most of the HTTPS websites with such behavior provide the HTTPS certificate with the original URL is in the CN (and not Punycode). HTTPS Categorization is not able to verify and categorize website if the CN contains non-English characters.

 

Scenario 4: HTTPS Inspection fails occasionally with "Internal System Error occurred" log from URL Filtering blade

Symptoms:

  • HTTPS Inspection fails occasionally and "Internal System Error occurred" log from URL Filtering blade is generated.
  • After enabling generation of core dump files per sk92764 / sk53363, core dump files for 'WSTLSD' process were generated in the /var/log/dump/usermode/ directory.

Known Limitations: 02497785, 01778991, 02555072, 02591749, 02648176

 

Configuration Issues

The following issues have different reasons and different configuration solutions:

Scenario 5: URL Filtering blade does not work as expected on VSX Gateway

Symptoms:

  • URL Filtering blade does not work as expected on VSX Gateway - URL categories are not blocked as configured in the policy.
  • Random traffic is blocked by URL Filtering blade with the following log: Internal System Error occurred, allowing request (as configured in engine settings). See SK64162 for more information
  • Connectivity tests based on sk83520 from VSX Gateway to Check Point servers fail.

Cause: VSX Gateway (context of VS0) is not configured to reach the Internet.


Show / Hide the Solution

Solution:

When configuring Virtual Systems to use the Application Control and URL Filtering Software Blades, you must configure the VSX Gateway to connect to the Application and URL Filtering Database. Make sure that the VSX Gateway (context of VS0) can connect to the Internet, because communication for updates and URL categories is only done from this Virtual System.

Refer to VSX Administration Guide (R77, R80.10) - Chapter 3 'Configuring VSX' - Using Application and URL Filtering with VSX.

 

Scenario 6: Custom application traffic over HTTPS is dropped by Application Control blade

Symptoms:

  • With HTTPS inspection blade disabled, custom application traffic is dropped with error in fw ctl zdebug drop debug output:

    fw_log_drop_ex: Packet proto=6 S.S.S.S:60768 -> D.D.D.D:443 dropped by fwpslglue_chain Reason: PSL Reject: ASPII_MT;
  • The "fw ctl debug -m APPI all" kernel debug shows:

    appi_rad_uf_cmi_handler_match_signature: ssl version not supported;
    appi_rad_uf_cmi_handler_match_signature: handling unsupported ssl version as fail close: dropping connection;
    appi_rad_uf_cmi_handler_match_cb: cmi action is not accept=3; appi_rad_uf_cmi_handler_match_cb: ended;

  • All the HTTPS (443) sites, servers will not be accessible. In SmartDashboard. 'Application & URL Filtering' tab - 'Advanced' - 'Engine Settings' - configured as 'Fail-close.
    Block log shows "Internal System Error occurred, blocking request (as configured in engine settings)".

  • When the "Categorize HTTPS sites" option is disabled, the traffic passes through as expected.

Cause: This custom application uses a non-RFC HTTPS implementation.


Show / Hide the Solution

Solution: To allow the non-RFC HTTPS traffic, set the value of the kernel parameter appi_rad_uf_unsupported_ssl_ver_do_drop_conn to 0 (zero). In such case, Security Gateway will handle connections with unsupported SSL version as "Fail-Open".

Note: In cluster environment, this procedure must be performed on all members of the cluster.

  • To check the current value of this kernel parameter:

    [Expert@HostName]# fw ctl get int appi_rad_uf_unsupported_ssl_ver_do_drop_conn
  • To disable this kernel parameter on-the-fly (does not survive reboot):

    [Expert@HostName]# fw ctl set int appi_rad_uf_unsupported_ssl_ver_do_drop_conn 0
  • To disable this kernel parameter permanently:

    Follow sk26202 - Changing the kernel global parameters for Check Point Security Gateway.

    For Gaia / SecurePlatform OS:

    1. Create the $FWDIR/boot/modules/fwkern.conf file (if it does not already exit):

      [Expert@HostName]# touch $FWDIR/boot/modules/fwkern.conf
    2. Edit the $FWDIR/boot/modules/fwkern.conf file in Vi editor:

      [Expert@HostName]# vi $FWDIR/boot/modules/fwkern.conf
    3. Add the following line (spaces and comments are not allowed):

      appi_rad_uf_unsupported_ssl_ver_do_drop_conn=0
    4. Save the changes and exit from Vi editor.

    5. Check the contents of the $FWDIR/boot/modules/fwkern.conf file:

      [Expert@HostName]# cat $FWDIR/boot/modules/fwkern.conf
    6. Reboot the Security Gateway.

    7. Verify that the new value was set:

      [Expert@HostName]# fw ctl get int appi_rad_uf_unsupported_ssl_ver_do_drop_conn

 

Scenario 7: Application Control policy fails to match rule with Access Role and "Internal System Error occurred" message is displayed

Symptoms:

  • Issue started after enabling and then disabling Identity Sharing.

  • Debug of APPI + all, NRB + all shows:
    {session} ... nrb_column_ip_identity_async_reply_cb: idapi async reply for column 'Source IP' (ret code 1);
    {session} ... nrb_column_ip_get_zone_rulenum_list: matching conn type 'Undefined';
    {session} ... nrb_column_ip_match: Async is required;
    {session} ... [ERROR]: nrb_rulebase_async_reply_callback: column 'Source IP' (1) match state is 1. returning error;
    {global} ... appi_global_policy_get_fail_action: fail action is REJECT;
    {policy} ... appi_transaction_rulebase_async_reply_callback: perform appi fail action [Reject];
            
    OR
    {session} ... nrb_column_ip_get_zone_rulenum_list: matching conn type 'Undefined'; 
    {session} ... nrb_column_ip_match: creating new identity handle; 
    {session} ... nrb_column_ip_match: Async is required; 
    {session} ... nrb_rulebase_default_match: column 'Source IP' (1) match state is NEED_ASYNC; 
    {session} ... nrb_rulebase_default_match: rulebase return match status 'NEED_ASYNC' ; 
    {connection} ... appi_transaction_exe_rulebase: rulebase match returned: NEED_ASYNC; 
    {global} ... appi_global_policy_get_fail_action: fail action is REJECT;

Cause: After Identity Sharing was enabled and then disabled, two tables were corrupted.


Show / Hide the Solution

Solution:

IMPORTANT: If Identity Sharing is enabled on more than one Security Gateway, and Identity Sharing was removed only from one of the Security Gateways, do not empty the tables on all Security Gateways, but only on the Security Gateway where Identity Sharing was disabled.

  1. Check if the following IA tables are empty or have values, by running next command:

    [Expert@Home]# fw tab -t pep_networks_to_pdp_db -s -t pep_net_reg -s

  2. If they have some values, clean them by running the following command:

    [Expert@Home]# fw tab -t pep_networks_to_pdp_db -t pep_net_reg -t pep_reported_network_masks_db -x -y

  3. Make sure to install the policy after deleting the tables, and check for the table sizes again to verify that they are empty:

    [Expert@Home]# fw tab -t pep_networks_to_pdp_db -s -t pep_net_reg -s

 


 

If neither Hotfix nor upgrade to R80.10 helped you, contact Check Point Support to troubleshoot the issue.
For faster resolution and verification, please provide the following information:

  • CPinfo file from the involved Security Management Server / Domain Management Server
  • CPinfo file from the involved Security Gateway / Cluster members
  • Screenshot and the complete text of the relevant log(s) from SmartView Tracker.
Applies To:
  • This article replaces sk113354 , sk106170 , sk117835 , sk119313 , sk102867 , sk107695 , sk108535 , sk109581 , sk103423 , sk103859 , sk101977 , sk93466 , sk95588 , sk98743 , sk109802 , sk98022 , sk102273 , sk114717

Give us Feedback
Please rate this document
[1=Worst,5=Best]
Comment