Support Center > Search Results > SecureKnowledge Details
What is the "Custom Intelligence Feeds" feature?
Solution

Custom Intelligence Feeds feature provides an ability to add custom cyber intelligence feeds into the Threat Prevention engine. It allows fetching feeds from a third-party server directly to the Security Gateway to be enforced by Anti-Virus and Anti-Bot blades.

The Custom Intelligence Feeds feature also assists customers with the operational and engineering management challenges they face handling indicators: Managing and monitoring of the custom intelligence feeds is done with minimal operational overhead.

Indicator is a pattern of relevant observable malicious activity in an operational cyber domain, with relevant information on how to interpret it and how to handle it. Indicators are derived from intelligence, self-analysis and/or governments, partners etc.

Observable is an event or a stateful property that can be observed in an operational cyber domain. For example: IP address, MD5 file signature, URL, Mail sender address.

Table of Contents

  • Supported Formats
  • Installation
  • Fetching new feed using CLI - ioc_feeds
  • Known feeds examples
  • Known Limitations
  • Troubleshooting

 

Supported Formats

Each indicator file can be either in CSV or STIX XML format.

CSV (*.csv) format

  • Delimiter between fields is ','
  • Delimiter between records is '\n'
  • Fields are plain text
  • Each record "must" contain the same number of comma-separated fields
  • Quoted fields are accepted.
  • The first line/record, prefixed with #, is a header containing column names in each of the fields.

Example:

#! DESCRIPTION = indi file,,,,,,
"#! REFERENCE = Indicator Bulletin; Feb 20, 2014",,,,,,
# FILE FORMAT:,,,,,,
"#      All lines beginning  ""#"" are comments",,,,,,
"#      All lines beginning  ""#!"" are metadata read by the SW",,,,,,
"# UNIQ-NAME,VALUE,TYPE,CONFIDENCE,SEVERITY,PRODUCT,COMMENT",,,,,,
observ1,8d9b6b8912a2ed175b77acd40cbe9a73,MD5,medium,medium,AV,FILENAME:WUC Invitation Letter Guests.doc
observ2,76700f862a0c241b8f4b754f76957bda,MD5,high,high,AV,FILENAME:essais~.swf|
NOTE:FWS type Flash file
observ7,http://somemaliciousdomain.com/uploadfiles/upload/exp.swf?info=
789c333432d333b4d4b330d133b7b230b03000001b39033b&infosize=00840000 
,URL,high,high,AV,IPV4ADDR:196.168.25.25
observ8,svr01.passport.ServeUser.com,Domain,low,high,AB,TCP:80|
IPV4ADDR:172.18.18.25|NOTE:Embedded EXE Remote C&C and Encoded Data
observ9,somemaliciousdomain2.com,Domain,,low,AV,TCP:8080|IPV4ADDR:172.22.14.10
observ10,http://www.bogusdomain.com/search?q=%24%2B%25&form=MOZSBR&pc=
MOZI,URL,low,low,AB,IPV4ADDR:172.25.1.5
observ11,http://somebogussolution.com/register/card/log.asp?isnew=-1&LocalInfo=
Microsoft%20Windows%20XP%20Service%20Pack%202&szHostName=
ADAM-E512679EFD&tmp3=tmp3,URL,medium,,AB,
observ14,172.16.47.44,IP,high,medium,AB,TCP:8080
observ15,172.16.73.69,IP,medium,medium,AV,TCP:443|NOTE:Related to Flash 
exploitation
observ16,abc@def.com,mail-to,,high,AV,"NOTE:truncated; samples have appended to 
the subject the string ""PH000000NNNNNNN"" where NNNNNNN is a varying number"
observ34,stamdomain.com,domain,,,AB,
observ35,stamdomain.com,mail-from,high,medium,AV,
observ37,xyz.com,mail-from,medium,medium,AB,
observ38,@xyz.com,mail-from,medium,medium,AB,    

 

Custom CSV format

Custom Intelligence Feeds feature supports different kinds of CSV structure files.

For the fetch to succeed, you must define the file's format, delimiter, and the comment lines to skip.

Syntax Notes:

  • The supported fields are: [name,value,type,confidence,severity,product,comment]
  • To use a field value from the original file (must be one of the supported fields listed above), you must specify its location in the csv row by using #index.
    For example, if you want to take the 3rd index in your csv file to be your observable's comment in the new file, use:

   --format [comment:#3]

  • To use a default value for all observables, use

--format [type:domain]

  • The Value field is mandatory and must be taken from the original file.
  • The Type field is mandatory and must be taken from the original file or be sent as a default value for all observables.
  • All other fields are optional and can either be taken from the original file or sent as a default value for all observables.
  • When the feed's resource is a remote source (transport equals HTTP or HTTPS) - every time the feed will be fetched, it will parse according to the format that has been specified for this feed.

 

Examples:

  • Original CSV structure is a list of IP addresses

ioc_feeds add --feed_name ip_list --transport http --resource "http://blocklist.greensnow.co/greensnow.txt" --format [value:1,type:ip]


  • Original CSV structure is a list of Domains

ioc_feeds add --feed_name domain_list --transport https --resource "https://isc.sans.edu/feeds/suspiciousdomains_High.txt" --format [type:domain,value:1] --comment "#, Site"


  • Original CSV structure is a list of IP addresses in CIDR format and comment lines are marked as ';'

ioc_feeds add --feed_name ip_cidr_list_with_delimiter --transport https --resource "https://www.spamhaus.org/drop/edrop.txt" --format [value:1,type:ip] --delimiter ";" --comment ";"


  • Original CSV structure is a list of IP addresses separated by '|' delimiter, and comment lines are marked as '#'

ioc_feeds add --feed_name ip_list_with_spaces --transport local_file --resource "/home/admin/ioc/ip_list_with_spaces.txt" --format [value:3,comment:#2,type:ip] --comment [#] --delimiter "|"


  • Original CSV structure is a list of different types separated by ',' delimiter, and comment lines are marked as '#' or 'Site'

ioc_feeds add --feed_name try_custom_csv --transport http --resource http://192.168.13.13/ioc/bad_csv_format.csv --format [type:#1,value:#3,name:#6,comment:#7,product:av] --comment [#, Site] --delimiter ,


  • Original CSV structure is a list of different types separated by ',' delimiter

ioc_feeds add --feed_name ioc_feed --transport http --resource "http://www.malwaredomainlist.com/updatecsv.php" --format [value:3,comment:#2,type:ip] --delimiter ,



STIX - Structured Threat Information eXpression

STIX™ is a collaborative community-driven effort to define and develop a standardized language to represent structured cyber threat information.
For more information, refer to STIX 1.x Archive Website.

Supported observables types

  • URL
  • Domain
  • IP
  • IP Range
  • MD5
  • Mail-subject
  • Mail-from
  • Mail-to
  • Mail-cc
  • Mail-reply-to


Installation

The following Download pakages are available:

Product
Download CPUSE Online Identifier
Hotfix on top of R80.20 GA (Take 101) (TGZ) Check_Point_R80.20_T101_Hotfix_sk132193_FULL.tgz
Hotfix on top of R80.20 + Jumbo Hotfix Take 47 up to Take 118 (TGZ) Check_Point_R80.20_Hotfix_sk132193_FULL.tgz 

 

Fetching new feed using CLI - ioc_feeds

Feed's resource can be:

  • URL - HTTP/HTTPS (--transport http --resource "http://10.0.0.1/my_feeds/stix_feed.xml")
    *Self-signed certificate HTTPS resource will propmt for user agreement to update the bundle. It is possible to skip the certificate verification by running "export EXT_IOC_NO_SSL_VALIDATION=1" on the gateway.
  • File on the machine (--transport local_file --resource "/home/admin/my_feed.csv")
  • Directory on the machine, which contains the same feed_format - (--transport local_directory --resource "/home/admin/my_feed_folder")

Parameter Description Example
push Push feeds now ioc_feeds push
show Prints all existing feeds ioc_feeds show
show --feed_name <feed>  Prints specific feed details  ioc_feeds show --feed_name local_feed
show_interval  Prints fetching interval  ioc_feeds show_interval 
set_interval sec

Set interval for fetching in seconds

*Feed fetching interval - same for all feeds

ioc_feeds set_interval 1000 
show_scanning_mode  Prints scanning mode  ioc_feeds show_scanning_mode 
set_scanning_mode  Set scanning mode - on/off  ioc_feeds set_scanning_mode off
add 

Adding new feed.

Mandatory fields:

--feed_name <feed>

--transport <http/https/local_file/local_directory>

--resource <url/full_path>

Optional fields:

--state <true/false> (active/inactive. default - True)

--feed_action <Prevent/Detect/Ask> (default – Prevent)

--user_name <user> (prompt for password)

--proxy <proxy:port>

--proxy <none> – don't use proxy

--proxy <proxy:port> - override gatewayproxy for feed

(not mentioning proxy flag - gateway proxy will be used)

--proxy_user_name <user> (prompt for password)

--test true
test feed fetching and parsing

Examples:

ioc_feeds add --feed_name local_feed --transport local_file --resource /home/admin/my_feed.csv

ioc_feeds add --feed_name remote_feed --transport http --resource 10.0.0.1/my_feeds/stix_feed.xml --proxy 127.10.10.1:8080 --state false –feed_action detect --user_name admin@checkpoint.com

modify 

Modify existing feed.
Fields that will not be mentioned will stay as they were before 

ioc_feeds modify --feed_name local_feed --state true
delete  Delete the existing feed  ioc_feeds delete --feed_name local_feed 

 

More examples:

  • Adding a new remote feed

[Expert@HostName:0]# ioc_feeds add --feed_name remote_csv_feed --transport http --resource "http://10.10.1.100/ioc/ioc_csv_file.csv" --feed_action Prevent

  • Adding a new local feed

[Expert@HostName:0]# ioc_feeds add --feed_name ioc_stix_file --transport local_file --resource "/home/admin/ioc/ioc_stix_file.xml"

  • Printing existing feeds

[Expert@HostName:0]# ioc_feeds show

  • Deleting a feed

[Expert@HostName:0]# ioc_feeds delete --feed_name ioc_stix_file

  • Test feed fetching and parsing

[Expert@HostName:0]# ioc_feeds add --feed_name remote_stix_file --transport http --resource "http://www.public_indicators.com/ioc_stix_file.xml" --test true

 

Known feeds examples (using the Custom CSV feature)

Description URL Command Line
Alienvault IP Reputation http://reputation.alienvault.com/reputation.data ioc_feeds add --feed_name reputation --transport http --resource "http://reputation.alienvault.com/reputation.data" --format [type:ip,value:#1,comment:#4] --delimiter "#"
C&C domains http://osint.bambenekconsulting.com/feeds/c2-dommasterlist.txt ioc_feeds add --feed_name c_and_c_domains --transport http --resource "http://osint.bambenekconsulting.com/feeds/c2-dommasterlist.txt" --format [type:domain,value:#1] --comment [#] --delimiter ","
C&C IPs http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist.txt ioc_feeds add --feed_name c_and_c_ips --transport http --resource "http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist.txt" --format [type:ip,value:#1] --comment [#] --delimiter ","
Talos IP Blacklist http://www.talosintelligence.com/documents/ip-blacklist ioc_feeds add --feed_name ip_blacklist --transport https --resource "https://www.talosintelligence.com/documents/ip-blacklist" --format [type:ip,value:#1]
Spam List http://www.ipspamlist.com/public_feeds.csv ioc_feeds add --feed_name spam_list --transport http --resource "http://www.ipspamlist.com/public_feeds.csv" --format [type:ip,value:#3,comment:#4] --comment ["#", "first_seen"] --delimiter ","
Cybercrime hash list http://cybercrime-tracker.net/ccamlist.php ioc_feeds add --feed_name hash_list --transport http --resource "http://cybercrime-tracker.net/ccamlist.php" --format [type:md5,value:#1]

 

Managing Custom Intelligence Feeds from the Security Management Server

This option is available starting from R80.30

  1. Run on one of your Security Gateways: 
    ioc_feeds export
  2. If the feeds were fetched successfully, move /home/admin/export_ioc.tar.gz file to the Security Management.
  3. From the GUI Command-line tool, run:
    put-file file-path "/home/admin" file-name "export_ioc.tar.gz" file-content @"/home/admin/export_ioc.tar.gz" --treat-value-as-file-by-prefix @ targets.1 "corporate-gateway" targets.2 "Backup-GW"
  4. From the GUI Command-line tool, run:
    run-script script-name "Import Custom feeds" script "ioc_feeds import /home/admin/export_ioc.tar.gz" targets.1 "corporate-gateway" targets.2 "Backup-GW"

 

Known Limitations

  • Observables of IP addresses and IP Ranges can hold IPv4 values only.

  • MD5 observables cannot be enforced by Anti-Bot Blade. If user does not enable Anti-Virus blade, there will be no enforcement.

 

Troubleshooting

  • Run $FWDIR/bin/ioc_feeder -d -f on the Security Gateway to fetch feeds in debug mode.

  • Verify that the $FWDIR/conf/ioc_feeder.conf configuration file exists and is not corrupt. If the file is corrupt, delete the feed and re-add with the proper changes
  • Verify that there are no errors in these debug files:
    $FWDIR/log/ioc_feeder.elg
    $FWDIR/log/ext_ioc_push.elg

  • For remote feeds (HTTP / HTTPS):
    • Web servers might return files with unauthorized or forbidden headers as a response. Make sure that file is fetched as expected and contains the correct information, whether it’s a STIX file or a CSV file, in: $FWDIR/external_ioc/feed_name_folder
    • For HTTPS remote feeds, if the certificate update process failed, you can skip the certificate verification. Run: export EXT_IOC_NO_SSL_VALIDATION=1 on the Security Gateway.
    • Check the Proxy configurations:
      • By default, the Security Gateway's proxy will be used
      • Override proxy configurations: --proxy <proxy:port>
      • Don't use proxy: --proxy <none>
This solution has been verified for the specific scenario, described by the combination of Product, Version and Symptoms. It may not work in other scenarios.

Give us Feedback
Please rate this document
[1=Worst,5=Best]
Comment