Custom Intelligence Feeds feature provides an ability to add custom cyber intelligence feeds into the Threat Prevention engine. It allows fetching feeds from a third-party server directly to the Security Gateway to be enforced by Anti-Virus and Anti-Bot blades.
The Custom Intelligence Feeds feature also assists customers with the operational and engineering management challenges they face handling indicators: Managing and monitoring of the custom intelligence feeds is done with minimal operational overhead.
Indicator is a pattern of relevant observable malicious activity in an operational cyber domain, with relevant information on how to interpret it and how to handle it. Indicators are derived from intelligence, self-analysis and/or governments, partners etc.
Observable is an event or a stateful property that can be observed in an operational cyber domain. For example: IP address, MD5 file signature, URL, Mail sender address.
Table of Contents
Each indicator file can be either in CSV or STIX XML format.
CSV (*.csv) format
- Delimiter between fields is ','
- Delimiter between records is '\n'
- Fields are plain text
- Each record "must" contain the same number of comma-separated fields
- Quoted fields are accepted.
- The first line/record, prefixed with #, is a header containing column names in each of the fields.
#! DESCRIPTION = indi file,,,,,,
"#! REFERENCE = Indicator Bulletin; Feb 20, 2014",,,,,,
# FILE FORMAT:,,,,,,
"# All lines beginning ""#"" are comments",,,,,,
"# All lines beginning ""#!"" are metadata read by the SW",,,,,,
observ1,8d9b6b8912a2ed175b77acd40cbe9a73,MD5,medium,medium,AV,FILENAME:WUC Invitation Letter Guests.doc
NOTE:FWS type Flash file
IPV4ADDR:172.18.18.25|NOTE:Embedded EXE Remote C&C and Encoded Data
observ15,172.16.73.69,IP,medium,medium,AV,TCP:443|NOTE:Related to Flash
observ16,firstname.lastname@example.org,mail-to,,high,AV,"NOTE:truncated; samples have appended to
the subject the string ""PH000000NNNNNNN"" where NNNNNNN is a varying number"
Custom CSV format
Custom Intelligence Feeds feature supports different kinds of CSV structure files.
For the fetch to succeed, you must define the file's format, delimiter, and the comment lines to skip.
- The supported fields are: [name,value,type,confidence,severity,product,comment]
- To use a field value from the original file (must be one of the supported fields listed above), you must specify its location in the csv row by using #index.
For example, if you want to take the 3rd index in your csv file to be your observable's comment in the new file, use:
- To use a default value for all observables, use
- The Value field is mandatory and must be taken from the original file.
- The Type field is mandatory and must be taken from the original file or be sent as a default value for all observables.
- All other fields are optional and can either be taken from the original file or sent as a default value for all observables.
- When the feed's resource is a remote source (transport equals HTTP or HTTPS) - every time the feed will be fetched, it will parse according to the format that has been specified for this feed.
- Original CSV structure is a list of IP addresses
ioc_feeds add --feed_name ip_list --transport http --resource "http://blocklist.greensnow.co/greensnow.txt" --format [value:1,type:ip]
- Original CSV structure is a list of Domains
ioc_feeds add --feed_name domain_list --transport https --resource "https://isc.sans.edu/feeds/suspiciousdomains_High.txt" --format [type:domain,value:1] --comment "#, Site"
- Original CSV structure is a list of IP addresses in CIDR format and comment lines are marked as ';'
ioc_feeds add --feed_name ip_cidr_list_with_delimiter --transport https --resource "https://www.spamhaus.org/drop/edrop.txt" --format [value:1,type:ip] --delimiter ";" --comment ";"
- Original CSV structure is a list of IP addresses separated by '|' delimiter, and comment lines are marked as '#'
ioc_feeds add --feed_name ip_list_with_spaces --transport local_file --resource "/home/admin/ioc/ip_list_with_spaces.txt" --format [value:3,comment:#2,type:ip] --comment [#] --delimiter "|"
- Original CSV structure is a list of different types separated by ',' delimiter, and comment lines are marked as '#' or 'Site'
ioc_feeds add --feed_name try_custom_csv --transport http --resource http://192.168.13.13/ioc/bad_csv_format.csv --format [type:#1,value:#3,name:#6,comment:#7,product:av] --comment [#, Site] --delimiter ,
- Original CSV structure is a list of different types separated by ',' delimiter
ioc_feeds add --feed_name ioc_feed --transport http --resource "http://www.malwaredomainlist.com/updatecsv.php" --format [value:3,comment:#2,type:ip] --delimiter ,
STIX - Structured Threat Information eXpression
STIX™ is a collaborative community-driven effort to define and develop a standardized language to represent structured cyber threat information.
For more information, refer to STIX 1.x Archive Website.
Supported observables types
- IP Range
- SHA1, (from R80.40)
- SHA256, (from R80.40)
The feature is integrated in version R80.30 and above.
Note: To import external Custom Intelligence Feeds using SmartConsole in versions R81 and higher, refer to: Threat Prevention R81 Administration Guide > Configuring Advanced Threat Prevention Settings > Configuring Threat Indicators > Importing External Custom Intelligence Feeds > Importing External Custom Intelligence Feeds in SmartConsole.
Fetching new feed using CLI - ioc_feeds
Note: Starting from R81, the ioc_feeds command will also support the "--yes" command line argument, which when specified, will avoid any user interactions on yes/no questions and assume the user answers 'yes'. This option is hidden and isn't visible in the command usage output. This allows the command to be scripted for automation.
Feed's resource can be:
- URL - HTTP/HTTPS (--transport http --resource "http://10.0.0.1/my_feeds/stix_feed.xml")
*Self-signed certificate HTTPS resource will propmt for user agreement to update the bundle. It is possible to skip the certificate verification by running "export EXT_IOC_NO_SSL_VALIDATION=1" on the gateway.
- File on the machine (--transport local_file --resource "/home/admin/my_feed.csv")
- Directory on the machine, which contains the same feed_format - (--transport local_directory --resource "/home/admin/my_feed_folder")
[Expert@HostName:0]# ioc_feeds add --feed_name remote_csv_feed --transport http --resource "http://10.10.1.100/ioc/ioc_csv_file.csv" --feed_action Prevent
[Expert@HostName:0]# ioc_feeds add --feed_name ioc_stix_file --transport local_file --resource "/home/admin/ioc/ioc_stix_file.xml"
Note for R80.20SP:
Local feed content is refreshed periodically based on the feed fetching interval (see “set_interval” above).
When you modify the local feed source file, you must distribute the updated version with “asg_cp2blades”.
# vi /home/admin/ioc/ioc_stix_file.xml
# asg_cp2blades /home/admin/ioc/ioc_stix_file.xml
[Expert@HostName:0]# ioc_feeds show
[Expert@HostName:0]# ioc_feeds delete --feed_name ioc_stix_file
- Test feed fetching and parsing
[Expert@HostName:0]# ioc_feeds add --feed_name remote_stix_file --transport http --resource "http://www.public_indicators.com/ioc_stix_file.xml" --test true
Known feeds examples (using the Custom CSV feature)
Managing Custom Intelligence Feeds from the Security Management Server
This option is available starting from R80.30
- Run on one of your Security Gateways:
- If the feeds were fetched successfully, move /home/admin/export_ioc.tar.gz file to the Security Management.
- From the GUI Command-line tool, run:
put-file file-path "/home/admin" file-name "export_ioc.tar.gz" file-content @"/home/admin/export_ioc.tar.gz" --treat-value-as-file-by-prefix @ targets.1 "corporate-gateway" targets.2 "Backup-GW"
- From the GUI Command-line tool, run:
run-script script-name "Import Custom feeds" script "ioc_feeds import /home/admin/export_ioc.tar.gz" targets.1 "corporate-gateway" targets.2 "Backup-GW"
- Run $FWDIR/bin/ioc_feeder -d -f on the Security Gateway to fetch feeds in debug mode.
- Verify that the $FWDIR/conf/ioc_feeder.conf configuration file exists and is not corrupt. If the file is corrupt, delete the feed and re-add with the proper changes
- Verify that there are no errors in these debug files:
- For remote feeds (HTTP / HTTPS):
- Web servers might return files with unauthorized or forbidden headers as a response. Make sure that file is fetched as expected and contains the correct information, whether it’s a STIX file or a CSV file, in: $FWDIR/external_ioc/feed_name_folder
- For HTTPS remote feeds, if the certificate update process failed, you can skip the certificate verification. Run: export EXT_IOC_NO_SSL_VALIDATION=1 on the Security Gateway.
- Check the Proxy configurations:
- By default, the Security Gateway's proxy will be used
- Override proxy configurations: --proxy <proxy:port>
- Don't use proxy: --proxy <none>
This solution has been verified for the specific scenario, described by the combination of Product, Version and Symptoms. It may not work in other scenarios.