Support Center > Search Results > SecureKnowledge Details
What is the "Custom Intelligence Feeds" feature? Technical Level

Custom Intelligence Feeds feature provides an ability to add custom cyber intelligence feeds into the Threat Prevention engine. It allows fetching feeds from a third-party server directly to the Security Gateway to be enforced by Anti-Virus and Anti-Bot blades.

The Custom Intelligence Feeds feature also assists customers with the operational and engineering management challenges they face handling indicators: Managing and monitoring of the custom intelligence feeds is done with minimal operational overhead.

Indicator is a pattern of relevant observable malicious activity in an operational cyber domain, with relevant information on how to interpret it and how to handle it. Indicators are derived from intelligence, self-analysis and/or governments, partners etc.

Observable is an event or a stateful property that can be observed in an operational cyber domain. For example: IP address, MD5 file signature, URL, Mail sender address.

Table of Contents

  • Supported Formats
  • Installation
  • Fetching new feed using CLI - ioc_feeds
  • Known feeds examples
  • Known Limitations
  • Troubleshooting

Supported Formats

Each indicator file can be either in CSV or STIX XML format.

CSV (*.csv) format

  • Delimiter between fields is ','
  • Delimiter between records is '\n'
  • Fields are plain text
  • Each record "must" contain the same number of comma-separated fields
  • Quoted fields are accepted.
  • The first line/record, prefixed with #, is a header containing column names in each of the fields.


#! DESCRIPTION = indi file,,,,,,
"#! REFERENCE = Indicator Bulletin; Feb 20, 2014",,,,,,
# FILE FORMAT:,,,,,,
"#      All lines beginning  ""#"" are comments",,,,,,
"#      All lines beginning  ""#!"" are metadata read by the SW",,,,,,
observ1,8d9b6b8912a2ed175b77acd40cbe9a73,MD5,medium,medium,AV,FILENAME:WUC Invitation Letter Guests.doc
NOTE:FWS type Flash file
IPV4ADDR:|NOTE:Embedded EXE Remote C&C and Encoded Data
observ15,,IP,medium,medium,AV,TCP:443|NOTE:Related to Flash 
observ16,,mail-to,,high,AV,"NOTE:truncated; samples have appended to 
the subject the string ""PH000000NNNNNNN"" where NNNNNNN is a varying number"


Custom CSV format

Custom Intelligence Feeds feature supports different kinds of CSV structure files.

For the fetch to succeed, you must define the file's format, delimiter, and the comment lines to skip.

Syntax Notes:

  • The supported fields are: [name,value,type,confidence,severity,product,comment]
  • To use a field value from the original file (must be one of the supported fields listed above), you must specify its location in the csv row by using #index.
    For example, if you want to take the 3rd index in your csv file to be your observable's comment in the new file, use:

   --format [comment:#3]

  • To use a default value for all observables, use

--format [type:domain]

  • The Value field is mandatory and must be taken from the original file.
  • The Type field is mandatory and must be taken from the original file or be sent as a default value for all observables.
  • All other fields are optional and can either be taken from the original file or sent as a default value for all observables.
  • When the feed's resource is a remote source (transport equals HTTP or HTTPS) - every time the feed will be fetched, it will parse according to the format that has been specified for this feed.



  • Original CSV structure is a list of IP addresses

ioc_feeds add --feed_name ip_list --transport http --resource "" --format [value:1,type:ip]

  • Original CSV structure is a list of Domains

ioc_feeds add --feed_name domain_list --transport https --resource "" --format [type:domain,value:1] --comment "#, Site"

  • Original CSV structure is a list of IP addresses in CIDR format and comment lines are marked as ';'

ioc_feeds add --feed_name ip_cidr_list_with_delimiter --transport https --resource "" --format [value:1,type:ip] --delimiter ";" --comment ";"

  • Original CSV structure is a list of IP addresses separated by '|' delimiter, and comment lines are marked as '#'

ioc_feeds add --feed_name ip_list_with_spaces --transport local_file --resource "/home/admin/ioc/ip_list_with_spaces.txt" --format [value:3,comment:#2,type:ip] --comment [#] --delimiter "|"

  • Original CSV structure is a list of different types separated by ',' delimiter, and comment lines are marked as '#' or 'Site'

ioc_feeds add --feed_name try_custom_csv --transport http --resource --format [type:#1,value:#3,name:#6,comment:#7,product:av] --comment [#, Site] --delimiter ,

  • Original CSV structure is a list of different types separated by ',' delimiter

ioc_feeds add --feed_name ioc_feed --transport http --resource "" --format [value:3,comment:#2,type:ip] --delimiter ,

    STIX - Structured Threat Information eXpression

    STIX™ is a collaborative community-driven effort to define and develop a standardized language to represent structured cyber threat information.
    For more information, refer to STIX 1.x Archive Website.

    Supported observables types

    • URL
    • Domain
    • IP
    • IP Range
    • MD5
    • Mail-subject
    • Mail-from
    • Mail-to
    • Mail-cc
    • Mail-reply-to
    • SHA1, (from R80.40)
    • SHA256, (from R80.40)


    The feature is integrated in version R80.30 and above.

    Note: To import external Custom Intelligence Feeds using SmartConsole in versions R81 and higher, refer to: Threat Prevention R81 Administration Guide > Configuring Advanced Threat Prevention Settings > Configuring Threat Indicators > Importing External Custom Intelligence Feeds > Importing External Custom Intelligence Feeds in SmartConsole.

    Fetching new feed using CLI - ioc_feeds

    Note: Starting from R81, the ioc_feeds command will also support the "--yes" command line argument, which when specified, will avoid any user interactions on yes/no questions and assume the user answers 'yes'. This option is hidden and isn't visible in the command usage output. This allows the command to be scripted for automation.

    Feed's resource can be:

    • URL - HTTP/HTTPS (--transport http --resource "")
      *Self-signed certificate HTTPS resource will propmt for user agreement to update the bundle. It is possible to skip the certificate verification by running "export EXT_IOC_NO_SSL_VALIDATION=1" on the gateway.
    • File on the machine (--transport local_file --resource "/home/admin/my_feed.csv")
    • Directory on the machine, which contains the same feed_format - (--transport local_directory --resource "/home/admin/my_feed_folder")

    Parameter Description Example
    push Push feeds now ioc_feeds push
    show Prints all existing feeds ioc_feeds show
    show --feed_name <feed>  Prints specific feed details  ioc_feeds show --feed_name local_feed
    show_interval  Prints fetching interval  ioc_feeds show_interval 
    set_interval sec

    Set interval for fetching in seconds

    *Feed fetching interval - same for all feeds

    ioc_feeds set_interval 1000 
    show_scanning_mode  Prints scanning mode  ioc_feeds show_scanning_mode 
    set_scanning_mode  Set scanning mode - on/off  ioc_feeds set_scanning_mode off

    Adding new feed.

    Mandatory fields:

    --feed_name <feed>

    --transport <http/https/local_file/local_directory>

    --resource <url/full_path>

    Optional fields:

    --state <true/false> (active/inactive. default - True)

    --feed_action <Prevent/Detect/Ask> (default – Prevent)

    --user_name <user> (prompt for password)

    --proxy <proxy:port>

    --proxy <none> – don't use proxy

    --proxy <proxy:port> - override gatewayproxy for feed

    (not mentioning proxy flag - gateway proxy will be used)

    --proxy_user_name <user> (prompt for password)

    --test true
    test feed fetching and parsing


    ioc_feeds add --feed_name local_feed --transport local_file --resource /home/admin/my_feed.csv

    ioc_feeds add --feed_name remote_feed --transport http --resource --proxy --state false –feed_action detect --user_name


    Modify existing feed.
    Fields that will not be mentioned will stay as they were before 

    ioc_feeds modify --feed_name local_feed --state true
    delete  Delete the existing feed  ioc_feeds delete --feed_name local_feed 


    More examples:

    • Adding a new remote feed

    [Expert@HostName:0]# ioc_feeds add --feed_name remote_csv_feed --transport http --resource "" --feed_action Prevent

    • Adding a new local feed

    [Expert@HostName:0]# ioc_feeds add --feed_name ioc_stix_file --transport local_file --resource "/home/admin/ioc/ioc_stix_file.xml"

    Note for R80.20SP:

    Local feed content is refreshed periodically based on the feed fetching interval (see “set_interval” above).
    When you modify the local feed source file, you must distribute the updated version with “asg_cp2blades”.
    # vi /home/admin/ioc/ioc_stix_file.xml
    # asg_cp2blades /home/admin/ioc/ioc_stix_file.xml

    • Printing existing feeds

    [Expert@HostName:0]# ioc_feeds show

    • Deleting a feed

    [Expert@HostName:0]# ioc_feeds delete --feed_name ioc_stix_file

    • Test feed fetching and parsing

    [Expert@HostName:0]# ioc_feeds add --feed_name remote_stix_file --transport http --resource "" --test true


    Known feeds examples (using the Custom CSV feature)

    Description URL Command Line
    Alienvault IP Reputation ioc_feeds add --feed_name reputation --transport http --resource "" --format [type:ip,value:#1,comment:#4] --delimiter "#"
    Domains ioc_feeds add --feed_name domains --transport https --resource "" --format [type:domain,value:#1]
    IPs ioc_feeds add --feed_name ips --transport https --resource "" --format [type:ip,value:#2] --comment [#] --delimiter ","
    Talos IP Blacklist ioc_feeds add --feed_name ip_blacklist --transport https --resource "" --format [type:ip,value:#1]
    Spam List ioc_feeds add --feed_name spam_list --transport http --resource "" --format [type:ip,value:#3,comment:#4] --comment ["#", "first_seen"] --delimiter ","
    Cybercrime hash list ioc_feeds add --feed_name hash_list --transport http --resource "" --format [type:md5,value:#1]


    Managing Custom Intelligence Feeds from the Security Management Server

    This option is available starting from R80.30

    1. Run on one of your Security Gateways: 
      ioc_feeds export
    2. If the feeds were fetched successfully, move /home/admin/export_ioc.tar.gz file to the Security Management.
    3. From the GUI Command-line tool, run:
      put-file file-path "/home/admin" file-name "export_ioc.tar.gz" file-content @"/home/admin/export_ioc.tar.gz" --treat-value-as-file-by-prefix @ targets.1 "corporate-gateway" targets.2 "Backup-GW"
      • From the GUI Command-line tool, run:
        run-script script-name "Import Custom feeds" script "ioc_feeds import /home/admin/export_ioc.tar.gz" targets.1 "corporate-gateway" targets.2 "Backup-GW"


        Known Limitations

        Observables of IP addresses and IP Ranges can hold IPv4 values only. Starting R81.00 IPV6 is supported as well.
        MD5 observables cannot be enforced by Anti-Bot Blade. If user does not enable Anti-Virus blade, there will be no enforcement.
        For R80.20SP, a Jumbo Hotfix Accumulator installation is required.
        Inbound traffic to a host behind the gateway does not get blocked.
        e.g: IP that is on the feed, sends ICMP Request to a host behind the gateway. This traffic does not get blocked. 

        In R81, this traffic is blocked. 
        Not supported on version R81 SP



        • Run $FWDIR/bin/ioc_feeder -d -f on the Security Gateway to fetch feeds in debug mode.

        • Verify that the $FWDIR/conf/ioc_feeder.conf configuration file exists and is not corrupt. If the file is corrupt, delete the feed and re-add with the proper changes
        • Verify that there are no errors in these debug files:

        • For remote feeds (HTTP / HTTPS):
          • Web servers might return files with unauthorized or forbidden headers as a response. Make sure that file is fetched as expected and contains the correct information, whether it’s a STIX file or a CSV file, in: $FWDIR/external_ioc/feed_name_folder
          • For HTTPS remote feeds, if the certificate update process failed, you can skip the certificate verification. Run: export EXT_IOC_NO_SSL_VALIDATION=1 on the Security Gateway.
          • Check the Proxy configurations:
            • By default, the Security Gateway's proxy will be used
            • Override proxy configurations: --proxy <proxy:port>
            • Don't use proxy: --proxy <none>
          This solution has been verified for the specific scenario, described by the combination of Product, Version and Symptoms. It may not work in other scenarios.

          Give us Feedback
          Please rate this document