Skip to content

jasonsford/intel_collector

Repository files navigation

intel_collector

intel collector is a Python library to query various sources of threat intelligence for data on domains, file hashes, and IP addresses. Responses are returned in JSON format and written to CSV.

CrowdStrike Falcon and Microsoft Defender for Endpoint customers can also query their tenant for the presence of indicators within their own environment.

Supported APIs

Free Resources

BinaryEdge (binaryedge.io)
Circl.lu (hashlookup.circl.lu)
Echotrail (echotrail.io)
Filescan.io (filescan.io)
GreyNoise Community API (api.greynoise.io)
Hybrid Analysis (hybrid-analysis.com)
LeakIX (leakix.net)
Netlas (app.netlas.io)
Onyphe Free Tier (onyphe.io)
PulseDive (pulsedive.com)
Shodan (shodan.io)
Stalkphish (stalkphish.io)
Stratosphere IPS (stratosphereips.org)
Triage (tria.ge)
Urlhaus (urlhaus-api.abuse.ch)
Urlscan.io (urlscan.io)
VirusTotal (virustotal.com)

Paid Resources

CrowdStrike Falcon Intel (api.crowdstrike.com)
Emerging Threats Intelligence (api.emergingthreats.net)
Microsoft Defender for Endpoint (api.securitycenter.windows.com)

Setting API keys

API keys are set from within the library for each intel source.

# binaryedge.py (BinaryEdge)
binaryedge_api_key = 'your binary edge api key'

# crwd.py (CrowdStrike)
crwd_client_id = 'your crowdstrike api client id'
crwd_client_secret = 'your crowdstrike api client secret'

# echotrail.py (Echotrail.io)
echotrail_api_key = 'your echotrail api key'

# etintel.py (Emerging Threats Intelligence)
etintel_api_key = 'your emerging threats intelligence api key'

# filescan.py (Filescan.io)
filescan_api_key = 'your filescan.io api key'

# greynoise.py (GreyNoise.io)
greynoise_api_key = 'your greynoise community api key'

# hybrid.py (Hybrid Analysis)
hybrid_api_key = 'your hybrid analysis api key'

# leakix.py (LeakIX)
leakix_api_key = 'your leakix api key'

# msde.py (Microsoft Defender for Endpoint)
msft_tenant_id = 'your M365 tenant id'
msft_client_id = 'your M365 client id'
msft_client_secret = 'your M365 client secret'

# netlas.py (Netlas.io)
netlas_api_key 'your netlas api key'

# onyphe.py (Onyphe)
onyphe_api_key = 'your onyphe api key'

# pulsedive.py (Pulsedive)
pulsedive_api_key = 'your pulsedive api key'

# shodanpy.py (Shodan)
shodan_api_key = 'your shodan api key'

# stalkphish.py (Stalkphish)
stalkphish_api_key = 'Token your stalkphish api key'

# triage.py (Tria.ge)
triage_api_key = 'your tria.ge api key'

# urlscan.py (Urlscan.io)
urlscan_api_key = 'your urlscan.io api key'

# virustotal.py (VirusTotal)
virustotal_api_key = 'your virustotal api key'

Disabling Modules

All modules are enabled by default. Modules within each function can be disabled if you don't have an API key or don't wish to utilize them. Add # to the beginning of these lines as needed:

# Free Resources 
import binaryedge
import circl
import echotrail
import filescan
import greynoise
import hybrid
import leakix
import netlas
import onyphe
import pulsedive
import shodanpy
import stalkphish
import strato
import triage
import urlhaus
import urlscan
import virustotal

# Paid Resources
import crwd
import msde
import etintel

find_domain

    results["Emerging Threats"] = etintel.domain(domain)        # Emerging Threats
    results["Microsoft"] = msde.domain(domain)                  # Microsoft Defender for Endpoint
    results["Netlas"] = netlas.iocs(domain)                     # Netlas.io
    results["Onyphe"] = onyphe.domain(domain)                   # Onyphe
    results["Pulsedive"] = pulsedive.iocs(domain)               # Pulsedive
    results["Shodan"] = shodanpy.domain(domain)                 # Shodan
    results["Tria.ge"] = triage.iocs(domain,'domain')           # Tria.ge
    results["URLhaus"] = urlhaus.iocs(domain,'host')            # Urlhaus
    results["URLScan"] = urlscan.domain(domain)                 # Urlscan.io
    results["VirusTotal"] = virustotal.domain(domain)           # VirusTotal

find_hash

    results["Circl.lu"] = circl.hash(hash,'md5')         # Circl.lu
    results["CrowdStrike"] = crwd.iocs(hash,'md5')       # CrowdStrike Falcon
    results["Echotrail"] = echotrail.hash(hash)          # Echotrail.io
    results["Emerging Threats"] = etintel.hash(hash)     # Emerging Threats
    results["FileScan.io"] = filescan.hash(hash,'md5')   # Filescan.io
    results["Hybrid Analysis"] = hybrid.hash(hash)       # Hybrid Analysis
    results["Microsoft"] = msde.hash(hash)               # Microsoft Defender for Endpoint            
    results["Tria.ge"] = triage.iocs(hash,'sha1')        # Tria.ge
    results["URLhaus"] = urlhaus.iocs(hash,'sha256')     # Urlhaus
    results["URLScan"] = urlscan.hash(hash)              # Urlscan.io
    results["VirusTotal"] = virustotal.hash(hash)        # VirusTotal

find_ip

    results["BinaryEdge"] = binaryedge.ip(ip)            # BinaryEdge.io
    results["CrowdStrike"] = crwd.iocs(ip,'ipv4')        # CrowdStrike Falcon
    results["Emerging Threats"] = etintel.ip(ip)         # Emerging Threats
    results["GreyNoise"] = greynoise.ip(ip)              # GreyNoise
    results["LeakIX"] = leakix.ip(ip)                    # LeakIX
    results["Microsoft"] = msde.ip(ip)                   # Microsoft Defender for Endpoint
    results["Netlas"] = netlas.iocs(ip)                  # Netlas.io
    results["Onyphe"] = onyphe.ip(ip)                    # Onyphe
    results["Pulsedive"] = pulsedive.iocs(ip)            # Pulsedive
    results["Shodan"] = shodanpy.ip(ip)                  # Shodan
    results["Stalkphish"] = stalkphish.ip(ip)            # Stalkphish
    results["Stratosphere IPS"] = strato.ip(ip)          # Stratosphere IPS
    results["Tria.ge"] = triage.iocs(ip,'ip')            # Tria.ge
    results["URLhaus"] = urlhaus.iocs(ip,'host')         # Urlhaus
    results["URLScan"] = urlscan.ip(ip)                  # Urlscan.io
    results["VirusTotal"] = virustotal.ip(ip)            # VirusTotal

Usage

# Import the library
from intel_collector import intel_collector

# Initialize client API keys and base URLs  
go = intel_collector()

# Get information on a domain
go.find_domain('bkdata.vn')

# Get information on an IP address
go.find_ip('103.161.17.242')

# Get information on a file hash
go.find_hash('870c31aa344b2950d0ea4849a472dafed312ecee8aa212c47bf543668bbee8e9')
go.find_hash('1e5bc9d7e413ddd7902c2932e418702b84d0cc07')
go.find_hash('177f3c8a2623d4efb41b0020d680be83')

Helpful hints for searching file hashes:

  • The Circl.lu API supports the following indicator types (md5, sha1, sha256)
  • The Crowdstrike Falcon API for custom IOCs supports the following indicator types (md5, sha256)
  • The Echotrail API supports the following indicator types (md5, sha256)
  • The ET Intel API supports the (md5) indicator type
  • The Filescan.io API supports the following indicator types (md5, sha1, sha256)
  • The Hybrid Analysis API supports the following indicator types (md5, sha1, sha256)
  • The Microsoft Defender for Endpoint API supports the following indicator types (sha1, sha256)
  • The Tria.ge API supports the following indicator types (md5, sha1, sha256, sha512)
  • The Urlhaus API supports the following indicator types (md5, sha256)
  • The Urlscan.io API supports the (sha256) indicator type
  • The VirusTotal API supports the following indicator types (md5, sha1, shad256)

Sample Output

go.find_domain('bkdata.vn')
    Found in Microsoft Defender for Endpoint - Domains
    Found in Netlas.io
    Found in Onyphe
    Found in Shodan
go.find_hash('870c31aa344b2950d0ea4849a472dafed312ecee8aa212c47bf543668bbee8e9')
    Response from Echotrail.io
    Found in Filescan.io
    Found in Hybrid Analysis
    Found in Microsoft Defender for Endpoint - Global File Stats
    Found in Tria.ge
    Found in VirusTotal - Files
go.find_ip('103.161.17.242')
    Found in Emerging Threats - Events
    Response from GreyNoise
    Found in Microsoft Defender for Endpoint - IP Stats
    Found in Onyphe
    Found in Shodan
    Found in Stalkphish
    Response from URLscan.io

Contributing

Pull requests are welcome. For major changes, please open an issue to discuss what you would like to change.

Authors

Jason Ford

License

GPLv3

About

Python library to query various sources of threat intelligence for data on domains, file hashes, and IP addresses.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages