Datalake2Splunk allows you to get Datalake data into Splunk for Orange Cyberdefense Datalake customers.
Datalake2splunk modular input builds a bulk-search using defined parameters and contacts Datalake API. It transforms the received JSON into a csv used as a lookup or can index Data into a custom index.
The IOC are available by using the lookups. Threat matches can be found by correlating with events log. Enterprise Security Threat Intelligence automates the threat matches job.
IOCs are first downloaded by the IDM and then sent to the Indexers to be indexed. When indexed, saved searches have to be enabled on the Search Head to aggregate the IOCs in lookups. Depending on what IOC needs to be used in lookups, the following search can be enabled:
- Datalake - Build Datalake <ioc_type> lookup
In addition, the macro datalake-index has be to configured with the index name where the IOCs are stored.
Lookups containing your regularly downloaded data are located under $SPLUNK_HOME/etc/apps/TA-datalake2splunk/lookups and named:
datalake_<ioc_type>.csv
All Lookup Definitions have been previously created and included in TA-Datalake2splunk (transforms.conf) with read permission for other apps.
From Enterprise Security main page click Configure -> Data Enrichment -> Threat Intelligence Management -> Sources
Click New and local to create a new intelligence entry and fill as follow:
- Name and Description: whatever name you choose
- Type: threatlist by default, (you can put “malware” type for datalake_file)
- URL is local path to lookup definition so lookup://datalake_<type>
- Weight is a risk_score multiplier. 1 to have no impact on assets and identities.
- Interval (in seconds) no need to have a smaller interval than what’s configured in the input. This control the frequency at which ES will update Intelligence KV Stores.
- Use Maximum age to control when splunk will delete the data from this lookup in Threat_Intelligence KV Stores. If you have very frequent pull, maximum age should remain low.
- Keep “Is Threat Intelligence” checked
This must be done for each type accepted by ES: ip, ip_range, domain, fqdn, url, file, email, ssl, regkey.
Next step is to check if ES has ingested the data.
There is a KV store for each type.
ip_intel for domain and ip, email_intel for email, http_intel for url, file_intel for file, certificate_intel for ssl and registry_intel for RegKey.
A lot of data is stacked in “description” field for this is the only way to add value to each line and will be parsed during Matching searches in order to add value to Threat Activity data later. You can use macro datalake_parser(description) to extract the data.
If you want to fully exploit collected data, you might want to modify the on-boarded Threat Activity search.
From Enterprise Security main page
Filter Correlationsearch in DA-ESS-Threat_Intelligence App and look for “Threat Activity Detected”
Edit the Search field
At the end add:
| `datalake_parser(threat_description)`
Intelligence data is searchable by any app.
You can call data thanks to the lookup definitions. Such as
|inputlookup datalake_domain
|inputlookup datalake_url
Now you’ll need to correlate your events with your logs. Either by using lookup command or inputlookup and check for a match.
You can easily get detailed data and correlate by using macro datalake_parser
Datalake IOC Stats offers insight on the content of the lookups such as most frequent threat types or sources. This is the base work to have a macro overview over IOC collection and a Splunk developer can edit those dashboards.
Dashboards are in development
Changed Regex extractions for IOC when indexed. It's a greedier regex because tags values sometimes contains brackets.
Splunklib updated for newest python SDK
Timeout increased for Bulk search pull
16 retries and 60 seconds wait between each try. So maximum time is 16 minutes for Datalake to provide bulk search result.
Increased timeout before aborting results. It is now 150 seconds.
Improved Internal logs
Indexed, Automatic Lookups or custom Lookup are now 3 separate storage options in Input menu.
IOC stats Dashboard for pre-built lookups.
Corrected savedsearches.conf
Added Readme.txt
Macros, Savedsearches and Dashboard added
Props for Datalake indexing
Lookup definitions for automatic lookups
Vetting changes for Splunk Cloud
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.