Datalake2Splunk allows you to get Datalake data into Splunk for Orange Cyberdefense Datalake customers.
Datalake2splunk modular input builds a bulk-search using defined parameters and contacts Datalake API. It transforms the received JSON into a csv used as a lookup or can index Data into a custom index.
The IOC are available by using the lookups. Threat matches can be found by correlating with events log. Enterprise Security Threat Intelligence automates the threat matches job.
o Name: a name for your input
o Interval: time in seconds between 2 gets. The process can take a while depending on the data volume, so we do not recommend under 1800.
o Index: If there is a need to index or if it is Splunk Cloud Classic Experience choose the right index otherwise whatever index you choose, no data will be indexed from this input. This is just a splunk requirement.
o Query Hash: enter your datalake query hash obtained via the URL /gui/search?query_hash=<>
o Token: choose which account (created in configuration) will be used to get Datalake information.
You’ll need to create one input per type and can have different Threat_Scores or Last Updated parameters for each Type. Each input will lead to the creation of one distinct lookup file named after ioc type.
On Splunk Cloud Classic experience, the data inputs are managed on the Input Data Manager which is not the Search Head. It means the standard data collection method described in the part “1. How it works” can’t work. An alternative mechanism has been built.
o Storage Type: choose if the IOCs should be indexed. Required for Splunk Cloud Classic Experience
IOCs are first downloaded by the IDM and then sent to the Indexers to be indexed. When indexed, saved searches have to be enabled on the Search Head to aggregate the IOCs in lookups. Depending on what IOC needs to be used in lookups, the following search can be enabled:
- Datalake - Build Datalake <ioc_type> lookup
In addition, the macro datalake-index has be to configured with the index name where the IOCs are stored.
Lookups containing your regularly downloaded data are located under $SPLUNK_HOME/etc/apps/TA-datalake2splunk/lookups and named:
datalake_<ioc_type>.csv
All Lookup Definitions have been previously created and included in TA-Datalake2splunk (transforms.conf) with read permission for other apps.
From Enterprise Security main page click Configure -> Data Enrichment -> Threat Intelligence Management -> Sources
Click New and local to create a new intelligence entry and fill as follow:
- Name and Description: whatever name you choose
- Type: threatlist by default, (you can put “malware” type for datalake_file)
- URL is local path to lookup definition so lookup://datalake_<type>
- Weight is a risk_score multiplier. 1 to have no impact on assets and identities.
- Interval (in seconds) no need to have a smaller interval than what’s configured in the input. This control the frequency at which ES will update Intelligence KV Stores.
- Use Maximum age to control when splunk will delete the data from this lookup in Threat_Intelligence KV Stores. If you have very frequent pull, maximum age should remain low.
- Keep “Is Threat Intelligence” checked
This must be done for each type accepted by ES: ip, ip_range, domain, fqdn, url, file, email, ssl, regkey.
Next step is to check if ES has ingested the data.
There is a KV store for each type.
ip_intel for domain and ip, email_intel for email, http_intel for url, file_intel for file, certificate_intel for ssl and registry_intel for RegKey.
A lot of data is stacked in “description” field for this is the only way to add value to each line and will be parsed during Matching searches in order to add value to Threat Activity data later. You can use macro datalake_parser(description)
to extract the data.
If you want to fully exploit collected data, you might want to modify the on-boarded Threat Activity search.
From Enterprise Security main page
Filter Correlationsearch in DA-ESS-Threat_Intelligence App and look for “Threat Activity Detected”
Edit the Search field
At the end add:
| `datalake_parser(threat_description)`
Intelligence data is searchable by any app.
You can call data thanks to the lookup_definitions. Such as
|inputlookup datalake_domain
|inputlookup datalake_url
Now you’ll need to correlate your events with your logs. Either by using lookup command or inputlookup and check for a match.
You can easily get detailed data and correlate by using macro datalake_parser
Datalake IOC Stats offers insight on the content of the lookups such as most frequent threat types or sources. This is the base work to have a macro overview over IOC collection and a Splunk developer can edit those dashboards.
Dashboards are in development
Corrected savedsearches.conf
Added Readme.txt
Macros, Savedsearches and Dashboard added
Props for Datalake indexing
Lookup definitions for automatic lookups
Vetting changes for Splunk Cloud
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.