Datalake2SplunkV3 allows you to get Datalake data into Splunk for Orange Cyberdefense Datalake customers.
Datalake2splunk modular input builds a bulk-search using defined parameters and contacts Datalake API. It transforms the received JSON into a csv used as a lookup or can index Data into a custom index.
The IOC are available by using the lookups. Threat matches can be found by correlating with events log. Enterprise Security Threat Intelligence automates the threat matches job.

Prerequisites

• This app has been developed and tested with Python 3.7. and 3.9 so the minimal Splunk version needed is 9.1.
• Internet access.
• Bulk-search account on Datalake.

General Usage

Configuration

• Account: you can whoose a name for the account and enter your Long Term TOken generated on Datalake profile.
• If needed, configure Proxy and provide your proxy configuration in order to allow internet access to Datalake2Splunk

Input

• Name: a name for your input
• Interval: time in seconds between 2 fetchs. The process can take a while depending on the data volume, so we do not recommend under 1800. This field also support a cron schedule
• Index: If there is a need to index or if it is Splunk Cloud Classic Experience choose the right index otherwise whatever index you choose, no data will be indexed from this input
• API URL: automatically filled with the latest Datalake URL but customizable for those who need preprod access
• Query Hash: enter your datalake query hash obtained via the URL /gui/search?query_hash=<>
• Token: choose which Token (created in configuration) will be used to get Datalake information
  On Splunk Cloud Classic experience, the data inputs are managed on the Input Data Manager which is not the Search Head
• Storage Type: choose if the IOCs should be indexed. Required for Splunk Cloud Classic Experience
  Lookups : The app creates csv named after atom_types
  Custom Lookup : a lookup named datalake_<input_name> is created containing all indicators.

Splunk Cloud Classic Experience

IOCs are first downloaded by the IDM and then sent to the Indexers to be indexed. When indexed, saved searches have to be enabled on the Search Head to aggregate the IOCs in lookups. Depending on what IOC needs to be used in lookups, the following search can be enabled:

Datalake - Build Datalake <ioc_type> lookup 

In addition, the macro datalake-index has be to configured with the index name where the IOCs are stored.

Use Datalake Intelligence

Lookups containing your regularly downloaded data are located under $SPLUNK_HOME/etc/apps/TA-datalake2splunk/lookups and named:
datalake_<ioc_type>.csv
All Lookup Definitions have been previously created and included in TA-Datalake2splunk (transforms.conf) with read permission for other apps.

with Enterprise Security

From Enterprise Security main page click Configure -> Threat Intelligence
Inside Threat Intelligence Sources page
Click New and local to create a new intelligence entry and fill as follow:
- Name and Description: whatever name you choose
- Type: threatlist by default, (you can put “malware” type for datalake_file)
- URL is local path to lookup definition so lookup://datalake_<type>
- Weight is a risk_score multiplier. 1 to have no impact on assets and identities.
- Interval (in seconds) no need to have a smaller interval than what’s configured in the input. This control the frequency at which ES will update Intelligence KV Stores
- Use Maximum age to control when splunk will delete the data from this lookup in Threat_Intelligence KV Stores. If you have very frequent pull, maximum age should remain low.
- Keep “Is Threat Intelligence” checked
- on parsing tab use the Fields field to indicate which columns as which field such as

src_user:$2

for the email lookup

Threat Intelligence types : https://help.splunk.com/en/splunk-enterprise-security-8/administer/8.1/threat-intelligence/supported-types-of-threat-intelligence-in-splunk-enterprise-security

This must be done for each type accepted by ES: ip, domain, url, file, email, certificate.
Next step is to check if ES has ingested the data.
There is a KV store for each type.
ip_intel for domain and ip, email_intel for email, http_intel for url, file_intel for file, certificate_intel for certificates.

Enrich

When indexed and transformed into lookup, a lot of data is stacked in “description” field for this is the only way to add value to each line and will be parsed during Matching searches in order to add value to Threat Activity data later. You can use macro datalake_parser(description) to extract the data.
If you want to fully exploit collected data, you might want to modify the on-boarded Threat Activity search.
From Enterprise Security main page
Filter Correlationsearch in DA-ESS-Threat_Intelligence App and look for “Threat Activity Detected”
Edit the Search field
At the end add:

| `datalake_parser(threat_description)`

Without ES

Intelligence data is searchable by any app.
You can call data thanks to the lookup definitions. Such as

|inputlookup datalake_domain
|inputlookup datalake_url

Now you’ll need to correlate your events with your logs. Either by using lookup command or inputlookup and check for a match.
You can easily get detailed data and correlate by using macro datalake_parser

Reports and Dashboards

Datalake IOC Stats offers insight on the content of the lookups such as most frequent threat types or sources. This is the base work to have a macro overview over IOC collection and a Splunk developer can edit those dashboards.