Add-on for ingesting DMARC XML aggregate reports into Splunk from an IMAP account or local directory, with mitigations against XML, GZ and ZIP-bombs.
| Splunk version | Linux | Windows |
|---|---|---|
| 8.0 | Yes | Yes |
| 8.1 | Yes | Yes |
| 8.2 | Yes | Yes |
| 9.0 | Yes | Likely but untested |
Additional requirements:
| Instance type | Supported | Required | Description |
|---|---|---|---|
| Search head | Yes | Yes | Install this add-on on your search head(s) where CIM compliance of DMARC aggregate reports is required |
| Indexer | Yes | No | This add-on should be installed on a heavy forwarder that does the index time parsing. There is no need to install this add-on on an indexer too. |
| Universal Forwarder | No | No | This add-on is not supported on a Universal Forwarder because it requires Python |
| Heavy Forwarder | Yes | Yes | Install this add-on on a heavy forwarder to ingest DMARC XML aggregate reports into Splunk. |
The following table lists support for distributed deployment roles in a Splunk deployment:
| Deployment role | Supported | Description |
|---|---|---|
| Search head deployer | Yes | Install this add-on on your search head deployer to enable CIM compliance of DMARC aggregate reports on a Search Head Cluster |
| Cluster Master | No | This add-on should be installed on a heavy forwarder that performs parsing at index time. There is no need to install this add-on on an indexer too. |
| Deployment Server | Depends | This add-on can be (1) deployed unconfigured to a client or (2) deployed preconfigured with a directory input. Due to the encrypted credentials it cannot be deployed preconfigured for IMAP inputs. |
The TA-dmarc supports the following input modes:
TA-dmarc can watch a folder where you drop DMARC aggregate reports manually or otherwise.
It will read files with .xml, .zip or .xml.gz extention, ingest them into Splunk. Any invalid .xml, .zip or .xml.gz files are ignored.
TA-dmarc will leave files untouched in the directory: it uses internal checkpointing to skip files that have been previously ingested.
TA-dmarc can fetch DMARC aggregate report attachments from mails on an IMAP server. It will process attachments in .xml, .zip or xml.gz format and ingest them into Splunk.
TA-dmarc will leave the mails on the server: it uses internal checkpointing to skip mails that have been previously ingested.
This add-on handles the following file formats in which aggregate reports are delivered:
Mitigations are in place against:
From the XML sample below, these fields are created:
| XML field | Splunk field name | Value |
|---|---|---|
| feedback/report_metadata/org_name | rpt_metadata_org_name | google.com |
| feedback/report_metadata/email | rpt_metadata_email | noreply-dmarc-support@google.com |
| feedback/report_metadata/extra_contact_info | rpt_metadata_extra_contact_info | https://support.google.com/a/answer/2466580 |
| feedback/report_metadata/report_id | rpt_metadata_report_id | 13190401177475355109 |
| feedback/report_metadata/date_range/begin | rpt_metadata_date_range_begin | 1506988800 |
| feedback/report_metadata/date_range/end | rpt_metadata_date_range_end | 1507075199 |
| feedback/policy_published/domain | policy_published_domain | example.com |
| feedback/policy_published/adkim | policy_published_adkim | r |
| feedback/policy_published/adpf | policy_published_aspf | r |
| feedback/policy_published/p | policy_published_p | none |
| feedback/policy_published/pct | policy_published_pct | 100 |
| feedback/record/row/source_ip | row_source_ip | 192.0.2.78 |
| feedback/record/row/count | row_count | 1 |
| feedback/record/row/policy_evaluated/disposition | row_policy_evaluated_disposition | none |
| feedback/record/row/policy_evaluated/dkim | row_policy_evaluated_dkim | fail |
| feedback/record/row/policy_evaluated/spf | row_policy_evaluated_spf | fail |
| feedback/record/identifiers/header_from | identifiers_header_from | example.com |
| feedback/record/auth_results/spf/domain | auth_result_spf_domain | example.com |
| feedback/record/auth_results/spf/domain | auth_result_spf_result | fail |
Besides the fields contained in the report, additional fields are mapped from the CIM Authentication datamodel, based on the XML sample below:
| Authentication datamodel field name | Value |
|---|---|
| action | failure |
| app | dmarc |
| dest | google.com |
| signature | Use of mail-from domain example.com |
| signature_id | 13190401177475355109 |
| src | resolved.name.if.available.test |
| src_ip | 192.0.2.78 |
| src_user | example.com |
| eventtype | dmarc_rua_spf_only |
| tag | authentication, insecure |
<?xml version="1.0" encoding="UTF-8" ?>
<feedback>
<report_metadata>
<org_name>google.com</org_name>
<email>noreply-dmarc-support@google.com</email>
<extra_contact_info>https://support.google.com/a/answer/2466580</extra_contact_info>
<report_id>13190401177475355109</report_id>
<date_range>
<begin>1506988800</begin>
<end>1507075199</end>
</date_range>
</report_metadata>
<policy_published>
<domain>example.com</domain>
<adkim>r</adkim>
<aspf>r</aspf>
<p>none</p>
<sp>none</sp>
<pct>100</pct>
</policy_published>
<record>
<row>
<source_ip>192.0.2.78</source_ip>
<count>1</count>
<policy_evaluated>
<disposition>none</disposition>
<dkim>fail</dkim>
<spf>fail</spf>
</policy_evaluated>
</row>
<identifiers>
<header_from>example.com</header_from>
</identifiers>
<auth_results>
<spf>
<domain>example.com</domain>
<result>fail</result>
</spf>
</auth_results>
</record>
</feedback>
The DMARC-imap input saves checkpointing data in KVstore.
To see its contents: |inputlookup ta_dmarc_checkpointer_lookup
If you want to reindex an entire mailbox, you can do so by deleting the TA-dmarc KVstore checkpointing data through this Splunk command:
|inputlookup ta_dmarc_checkpointer_lookup
|search state!="*input=dmarc_imap, server=imap.gmail.com*"
|outputlookup ta_dmarc_checkpointer_lookup`
If you want to reindex a single DMARC report, you can do so by deleting its corresponding record from KVstore:
| inputlookup ta_dmarc_checkpointer_lookup
| search state!="*Report-ID: 3596274322387252907*"
| outputlookup ta_dmarc_checkpointer_lookup
Reindexing a DMARC report from a directory input is left as an excercise for the reader.
The following software components are used in this add-on:
This is an open source project without warranty of any kind. No support is provided. However, a public repository and issue tracker are available at https://github.com/jorritfolmer/TA-dmarc
Fixed exception in directory input. (Thanks to Georgi Georgiev for providing a patch)
Fixed resolving of IP addresses in src field.
(Thanks to Martin Wright for reporting this issue)
Added support for Splunk 8.x and Python 3.x
The event output format changed to JSON. This is the new default because JSON is a better match for the structured aggregate reports than a flat key=value (KV) format. See for example issue #4 on Github. If you want your inputs to to keep using the KV format:
Note that KV format output is likely to be deprecated in future versions, and enhancements like multiple XSD validation aren't available in KV output format.
Improved checkpointing:
Addressed concerns from Splunk Cert Admin:
NOTES BEFORE UPGRADING FROM v1.x TO 2.0.0:
Because of the changes to KVstore logic, the add-on will re-index every report on IMAP, leading to duplicate events.
Added non-multipart support for mail messages on IMAP
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.