Ingesting Data from Splunk to Microsoft Fabric using Microsoft Fabric Addon for Splunk

Microsoft Fabric Add-On for Splunk allows users to ingest logs from splunk platform using the kusto python sdk.
Learn More about Microsoft Fabric at https://learn.microsoft.com/en-us/fabric/

Details

Details on pre-requisites, configuring the add-on and viewing the data in Microsoft Fabric is covered in this section.

Background

When we add data to Splunk, the Splunk indexer processes it and stores it in a designated index (either, by default, in the main index or custom index). Searching in Splunk involves using the indexed data for the purpose of creating metrics, dashboards and alerts. This Splunk add-on triggers an action based on the alert in Splunk. We can use Alert actions to send data to Microsoft Fabric using the specified addon.

This add-on uses kusto python sdk(https://learn.microsoft.com/en-us/azure/data-explorer/kusto/api/python/kusto-python-client-library) to send log data to Microsoft Fabric. Hence, this addon supports queued mode of ingestion by default. This addon has a durable feature as well which helps to minimize data loss during any unexpected network error scenarios. But durability in ingestion comes at the cost of throughput, so it is advised to use this option judiciously.

Prerequisites

1. A Splunk instance (latest release) with the required installation privileges to configure add-ons.
2. Azure Service Principal such as (clientId, clientSecret and TenantId).

Step 1: Install the Microsoft Fabric Addon

1. Download the Microsoft Fabric Addon from the Splunkbase website
2. Log in to your Splunk instance as an administrator.
3. Navigate to "Apps" and click on "Manage Apps."
4. Click on "Install app from file" and select the downloaded Microsoft Fabric for Splunk file.
5. Follow the prompts to complete the installation

After installation of the Splunk Addon for alerts, it should be visible in the Dashboard -> Alert Actions

Step 2: Create Splunk Index

1. Log in to your Splunk instance.
2. Navigate to "Settings" and click on "Indexes."
3. Click on "New Index" to create a new index.
4. Provide a name for the index and configure the necessary settings (e.g., retention period, data model, etc.).
5. Save the index configuration.

Step 3: Create Real Time Analytics KQL Database in Microsoft Fabric

1. Login to Microsoft Fabric, navigate to Synapse Real Time Analytics.
2. Click on KQL Database and create database.
3. The created database opens up and we need to create a KQL table for ingestion.

Step 4: Configure Splunk Addon for Microsoft Fabric

1. In Splunk dashboard, Enter your search query in the Search bar based on which alerts will be generated and this alert data will be ingested to Microsoft Fabric.
2. Click on Save As and select Alert.
3. Provide a name for the alert and provide the interval at which the alert should be triggered.
4. Select the alert action as "Send to Microsoft Fabric"
   
5. Configure the Azure Data Explorer connection details such as application client Id, application client secret, cluster name, database name, table name.
   
6. When the alert is created, it should be visible in Splunk Dashboard -> Alerts
   

Step 5: Verify the data in Microsoft Fabric

1. Once the alert is triggered in Splunk, the data will be ingested to Microsoft Fabric.
2. Start monitoring the addon logs in Splunk by navigating to Settings -> Alert Actions -> View log events.
3. Verify the data in Microsoft Fabric using the database and table name in the previous step.
   

Azure Data Explorer Addon Parameters

The following is the list of parameters which need to be entered/selected while configuring the addon
1. Microsoft Fabric Cluster Ingestion URL: Represents the ingestion URL of the Microsoft Fabric KQL cluster.
2. Azure Application Client Id: Represents the Azure Application Client Id credentials required to access Microsoft Fabric.
3. Azure Application Client secret: Represents the Azure Application Client secret credentials required to access Microsoft Fabric.
4. Azure Application Tenant Id: Represents the Azure Application Tenant Id required to access Microsoft Fabric.
5. Microsoft Fabric Database Name: This represents the name of the database created in the Microsoft Fabric KQL cluster, where we want our data to be ingested.
6. Microsoft Fabric Table Name: This represents the name of the table inside the database created in the Microsoft Fabric KQL cluster, where we want our data to be ingested.
7. Microsoft Fabric Table Mapping Name: This represents the Microsoft Fabric KQL table mapping name used to map the incoming data to created Microsoft Fabric KQL table.
8. Remove Extra Fields : This represents whether we want to remove empty fields in the splunk event payload
9. Durable Mode : This property specifies whether durability mode is required during ingestion. When set to true, the ingestion throughput is impacted.