Configure your collectd agents to send metrics using the "write_http" plugin to a Splunk HTTP Event Collector with the built-in sourcetype called “collectd_http”
i.e. “Analytics for Linux” doesn’t require a separate Splunk Technology Add-on (TA) for ingestion of metrics.
e.g. collectd:
indexes.conf :-
[collectd]
datatype = metric
homePath = $SPLUNK_DB/collectd/db
coldPath = $SPLUNK_DB/collectd/colddb
thawedPath = $SPLUNK_DB/collectd/thaweddb
e.g. replace hec_token below:
inputs.conf :-
[http://Collectd]
token=hec_token
disabled=0
index=collectd
source=collectd token
sourcetype=collectd_http
an example configuration file has been included in this app:
i.e. $SPLUNK_HOME/etc/apps/sh_collectd/examples/collectd.conf
Note: You must replace splunk_server & hec_token in the Node definition, e.g.
<Plugin write_http>
<Node "node1">
URL "https://splunk_server:8088/services/collector/raw"
Header "Authorization: Splunk hec_token"
Format "JSON"
VerifyPeer false
VerifyHost false
Metrics true
StoreRates true
</Node>
</Plugin>
/etc/init.d/collectd start
Developer: Luke Harris (Data Analytics Practice Lead at Katana1)
Web: https://katana1.com
Twitter: https://twitter.com/skywalka
https://www.datagods.com.au/news/splunk-metrics-ftw
Issues & Pull Requests are welcome :)
https://bitbucket.org/katana1/analyticsforlinux
This app has been tested with the following versions of collectd:
Ubuntu: collectd 5.6.3.2, 5.7.2.52, & 5.8.0.19
RHEL 7: collectd 5.8.0-1
CentOS 7: collectd 5.7.1-2
Amazon Linux: collectd 5.7.1-3.19
Mac OS X: collectd 5.8.0
Note: collectd version 5.6 or higher is required.
Minimum Version 7.x
http://docs.splunk.com/Documentation/Splunk/latest/Metrics/GetMetricsInCollectd
1/ Run the following search to confirm that metrics are being indexed :-
| mcatalog values(metric_name)
2/ Add the collectd index to "Indexes searched by default" :-
3/ Ensure that the sourcetype is set to "collectd_http" :-
Example error in splunkd.log:
WARN IndexProcessor - Index Processor: Metric value=unset is not valid for source=collectd_linux, sourcetype=hec, host=foo, index=collectd. Metric event data with an invalid metric value would not be indexed. Ensure the input metric data is not malformed.
4/ Ensure that collectd can connect to the network using TCP by turning on the "collectd_tcp_network_connect" boolean for SELinux which is disabled by default :-
Example error in /var/log/messages or /var/log/syslog:
write_http plugin: curl_easy_perform failed with status 7: Failed to connect to 10.11.22.33: Permission denied
Fix:
# setsebool -P collectd_tcp_network_connect 1
This app was inspired by the 'Collectd App for Splunk Enterprise' by Nexinto GmbH
Added Overview dashboard, Added CPU Overview dashboard, Fixed Bugs in Storage Trends dashboard, & Added Ping dashboard
Added a host multi-select dropdown to the Overview and CPU Overview dashboards
Added Overview dashboard, Added CPU Overview dashboard, & Fixed Bugs in Storage Trends dashboard
Updated Storage Performance chart with per host metrics, Added Index dropdown to Metrics dashboards, & Updated CPU Usage dashboard with Cores dropdown
Updated configs in readiness for app certification
New dashboards for nginx & Apache web servers
Added the Metrics Navigator dashboard to dynamically display multiple charts of metrics
Added new dashboards to explore and compare metrics
Version 2.0.1 - Removed erroneous configs
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.