Splunk SAI and Metrics

I recently had a play with Splunk SAI and I wanted to monitor a number of Linux Systems.

My options to collect metric data from the systems was to use collectd and send the data to HEC OR use Luke Harris’s TA-Metrics add-on https://splunkbase.splunk.com/app/4856/ and send the data to the indexers.

I opted for the metrics add-on as this seemed so much easier, the SAI has made it easier to deploy collectd + agent, but this TA to me was even easier, and it supports SAI.

I created a linux_metrics index in Splunk

homePath = $SPLUNK_DB /linux_metrics/db
coldPath = $SPLUNK_DB /linux_metrics/colddb
thawedPath = $SPLUNK_DB/linux_metrics/thaweddb
datatype = metric
frozenTimePeriodInSecs = 2419200

The inputs.conf from the default folder is already set for you, its polls every 5 minutes for various metrics. I then created a local folder in the TA-linux-metrics folder and copied the process_mon.conf with below configuration.

allowlist = CROND,run*,systemd*,chronyd,rsyslogd,auditd,journal,su,splunk,gnome-session,NetworkManager,dnsmasq-dhcp,dnsmasq,nm-dispatcher,snmpd,network,crond,accounts-daemon,gdm

I got the above process list from using the nix TA https://splunkbase.splunk.com/app/833/  I exported the process’s into a CSV file using a simple SPL as below

index=linux process=* sourcetype=top
| dedup process
| table process

After the config, I used the deployment server to deploy the TA-metrics to all the Linux systems, I could then see the data in the Analytics workspace in Splunk.

I installed SAI on to the Search Head – https://splunkbase.splunk.com/app/3975/

I install the SAI Infrastructure add onto the indexers https://splunkbase.splunk.com/app/4217/

Configured the macro sai_metrics_indexes to point to the linux_metrics index and I could then see metrics data with then entities in SAI App.

A Bigup to Luke’s TA – makes collecting metrics it so much easier and the SAI is great to monitor OS systems.