I recently had a play with Splunk SAI and I wanted to monitor a number of Linux Systems.
My options to collect metric data from the systems was to use collectd and send the data to HEC OR use Luke Harris’s TA-Metrics add-on https://splunkbase.splunk.com/app/4856/ and send the data to the indexers.
I opted for the metrics add-on as this seemed so much easier, the SAI has made it easier to deploy collectd + agent, but this TA to me was even easier, and it supports SAI.
I created a linux_metrics index in Splunk
[linux_metrics]
homePath = $SPLUNK_DB /linux_metrics/db
coldPath = $SPLUNK_DB /linux_metrics/colddb
thawedPath = $SPLUNK_DB/linux_metrics/thaweddb
datatype = metric
frozenTimePeriodInSecs = 2419200
The inputs.conf from the default folder is already set for you, its polls every 5 minutes for various metrics. I then created a local folder in the TA-linux-metrics folder and copied the process_mon.conf with below configuration.
[process_mon]
allowlist = CROND,run*,systemd*,chronyd,rsyslogd,auditd,journal,su,splunk,gnome-session,NetworkManager,dnsmasq-dhcp,dnsmasq,nm-dispatcher,snmpd,network,crond,accounts-daemon,gdm
I got the above process list from using the nix TA https://splunkbase.splunk.com/app/833/ I exported the process’s into a CSV file using a simple SPL as below
index=linux process=* sourcetype=top
| dedup process
| table process
After the config, I used the deployment server to deploy the TA-metrics to all the Linux systems, I could then see the data in the Analytics workspace in Splunk.
I installed SAI on to the Search Head – https://splunkbase.splunk.com/app/3975/
I install the SAI Infrastructure add onto the indexers https://splunkbase.splunk.com/app/4217/
Configured the macro sai_metrics_indexes to point to the linux_metrics index and I could then see metrics data with then entities in SAI App.
A Bigup to Luke’s TA – makes collecting metrics it so much easier and the SAI is great to monitor OS systems.
iOpsMon 