Splunk Quick Health Check

Placeholder Image

During various Splunk projects, I found some customers were not using DMC, as it was not part of the deployment architecture, so I put together a number of health check searches, which then led to this simple app.

Therefore, it is useful for such situations, and not intended to take place over the DMC, which is the preferred health checker.

You can deploy the DMC on Master Nodes for Cluster deployment scenarios

You can deploy the DMC on non-clustered scenario’s, on  a SH, Licence Master, Deployment Server <= 50 nodes.

My App is normally deployed onto a SH


Here is the link:-



Watch List In Splunk

Placeholder Image

This is a simple way to put someone on a watchlist, it is not only external threats, but internal ones as well, and this is useful to see if anyone is exfiltrating sensitive files or trying to connect to un-authorised hosts.

Try to look, logs without Splunk, this will take you ages, life will have evolved by then.

In one click you get a digital footprint of the person on the watch list, you get a quick check of login success’s and failure’s,  a list of the commands they have been running, and the hosts they have been trying to login into.

Create a lookup, authorised_users.csv, add to lookup folder, and then create a simple dashboard, which uses drop down input from the csv file and create the widgets as required.



Create a token base on the user, and run search examples such as below.

| from datamodel: “Authentication”.”Failed_Authentication” | search user=$user_token$ | stats count



Splunk & CIS Top 20 Security Monitoring

Placeholder Image
The purpose of this blog is to provide a quick overview on one can exploit Splunk and the Centre for Internet Security Critical Security Controls for Effective Cyber Defence best practises, otherwise known as CIS top 20.  The best practises consist of 20 key security controls (CSC), that an organisation could use to block or mitigate cyber-attacks and improve their security posture overall. The CIS CSC are ranked in order of overall importance and application to a corporate security strategy. For example, the first two controls, surrounding known inventory, are at the top of the list and are foundational in nature, ranking “very high” for attack mitigation.

The published top 20 controls are as follows:
CSC 1: Inventory of Authorized and Unauthorized Devices
CSC 2: Inventory of Authorized and Unauthorized Software
CSC 3: Secure Configurations for Hardware and Software on Mobile Devices, Laptops, Workstations, and Servers
CSC 4: Continuous Vulnerability Assessment and Remediation
CSC 5: Controlled Use of Administrative Privileges
CSC 6: Maintenance, Monitoring, and Analysis of Audit Logs
CSC 7: Email and Web Browser Protections
CSC 8: Malware Defences
CSC 9: Limitation and Control of Network Ports, Protocols, and Services
CSC 10: Data Recovery Capability
CSC 11: Secure Configurations for Network Devices such as Firewalls, Routers, and Switches
CSC 12: Boundary Defence
CSC 13: Data Protection
CSC 14: Controlled Access Based on the Need to Know
CSC 15: Wireless Access Control
CSC 16: Account Monitoring and Control
CSC 17: Security Skills Assessment and Appropriate Training to Fill Gaps
CSC 18: Application Software Security
CSC 19: Incident Response and Management
CSC 20: Penetration Tests and Red Team Exercises

  • Splunk software can be used to build and operate security operations centres of any size
  • Support the full range of information security operations, including posture assessment, monitoring, alert and incident handling, CSIRT, breach analysis and response, and event correlation
  • Out-of-the-box support for SIEM and security use cases
  • Detect known and unknown threats, investigate threats, determine compliance and use advanced security analytics for detailed insight
  • Proven integrated, big data-based security intelligence platform
  • Use ad hoc searches for advanced breach analysis
  • On-premises, cloud, and hybrid on-premises and cloud deployment options
  • Improve operational efficiency with automated and human-assisted decisions by using Splunk as a security nerve centre
  • Indexes data from any machine data source
  • Searches through machine data from a centralized console
  • Allows the security professional to add tags, create event types and correlate the incoming data with business context
  • Proactively monitors and alerts on security incidents, with automatic remediation of security issues – for example, changing a firewall rule in response to Splunk search results
  • Allows for the creation of reports, dashboards and other forms of analytics to communicate security information throughout the organization

So How Can Splunk Help?
Splunk has a free App that one can use to help provide security compliance, this is based on the CIS top 20 security controls as published by the Centre for Internet Security (CIS).



The app provides a data-agnostic framework which exploits Splunk’s data analytics features and functions such as data model, otherwise known as the Common information model (CIM). This features allows Splunk to be data-agnostic as it normalises data for common fields, such as source IP, user etc.

The App has been developed for anyone in the IT security segment, and Splunk Administrators. It relies on data being ingested into Splunk, this would be as part of the Splunk design and assessment process, such as identifying which sources of data should be ingested, example being Windows Active Directory Logs, Firewall Logs and so forth.
Once this data has been ingested and its CIM compliant, the app runs searches and populates the dashboards for CIS compliancy, this will provide Security teams with insights as to their compliancy status and improve their overall security posture.
There are hundreds of free Splunk TA’s (Splunk Technical Add-ons), these provide field extractions, lookups, tags, and event-types, these all help the Splunk CIS app for presenting the data.

If any customisation is required, lookup files and TA’s can be developed for any application that is required for CIS compliancy.

Splunk supports the controls in four ways:

As Splunk software ingests data, it can generate reports and dashboards that show compliance or non-compliance with controls. Incidents of non-compliance can generate alerts to SOC personnel.

In the case of an attack or non-compliance, Splunk software can carry out recommended actions to meet controls. With version 6.0 of the CIS CSC, Splunk software becomes even more critical, since control 14 surrounding audit logs has been promoted to position

Verification & Execution:
Data from third-party sources can be correlated with data ingested in Splunk software to meet the control.

The Splunk platform provides flexible features that help security professionals with controls that are largely policy and process based.

Mapping Example – CSC 5

Controlled use of admin privileges can be accomplished with a number of toolsets that restrict the use of administrative accounts. The simplest methods are OS-level tools, like sudo, and controls that can be put in place with vendor-supplied tools like Active Directory, so with this in mind you want to comply with CSC 5: Controlled Use of Administrative Privileges to your IT environment.
Splunk can help by consuming authentication logs from across the technology environment that detail account activity, including how accounts are being accessed and from where. Authentication logs come from, but are not limited to: host devices, domain controllers, directory servers, network devices, application logs and many others. All of this data will be ingested into Splunk software for searching and correlation.
Any use of known administrative accounts like “Administrator” and “root” and “sa” can easily be searched across the entire environment and reported or alerted upon.
The below is an example of a dashboard showing Successful Logins from 10 Most Rare Users – Privileged Accounts


So, Get Splunking, and the CIS App if you need to implement the CIS CSC, Splunk makes all data in your organization security relevant, as data is indexed by Splunk Enterprise, it becomes instantly searchable and security professionals can easily correlate all of these seemingly disparate data sources. Furthermore, the different data types can be seen in the context of data locked in business systems, which is often the key factor in determining correct root causes. Security professionals can then build dashboards and reports on top of the data, and set up actions and alerts to be executed on specific thresholds. In addition, any analysis can be operationalized to proactively protect your organization from an emerging threat.

Check to see who’s copying DATA from your restricted Linux servers!

Placeholder Image

There’s a lot of insiders that do a lot of copying and this is one method that could help you observe who’s copying the data when they shouldn’t be init!.

 Install the https://splunkbase.splunk.com/app/833/ which is the TA for collecting Linux OS data onto your restricted Linux servers. Once this has been ingested into Splunk, check the sourcetype and ensure the data is correct and the parsing is good.

TA version used there was 5.2.4

SPL = index=linux sourcetype=bash_history


If the data looks good, create a table, run a simple SPL search to check for any copy or running sudo command for this sourcetype.

SPL = index=linux sourcetype=bash_history sudo OR cp  | table _time,  user_name,  host, bash_command


From this you could enhance the table with some colours and see which user has been a very naughty boy OR girl!!!!


You could do other tables or charts which show data being deleted, which could be a disgruntled employee wanting to do some damage before they leave.





Splunk Stream Is Cool

Placeholder Image

Splunk steam once configured can monitor many protocols over the wire, so I wanted to see what I could get into Splunk.

I configured the stream app https://splunkbase.splunk.com/app/1809/  which includes a binary that captures the packets onto a number of test servers running the universal forwarder. In the real world you may use a tap port or use the independent Stream Forwarder which uses HEC, so you could ingest network data straight to it.

My config was on some test servers to capture the packets via the streamfwd binary.

Follow the Stream documentation for the config: https://docs.splunk.com/Documentation/StreamApp/7.1.2/DeployStreamApp/InstallSplunkAppforStream

Also deploy the Stream App onto the search head which provides the dashboards / props /transforms and configuration of the Stream App

So I wanted a simple check on icmp traffic, so I enabled the icmp protocol in the config in the Stream app Configuration > Configure Streams


I ran some ping checks and could see the data via a basic SPL:

index=dc_stream sourcetype=”stream:icmp” | table src_ip, dest_ip,  protocol_stack, bytes, bytes_in, bytes_out



I created a simple chart to see the data and which destination has had most icmp packets

SPL: index=dc_stream sourcetype=”stream:icmp” | timechart sum(bytes) as total_bytes by dest_ip


So this demonstrates how one can capture wire data and then run some SPL to get stats on network traffic your interested in.

Here’s some other stream data dashboards examples that you get.

DNS is a good, you could see how active the DNS server is.stream5



This app is helpful in getting wiredata into Splunk – go check it out



Windows Services Monitor In Splunk

Placeholder Image

This is a quick way of monitoring your Windows Services

So after ingesting Windows data via the https://splunkbase.splunk.com/app/742/ version 4.8.4, I wanted to see what services were set to auto but not running.

I wanted to ensure any important services were up and actually running, so by running the below search I could capture these services from some test hosts.

SPL: index=windows sourcetype=WinHostMon  Name=* StartMode=Auto State=Stopped | stats values(DisplayName) by host

From the search I could see the SNMP and Firewall services were stopped but should be running.

The below is part config from the Windows Add-On inputs.conf which collects the data. Set these to every 300 seconds (5 mins). Once configured deploy it to some Windows test nodes which run the Universal Forwarder and do some search tests.


Ensure you have deployed the add-on props and transforms to the search heads / indexers for the parsing, otherwise you won’t see the field names.

###### Host monitoring ######
interval = 300
disabled = 0
type = Process
index = windows

interval = 300
disabled = 0
type = Service
index = windows



Monitor Linux RPM Installs With Splunk

Placeholder Image

This Splunk config will help you monitor which software packages are being installed on your critical Linux servers.

Watch for RPM packages being installed on some critical Linux Centos/RHEL servers, it could be an indication of someone not following change control or you could use it to monitoring change control and many other use cases to monitor such an event.

Before configuring the below you will need to ensure you have setup Splunk, indexes, uf’s and have some test Linux servers. You also need to have Splunk admin level skills, or be an experienced Splunker.

This config was performed on Centos 7.x servers and Splunk 7.x


  • A few fields were created using regex, this was done after analysing the logs
  • New tags and event types are created for this config.

Fields: action/ software

Eventypes: yum_packages
Tags: Installed /installed

Configure an inputs / eventypes / tags to monitor the yum log file:

inputs.conf (Deploy to the UF/Linux Server)

whitelist = (yum.log)
sourcetype = linux_yum
index = syslog
disabled = 0

props.conf (Deploy to the Search Head / Indexers)

TIME_FORMAT=%b %d %H:%M:%S
LINE_BREAKER =([\r\n]+)
KV_MODE = auto

#Extract and action field which is Installed and the software field which is the RPM package installed.
EXTRACT-action = ^(?:[^ \n]* )\d+\s\d+:\d+:\d+\s(?P<action>\w+)
EXTRACT-software = ^(?:[^ \n]* )\d+\s\d+:\d+:\d+\sInstalled:\s\d+:(?P<software>.*)

#normalise the action field as status
FIELDALIAS-action = action as status

Add the below to add event types and tags for the linux_yum sourcetype, this will help with CIM model compliance.

eventtypes.conf (Deploy to the Search Head / Indexers)

search = sourcetype= sourcetype=linux_yum
#tags = installed Installed

tags.conf (Deploy to the Search Head / Indexers)

Installed = enabled
installed = enabled

After the data has been ingested, install some test RPM packages and run the below search, you should get a similar output as in the screenshot.

index=syslog sourcetype=linux_yum action=”Installed”
| rename software as installed_software_rpm
| fields _time, host, action, installed_software_rpm
| eval date=strftime(_time, “%d/%m/%Y %H:%M:%S”)
| stats count by date, action, host, installed_software_rpm