There’s a lot of insiders that do a lot of copying and this is one method that could help you observe who’s copying the data when they shouldn’t be init!.

 Install the https://splunkbase.splunk.com/app/833/ which is the TA for collecting Linux OS data onto your restricted Linux servers. Once this has been ingested into Splunk, check the sourcetype and ensure the data is correct and the parsing is good.

TA version used there was 5.2.4

SPL = index=linux sourcetype=bash_history

If the data looks good, create a table, run a simple SPL search to check for any copy or running sudo command for this sourcetype.

SPL = index=linux sourcetype=bash_history sudo OR cp  | table _time,  user_name,  host, bash_command

From this you could enhance the table with some colours and see which user has been a very naughty boy OR girl!!!!

You could do other tables or charts which show data being deleted, which could be a disgruntled employee wanting to do some damage before they leave.

Done!

 

 

 

Splunk steam once configured can monitor many protocols over the wire, so I wanted to see what I could get into Splunk.

I configured the stream app https://splunkbase.splunk.com/app/1809/  which includes a binary that captures the packets onto a number of test servers running the universal forwarder. In the real world you may use a tap port or use the independent Stream Forwarder which uses HEC, so you could ingest network data straight to it.

My config was on some test servers to capture the packets via the streamfwd binary.

Follow the Stream documentation for the config: https://docs.splunk.com/Documentation/StreamApp/7.1.2/DeployStreamApp/InstallSplunkAppforStream

Also deploy the Stream App onto the search head which provides the dashboards / props /transforms and configuration of the Stream App

So I wanted a simple check on icmp traffic, so I enabled the icmp protocol in the config in the Stream app Configuration > Configure Streams

I ran some ping checks and could see the data via a basic SPL:

index=dc_stream sourcetype=”stream:icmp” | table src_ip, dest_ip,  protocol_stack, bytes, bytes_in, bytes_out

 

I created a simple chart to see the data and which destination has had most icmp packets

SPL: index=dc_stream sourcetype=”stream:icmp” | timechart sum(bytes) as total_bytes by dest_ip

So this demonstrates how one can capture wire data and then run some SPL to get stats on network traffic your interested in.

Here’s some other stream data dashboards examples that you get.

DNS is a good, you could see how active the DNS server is.

 

Done.

This app is helpful in getting wiredata into Splunk – go check it out

https://splunkbase.splunk.com/app/4372/

 

This Splunk config will help you monitor which software packages are being installed on your critical Linux servers.

Watch for RPM packages being installed on some critical Linux Centos/RHEL servers, it could be an indication of someone not following change control or you could use it to monitoring change control and many other use cases to monitor such an event.

Before configuring the below you will need to ensure you have setup Splunk, indexes, uf’s and have some test Linux servers. You also need to have Splunk admin level skills, or be an experienced Splunker.

This config was performed on Centos 7.x servers and Splunk 7.x

Config:

• A few fields were created using regex, this was done after analysing the logs
• New tags and event types are created for this config.

Fields: action/ software

Eventypes: yum_packages
Tags: Installed /installed

Configure an inputs / eventypes / tags to monitor the yum log file:

inputs.conf (Deploy to the UF/Linux Server)

[monitor:///var/log/yum.log]
whitelist = (yum.log)
sourcetype = linux_yum
index = syslog
disabled = 0

props.conf (Deploy to the Search Head / Indexers)

[linux_yum]
CHARSET=UTF-8
TIME_PREFIX=^
MAX_TIMESTAMP_LOOKAHEAD=17
TIME_FORMAT=%b %d %H:%M:%S
REPORT-syslog=syslog-extractions
SHOULD_LINEMERGE=false
LINE_BREAKER =([\r\n]+)
KV_MODE = auto
NO_BINARY_CHECK = true
TRUNCATE = 9999
TRANSFORMS=syslog-host
disabled=false

#Extract and action field which is Installed and the software field which is the RPM package installed.
EXTRACT-action = ^(?:[^ \n]* )\d+\s\d+:\d+:\d+\s(?P<action>\w+)
EXTRACT-software = ^(?:[^ \n]* )\d+\s\d+:\d+:\d+\sInstalled:\s\d+:(?P<software>.*)

#normalise the action field as status
FIELDALIAS-action = action as status

Add the below to add event types and tags for the linux_yum sourcetype, this will help with CIM model compliance.

eventtypes.conf (Deploy to the Search Head / Indexers)

[yum_packages]
search = sourcetype= sourcetype=linux_yum
#tags = installed Installed

tags.conf (Deploy to the Search Head / Indexers)

[eventtype=yum_packages]
Installed = enabled
installed = enabled

After the data has been ingested, install some test RPM packages and run the below search, you should get a similar output as in the screenshot.

index=syslog sourcetype=linux_yum action=”Installed”
| rename software as installed_software_rpm
| fields _time, host, action, installed_software_rpm
| eval date=strftime(_time, “%d/%m/%Y %H:%M:%S”)
| stats count by date, action, host, installed_software_rpm

Done

Here’s a simple Splunk port scanning script I put together – its helped me when the ports required have not been opened on clusters members (indexers/search heads) and I was getting connection failed errors – so I thought I’d share this for those that may need to quickly check the Splunk port status in a multiple Splunk server enviroment – you can change the ports for your enviroment, should they have been changed from the default.

https://github.com/iopsmon/splunk_port_scanner

 

There’s a plethora of Windows Security events you can monitor, so the task can become overwhelming right, which ones etc, so these are just a few of the key ones
to start with and then you expand them as you go on, of course you can enable and ingest all of them, but you’ll need to keep an eye on the volume and daily quota limit as Windows logs can be very noisy.

So look at the ones below, and monitor these as a quick start set.

 

Good Microsoft Article On Which Events:
https://docs.microsoft.com/en-gb/windows-server/identity/ad-ds/plan/security-best-practices/audit-policy-recommendations#events-to-monitor

4719 System audit policy was changed.
4964 Special groups have been assigned to a new logon.
1102 The audit log was cleared.
4706 A new trust was created to a domain.
4724 An attempt was made to reset an account’s password.
4739 Domain Policy was changed.
4608 Windows Is Starting Up.
4609 Windows Is Shutting Down.
4625 An account failed to log on.
4648 Privilege Escalation.
4697 Attempt To Install a Service.
4700 A scheduled task was enabled.
4720 A user account was created.
4722 A user account was enabled
4723 An attempt was made to change an account’s password.
4725 A user account was disabled.
4726 A user account was deleted.
4728 A member was added to a security-enabled global group.
4731 A security-enabled local group was created.
4732 A member was added to a security-enabled local group.
4740 A user account was locked out.
4743 A computer account was deleted.
4767 A user account was unlocked.

This is is simple Windows dashboard example you can produce from the security events

 

Here are some useful Splunk links

[Splunk Apps and TA’s]
https://splunkbase.splunk.com/

[Splunk Disk Capacity Sizing]
https://splunk-sizing.appspot.com/

[Splunk Forum (Great place for questions and answers)]
https://answers.splunk.com/index.html

[Splunk data on boarding guide – good for understanding about data]
https://conf.splunk.com/files/2017/slides/data-onboarding-where-do-i-begin.pdf

[Splunk wiki]
http://wiki.splunk.com/Main_Page

[Splunk Reference]
https://www.splunk.com/pdfs/solution-guides/splunk-quick-reference-guide.pdf

[Splunk Quick Command Reference]
https://wiki.splunk.com/images/a/a3/Splunk_4.x_cheatsheet.pdf

[Splunk hot/cold/warm data process]
https://conf.splunk.com/files/2017/slides/splunk-data-life-cycle-determining-when-and-where-to-roll-data.pdf

[Data Onboarding Cheet Sheet]
https://www.aplura.com/wp-content/uploads/2016/09/data_onboarding_cheat_sheet_v2.pdf