Check to see who’s copying DATA from your restricted Linux servers!

Placeholder Image

There’s a lot of insiders that do a lot of copying and this is one method that could help you observe who’s copying the data when they shouldn’t be init!.

 Install the https://splunkbase.splunk.com/app/833/ which is the TA for collecting Linux OS data onto your restricted Linux servers. Once this has been ingested into Splunk, check the sourcetype and ensure the data is correct and the parsing is good.

TA version used there was 5.2.4

SPL = index=linux sourcetype=bash_history

cp1

If the data looks good, create a table, run a simple SPL search to check for any copy or running sudo command for this sourcetype.

SPL = index=linux sourcetype=bash_history sudo OR cp  | table _time,  user_name,  host, bash_command

cp2

From this you could enhance the table with some colours and see which user has been a very naughty boy OR girl!!!!

cp3

You could do other tables or charts which show data being deleted, which could be a disgruntled employee wanting to do some damage before they leave.

Done!

 

 

 

Splunk Stream Is Cool

Placeholder Image

Splunk steam once configured can monitor many protocols over the wire, so I wanted to see what I could get into Splunk.

I configured the stream app https://splunkbase.splunk.com/app/1809/  which includes a binary that captures the packets onto a number of test servers running the universal forwarder. In the real world you may use a tap port or use the independent Stream Forwarder which uses HEC, so you could ingest network data straight to it.

My config was on some test servers to capture the packets via the streamfwd binary.

Follow the Stream documentation for the config: https://docs.splunk.com/Documentation/StreamApp/7.1.2/DeployStreamApp/InstallSplunkAppforStream

Also deploy the Stream App onto the search head which provides the dashboards / props /transforms and configuration of the Stream App

So I wanted a simple check on icmp traffic, so I enabled the icmp protocol in the config in the Stream app Configuration > Configure Streams

stream2

I ran some ping checks and could see the data via a basic SPL:

index=dc_stream sourcetype=”stream:icmp” | table src_ip, dest_ip,  protocol_stack, bytes, bytes_in, bytes_out

stream3

 

I created a simple chart to see the data and which destination has had most icmp packets

SPL: index=dc_stream sourcetype=”stream:icmp” | timechart sum(bytes) as total_bytes by dest_ip

stream4

So this demonstrates how one can capture wire data and then run some SPL to get stats on network traffic your interested in.

Here’s some other stream data dashboards examples that you get.

DNS is a good, you could see how active the DNS server is.stream5

 

Done.

This app is helpful in getting wiredata into Splunk – go check it out

https://splunkbase.splunk.com/app/4372/

 

Windows Services Monitor In Splunk

Placeholder Image

This is a quick way of monitoring your Windows Services

So after ingesting Windows data via the https://splunkbase.splunk.com/app/742/ version 4.8.4, I wanted to see what services were set to auto but not running.

I wanted to ensure any important services were up and actually running, so by running the below search I could capture these services from some test hosts.

SPL: index=windows sourcetype=WinHostMon  Name=* StartMode=Auto State=Stopped | stats values(DisplayName) by host

From the search I could see the SNMP and Firewall services were stopped but should be running.

The below is part config from the Windows Add-On inputs.conf which collects the data. Set these to every 300 seconds (5 mins). Once configured deploy it to some Windows test nodes which run the Universal Forwarder and do some search tests.

win_services_stopped

Ensure you have deployed the add-on props and transforms to the search heads / indexers for the parsing, otherwise you won’t see the field names.

###### Host monitoring ######
[WinHostMon://Process]
interval = 300
disabled = 0
type = Process
index = windows

[WinHostMon://Service]
interval = 300
disabled = 0
type = Service
index = windows

Done.

 

Monitor Linux RPM Installs With Splunk

Placeholder Image

This Splunk config will help you monitor which software packages are being installed on your critical Linux servers.

Watch for RPM packages being installed on some critical Linux Centos/RHEL servers, it could be an indication of someone not following change control or you could use it to monitoring change control and many other use cases to monitor such an event.

Before configuring the below you will need to ensure you have setup Splunk, indexes, uf’s and have some test Linux servers. You also need to have Splunk admin level skills, or be an experienced Splunker.

This config was performed on Centos 7.x servers and Splunk 7.x

Config:

  • A few fields were created using regex, this was done after analysing the logs
  • New tags and event types are created for this config.

Fields: action/ software

Eventypes: yum_packages
Tags: Installed /installed

Configure an inputs / eventypes / tags to monitor the yum log file:

inputs.conf (Deploy to the UF/Linux Server)

[monitor:///var/log/yum.log]
whitelist = (yum.log)
sourcetype = linux_yum
index = syslog
disabled = 0

props.conf (Deploy to the Search Head / Indexers)

[linux_yum]
CHARSET=UTF-8
TIME_PREFIX=^
MAX_TIMESTAMP_LOOKAHEAD=17
TIME_FORMAT=%b %d %H:%M:%S
REPORT-syslog=syslog-extractions
SHOULD_LINEMERGE=false
LINE_BREAKER =([\r\n]+)
KV_MODE = auto
NO_BINARY_CHECK = true
TRUNCATE = 9999
TRANSFORMS=syslog-host
disabled=false

#Extract and action field which is Installed and the software field which is the RPM package installed.
EXTRACT-action = ^(?:[^ \n]* )\d+\s\d+:\d+:\d+\s(?P<action>\w+)
EXTRACT-software = ^(?:[^ \n]* )\d+\s\d+:\d+:\d+\sInstalled:\s\d+:(?P<software>.*)

#normalise the action field as status
FIELDALIAS-action = action as status

Add the below to add event types and tags for the linux_yum sourcetype, this will help with CIM model compliance.

eventtypes.conf (Deploy to the Search Head / Indexers)

[yum_packages]
search = sourcetype= sourcetype=linux_yum
#tags = installed Installed

tags.conf (Deploy to the Search Head / Indexers)

[eventtype=yum_packages]
Installed = enabled
installed = enabled

After the data has been ingested, install some test RPM packages and run the below search, you should get a similar output as in the screenshot.

index=syslog sourcetype=linux_yum action=”Installed”
| rename software as installed_software_rpm
| fields _time, host, action, installed_software_rpm
| eval date=strftime(_time, “%d/%m/%Y %H:%M:%S”)
| stats count by date, action, host, installed_software_rpm

software

Done

Splunk Ports Check Scanner

Placeholder Image

Here’s a simple Splunk port scanning script I put together – its helped me when the ports required have not been opened on clusters members (indexers/search heads) and I was getting connection failed errors – so I thought I’d share this for those that may need to quickly check the Splunk port status in a multiple Splunk server enviroment – you can change the ports for your enviroment, should they have been changed from the default.

https://github.com/iopsmon/splunk_port_scanner

 

Splunk And Windows Security Logs

Placeholder Image

There’s a plethora of Windows Security events you can monitor, so the task can become overwhelming right, which ones etc, so these are just a few of the key ones
to start with and then you expand them as you go on, of course you can enable and ingest all of them, but you’ll need to keep an eye on the volume and daily quota limit as Windows logs can be very noisy.

So look at the ones below, and monitor these as a quick start set.

 

Good Microsoft Article On Which Events:
https://docs.microsoft.com/en-gb/windows-server/identity/ad-ds/plan/security-best-practices/audit-policy-recommendations#events-to-monitor

4719 System audit policy was changed.
4964 Special groups have been assigned to a new logon.
1102 The audit log was cleared.
4706 A new trust was created to a domain.
4724 An attempt was made to reset an account’s password.
4739 Domain Policy was changed.
4608 Windows Is Starting Up.
4609 Windows Is Shutting Down.
4625 An account failed to log on.
4648 Privilege Escalation.
4697 Attempt To Install a Service.
4700 A scheduled task was enabled.
4720 A user account was created.
4722 A user account was enabled
4723 An attempt was made to change an account’s password.
4725 A user account was disabled.
4726 A user account was deleted.
4728 A member was added to a security-enabled global group.
4731 A security-enabled local group was created.
4732 A member was added to a security-enabled local group.
4740 A user account was locked out.
4743 A computer account was deleted.
4767 A user account was unlocked.

This is is simple Windows dashboard example you can produce from the security events

win

 

Good Splunk Links

Placeholder Image

Here are some useful Splunk links

[Splunk Apps and TA’s]
https://splunkbase.splunk.com/

[Splunk Disk Capacity Sizing]
https://splunk-sizing.appspot.com/

[Splunk Forum (Great place for questions and answers)]
https://answers.splunk.com/index.html

[Splunk data on boarding guide – good for understanding about data]
https://conf.splunk.com/files/2017/slides/data-onboarding-where-do-i-begin.pdf

[Splunk wiki]
http://wiki.splunk.com/Main_Page

[Splunk Reference]
https://www.splunk.com/pdfs/solution-guides/splunk-quick-reference-guide.pdf

[Splunk Quick Command Reference]
https://wiki.splunk.com/images/a/a3/Splunk_4.x_cheatsheet.pdf

[Splunk hot/cold/warm data process]
https://conf.splunk.com/files/2017/slides/splunk-data-life-cycle-determining-when-and-where-to-roll-data.pdf

[Data Onboarding Cheet Sheet]
https://www.aplura.com/wp-content/uploads/2016/09/data_onboarding_cheat_sheet_v2.pdf