Splunk steam once configured can monitor many protocols over the wire, so I wanted to see what I could get into Splunk.

I configured the stream app https://splunkbase.splunk.com/app/1809/  which includes a binary that captures the packets onto a number of test servers running the universal forwarder. In the real world you may use a tap port or use the independent Stream Forwarder which uses HEC, so you could ingest network data straight to it.

My config was on some test servers to capture the packets via the streamfwd binary.

Follow the Stream documentation for the config: https://docs.splunk.com/Documentation/StreamApp/7.1.2/DeployStreamApp/InstallSplunkAppforStream

Also deploy the Stream App onto the search head which provides the dashboards / props /transforms and configuration of the Stream App

So I wanted a simple check on icmp traffic, so I enabled the icmp protocol in the config in the Stream app Configuration > Configure Streams

I ran some ping checks and could see the data via a basic SPL:

index=dc_stream sourcetype=”stream:icmp” | table src_ip, dest_ip,  protocol_stack, bytes, bytes_in, bytes_out

 

I created a simple chart to see the data and which destination has had most icmp packets

SPL: index=dc_stream sourcetype=”stream:icmp” | timechart sum(bytes) as total_bytes by dest_ip

So this demonstrates how one can capture wire data and then run some SPL to get stats on network traffic your interested in.

Here’s some other stream data dashboards examples that you get.

DNS is a good, you could see how active the DNS server is.

 

Done.

This app is helpful in getting wiredata into Splunk – go check it out

https://splunkbase.splunk.com/app/4372/

 

This Splunk config will help you monitor which software packages are being installed on your critical Linux servers.

Watch for RPM packages being installed on some critical Linux Centos/RHEL servers, it could be an indication of someone not following change control or you could use it to monitoring change control and many other use cases to monitor such an event.

Before configuring the below you will need to ensure you have setup Splunk, indexes, uf’s and have some test Linux servers. You also need to have Splunk admin level skills, or be an experienced Splunker.

This config was performed on Centos 7.x servers and Splunk 7.x

Config:

• A few fields were created using regex, this was done after analysing the logs
• New tags and event types are created for this config.

Fields: action/ software

Eventypes: yum_packages
Tags: Installed /installed

Configure an inputs / eventypes / tags to monitor the yum log file:

inputs.conf (Deploy to the UF/Linux Server)

[monitor:///var/log/yum.log]
whitelist = (yum.log)
sourcetype = linux_yum
index = syslog
disabled = 0

props.conf (Deploy to the Search Head / Indexers)

[linux_yum]
CHARSET=UTF-8
TIME_PREFIX=^
MAX_TIMESTAMP_LOOKAHEAD=17
TIME_FORMAT=%b %d %H:%M:%S
REPORT-syslog=syslog-extractions
SHOULD_LINEMERGE=false
LINE_BREAKER =([\r\n]+)
KV_MODE = auto
NO_BINARY_CHECK = true
TRUNCATE = 9999
TRANSFORMS=syslog-host
disabled=false

#Extract and action field which is Installed and the software field which is the RPM package installed.
EXTRACT-action = ^(?:[^ \n]* )\d+\s\d+:\d+:\d+\s(?P<action>\w+)
EXTRACT-software = ^(?:[^ \n]* )\d+\s\d+:\d+:\d+\sInstalled:\s\d+:(?P<software>.*)

#normalise the action field as status
FIELDALIAS-action = action as status

Add the below to add event types and tags for the linux_yum sourcetype, this will help with CIM model compliance.

eventtypes.conf (Deploy to the Search Head / Indexers)

[yum_packages]
search = sourcetype= sourcetype=linux_yum
#tags = installed Installed

tags.conf (Deploy to the Search Head / Indexers)

[eventtype=yum_packages]
Installed = enabled
installed = enabled

After the data has been ingested, install some test RPM packages and run the below search, you should get a similar output as in the screenshot.

index=syslog sourcetype=linux_yum action=”Installed”
| rename software as installed_software_rpm
| fields _time, host, action, installed_software_rpm
| eval date=strftime(_time, “%d/%m/%Y %H:%M:%S”)
| stats count by date, action, host, installed_software_rpm

Done

Here’s a simple Splunk port scanning script I put together – its helped me when the ports required have not been opened on clusters members (indexers/search heads) and I was getting connection failed errors – so I thought I’d share this for those that may need to quickly check the Splunk port status in a multiple Splunk server enviroment – you can change the ports for your enviroment, should they have been changed from the default.

https://github.com/iopsmon/splunk_port_scanner

 

There’s a plethora of Windows Security events you can monitor, so the task can become overwhelming right, which ones etc, so these are just a few of the key ones
to start with and then you expand them as you go on, of course you can enable and ingest all of them, but you’ll need to keep an eye on the volume and daily quota limit as Windows logs can be very noisy.

So look at the ones below, and monitor these as a quick start set.

 

Good Microsoft Article On Which Events:
https://docs.microsoft.com/en-gb/windows-server/identity/ad-ds/plan/security-best-practices/audit-policy-recommendations#events-to-monitor

4719 System audit policy was changed.
4964 Special groups have been assigned to a new logon.
1102 The audit log was cleared.
4706 A new trust was created to a domain.
4724 An attempt was made to reset an account’s password.
4739 Domain Policy was changed.
4608 Windows Is Starting Up.
4609 Windows Is Shutting Down.
4625 An account failed to log on.
4648 Privilege Escalation.
4697 Attempt To Install a Service.
4700 A scheduled task was enabled.
4720 A user account was created.
4722 A user account was enabled
4723 An attempt was made to change an account’s password.
4725 A user account was disabled.
4726 A user account was deleted.
4728 A member was added to a security-enabled global group.
4731 A security-enabled local group was created.
4732 A member was added to a security-enabled local group.
4740 A user account was locked out.
4743 A computer account was deleted.
4767 A user account was unlocked.

This is is simple Windows dashboard example you can produce from the security events

 

Here are some useful Splunk links

[Splunk Apps and TA’s]
https://splunkbase.splunk.com/

[Splunk Disk Capacity Sizing]
https://splunk-sizing.appspot.com/

[Splunk Forum (Great place for questions and answers)]
https://answers.splunk.com/index.html

[Splunk data on boarding guide – good for understanding about data]
https://conf.splunk.com/files/2017/slides/data-onboarding-where-do-i-begin.pdf

[Splunk wiki]
http://wiki.splunk.com/Main_Page

[Splunk Reference]
https://www.splunk.com/pdfs/solution-guides/splunk-quick-reference-guide.pdf

[Splunk Quick Command Reference]
https://wiki.splunk.com/images/a/a3/Splunk_4.x_cheatsheet.pdf

[Splunk hot/cold/warm data process]
https://conf.splunk.com/files/2017/slides/splunk-data-life-cycle-determining-when-and-where-to-roll-data.pdf

[Data Onboarding Cheet Sheet]
https://www.aplura.com/wp-content/uploads/2016/09/data_onboarding_cheat_sheet_v2.pdf

I recently had to a do a 5 minute presentation on my Splunk ES course, I chose lookups as the subject, so I thought I ‘d add this to my blog.

“lookups NOT hookups” (was my presentation title)

Anyway, this simple article will show you how to set up a simple lookup within Splunk.

The search is for an Windows event id which runs a lookup function on the filed name “Account_Name” and if a match is found in the lookpup, the fields specified in the OUTPUT function will be displayed in the fields display in Splunk, from this you can create dashboards tables etcs.

Time to get some data in!!! –

Configure some UF’s on Windows servers, and ingest security logs – see the security essentials app https://splunkbase.splunk.com/app/3435/ it will help you get the data in.

Tasks: (After Ingesting the Windows logs)

a. Run a search and check the fields – note the fields of interest

index=wineventlog (type your index details)

Look for the Account_Name this field contains the windows domain name and is of interest.

b. Create lookup csv file (notice the Account_Name field), this must match the
csv file headers.
Lookup Example csv file

Account_Name,first_name,last_name,account_status
donna,donna,kebab,Disabled
user1,user1,lazy,Enabled
wendy,wendy,moss,Enabled
jane,jane,simms,Enabled
instructor,instructor,splunk,Enabled
space,space,cadet,Enabled

Add the csv file to Splunk / lookups / define the lookup name , then run the command to see the data from the csv file

| inputslookup <csv filename> (If you can see the data move onto to the next step)

c. Run the below search, which will add names once its Account_Name is matched and is disabled.

Before you run the serach disable one of the accounts in the csv file and the event 4738 should be generated in Windows(im using this as an example)

index=wineventlog sourcetype=WinEventLog:Security EventCode=4738 “Account Disabled”
| lookup dc_ad_accounts Account_Name OUTPUT first_name, last_name, account_status

you should see new fields first_name and last_name and they should contain the values frojm the csv file

Now Add a table / dashboard

index=wineventlog sourcetype=WinEventLog:Security EventCode=4738 “Account Disabled”
| lookup dc_ad_accounts Account_Name OUTPUT first_name, last_name, account_status
| table _time, first_name, last_name, account_status

Done!