Monitor Linux RPM Installs With Splunk

Placeholder Image

This Splunk config will help you monitor which software packages are being installed on your critical Linux servers.

Watch for RPM packages being installed on some critical Linux Centos/RHEL servers, it could be an indication of someone not following change control or you could use it to monitoring change control and many other use cases to monitor such an event.

Before configuring the below you will need to ensure you have setup Splunk, indexes, uf’s and have some test Linux servers. You also need to have Splunk admin level skills, or be an experienced Splunker.

This config was performed on Centos 7.x servers and Splunk 7.x

Config:

  • A few fields were created using regex, this was done after analysing the logs
  • New tags and event types are created for this config.

Fields: action/ software

Eventypes: yum_packages
Tags: Installed /installed

Configure an inputs / eventypes / tags to monitor the yum log file:

inputs.conf (Deploy to the UF/Linux Server)

[monitor:///var/log/yum.log]
whitelist = (yum.log)
sourcetype = linux_yum
index = syslog
disabled = 0

props.conf (Deploy to the Search Head / Indexers)

[linux_yum]
CHARSET=UTF-8
TIME_PREFIX=^
MAX_TIMESTAMP_LOOKAHEAD=17
TIME_FORMAT=%b %d %H:%M:%S
REPORT-syslog=syslog-extractions
SHOULD_LINEMERGE=false
LINE_BREAKER =([\r\n]+)
KV_MODE = auto
NO_BINARY_CHECK = true
TRUNCATE = 9999
TRANSFORMS=syslog-host
disabled=false

#Extract and action field which is Installed and the software field which is the RPM package installed.
EXTRACT-action = ^(?:[^ \n]* )\d+\s\d+:\d+:\d+\s(?P<action>\w+)
EXTRACT-software = ^(?:[^ \n]* )\d+\s\d+:\d+:\d+\sInstalled:\s\d+:(?P<software>.*)

#normalise the action field as status
FIELDALIAS-action = action as status

Add the below to add event types and tags for the linux_yum sourcetype, this will help with CIM model compliance.

eventtypes.conf (Deploy to the Search Head / Indexers)

[yum_packages]
search = sourcetype= sourcetype=linux_yum
#tags = installed Installed

tags.conf (Deploy to the Search Head / Indexers)

[eventtype=yum_packages]
Installed = enabled
installed = enabled

After the data has been ingested, install some test RPM packages and run the below search, you should get a similar output as in the screenshot.

index=syslog sourcetype=linux_yum action=”Installed”
| rename software as installed_software_rpm
| fields _time, host, action, installed_software_rpm
| eval date=strftime(_time, “%d/%m/%Y %H:%M:%S”)
| stats count by date, action, host, installed_software_rpm

software

Done

Splunk Ports Check Scanner

Placeholder Image

Here’s a simple Splunk port scanning script I put together – its helped me when the ports required have not been opened on clusters members (indexers/search heads) and I was getting connection failed errors – so I thought I’d share this for those that may need to quickly check the Splunk port status in a multiple Splunk server enviroment – you can change the ports for your enviroment, should they have been changed from the default.

https://github.com/iopsmon/splunk_port_scanner

 

Splunk And Windows Security Logs

Placeholder Image

There’s a plethora of Windows Security events you can monitor, so the task can become overwhelming right, which ones etc, so these are just a few of the key ones
to start with and then you expand them as you go on, of course you can enable and ingest all of them, but you’ll need to keep an eye on the volume and daily quota limit as Windows logs can be very noisy.

So look at the ones below, and monitor these as a quick start set.

 

Good Microsoft Article On Which Events:
https://docs.microsoft.com/en-gb/windows-server/identity/ad-ds/plan/security-best-practices/audit-policy-recommendations#events-to-monitor

4719 System audit policy was changed.
4964 Special groups have been assigned to a new logon.
1102 The audit log was cleared.
4706 A new trust was created to a domain.
4724 An attempt was made to reset an account’s password.
4739 Domain Policy was changed.
4608 Windows Is Starting Up.
4609 Windows Is Shutting Down.
4625 An account failed to log on.
4648 Privilege Escalation.
4697 Attempt To Install a Service.
4700 A scheduled task was enabled.
4720 A user account was created.
4722 A user account was enabled
4723 An attempt was made to change an account’s password.
4725 A user account was disabled.
4726 A user account was deleted.
4728 A member was added to a security-enabled global group.
4731 A security-enabled local group was created.
4732 A member was added to a security-enabled local group.
4740 A user account was locked out.
4743 A computer account was deleted.
4767 A user account was unlocked.

This is is simple Windows dashboard example you can produce from the security events

win

 

Good Splunk Links

Placeholder Image

Here are some useful Splunk links

[Splunk Apps and TA’s]
https://splunkbase.splunk.com/

[Splunk Disk Capacity Sizing]
https://splunk-sizing.appspot.com/

[Splunk Forum (Great place for questions and answers)]
https://answers.splunk.com/index.html

[Splunk data on boarding guide – good for understanding about data]
https://conf.splunk.com/files/2017/slides/data-onboarding-where-do-i-begin.pdf

[Splunk wiki]
http://wiki.splunk.com/Main_Page

[Splunk Reference]
https://www.splunk.com/pdfs/solution-guides/splunk-quick-reference-guide.pdf

[Splunk Quick Command Reference]
https://wiki.splunk.com/images/a/a3/Splunk_4.x_cheatsheet.pdf

[Splunk hot/cold/warm data process]
https://conf.splunk.com/files/2017/slides/splunk-data-life-cycle-determining-when-and-where-to-roll-data.pdf

[Data Onboarding Cheet Sheet]
https://www.aplura.com/wp-content/uploads/2016/09/data_onboarding_cheat_sheet_v2.pdf

“Lookups NOT Hookups”

Placeholder Image

I recently had to a do a 5 minute presentation on my Splunk ES course, I chose lookups as the subject, so I thought I ‘d add this to my blog.

“lookups NOT hookups” (was my presentation title)

Anyway, this simple article will show you how to set up a simple lookup within Splunk.

The search is for an Windows event id which runs a lookup function on the filed name “Account_Name” and if a match is found in the lookpup, the fields specified in the OUTPUT function will be displayed in the fields display in Splunk, from this you can create dashboards tables etcs.

Time to get some data in!!! –

Configure some UF’s on Windows servers, and ingest security logs – see the security essentials app https://splunkbase.splunk.com/app/3435/ it will help you get the data in.

Tasks: (After Ingesting the Windows logs)

a. Run a search and check the fields – note the fields of interest

index=wineventlog (type your index details)

Look for the Account_Name this field contains the windows domain name and is of interest.

b. Create lookup csv file (notice the Account_Name field), this must match the
csv file headers.
Lookup Example csv file

Account_Name,first_name,last_name,account_status
donna,donna,kebab,Disabled
user1,user1,lazy,Enabled
wendy,wendy,moss,Enabled
jane,jane,simms,Enabled
instructor,instructor,splunk,Enabled
space,space,cadet,Enabled

Add the csv file to Splunk / lookups / define the lookup name , then run the command to see the data from the csv file

| inputslookup <csv filename> (If you can see the data move onto to the next step)

c. Run the below search, which will add names once its Account_Name is matched and is disabled.

Before you run the serach disable one of the accounts in the csv file and the event 4738 should be generated in Windows(im using this as an example)

index=wineventlog sourcetype=WinEventLog:Security EventCode=4738 “Account Disabled”
| lookup dc_ad_accounts Account_Name OUTPUT first_name, last_name, account_status

you should see new fields first_name and last_name and they should contain the values frojm the csv file

Now Add a table / dashboard

index=wineventlog sourcetype=WinEventLog:Security EventCode=4738 “Account Disabled”
| lookup dc_ad_accounts Account_Name OUTPUT first_name, last_name, account_status
| table _time, first_name, last_name, account_status

Done!

Splunk Index Cluster Config

Placeholder Image

This is how to setup a Splunk cluster, search heads and replicate an index.

(The below is a very simple setup and good for lab work, since passing my Core Implementation I would advise you to apply best practise when setting up a splunk cluster, typically you need three indexers and three search heads as minimum, you can also use the base configs and not use the gui method. If you dont have the resources for cluster then use the distributed method)

My Lab Environment:

  • 5 x RHEL6.5 Servers
  • Splunk 7.1 Enterprise
  • 1 x master (This controls the peers)
  • 2 x index servers (This is where the index files (log data is stored)
  • 2 x search heads (These are the servers used for searching the data)

After building the RHEL servers and installing Splunk 7.1, you start by configuring the master server.

Step 1 On the master server / login as admin and go to settings > Indexer clustering > click on Enable Indexer clustering and set to Master Mode.

spc1

Step 2 Set as below:

  • Replication Factor = 2 (this is because there are 2 nodes acting as peers)
  • Search Factor = 2
  • Security Key = (Choose a password)
  • Cluster Name = splunk_lab

spc2

Press enable and Splunk will restart.

Step 3 Login to the Splunk peer node 1 server as admin  and go to settings > Indexer clustering > click on Enable Indexer clustering and set to Peer mode

spc3

Step 4 Then set the below

  • Master URI = (Master FQDN)
  • Peer replication port 8080
  • Security Key = (Password from master config)

spc4

Step 5 Press enable peer node and press Restart now. (The service will restart)

Step 6 Do the same for the second peer node

Step 7 Login tothe master node and check the config. (Settings > Indexer clustering)

spc5

Step 8 Configure the index of your choice to be replicated – I created one as below On the master (opscx1 server) go to the folder /opt/splunk/etc/master-apps/_cluster/local  and create a indexes.conf file and add the below.

[dc_security]
repFactor=auto
homePath=/var/splunkdata/security/db
coldPath=/var/splunkdata/security/colddb
thawedPath=/var/splunkdata/security/thawedDb

Step 9 Push the configuration from the master node

On the master node, in Splunk Web, click Settings > Indexer Clustering.
The Master Node dashboard opens.

Click Edit > Configuration Bundle Actions.

Click Validate and Check Restart > Validate and Check Restart.
A message appears that indicates bundle validation and check restart success or failure

spc6

Step 10 Generate some data using my scripts and load the file or create a monitor -see my earlier post on how to do this.

https://github.com/iopsmon/log_data_generator

Step 11 From the Master check the data (you would normally configure the search heads for normal use, this is just to check the data)

spc7

Step 12 Check the indexes file on the peer nodes it should be as the one on the master

/opt/splunk/etc/slave-apps/_cluster/local

cat /opt/splunk/etc/slave-apps/_cluster/local/indexes.conf
[dc_security]
repFactor=auto
homePath=/var/splunkdata/security/db
coldPath=/var/splunkdata/security/colddb
thawedPath=/var/splunkdata/security/thawedDb

Step 13 Add search heads

Login to one of the search heads  and go to > Settings > Indexer Clustering > enable Indexer clustering

spc9

Press next and add the below
Master node = FQDN of your master node
Security Key = (From master node config)

spc10

Enable search head node and restart Splunk
Log back into the search head and you should get a similar screen shot as below

spc11

Run a search on your index and you should get the results as before

index=”dc_security” (This is the name of your index)

spc13

Do the same on the other search head server, then login to the master and you should see both search heads.

spc14

Done