Splunk Index Cluster Config

Placeholder Image

This is how to setup a Splunk cluster, search heads and replicate an index.

Environment:

  • 5 x RHEL6.5 Servers
  • Splunk 7.1 Enterprise
  • 1 x master (This controls the peers)
  • 2 x index servers (This is where the index files (log data is stored)
  • 2 x search heads (These are the servers used for searching the data)

After building the RHEL servers and installing Splunk 7.1, you start by configuring the master server.

Step 1 On the master server / login as admin and go to settings > Indexer clustering > click on Enable Indexer clustering and set to Master Mode.

spc1

Step 2 Set as below:

  • Replication Factor = 2 (this is because there are 2 nodes acting as peers)
  • Search Factor = 2
  • Security Key = (Choose a password)
  • Cluster Name = splunk_lab

spc2

Press enable and Splunk will restart.

Step 3 Login to the Splunk peer node 1 server as admin  and go to settings > Indexer clustering > click on Enable Indexer clustering and set to Peer mode

spc3

Step 4 Then set the below

  • Master URI = (Master FQDN)
  • Peer replication port 8080
  • Security Key = (Password from master config)

spc4

Step 5 Press enable peer node and press Restart now. (The service will restart)

Step 6 Do the same for the second peer node

Step 7 Login tothe master node and check the config. (Settings > Indexer clustering)

spc5

Step 8 Configure the index of your choice to be replicated – I created one as below On the master (opscx1 server) go to the folder /opt/splunk/etc/master-apps/_cluster/local  and create a indexes.conf file and add the below.

[dc_security]
repFactor=auto
homePath=/var/splunkdata/security/db
coldPath=/var/splunkdata/security/colddb
thawedPath=/var/splunkdata/security/thawedDb

Step 9 Push the configuration from the master node

On the master node, in Splunk Web, click Settings > Indexer Clustering.
The Master Node dashboard opens.

Click Edit > Configuration Bundle Actions.

Click Validate and Check Restart > Validate and Check Restart.
A message appears that indicates bundle validation and check restart success or failure

spc6

Step 10 Generate some data using my scripts and load the file or create a monitor -see my earlier post on how to do this.

https://github.com/iopsmon/log_data_generator

Step 11 From the Master check the data (you would normally configure the search heads for normal use, this is just to check the data)

spc7

Step 12 Check the indexes file on the peer nodes it should be as the one on the master

/opt/splunk/etc/slave-apps/_cluster/local

cat /opt/splunk/etc/slave-apps/_cluster/local/indexes.conf
[dc_security]
repFactor=auto
homePath=/var/splunkdata/security/db
coldPath=/var/splunkdata/security/colddb
thawedPath=/var/splunkdata/security/thawedDb

Step 13 Add search heads

Login to one of the search heads  and go to > Settings > Indexer Clustering > enable Indexer clustering

spc9

Press next and add the below
Master node = FQDN of your master node
Security Key = (From master node config)

spc10

Enable search head node and restart Splunk
Log back into the search head and you should get a similar screen shot as below

spc11

Run a search on your index and you should get the results as before

index=”dc_security” (This is the name of your index)

spc13

Do the same on the other search head server, then login to the master and you should see both search heads.

spc14

Done

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s