This is how to setup a Splunk cluster, search heads and replicate an index.
(The below is a very simple setup and good for lab work, since passing my Core Implementation I would advise you to apply best practise when setting up a splunk cluster, typically you need three indexers and three search heads as minimum, you can also use the base configs and not use the gui method. If you dont have the resources for cluster then use the distributed method)
My Lab Environment:
- 5 x RHEL6.5 Servers
- Splunk 7.1 Enterprise
- 1 x master (This controls the peers)
- 2 x index servers (This is where the index files (log data is stored)
- 2 x search heads (These are the servers used for searching the data)
After building the RHEL servers and installing Splunk 7.1, you start by configuring the master server.
Step 1 On the master server / login as admin and go to settings > Indexer clustering > click on Enable Indexer clustering and set to Master Mode.
Step 2 Set as below:
- Replication Factor = 2 (this is because there are 2 nodes acting as peers)
- Search Factor = 2
- Security Key = (Choose a password)
- Cluster Name = splunk_lab
Press enable and Splunk will restart.
Step 3 Login to the Splunk peer node 1 server as admin and go to settings > Indexer clustering > click on Enable Indexer clustering and set to Peer mode
Step 4 Then set the below
- Master URI = (Master FQDN)
- Peer replication port 8080
- Security Key = (Password from master config)
Step 5 Press enable peer node and press Restart now. (The service will restart)
Step 6 Do the same for the second peer node
Step 7 Login tothe master node and check the config. (Settings > Indexer clustering)
Step 8 Configure the index of your choice to be replicated – I created one as below On the master (opscx1 server) go to the folder /opt/splunk/etc/master-apps/_cluster/local and create a indexes.conf file and add the below.
[dc_security]
repFactor=auto
homePath=/var/splunkdata/security/db
coldPath=/var/splunkdata/security/colddb
thawedPath=/var/splunkdata/security/thawedDb
Step 9 Push the configuration from the master node
On the master node, in Splunk Web, click Settings > Indexer Clustering.
The Master Node dashboard opens.
Click Edit > Configuration Bundle Actions.
Click Validate and Check Restart > Validate and Check Restart.
A message appears that indicates bundle validation and check restart success or failure
Step 10 Generate some data using my scripts and load the file or create a monitor -see my earlier post on how to do this.
https://github.com/iopsmon/log_data_generator
Step 11 From the Master check the data (you would normally configure the search heads for normal use, this is just to check the data)
Step 12 Check the indexes file on the peer nodes it should be as the one on the master
/opt/splunk/etc/slave-apps/_cluster/local
cat /opt/splunk/etc/slave-apps/_cluster/local/indexes.conf
[dc_security]
repFactor=auto
homePath=/var/splunkdata/security/db
coldPath=/var/splunkdata/security/colddb
thawedPath=/var/splunkdata/security/thawedDb
Step 13 Add search heads
Login to one of the search heads and go to > Settings > Indexer Clustering > enable Indexer clustering
Press next and add the below
Master node = FQDN of your master node
Security Key = (From master node config)
Enable search head node and restart Splunk
Log back into the search head and you should get a similar screen shot as below
Run a search on your index and you should get the results as before
index=”dc_security” (This is the name of your index)
Do the same on the other search head server, then login to the master and you should see both search heads.
Done
iOpsMon 