Splunk Data Onboarding | NuHarbor Security

Overview

Thanks for taking a few minutes to read our post onboarding custom data into Splunk.  For support with Splunk or Splunk Managed Services please contact us here.

This guide is to help Splunk Admins onboard new data that do not have a Splunkbase TA available.  It will cover how to properly onboard data at the index level, and introduce search time considerations for Common Information Model  (CIM) mapping.  Throughout this example we will look at IIS logs, props, transforms, inputs, and the nullQueue.

1. Understand Your Data

Before you can properly onboard data, you will need a sample to understand the file format, timestamps, and ‘what an event is’.  An event can be one line, or can it be multi-line like a JAVA debug dump.

Sample Data – Location on disk: C:\inetpub\logs\LogFiles\W3SVC2\u_ex180277.log

Sample IIS Log

#Software: Microsoft Internet Information Services 8.5
#Version: 1.0
#Date: 2016-06-30 14:44:43
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
2016-06-30 14:44:43 172.30.16.210 GET / - 80 - 96.84.222.21 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/51.0.2704.103+Safari/537.36 - 200 0 0 546
2016-06-30 14:44:43 172.30.16.210 GET /favicon.ico - 80 - 96.84.222.21 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/51.0.2704.103+Safari/537.36 http://54.224.49.25/ 404 0 2 187
2016-06-30 14:46:33 172.30.16.210 GET / - 80 - 96.84.222.21 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/51.0.2704.103+Safari/537.36 - 200 0 0 62
2016-06-30 14:46:34 172.30.16.210 GET /page1.html - 80 - 96.84.222.21 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/51.0.2704.103+Safari/537.36 http://54.224.49.25/ 200 0 0 62
2016-06-30 14:46:37 172.30.16.210 GET /page2.html - 80 - 96.84.222.21 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/51.0.2704.103+Safari/537.36 http://54.224.49.25/page1.html 200 0 0 62
2016-06-30 14:46:37 172.30.16.210 GET /page3.html - 80 - 96.84.222.21 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/51.0.2704.103+Safari/537.36 http://54.224.49.25/page2.html 200 0 0 62

Key Observations from the above sample IIS log:

  • Each line appears to be its own event
  • Timestamps are at the beginning of each event
  • Events actually start at line 5, the first 4 lines are just header information
  • The file is structured (tab delimited)

2. Get Control of Standard Props.conf Configuration

Props.conf controls many things.  We will focus on index time operations in this section.  Index time operations happen during the parsing pipeline and before the data is written to disk.  Once data is written to disk, it cannot be changed or modified so ensuring these are correct is the first step of onboarding data.

Props.conf Example

[iis_example]
TIME_PREFIX = ^ 
TIME_FORMAT = %Y-%m-%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 20
LINE_BREAKER = ([\r\n]+)\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} 
SHOULD_LINEMERGE = FALSE 
TRUNCATE = 9999

In the example above, we are:

  • creating a new sourcetype called iis_example (line 1)
  • telling the indexer where the timestamp is located (line 2).  In this case the timestamp is at the start of the line, which is denoted by the ^ character.
  • telling Splunk the format of the timestamp (line 3).  While Splunk is pretty good about determining this on its own, this makes it quicker because it does not have to guess – leading to better performance on indexers.  This is in strptime format.
  • telling Splunk how many characters from TIME_PREFIX to read to find the full timestamp (line 4).  Ensure you account for situations where date can be a single digit (example the date is Feb 8 2018 instead of Feb 08 2018).
  • telling Splunk when a new event starts (line 5).  ([\r\n]+) removes all hard return characters and denotes the beginning of a new line.  You want to be as specific as possible (in this case we’re also looking for a date after the hard return) to ensure it only breaks when it should.  If you think of a JAVA debug dump, it can be 100’s of lines long, but usually starts with a pre-determined timestamp format.  This is done using regular expression format.
  • telling Splunk not to paste lines together (line 6).  Making Splunk ‘glue’ events back together before writing the event to disk is very costly on performance.  This should always be FALSE if you have LINE_BREAKER set correctly.
  • telling Splunk how many characters an event can be (line 7).  When Splunk reaches this amount of characters, it automatically breaks the event and starts a new event.  Ensure this is large enough to cover the max event length, not doing so will truncate the event prematurely and have adverse effect on your ability to search the data.

Helpful References:

Splunk Quick Reference Guide has common regular expression and date/time formatting examples.  Download the guide here: https://www.splunk.com/pdfs/solution-guides/splunk-quick-reference-guide.pdf

There is a very helpful way to verify the expressions and date/time regexes you end up createing by going to https://regex101.com/. From there you can test your regex against the event string of interest and see if it is correct.

3. Work on Advanced Props.conf Configuration: nullQueue

One of the observations above is the events don’t actually start until line 5.  If we only used what the props listed above, we would end up with the header rows (‘junk’ data) being indexed as an event.  This section works through how to tell Splunk to ignore these rows and not index the data.

First, create a regex to identify these rows.  We can see these rows all start with a #, so we can use that as the identifier to tell Splunk to ignore these rows.  We want to be as specific as possible to ensure we do not toss out good data – so in this case our transforms.conf (line 2) is looking for rows that starts with a # followed by 1 or more letters.  This would match #Software, #Version, #Date, and #Fields but wont’ match any # in the event itself because the event does not being with #.

Next we want to set the queue to a special value ‘nullQueue’ which instructs Splunk to discard the data prior to writing it to disk.  Remember, your license is based on the amount of data written to disk, so this will not count against your license.

transforms.conf

[eliminate-iis-header-row]
REGEX=^#\w+ 
DEST_KEY=queue 
FORMAT=nullQueue

Lastly, we need to update the iis_example sourcetype in props.conf to use the new transform (line 10 below)

Updated props.conf

[iis_example] 
TIME_PREFIX = ^ 
TIME_FORMAT = %Y-%m-%d %H:%M:%S 
MAX_TIMESTAMP_LOOKAHEAD = 20 
LINE_BREAKER = ([\r\n]+)\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} 
SHOULD_LINEMERGE = FALSE 
TRUNCATE = 9999 

# Apply Transform
TRANSFORMS-remove-header=eliminate-iis-header-row

At this point, we can deploy the props and transforms to the indexers.  We need to ensure these are deployed prior to sending any data.

4. Send the Data to the Indexers

The next step is to send the data to the indexers.  This assumes you already know how to use a deployment server, understand server classes, and have outputs.conf setup correctly.  This will focus just on the inputs.conf file.

We know the location of this file is C:\inetpub\logs\LogFiles\W3SVC2\u_ex180277.log so we want to onboard all data in the C:\inetpub\logs\LogFiles\W3SVC2 directory as this new iis_example sourcetype (stanza name in props.conf) and put it into an index called web_logs.

inputs.conf

[monitor://C:\inetpub\logs\LogFiles\W3SVC2\*.log] 
index=web_logs
sourcetype=iis_example
disabled=false

After this is deployed, we can validate the data is coming in and parsing correctly from the search head.  You may need to wait for all of the data to be indexed to validate time zones.  Consider setting your time to ‘All Time’ for validation.

Example Search

index=web_logs sourcetype=iis_example

Garbage In – Garbage Out

Validation is a crucial part of onboarding data.  The major things you want to look for are:

  • Is the timestamp correct – if the _time is not the same as the timestamp in the event, you likely have an issue with your TIME_FORMAT or MAX_TIMESTAMP_LOOKAHEAD
  • Is the data in the past or the future – you may have a time zone issue and need to define that in your props.conf (TZ=XXX)
  • Are events breaking in the correct location or is every line of a JAVA debug dump its own event

5. Search your Data with Knowledge Objects

The major point of this guide have already been covered above but the last step is to make the data usable by creating fields.  Just to name a few ways of doing so, this can be done using regular expression, evaluations, and field aliases.  You can add additional context to the data such as showing a human readable status code (while IIS subject matter experts will quickly know status 404 is “page not found”, end users may not know this) with lookup tables.  You may also want to normalize the data into a CIM data model using eventtypes and tags so that you can view all of your Web data regardless of technology (IIS, Apache, Tomcat, WebLogic, etc) in the same dashboard.

Thanks for taking a few minutes to read our post onboarding custom data into Splunk.  For support with Splunk or Splunk Managed Services please contact us here.

Pin It on Pinterest

Share This