Splunk 5.0.2 Deploy Forwarders

Splunk 5.0.2

Distributed Deployment Manual

Generated: 5/08/2013 9:26 pm

Copyright © 2013 Splunk, Inc. All Rights Reserved

Table of ContentsOverview...............................................................................................................1

Distributed Splunk overview......................................................................1How data moves through Splunk: the data pipeline...................................5Scale your deployment: Splunk components.............................................7 Components and roles............................................................................10

Estimate hardware requirements.....................................................................14 Hardware capacity planning for a distributed Splunk deployment..........14 Distribute indexing and searching...........................................................15 How Splunk looks through your data......................................................16 Reference hardware................................................................................18 Accommodate concurrent users and searches.......................................20 How Splunk apps affect resource requirements.....................................24 Summary of performance recommendations..........................................24

Forward data......................................................................................................26 About forwarding and receiving...............................................................26 Types of forwarders................................................................................27 Forwarder deployment topologies...........................................................31

Configure forwarding........................................................................................35 Set up forwarding and receiving..............................................................35 Enable a receiver....................................................................................37 Configure forwarders with outputs.conf...................................................42 Protect against loss of in-flight data........................................................50

Use the forwarder to create deployment topologies......................................58 Consolidate data from multiple machines...............................................58 Set up load balancing..............................................................................59 Route and filter data................................................................................64 Forward data to third-party systems........................................................79

Deploy the universal forwarder........................................................................84 Introducing the universal forwarder.........................................................84 Universal forwarder deployment overview..............................................86 Deploy a Windows universal forwarder via the installer GUI..................91 Deploy a Windows universal forwarder via the command line................98 Remotely deploy a Windows universal forwarder with a static

configuration...........................................................................................110

i

Table of ContentsDeploy the universal forwarder

Deploy a *nix universal forwarder manually..........................................111 Remotely deploy a *nix universal forwarder with a static

configuration...........................................................................................117 Make a universal forwarder part of a system image.............................124 Migrate a Windows light forwarder........................................................126 Migrate a *nix light forwarder................................................................128 Supported CLI commands....................................................................129

Deploy heavy and light forwarders................................................................131 Deploy a heavy or light forwarder.........................................................131 Heavy and light forwarder capabilities..................................................136

Search across multiple indexers....................................................................139 About distributed search.......................................................................139 Install a dedicated search head............................................................144 Configure distributed search.................................................................145Mount the knowledge bundle.................................................................151 Configure search head pooling.............................................................160 How authorization works in distributed searches..................................168 Use distributed search..........................................................................171 Troubleshoot distributed search............................................................172

Monitor your deployment................................................................................176 About Splunk Deployment Monitor App................................................176

Deploy configuration updates and apps across your environment...........177 About deployment server......................................................................177 Plan a deployment................................................................................180 Configure deployment clients................................................................182 Define server classes............................................................................187 Deploy apps and configurations............................................................196 Extended example: deploy configurations to several forwarders..........199 Example: add an input to forwarders....................................................207 Example: deploy an app........................................................................208 Deploy in multi-tenant environments.....................................................209

ii

Table of ContentsUpgrade your deployment..............................................................................212

Upgrade your distributed environment...................................................212 Upgrade the Windows universal forwarder...........................................218 Upgrade the universal forwarder for *nix systems................................221

iii

Overview

Distributed Splunk overview

This manual describes how to distribute various components of Splunkfunctionality across multiple machines. By distributing Splunk, you can scale itsfunctionality to handle the data needs for enterprises of any size and complexity.

In single-machine deployments, one instance of Splunk handles the entireend-to-end process, from data input through indexing to search. Asingle-machine deployment can be useful for testing and evaluation purposesand might serve the needs of department-sized environments. For largerenvironments, however, where data originates on many machines and wheremany users need to search the data, you'll want to distribute functionality acrossmultiple Splunk instances. This manual describes how to deploy and use Splunkin such a distributed environment.

How Splunk scales

Splunk performs three key functions as it moves data through the data pipeline.First, Splunk consumes data from files, the network, or elsewhere. Then itindexes the data. (Actually, it first parses and then indexes the data, but forpurposes of this discussion, we consider parsing to be part of the indexingprocess.) Finally, it runs interactive or scheduled searches on the indexed data.

You can split this functionality across multiple specialized instances of Splunk,ranging in number from just a few to thousands, depending on the quantity ofdata you're dealing with and other variables in your environment. You might, forexample, create a deployment with many Splunk instances that only consumedata, several other instances that index the data, and one or more instances thathandle search requests. These specialized instances of Splunk are knowncollectively as components. There are several types of components.

For a typical mid-size deployment, for example, you can deploy lightweightversions of Splunk, called forwarders, on the machines where the dataoriginates. The forwarders consume data locally and then forward the dataacross the network to another Splunk component, called the indexer. Theindexer does the heavy lifting; it indexes the data and runs searches. It shouldreside on a machine by itself. The forwarders, on the other hand, can easilyco-exist on the machines generating the data, because the data-consumingfunction has minimal impact on machine performance. This diagram shows

1

several forwarders sending data to a single indexer:

As you scale up, you can add more forwarders and indexers. For a largerdeployment, you might have hundreds of forwarders sending data to a number ofindexers. You can use load balancing on the forwarders, so that they distributetheir data across some or all of the indexers. Not only does load balancing helpwith scaling, but it also provides a fail-over capability if one of the indexers goesdown. The forwarders automatically switch to sending their data to any indexersthat remain alive. In this diagram, each forwarder load-balances its data acrosstwo indexers:

To coordinate and consolidate search activities across multiple indexers, you canalso separate out the functions of indexing and searching. In this type ofdeployment, called distributed search, each indexer just indexes data andperforms searches across its own indexes. A Splunk instance dedicated tosearch management, called the search head, coordinates searches across theset of indexers, consolidating the results and presenting them to the user:

2

For the largest environments, you can deploy a pool of several search headssharing a single configuration set. With search head pooling, you cancoordinate simultaneous searches across a large number of indexers:

These diagrams illustrate a few basic deployment topologies. You can actuallycombine the Splunk functions of data input, indexing, and search in a greatvariety of ways. For example, you can set up the forwarders so that they routedata to multiple indexers, based on specified criteria. You can also configureforwarders to process data locally before sending the data on to an indexer forstorage. In another scenario, you can deploy a single Splunk instance that servesas both search head and indexer, searching across not only its own indexes butthe indexes on other Splunk indexers as well. You can mix-and-match Splunkcomponents as needed. The possible scenarios are nearly limitless.

This manual describes how to scale a deployment to fit your exact needs,whether you're managing data for a single department or for a global enterprise...or for anything in between.

Use index clusters

Starting with version 5.0, you can group Splunk indexers into clusters. Clustersare groups of Splunk indexers configured to replicate each others' data, so that

3

the system keeps multiple copies of all data. This process is known as indexreplication. By maintaining multiple, identical copies of Splunk data, clustersprevent data loss while promoting data availability for searching.

Splunk clusters feature automatic failover from one indexer to the next. Thismeans that, if one or more indexers fail, incoming data continues to get indexedand indexed data continues to be searchable.

Besides enhancing data availability, clusters have other key features that youshould consider when you're scaling a deployment. For example, they include acapability to coordinate configuration updates easily across all indexers in thecluster. They also include a built-in distributed search capability. For moreinformation on clusters, see "About clusters and index replication" in theManaging Indexers and Clusters manual.

Manage your Splunk deployment

Splunk provides a few key tools to help manage a distributed deployment:

Deployment server. This Splunk component provides a way to centrallymanage configurations and content updates across your entiredeployment. See "About deployment server" for details.

•

Deployment monitor. This app can help you manage and troubleshootyour deployment. It tracks the status of your forwarders and indexers andprovides early warning if problems develop. Read the Deploy and UseSplunk Deployment Monitor App manual for details.

•

What comes next

The rest of this Overview section covers:

How data moves through Splunk: the data pipeline• Scale your deployment: Splunk components• Components and roles•

It starts by describing the data pipeline, from the point that the data enters Splunkto when it becomes available for users to search on. Next, the overviewdescribes how Splunk functionality can be split into modular components. It thencorrelates the available Splunk components with their roles in facilitating the datapipeline.

The remaining sections of this manual describe the Splunk components in detail,explaining how to use them to create a distributed Splunk deployment.

4

For information on capacity planning based on the scale of your deployment,read "Hardware capacity planning for your Splunk deployment" in the Installationmanual.

How data moves through Splunk: the data pipeline

Data in Splunk transitions through several phases, as it moves along the datapipeline from its origin in sources such as logfiles and network feeds to itstransformation into searchable events that encapsulate valuable knowledge. Thedata pipeline includes these segments:

Input• Parsing• Indexing• Search•

You can assign each of these segments to a different Splunk instance, asdescribed here.

This diagram outlines the data pipeline:

Splunk instances participate in one or more segments of the data pipeline, asdescribed in "Scale your deployment".

5

Note: The diagram represents a simplified view of the indexing architecture. Itprovides a functional view of the architecture and does not fully describe Splunkinternals. In particular, the parsing pipeline actually consists of three pipelines:parsing, merging, and typing, which together handle the parsing function. Thedistinction can matter during troubleshooting, but does not ordinarily affect howyou configure or deploy Splunk.

Input

In the input segment, Splunk consumes data. It acquires the raw data streamfrom its source, breaks it into 64K blocks, and annotates each block with somemetadata keys. The keys apply to the entire input source overall. They includethe host, source, and source type of the data. The keys can also include valuesthat are used internally by Splunk, such as the character encoding of the datastream, and values that control later processing of the data, such as the indexinto which the events should be stored.

During this phase, Splunk does not look at the contents of the data stream, sothe keys apply to the entire source, not to individual events. In fact, at this point,Splunk has no notion of individual events at all, only of a stream of data withcertain global properties.

Parsing

During the parsing segment, Splunk examines, analyzes, and transforms thedata. This is also known as event processing. It is during this phase that Splunkbreaks the data stream into individual events.The parsing phase has manysub-phases:

Breaking the stream of data into individual lines.• Identifying, parsing, and setting timestamps.• Annotating individual events with metadata copied from the source-widekeys.

•

Transforming event data and metadata according to Splunk regextransform rules.

•

Indexing

During indexing, Splunk takes the parsed events and writes them to the index ondisk. It writes both compressed raw data and the corresponding index files.

For brevity, parsing and indexing are often referred together as the indexing

6

process. At a high level, that's fine. But when you need to look more closely atthe actual processing of data, it can be important to consider the two segmentsindividually.

A detailed diagram that depicts the indexing pipelines and explains how indexingworks can be found in "How Indexing Works" in the Community Wiki.

Search

Splunk's search function manages all aspects of how the user sees and uses theindexed data, including interactive and scheduled searches, reports and charts,dashboards, and alerts. As part of its search function, Splunk stores user-createdknowledge objects, such as saved searches, event types, views, and fieldextractions.

For more information on the various steps in the pipeline, see "How indexingworks" in the Managing Indexers and Clusters manual.

Scale your deployment: Splunk components

To accommodate your deployment topology and performance requirements, youcan allocate the different Splunk roles, such as data input and indexing, toseparate Splunk instances. For example, you can have instances that just gatherdata inputs, which they then forward to another, central instance for indexing. Oryou can distribute indexing across several instances that coordinate with aseparate instance that processes all search requests. To facilitate the distributionof roles, Splunk can be configured into a range of separate component types,each mapping to one or more of the roles. You create most components byenabling or disabling specific functions of the full Splunk instance.

These are the Splunk component types available for use in a distributedenvironment:

Indexer• Forwarder• Search head• Deployment server•

All components are variations of the full Splunk instance, with certain featureseither enabled or disabled, except for the universal forwarder, which is its ownexecutable.

7

Indexers

The indexer is the Splunk component that creates and manages indexes. Theprimary functions of an indexer are:

Indexing incoming data.• Searching the indexed data.•

In single-machine deployments consisting of just one Splunk instance, theindexer also handles the data input and search management functions.

For larger-scale needs, indexing is split out from the data input function andsometimes from the search management function as well. In these larger,distributed deployments, the Splunk indexer might reside on its own machine andhandle only indexing (usually along with parsing), along with searching of itsindexed data. In those cases, other Splunk components take over thenon-indexing/searching roles. Forwarders consume the data, indexers index andsearch the data, and search heads coordinate searches across the set ofindexers.

For information on indexers, see the Managing Indexers and Clusters manual,starting with the topic "About indexes and indexers".

Forwarders

One role that's typically split off from the indexer is the data input function. Forinstance, you might have a group of Windows and Linux machines generatingdata that needs to go to a central Splunk indexer for consolidation. Usually thebest way to do this is to install a lightweight instance of Splunk, known as aforwarder, on each of the data-generating machines. These forwarders managethe data input and send the resulting data streams across the network to aSplunk indexer, which resides on its own machine. There are two types offorwarders:

Universal forwarders. These have a very light footprint and forward onlyunparsed data.

•

Heavy forwarders. These have a larger footprint but can parse, and evenindex, data before forwarding it.

•

Note: There is also a third type of forwarder, the light forwarder. The lightforwarder is essentially obsolete, having being replaced in release 4.2 by theuniversal forwarder, which provides similar functionality in a smaller footprint.

8

For information on forwarders, start with the topic "About forwarding andreceiving".

Search heads

In situations where you have a large amount of indexed data and numeroususers concurrently searching on it, it can make sense to distribute the indexingload across several indexers, while offloading the search query function to aseparate machine. In this type of scenario, known as distributed search, one ormore Splunk components called search heads distribute search requests acrossmultiple indexers.

For information on search heads, see "About distributed search".

Deployment server

To update a distributed deployment, you can use Splunk's deployment server.The deployment server lets you push out configurations and content to sets ofSplunk instances (referred to, in this context, as deployment clients), groupedaccording to any useful criteria, such as OS, machine type, application area,location, and so on. The deployment clients are usually forwarders or indexers.For example, once you've made and tested an updated configuration on a localLinux forwarder, you can push the changes to all the Linux forwarders in yourdeployment.

The deployment server can cohabit a Splunk instance with another Splunkcomponent, either a search head or an indexer, if your deployment is small (lessthan around 30 deployment clients). It should run on its own Splunk instance inlarger deployments. For more information, see this tech note on the CommunityWiki.

For detailed information on the deployment server, see "About deploymentserver".

Deployment monitor

Although it's actually an app, not a Splunk component, the deployment monitorhas an important role to play in distributed environments. Distributeddeployments can scale to forwarders numbering into the thousands, sendingdata to many indexers, which feed multiple search heads. To view andtroubleshoot these distributed deployments, you can use the deploymentmonitor, which provides numerous views into the state of your forwarders and

9

indexers.

For detailed information on the deployment monitor, read the Deploy and UseSplunk Deployment Monitor App manual.

Where to go next

While the fundamental issues of indexing and event processing remain thesame no matter what the size or nature of your distributed deployment, it isimportant to take into account deployment needs when planning your indexingstrategy. To do that effectively, you must also understand how components mapto Splunk roles.

For information on hardware requirements for scaling your deployment, see"Hardware capacity planning for your Splunk deployment".

Components and roles

Each segment of the data pipeline directly corresponds to a role that one ormore Splunk components can perform. For instance, data input is a Splunk role.Either an indexer or a forwarder can perform the data input role. For moreinformation on the data pipeline, look here.

How components support the data pipeline

This table correlates the pipeline segments and Splunk roles with thecomponents that can perform them:

Data pipelinesegment Splunk role Splunk components that can perform this

role

Data input Data inputindexeruniversal forwarderheavy forwarder

Parsing Parsingindexerheavy forwarder

Indexing Indexing indexer

Search Searchindexersearch head

n/a deployment server

10

Managing distributedupdates

n/a Troubleshootingdeployments

deployment monitor app (not actually acomponent, but rather a key feature formanaging distributed environments)

As you can see, some roles can be filled by diffferent components depending onthe situation. For instance, data input can be handled by an indexer insingle-machine deployments, or by a forwarder in larger deployments.

For more information on components, look here.

Components in action

These are some of the common ways in which Splunk functionality is distributedand managed.

Forward data to an indexer

In this deployment scenario, forwarders handle data input, collecting data andsend it on to a Splunk indexer. Forwarders come in two flavors:

Universal forwarders. These maintain a small footprint on their hostmachine. They perform minimal processing on the incoming data streamsbefore forwarding them on to an indexer, also known as the receiver.

•

Heavy forwarders. These retain much of the functionality of a full Splunkinstance. They can parse data before forwarding it to the receivingindexer. (See "How data moves through Splunk" for the distinctionbetween parsing and indexing.)

•

Both types of forwarders tag data with metadata such as host, source, andsource type, before forwarding it on to the indexer.

Note: There is also a third type of forwarder, the light forwarder. The lightforwarder is essentially obsolete, having being replaced in release 4.2 by theuniversal forwarder, which provides similar functionality in a smaller footprint.

Forwarders allow you to use resources efficiently while processing largequantities or disparate types of data. They also enable a number of interestingdeployment topologies, by offering capabilities for load balancing, datafiltering, and routing.

11

For an extended discussion of forwarders, including configuration and detaileduse cases, see "About forwarding and receiving".

Search across multiple indexers

In distributed search, Splunk instances send search requests to other Splunkinstances and merge the results back to the user. This is useful for a number ofpurposes, including horizontal scaling, access control, and managinggeo-dispersed data.

The Splunk instance that manages search requests is called the search head.The instances that maintain the indexes and perform the actual searching areindexers, called search peers in this context.

For an extended discussion of distributed search, including configuration anddetailed use cases, see "About distributed search".

Manage distributed updates

When dealing with distributed deployments consisting potentially of manyforwarders, indexers, and search heads, the Splunk deployment server simplifiesthe process of configuring and updating Splunk components, mainly forwardersand indexers. Using the deployment server, you can group the components(referred to as deployment clients in this context) into server classes, making itpossible to push updates based on common characteristics.

A server class is a set of Splunk instances that share configurations. Serverclasses are typically grouped by OS, machine type, application area, location, orother useful criteria. A single deployment client can belong to multiple serverclasses, so a Linux universal forwarder residing in the UK, for example, mightbelong to a Linux server class and a UK server class, and receive configurationsettings appropriate to each.

For an extended discussion of deployment management, see "About deploymentserver".

View and troubleshoot your deployment

Use the deployment monitor app to view the status of your Splunk componentsand troubleshoot them. The deployment monitor is functionally part of the searchrole, so it resides with either the indexer or a search head. It looks at internalevents generated by Splunk forwarders and indexers.

12

The home page for this app is a dashboard that provides charts with basic statson index throughput and forwarder connections over time. It also includeswarnings for unusual conditions, such as forwarders that appear to be missingfrom the system or indexers that aren't currently indexing any data.

The charts, warnings, and other information on this page provide an easy way tomonitor potentially serious conditions. The page itself provides guidance on whateach type of warning means.

The deployment monitor also provides pages that consolidate data for allforwarders and indexers in your deployment. You can drilldown further, to obtaindetailed information on any forwarder or indexer.

For more information on the deployment monitor, read the Deploy and UseSplunk Deployment Monitor App manual.

For more information

In summary, these are the fundamental components and features of a Splunkdistributed environment:

Indexers. See "About indexes and indexers" in the Managing Indexersand Clusters manual.

•

Forwarders. See "About forwarding and receiving" in this manual.• Search heads. See "About distributed search" in this manual.• Deployment server. See "About deployment server" in this manual.• Deployment monitor. Read the Deploy and Use Splunk DeploymentMonitor App manual.

•

For guidance on where to configure various Splunk settings, see "Configurationparameters and the data pipeline" in the Admin manual. That topic lists keyconfiguration settings and the data pipeline segments they act upon. If you knowwhich components in your Splunk topology handle which segments of the datapipeline, you can use that topic to determine where to configure the varioussettings. For example, if you use a search head to handle the search segment,you'll need to configure any search-related settings on the search head and noton your indexers.

13

Estimate hardware requirements

Hardware capacity planning for a distributedSplunk deployment

Overview

If you have larger indexing or searching requirements, run Splunk apps orsolutions that generate or execute a lot of saved searches, or regularly employI/O-intensive searches, then you should scale your Splunk deployment toaddress the increased resource overhead that those operations incur. For anoverview of what a distributed Splunk deployment is, review "Distributed Splunkoverview" in this manual.

In many cases, this involves using distributed search to run searches in parallelacross multiple indexers at once. You can gather data from machines usingSplunk forwarders and, optionally, configure those servers to send data tomultiple indexers at once to reduce search time. For information on the individualelements of a Splunk deployment, read "Components of a Splunk deployment" inthe Installation Manual.

Estimate hardware requirements

While determining the hardware requirements for your distributed Splunkdeployment, there are a number of things you must consider.

You must understand how various Splunk activities affect the resource overheadrequired to perform them. Each of the following activities has a direct impact onthe overall performance of Splunk:

The amount of data you index.• The number of concurrent Splunk users.• The number of saved searches you run.• The types of search you employ.• The number of apps or solutions you implement.• When you run apps, whether or not those apps execute a large number ofsaved searches.

•

When you add more indexers to a Splunk deployment, you increase the amountof available indexing capacity by reducing the indexing overhead per server.

14

Consequently, reduced indexing overhead also means reduced search time.

But that is only half the story. While Splunk scales across multiple indexers, theamount of indexing throughput becomes less important as either the number ofconcurrent users or saved searches increases. Additionally, depending on thekinds of searches you employ against your data, the resource needs forsearching can become as important as the resource needs for indexing.

For additional information on estimating your hardware requirements, read thefollowing topics, all in this manual:

"Distribute indexing and searching" - for details on how to beginstructuring your distributed environment.

•

"How Splunk looks through your data" - to learn what the different searchtypes are, and how they impact performance on an indexer.

•

"Reference hardware" - to learn about the reference servers and theindexing and searching performance they are capable of.

•

"Accommodate concurrent users and searches" - for various scenarios onaddressing search performance.

•

"How Splunk apps affect resource requirements" - for additionalinformation on how Splunk apps consume computing resources.

•

Considerations for clusters

There are some additional hardware issues to consider if you're implementingSplunk clusters. See "System requirements and other deploymentconsiderations" in the Managing Indexers and Clusters manual.

Distribute indexing and searching

This topic discusses the concepts and hardware requirements for distributing theindexing and searching components of your Splunk deployment.

Concepts of distributed indexing and searching

You scale your Splunk deployment by dedicating searching and indexing acrossmultiple servers. Indexers bring in, store, and search the data. Search headsmanage search requests and present results.

Since indexers require much more disk I/O throughput than search heads do,you give your environment more indexing capacity by reducing the overhead

15

required for searching. The key points to remember are:

The more indexers you add to the deployment, the faster data isconsumed and prepared for searches.

•

The more search heads you add to the deployment, the faster you areable to find the data you indexed.

•

Considerations for search performance vs. indexingperformance

While the two points shown above are best practice for improving indexingspeed, there are some important caveats to note as well, particularly when itcomes to search speed.

As your indexers consume data, they store it in buckets - individual elements ofan index. As more data comes in, the number of buckets increases. An increasednumber of buckets - particularly those which hold smaller amounts of data - canimpact search speed because of the throughput required to navigate throughthose buckets for the data that you're searching.

Additionally, as the number of buckets increases, Splunk must manage thebuckets on an individual system. Splunk does this by "rolling" buckets - thusmaking room for new incoming data. This procedure takes up I/O cycles as well -cycles that could be used to fetch events for search requests.

The key points to understand are:

You can't necessarily improve search performance simply by addingsearch heads to your distributed deployment. A mix of search heads andindexers is vital.

•

The number and types of search also impact indexer performance. Somesearch types tax an indexer's CPU, others apply pressure to the disksubsystem.

•

More detail about how to plan for simultaneous searches is found in"Accommodate concurrent users and searches" in this manual.

How Splunk looks through your data

As you scale your deployment up, the importance of understanding the differenttypes of search and how they impact Splunk's performance increases. This

16

knowledge helps you determine how many indexers and search heads youshould add to your distributed deployment.

Search types: the details

There are four basic types of search that you can invoke against data stored in aSplunk index. Each of these search types impacts the Splunk indexer in adifferent way. The search types are:

Dense. A dense search is a search that returns a large percentage (10% ormore) of matching results for a given set of data in a given period of time. Areference server should be able to fetch up to 50,000 matching events persecond for a dense search. Dense searches usually tax a server's CPU first,because of the overhead required to decompress the raw data stored in a Splunkindex.

Sparse. Sparse searches return smaller numbers of results for a given set ofdata in a given period of time (anywhere from .01 to 1%) than dense searchesdo. A reference indexer should be able to fetch up to 5,000 matching events persecond when executing a sparse search.

Super-sparse. A super-sparse search is a "needle in the haystack" search thatretrieves only a very small number of results across the same set of data withinthe same time period as the other searches. A super-sparse search is very I/Ointensive because the indexer must look through all of the buckets of an index tofind the desired results. This can take up to two seconds per searched bucket. Ifyou have a large amount of data stored on your indexer, there are a lot ofbuckets, and a super-sparse search can take a very long time to complete.

Rare. Rare searches are like super-sparse searches in that they match just ahandful of results across a number of index buckets. The major difference withrare searches is that bloom filters - data structures that test whether or not anelement is a member of a set - significantly reduce the number of buckets thatneed to be searched by eliminating those buckets which do not contain eventsthat match the search request. This allows a rare search to complete anywherefrom 20 to 100 times faster than a super-sparse search, for the same amount ofdata searched.

Summary

The following table summarizes the different search types. Note that for denseand sparse searches, Splunk measures performance based on number ofmatching events, while with super-sparse and rare searches, performance is

17

measured based on total indexed volume.

Searchtype

Description Ref. indexerthroughput

Performanceimpact

DenseDense searches return a largepercentage of results for a given set ofdata in a given period of time.

Up to 50,000matching eventsper second

GenerallyCPU-bound

Sparse

Sparse searches return a smaller amountof results for a given set of data in agiven period of time than dense searchesdo.

Up to 5,000matching eventsper second

GenerallyCPU-bound

Super-sparse

Super-sparse searches return a verysmall number of results from each indexbucket which match the search.Depending on how large the set ofdata is, these types of search cantake a long period of time.

Up to 2 secondsper index bucket

Primarily I/Obound

Rare

Rare searches are similar tosuper-sparse searches, but are assistedby bloom filters which helpeliminate index buckets that donot match the search request.Rare searches return resultsanywhere from 20 to 100 timesfaster than a super-sparse searchdoes.

From 10 to 50index buckets persecond

Primarily I/Obound

Reference hardware

At higher data indexing rates and/or user counts, you must take into account thediffering needs of indexers and search heads. Dedicated search heads do notneed extremely fast disk throughput, nor do they need much local storage. Theydo, however, require far more CPU resources than indexers do.

The reference indexer for a distributed Splunk deployment is somewhat differentfrom a reference indexer in a single-server deployment. Of particular note is thedisk subsystem: a reference indexer in a distributed deployment has significantlylarger disk throughput requirements over a single-server reference indexer. Themain reason for this is that an indexer in a distributed deployment indexes moredata and handles more search requests than one in a single-server deployment.

18

Following are the recommendations for both search heads and indexers in adistributed deployment:

Dedicated search head

Intel 64-bit chip architecture• 4 CPUs, 4 cores per CPU, 2.5-3 Ghz per core• 8 GB RAM• 2 x 300 GB, 10,000 RPM SAS hard disks, configured in RAID 0• Standard 1Gb Ethernet NIC, optional 2nd NIC for a management network• Standard 64-bit Linux or Windows distribution•

Note: Since search heads require raw computing power and are likely to becomeCPU-bound, it is better to add additional CPU cores to a search head if fasterperformance per server is desired. The guideline of 1 core per active Splunk userstill applies. Additionally:

Don't forget to account for scheduled searches in your CPU allowance. Atypical search request requires 1 CPU core.

•

Depending on the type of search you use, you might need to add CPUcores to account for the increased load those search types cause.

•

Indexer

Indexers in a distributed deployment configuration have higher disk I/Obandwidth requirements than indexers in a non-distributed environment. This isbecause indexers must both write new data and service the remote requests ofsearch heads.

Intel 64-bit chip architecture• 2 CPUs, 4 cores per CPU, 2.5-3Ghz per core• 8 GB RAM• Disk subsystem capable of 1200 average input/output operations persecond (IOPS) (for example: 12 x 300GB, 15,000 RPM SAS hard disks,configured in RAID 1+0

•

Standard 1Gb Ethernet NIC, optional 2nd NIC for a management network• Standard 64-bit Linux or Windows distribution•

At higher daily volumes, local disk will likely not provide cost-effective storage forthe time frames where speedy search is desired. In these cases, we suggestdeploying fast attached storage or networked storage, such as storage areanetworks (SAN) over fiber. While there are too many types of storage torecommend, consider these guidelines when planning your storage

19

infrastructure:

Indexers do many bulk reads.• Indexers do many disk seeks.•

Therefore:

More disks (specifically, more spindles) are better for indexingperformance.

•

Total throughput of the entire system is important, however:• The ratio of disks to disk controllers in a particular system should behigher, similar to how you configure a database server.

•

Ratio of indexers to search heads

Technically, there is no practical Splunk limitation on the number of search headsan indexer can support, or the number of indexers a search head can searchagainst. However, systems limitations suggest a ratio of approximately 8 to 1 inmost use cases. That is a rough guideline - if you have many searcherscompared to your total data volume, then more search heads will increasesearch efficiency. In some cases, the best use of a separate search head is topopulate summary indexes. This search head then acts like an indexer to theprimary search head that users log into.

Accommodate concurrent users and searches

This topic discusses how to accommodate many simultaneous searches, searchtypes, and concurrent users with your distributed Splunk deployment.

Overview

The biggest performance impacts on a Splunk deployment are the number ofconcurrent users, the number of concurrent searches, and the types of searchesyou employ during the course of the deployment's operation. Each of theseelements affects the deployment in different ways.

How concurrent users and searches affect performance

When a user submits a search request, the search request may take up to oneCPU core on each indexer to process while it is running. Any additional searcheslaunched by that user also account for one CPU core.

20

The type of search the user invokes also impacts resource usage. The additionalamount of CPU or disk usage varies depending on the search type. Foradditional information about Splunk search types, read "How Splunk looksthrough your data" in this chapter.

How to maximize search performance

The best way to address the resource overhead for many concurrent searches isto configure the environment to handle search requests completely withinavailable physical server memory. While this is important for the servers that youdesignate as search heads, it's particularly important for indexers in yourdeployment, as indexers must both index incoming data and search existingdata.

For example, in a deployment that has up to 48 concurrent searches happeningat a time, a single search that uses 200 MB of memory translates to nearly 10GB of memory required to satisfy the 48 concurrent search requests. The amountof available memory is a very important statistic - while performance on anindexer only declines gradually with increased CPU usage from concurrentsearch jobs, it drops dramatically when the server exhausts all available physicalmemory. A single reference indexer could not handle 48 concurrent searches - atthe very least, additional memory is required for satisfactory performance.

Notwithstanding the memory constraints, a search's run time increasesproportionally as the number of free CPU cores on a system decreases. Forexample, on an idle system with 8 available cores, the first eight searches toarrive get serviced immediately by each core, and complete within a short periodof time (for the purposes of this example, 10 seconds).

If there are 48 searches running concurrently, then completion time increasessignificantly, as shown in the calculation below:

No. ofconcurrentsearches

/ No. ofavail.cores

= No. ofsearches per

core

x No. of sec. perindividual search

= Totaltime(sec.)per

search48 8 6 10 60

Since indexers do the bulk of the work in search operations (reading data offdisk, decompressing it, extracting knowledge and reporting), it's best practice toadd indexers to decrease the amount of time per search. In this example, toreturn to the performance level of 10-second searches, you must deploy 6

21

indexers with eight cores each:


/ No. ofavail.cores

= No. ofcores per

search


= Total time(sec.) per

search8 48 6 10 1.6


/ No. ofavail.cores

= No. ofcores per

search



search48 48 1 10 10

Note: In this example, one search head to service search requests is okay. Itmight be appropriate, however, to set aside a second search head to createsummary indexes.)

In many cases, though, the system isn't idle before searches arrive. For example,if an indexer is indexing 150 GB/day of data at peak times, then up to 4 of the 8cores are in use indexing the data. In this case, search times increasesignificantly:


/ No. ofavail.cores

= No. ofcores per

search



search4 4 1 10 10


/ No. ofavail.cores


core



search48 4 12 10 120

Increasing the number of cores per server does decrease the amount of timetaken per search, but is not the most effective way to streamline searchoperations. One system with 16 cores has the following performance (again,assuming an idle system before searches arrive):


/ No. ofavail.cores

= No. ofcores per

search



search16 16 1 10 10

22


/ No. ofavail.cores


core



search48 16 3 10 30

With two 8-core servers, the performance profile becomes the following:


/ No. ofavail.cores

= No. ofcores per

search



search8 16 2 10 5


/ No. ofavail.cores

= No. ofcores per

search



search16 16 1 10 10


/ No. ofavail.cores


core



search48 16 3 10 30

Two 8-core servers cost only slightly more than one 16-core server. In addition,two 8-core servers provide significantly more available disk throughput than one16-core server does, based on spindle count. This is especially important forindexers because of the high disk bandwidth that they require.

Adding indexers reduces the indexing load on any system and frees CPU coresfor searching. Also, since the performance of almost all types of search scaleswith the number of indexers, searches will be faster, which mitigates the effect ofreduced performance from sharing resources amongst both indexing andsearching. Increasing search speed reduces the chance of concurrent searcheswith concurrent users.

In real-world situations with hundreds of users, each user will run a search everyfew minutes, though not at the exact same time as other users. By addingindexers and reducing the search time, you reduce the concurrency factor andlower the concurrency-related I/O and memory contention.

23

How Splunk apps affect resource requirements

This topic discusses how Splunk apps and solutions affect hardwarerequirements in a distributed Splunk deployment.

If you use a Splunk app or solution that gets knowledge by executing a largernumber of saved searches, then you can easily overwhelm a single-serverSplunk deployment. This is because multiple searches quickly exhaust availableCPU resources on an indexer. (Review "Accommodate many simultaneoussearches" in this manual for specifics on how Splunk deals with multipleconcurrent searches.)

When installing an app or solution, read the system requirements outlined in thatapp or solution's documentation. If that information is not available, contact theauthors of the app or solution to get specific details about what is needed to runthe app properly.

Summary of performance recommendations

The table below depicts the performance recommendations based on thereference servers described earlier in this chapter. For specifics on thosereference servers, read "Reference hardware."

Important: The table shows approximate guidelines only. You should modifythese figures based on your specific use case. If you need additional guidance,contact Splunk. You might want to engage a member of Professional Servicesdepending on the deployment's initial size.

DailyVolume

Number ofSearch Users

RecommendedIndexers

RecommendedSearch Heads

< 2 GB/day < 2 1, shared N/A

2 to 100GB/day up to 4 1, dedicated N/A

100 to 200GB/day up to 8 2 1

200 to 300GB/day up to 12 3 1

300 to 400GB/day up to 8 4 1

up to 16 5 2

24

400 to 500GB/day

500 GB to 1TB/day up to 24 10 2

1 TB to 20TB/day up to 100 100 24

20 TB to 60TB/day up to 100 300 32

Answers

Have questions? Visit Splunk Answers to see what questions and answers otherSplunk users had about hardware and Splunk.

25

Forward data

About forwarding and receiving

You can forward data from one Splunk instance to another Splunk server or evento a non-Splunk system. The Splunk instance that performs the forwarding istypically a smaller footprint version of Splunk, called a forwarder.

A Splunk instance that receives data from one or more forwarders is called areceiver. The receiver is usually a Splunk indexer, but can also be anotherforwarder, as described here.

This diagram shows three forwarders sending data to a single Splunk receiver(an indexer), which then indexes the data and makes it available for searching:

Forwarders represent a much more robust solution for data forwarding than rawnetwork feeds, with their capabilities for:

Tagging of metadata (source, source type, and host)• Configurable buffering• Data compression• SSL security• Use of any available network ports•

The forwarding and receiving capability makes possible all sorts of interestingSplunk topologies to handle functions like data consolidation, load balancing,and data routing. For more information on the types of deployment topologiesthat you can create with forwarders, see "Forwarder deployment topologies".

Splunk provides a number of types of forwarders to meet various needs. Theseare described in "Types of forwarders".

26

Types of forwarders

There are three types of forwarders:

The universal forwarder is a streamlined, dedicated version of Splunkthat contains only the essential components needed to forward data toreceivers.

•

A heavy forwarder is a full Splunk instance, with some features disabledto achieve a smaller footprint.

•

A light forwarder is also a full Splunk instance, with most featuresdisabled to achieve as small a footprint as possible. The universalforwarder, with its even smaller footprint yet similar functionality,supersedes the light forwarder for nearly all purposes.

•

Note: The light forwarder has been deprecated in Splunk version 6.0. For a list ofall deprecated features, see the topic "Deprecated features" in the ReleaseNotes.

In nearly all respects, the universal forwarder represents the best tool forforwarding data to indexers. Its main limitation is that it forwards only unparseddata, as described later in this topic. Therefore, you cannot use it to route databased on event contents. For that, you must use a heavy forwarder. You alsocannot index data locally on a universal forwarder; only a heavy forwarder canindex and forward.

The universal forwarder

The universal forwarder is Splunk's new lightweight forwarder. You use it togather data from a variety of inputs and forward the data to a Splunk server forindexing and searching. You can also forward data to another forwarder, as anintermediate step before sending the data onwards to an indexer.

The universal forwarder's sole purpose is to forward data. Unlike a full Splunkinstance, you cannot use the universal forwarder to index or search data. Toachieve higher performance and a lighter footprint, it has several limitations:

The universal forwarder has no searching, indexing, or alerting capability.• The universal forwarder does not parse data.• Unlike full Splunk, the universal forwarder does not include a bundledversion of Python.

•

27

For details on the universal forwarder's capabilities, see "Introducing theuniversal forwarder".

Note: The universal forwarder is a separately downloadable piece of software.Unlike the heavy and light forwarders, you do not enable it from a full Splunkinstance. To learn how to download, install, and deploy a universal forwarder,see "Universal forwarder deployment overview".

Heavy and light forwarders

While the universal forwarder is generally the preferred way to forward data, youmight have reason (legacy-based or otherwise) to use heavy or light forwardersas well. Unlike the universal forwarder, which is an entirely separate, streamlinedexecutable, both heavy and light forwarders are actually full Splunk instanceswith certain features disabled. Heavy and light forwarders differ in capability andthe corresponding size of their footprints.

A heavy forwarder (sometimes referred to as a "regular forwarder") has asmaller footprint than a Splunk indexer but retains most of the capability, exceptthat it lacks the ability to perform distributed searches. Much of its defaultfunctionality, such as Splunk Web, can be disabled, if necessary, to reduce thesize of its footprint. A heavy forwarder parses data before forwarding it and canroute data based on criteria such as source or type of event.

One key advantage of the heavy forwarder is that it can index data locally, aswell as forward data to another Splunk instance. You must turn this capabilityon; it's disabled by default. See "Configure forwarders with outputs.conf" in thismanual for details.

A light forwarder has a smaller footprint with much more limited functionality. Itforwards only unparsed data. Starting with 4.2, it has been superseded by theuniversal forwarder, which provides very similar functionality in a smallerfootprint. The light forwarder continues to be available mainly to meet any legacyneeds. We recommend that you always use the universal forwarder to forwardunparsed data. When you install a universal forwarder, the installer gives you theopportunity to migrate checkpoint settings from any (version 4.0 or greater) lightforwarder residing on the same machine. See "Introducing the universalforwarder" for a more detailed comparison of the universal and light forwarders.

For detailed information on the capabilities of heavy and light forwarders, see"Heavy and light forwarder capabilities".

28

To learn how to enable and deploy a heavy or light forwarder, see "Deploy aheavy or light forwarder".

Forwarder comparison

This table summarizes the similarities and differences among the three types offorwarders:

Features andcapabilities

Universalforwarder Light forwarder Heavy forwarder

Type of Splunkinstance Dedicated executable Full Splunk, with most

features disabledFull Splunk, with somefeatures disabled

Footprint (memory,CPU load) Smallest Small

Medium-to-large(depending onenabled features)

Bundles Python? No Yes Yes

Handles data inputs?All types (but scriptedinputs might requirePython installation)

All types All types

Forwards to Splunk? Yes Yes Yes

Forwards to 3rd partysystems? Yes Yes Yes

Serves asintermediateforwarder?

Yes Yes Yes

Indexeracknowledgment(guaranteed delivery)?

Optional Optional (version 4.2+) Optional (version 4.2+)

Load balancing? Yes Yes Yes

Data cloning? Yes Yes Yes

Per-event filtering? No No Yes

Event routing? No No Yes

Event parsing? No No Yes

Local indexing? No No

Optional, by settingindexAndForward

attribute inoutputs.conf

Searching/alerting? No No Optional

Splunk Web? No No Optional

29

For detailed information on specific capabilities, see the rest of this topic, as wellas the other forwarding topics in the manual.

Types of forwarder data

Forwarders can transmit three types of data:

Raw• Unparsed• Parsed•

The type of data a forwarder can send depends on the type of forwarder it is, aswell as how you configure it. Universal forwarders and light forwarders can sendraw or unparsed data. Heavy forwarders can send raw or parsed data.

With raw data, the data stream is forwarded as raw TCP; it is not converted intoSplunk's communications format. The forwarder just collects the data andforwards it on. This is particularly useful for sending data to a non-Splunk system.

With unparsed data, a universal forwarder performs only minimal processing. Itdoes not examine the data stream, but it does tag the entire stream withmetadata to identify source, source type, and host. It also divides the data streaminto 64K blocks and performs some rudimentary timestamping on the stream, foruse by the receiving indexer in case the events themselves have no discernibletimestamps. The universal forwarder does not identify, examine, or tag individualevents.

With parsed data, a heavy forwarder breaks the data into individual events,which it tags and then forwards to a Splunk indexer. It can also examine theevents. Because the data has been parsed, the forwarder can performconditional routing based on event data, such as field values.

The parsed and unparsed formats are both referred to as cooked data, todistinguish them from raw data. By default, forwarders send cooked data — inthe universal forwarder's case, unparsed data, and in the heavy forwarder's case,parsed data. To send raw data instead, set the sendCookedData=falseattribute/value pair in outputs.conf.

Forwarders and indexes

Forwarders forward and route data on an index-by-index basis. By default, theyforward all external data, as well as data for the _audit internal index. In somecases, they also forward data for the _internal internal index. You can change

30

this behavior as necessary. For details, see "Filter data by target index".

Forwarder deployment topologies

You can deploy Splunk forwarders in a wide variety of scenarios. This topicprovides an overview of some of the most useful types of topologies that you cancreate with forwarders. For detailed information on how to configure variousdeployment topologies, refer to the topics in the section "Use the forwarder tocreate deployment topologies".

Data consolidation

Data consolidation is one of the most common topologies, with multipleforwarders sending data to a single Splunk server. The scenario typicallyinvolves universal forwarders forwarding unparsed data from workstations orproduction non-Splunk servers to a central Splunk server for consolidation andindexing. With their lighter footprint, universal forwarders have minimal impact onthe performance of the systems they reside on. In other scenarios, heavyforwarders can send parsed data to a central Splunk indexer.

Here, three universal forwarders are sending data to a single Splunk indexer:

For more information on data consolidation, read "Consolidate data from multiplemachines".

Load balancing

Load balancing simplifies the process of distributing data across several Splunkindexers to handle considerations such as high data volume, horizontal scalingfor enhanced search performance, and fault tolerance. In load balancing, theforwarder routes data sequentially to different indexers at specified intervals.

31

Splunk forwarders perform automatic load balancing, in which the forwarderswitches receivers at set time intervals. If parsing is turned on (for a heavyforwarder), the switching will occur at event boundaries.

In this diagram, three universal forwarders are each performing load balancingbetween two indexers:

For more information on load balancing, read "Set up load balancing".

Routing and filtering

In data routing, a forwarder routes events to specific Splunk or third-partyservers, based on criteria such as source, source type, or patterns in the eventsthemselves. Routing at the event level requires a heavy forwarder.

A forwarder can also filter and route events to specific queues, or discard themaltogether by routing to the null queue.

Here, a heavy forwarder routes data to three Splunk indexers based on eventpatterns:

For more information on routing and filtering, read "Route and filter data".

32

Forwarders and clusters

You can use forwarders to send data to peer nodes in a cluster. It isrecommended that you use load-balanced forwarders for that purpose.

This diagram shows two load-balanced forwarders sending data to a cluster:

To learn more about forwarders and clusters, read "Use forwarders to get yourdata" in the Managing Indexers and Clusters Manual. To learn more aboutclusters in general, read "About clusters and index replication".

Forwarding to non-Splunk systems

You can send raw data to a third-party system such as a syslog aggregator. Youcan combine this with data routing, sending some data to a non-Splunk systemand other data to one or more Splunk servers.

Here, three forwarders are routing data to two Splunk servers and a non-Splunksystem:

33

For more information on forwarding to non-Splunk systems, read "Forward datato third-party systems".

Intermediate forwarding

To handle some advanced use cases, you might want to insert an intermediateforwarder between a group of forwarders and the indexer. In this type ofscenario, the end-point forwarders send data to a consolidating forwarder, whichthen forwards the data on to an indexer, usually after indexing it locally.

Typical use cases are situations where you need an intermediate index, either for"store-and-forward" requirements or to enable localized searching. (In this case,you would need to use a heavy forwarder.) You can also use an intermediateforwarder if you have some need to limit access to the indexer machine; forinstance, for security reasons.

To enable intermediate forwarding, you need to configure the forwarder as a botha forwarder and a receiver. For information on how to configure a receiver, read"Enable a receiver".

34

Configure forwarding

Set up forwarding and receiving

Once you've determined your Splunk forwarder deployment topology and whattype of forwarder is necessary to implement it, the steps for setting up forwardingand receiving are straightforward. This topic outlines the key steps and provideslinks to the detailed topics.

To set up forwarding and receiving, you need to perform two basic actions, in thisorder:

1. Set up one or more Splunk indexers as receivers. These will receive the datafrom the forwarders.

2. Set up one or more Splunk forwarders. These will forward data to thereceivers.

The remainder of this topic lists the key steps involved, with links to moredetailed topics. The procedures vary somewhat according to whether theforwarder is a universal forwarder or a heavy/light forwarder. Universalforwarders can sometimes be installed and configured in a single step.Heavy/light forwarders are first installed as full Splunk instances and thenconfigured as forwarders.

Note: This topic assumes that your receivers are indexers. However, in somescenarios, discussed elsewhere, a forwarder also serves as receiver. The set-upis basically much the same for any kind of receiver.


When using forwarders to send data to peer nodes in a cluster, you set upforwarding and receiving a bit differently from the description in this topic. Tolearn more about forwarders and clusters, read "Use forwarders to get your data"in the Managing Indexers and Clusters Manual.

Set up forwarding and receiving: universal forwarders

1. Install the full Splunk instances that will serve as receivers. See the InstallationManual for details.

35

2. Use Splunk Web or the CLI to enable receiving on the instances designated asreceivers. See "Enable a receiver" in this manual.

3. Install, configure, and deploy the universal forwarders. Depending on yourforwarding needs, there are a number of best practices deployment scenarios.See "Universal forwarder deployment overview" for details. Some of thesescenarios allow you to configure the forwarder during the installation process.

4. If you have not already done so during installation, you must specify datainputs for each universal forwarder. See "What Splunk can index" in the GettingData In manual.

Note: Since the universal forwarder does not include Splunk Web, you mustconfigure inputs through either the CLI or inputs.conf; you cannot configure withSplunk Manager.

5. If you have not already done so during installation, you must specify theuniversal forwarders' output configurations. You can do so through the CLI or byediting the outputs.conf file. You get the greatest flexibility by editingoutputs.conf. For details, see the other topics in this section, including"Configure forwarders with outputs.conf".

6. Test the results to confirm that forwarding, along with any configuredbehaviors like load balancing or filtering, is occurring as expected.

Set up forwarding and receiving: heavy or light forwarders


1. Install the full Splunk instances that will serve as forwarders and receivers.See the Installation Manual for details.

2. Use Splunk Web or the CLI to enable receiving on the instances designated asreceivers. See "Enable a receiver" in this manual.

3. Use Splunk Web or the CLI to enable forwarding on the instances designatedas forwarders. See "Deploy a heavy or light forwarder" in this manual.

4. Specify data inputs for the forwarders in the usual manner. See "What Splunkcan index" in the Getting Data In manual.

36

5. Specify the forwarders' output configurations. You can do so through SplunkManager, the CLI, or by editing the outputs.conf file. You get the greatestflexibility by editing outputs.conf. For details, see "Deploy a heavy or lightforwarder", as well as the other topics in this section, including "Configureforwarders with outputs.conf".

6. Test the results to confirm that forwarding, along with any configuredbehaviors like load balancing or routing, is occurring as expected.

Manage your forwarders

In environments with multiple forwarders, you might find it helpful to use thedeployment server to update and manage your forwarders. See "Aboutdeployment server" in this manual.

To view the status of your forwarders, you can use the deployment monitor.

Enable a receiver

To enable forwarding and receiving, you configure both a receiver and aforwarder. The receiver is the Splunk instance receiving the data; the forwardersends data to the receiver.

Depending on your needs (for example to enable load balancing), you mighthave multiple receivers for each forwarder. Conversely, a single receiver usuallyreceives data from many forwarders.

The receiver is either a Splunk indexer (the typical case) or another forwarder(referred to as an "intermediate forwarder") configured to receive data fromforwarders.

You must set up the receiver first. You can then set up forwarders to send data tothat receiver.

Compatibility between forwarders and indexers

These are the compatibility restrictions between versions of forwarders andindexers:

4.2+/5.0+ forwarders (universal/light/heavy) are backwards compatibledown to 4.2+ indexers. For example, a 4.3 forwarder can send data to a

•

37

4.2 indexer but not to a 4.1 indexer.

Pre-4.2 forwarders are backwards compatible down to 4.0 indexers.•

All indexers are backwards compatible with any forwarder and can receivedata from any earlier version forwarder. For example, a 4.2 indexer canreceive data from a 4.1 forwarder.

•

For each app, check Splunkbase for version compatibility.

Note: Splunk recommends that the indexer version should be the same or newerthan the version of the forwarders sending to it. Although we strive to ensurebackward compatibility, it is not always possible.

Set up receiving

Before enabling a Splunk instance (either an indexer or a forwarder) as areceiver you must, of course, first install it.

You can then enable receiving on a Splunk instance through Splunk Web, theCLI, or the inputs.conf configuration file.

Set up receiving with Splunk Web

Use Splunk Manager to set up a receiver:

1. Log into Splunk Web as admin on the server that will be receiving data from aforwarder.

2. Click the Manager link in the upper right corner.

3. Select Forwarding and receiving in the Data area.

4. Click Add new in the Receive data section.

5. Specify which TCP port you want the receiver to listen on (the listening port,also known as the receiving port). For example, if you enter "9997," the receiverwill receive data on port 9997. By convention, receivers listen on port 9997, butyou can specify any unused port. You can use a tool like netstat to determinewhat ports are available on your system. Make sure the port you select is not inuse by splunkweb or splunkd.

6. Click Save. You must restart Splunk to complete the process.

38

Set up receiving with Splunk CLI

To access the CLI, first navigate to $SPLUNK_HOME/bin/. This is unnecessary ifyou have added Splunk to your path.

To enable receiving, enter:

./splunk enable listen <port> -auth <username>:<password>

For <port>, substitute the port you want the receiver to listen on (the receivingport). For example, if you enter "9997," the receiver will receive data on port9997. By convention, receivers listen on port 9997, but you can specify anyunused port. You can use a tool like netstat to determine what ports areavailable on your system. Make sure the port you select is not in use bysplunkweb or splunkd.

To disable receiving, enter:

./splunk disable listen -port <port> -auth <username>:<password>

Set up receiving with the configuration file

You can enable receiving on your Splunk instance by configuring inputs.conf in$SPLUNK_HOME/etc/system/local.

For most purposes, you just need to add a [splunktcp] stanza that specifies thereceiving port. In this example, the receiving port is 9997:

[splunktcp://9997]

For further details, refer to the inputs.conf spec file.

To configure a universal forwarder as an intermediate forwarder (a forwarder thatfunctions also as a receiver), use this method.

Searching data received from a forwarder running on adifferent operating system

In most cases, a Splunk instance receiving data from a forwarder on a differentOS will need to install the app for that OS. However, there are numeroussubtleties that affect this; read on for the details.

39

Forwarding and indexing are OS-independent operations. Splunk supports anycombination of forwarders and receivers, as long as each is running on a certifiedOS. For example, a Linux receiver can index data from a Windows universalforwarder.

Once data has been forwarded and indexed, the next step is to search orperform other knowledge-based activities on the data. At this point, the Splunkinstance performing such activities might need information about the OS whosedata it is examining. You typically handle this by installing the app specific to thatOS. For example, if you want a Linux Splunk instance to search OS-specific dataforwarded from Windows, you will ordinarily want to install the Windows app onthe Linux instance.

If the data you're interested in is not OS-specific, such as web logs, then you donot need to install the Splunk OS app.

In addition, if the receiver is only indexing the data, and an external search headis performing the actual searches, you do not need to install the OS app on thereceiver, but you might need to install it on the search head. As an alternative,you can use a search head running the OS. For example, to search dataforwarded from Windows to a Linux receiver, you can use a Windows searchhead pointing to the Linux indexer as a remote search peer. For more informationon search heads, see "Set up distributed search".

Important: After you have downloaded the relevant OS app, remove itsinputs.conf file before enabling the app, to ensure that its default inputs are notadded to your indexer. For the Windows app, the location is:%SPLUNK_HOME%\etc\apps\windows\default\inputs.conf.

In summary, you only need to install the app for the forwarder's OS on thereceiver (or search head) if it will be performing searches on the forwarded OSdata.

Troubleshoot forwarder to receiver connectivity

Confusing the receiver's receiver and management ports

As part of setting up a forwarder, you specify the receiver's hostname/IP_addressand port. The forwarder uses these to send data to the receiver. Be sure tospecify the port that was designated as the receiving port at the time the receiverwas configured. If you mistakenly specify the receiver's management port, thereceiver will generate an error similar to this:

40

splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error =error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocolsplunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - ACCEPT_RESULT=-1VERIFY_RESULT=0splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error for fdfrom HOST:localhost.localdomain, IP:127.0.0.1, PORT:53075splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error =error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocolsplunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - ACCEPT_RESULT=-1VERIFY_RESULT=0splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error for fdfrom HOST:localhost.localdomain, IP:127.0.0.1, PORT:53076splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error =error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocolsplunkd.log:03-01-2010 13:35:28.654 ERROR TcpInputFd - ACCEPT_RESULT=-1VERIFY_RESULT=0splunkd.log:03-01-2010 13:35:28.654 ERROR TcpInputFd - SSL Error for fdfrom HOST:localhost.localdomain, IP:127.0.0.1, PORT:53077splunkd.log:03-01-2010 13:35:28.654 ERROR TcpInputFd - SSL Error =error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocolsplunkd.log:03-01-2010 13:35:28.654 ERROR TcpInputFd - ACCEPT_RESULT=-1VERIFY_RESULT=0

Closed receiver socket

If a receiving indexer's queues become full, it will close the receiver socket, toprevent additional forwarders from connecting to it. If a forwarder withload-balancing enabled can no longer forward to that receiver, it will send its datato another indexer on its list. If the fowarder does not employ load-balancing, itwill hold the data until the problem is resolved.

The receiver socket will reopen automatically when the queue gets unclogged.

Typically, a receiver gets behind on the dataflow because it can no longer writedata due to a full disk or because it is itself attempting to forward data to anotherSplunk instance that is not accepting data.

The following warning message will appear in splunkd.log if the socket getsblocked:

Stopping all listening ports. Queues blocked for more than N seconds.

This message will appear when the socket reopens:

Started listening on tcp ports. Queues unblocked.

41

Answers

Have questions? Visit Splunk Answers and see what questions and answers theSplunk community has around configuring forwarding.

Configure forwarders with outputs.conf

The outputs.conf file defines how forwarders send data to receivers. You canspecify some output configurations at installation time (Windows universalforwarders only) or through Splunk Web (heavy/light forwarders only) or the CLI,but most advanced configuration settings require that you directly editoutputs.conf. The topics describing various topologies, such as load balancingand data routing, provide detailed examples on configuring outputs.conf tosupport those topologies.

Important: Although outputs.conf is a critical file for configuring forwarders, itspecifically addresses the outputs from the forwarder. To specify the inputs to aforwarder, you must separately configure the inputs, as you would for any Splunkinstance. For details on configuring inputs, see "Add data and configure inputs" inthe Getting Data In manual.

Types of outputs.conf files

A single forwarder can have multiple outputs.conf files (for instance, one locatedin an apps directory and another in /system/local). No matter how manyoutputs.conf files the forwarder has and where they reside, the forwardercombines all their settings, using the rules of location precedence, as describedin "Configuration file precedence". Your installation will contain both default andcustom outputs.conf files.

Default versions

Splunk ships with these default versions of outputs.conf:

On the universal forwarder: The universal forwarder has two defaultoutputs.conf files, one in $SPLUNK_HOME/etc/system/default and theother in $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default. Thedefault version in the SplunkUniversalForwarder app has precedenceover the version under /etc/system.

•

On heavy and light forwarders: These have a single defaultoutputs.conf file, located in $SPLUNK_HOME/etc/system/default.

•

42

Important: Do not touch default versions of any configuration files, for reasonsexplained in "About configuration files".

Custom versions

When you configure forwarding behavior, those changes get saved in customversions of outputs.conf. There are several ways you can specify forwardingbehavior:

While installing the forwarder (Windows universal forwarder only)• By running CLI commands• By using Splunk Manager (heavy/light forwarders only)• By directly editing an outputs.conf file•

Splunk automatically creates or edits custom versions of outputs.conf inresponse to the first three methods. The locations of those versions vary,depending on the type of forwarder and other factors:

The universal forwarder. If you use the CLI to make changes touniversal forwarder output behavior, it creates or edits a copy ofoutputs.conf in $SPLUNK_HOME/etc/system/local. However, the Windowsinstallation process writes configuration changes to an outputs.conf filelocated in the MSICreated app. For more information on configuring theuniversal forwarder, look here.

•

Heavy and light forwarders. When you enable a heavy/light forwarderthrough Splunk Web or the CLI, Splunk creates an outputs.conf file in thedirectory of the currently running app. For example, if you're working in thesearch app, Splunk places the file in$SPLUNK_HOME/etc/apps/search/local/. You can then edit it there.

•

In addition to any outputs.conf files that you create and edit indirectly (forexample, through the CLI), you can also create or edit an outputs.conf filedirectly. It is recommended that you work with just a single copy of the file, whichyou place in $SPLUNK_HOME/etc/system/local/. (If a copy of the file alreadyexists in that directory, because of configuration changes made through the CLI,just edit that copy.) For purposes of distribution and management simplicity, youcan combine settings from all non-default versions into a single customoutputs.conf file.

After making changes to outputs.conf, you must restart the forwarder for thechanges to take effect.

43

For detailed information on outputs.conf, look here for the spec and examples.

Configuration levels

There are two types of output processors: tcpout and syslog. You can configurethem at three levels of stanzas:

Global. At the global level, you specify any attributes that you want toapply globally, as well as certain attributes only configurable at thesystem-wide level for the output processor. This stanza is optional.

•

Target group. A target group defines settings for one or more receivingindexers. There can be multiple target groups per output processor. Mostconfiguration settings can be specified at the target group level.

•

Single server. You can specify configuration values for single servers(receivers) within a target group. This stanza type is optional.

•

Configurations at the more specific level take precedence. For example, if youspecify compressed=true for a target group, the forwarder will send the servers inthat target group compressed data, even if compressed is set to "false" for theglobal level.

Note: This discussion focuses on the tcpout processor, which uses the [tcpout]header. For the syslog output processor, see "Forward data to third-partysystems" for details.

Global stanza

Here you set any attributes that you want to apply globally. This stanza is notrequired. However, there are several attributes that you can set only at the globallevel, including defaultGroup and indexAndForward.

The global stanza for the tcpout procesor is specified with the [tcpout] header.

Here's an example of a global tcpout stanza:

[tcpout]defaultGroup=indexer1indexAndForward=true

This global stanza includes two attribute/value pairs:

defaultGroup=indexer1 This tells the forwarder to send all data to the•

44

"indexer1" target group. See "Default target groups" for more information.indexAndForward=true This tells the forwarder to index the data locally,as well as forward the data to receiving indexers in the target groups. If setto "false" (the default), the forwarder just forwards data but does not indexit. This attribute is only available for heavy forwarders; universal and lightforwarders cannot index data.

•

Default target groups

To set default groups for automatic forwarding, include the defaultGroup attributeat the global level, in your [tcpout] stanza:

[tcpout]defaultGroup= <target_group1>, <target_group2>, ...

The defaultGroup specifies one or more target groups, defined later intcpout:<target_group> stanzas. The forwarder will send all events to thespecified groups.

If you do not want to forward data automatically, don't set the defaultGroupattribute. (Prior to 4.2, you were required to set the defaultGroup to some value.This is no longer necessary.)

For some examples of using the defaultGroup attribute, see "Route and filterdata".

Target group stanza

The target group identifies a set of receivers. It also specifies how the forwardersends data to those receivers. You can define multiple target groups.

Here's the basic pattern for the target group stanza:

[tcpout:<target_group>]server=<receiving_server1>, <receiving_server2>, ...<attribute1> = <val1><attribute2> = <val2>...

To specify a receiving server in a target group, use the format<ipaddress_or_servername>:<port>, where <port> is the receiving server'sreceiving port. For example, myhost.Splunk.com:9997. You can specify multiplereceivers and the forwarder will load balance among them.

45

Note: Starting with Splunk version 4.3, you can use an IPv6 address whenspecifying the receiving indexer. For more information, see "Configure Splunk forIPv6" in the Admin manual.

See "Define typical deployment topologies", later in this topic, for information onhow to use the target group stanza to define several deployment topologies.

Single-server stanza

You can define a specific configuration for an individual receiving indexer.However, the receiver must also be a member of a target group.

When you define an attribute at the single-server level, it takes precedence overany definition at the target group or global level.

Here is the syntax for defining a single-server stanza:

[tcpout-server://<ipaddress_or_servername>:<port>]<attribute1> = <val1><attribute2> = <val2>...

Example

The following outputs.conf example contains three stanzas for sending tcpout toSplunk receivers:

Global settings. In this example, there is one setting, to specify adefaultGroup.

•

Settings for a single target group consisting of two receivers. Here, we arespecifying a load-balanced target group consisting of two receivers.

•

Settings for one receiver within the target group. In this stanza, you canspecify any settings specific to the mysplunk_indexer1 receiver.

•

[tcpout]defaultGroup=my_indexers

[tcpout:my_indexers]server=mysplunk_indexer1:9997, mysplunk_indexer2:9996

[tcpout-server://mysplunk_indexer1:9997]

46

Define typical deployment topologies

This section shows how you can configure a forwarder to support several typicaldeployment topologies. See the other topics in the "Forward data" section of thisbook for information on configuring forwarders for other topologies.

Load balancing

To perform load balancing, specify one target group with multiple receivers. Inthis example, the target group consists of three receivers:

[tcpout:my_LB_indexers]server=10.10.10.1:9997,10.10.10.2:9996,10.10.10.3:9995

The forwarder will load balance between the three receivers listed. If one receivergoes down, the forwarder automatically switches to the next one available.

Data cloning

To perform data cloning, specify multiple target groups, each in its own stanza.In data cloning, the forwarder sends copies of all its events to the receivers in twoor more target groups. Data cloning usually results in similar, but not necessarilyexact, copies of data on the receiving indexers. Here's an example of how youset up data cloning:

[tcpout]defaultGroup=indexer1,indexer2

[tcpout:indexer1]server=10.1.1.197:9997

[tcpout:indexer2]server=10.1.1.200:9997

The forwarder will send duplicate data streams to the servers specified in boththe indexer1 and indexer2 target groups.

Data cloning with load balancing

You can combine load balancing with data cloning. For example:

[tcpout]

47

defaultGroup=cloned_group1,cloned_group2

[tcpout:cloned_group1]server=10.10.10.1:9997, 10.10.10.2:9997, 10.10.10.3:9997

[tcpout:cloned_group2]server=10.1.1.197:9997, 10.1.1.198:9997, 10.1.1.199:9997,10.1.1.200:9997

The forwarder will send full data streams to both cloned_group1 andcloned_group2. The data will be load-balanced within each group, rotatingamong receivers every 30 seconds (the default frequency).

Note: For syslog and other output types, you must explicitly specify routing asdescribed here: "Route and filter data".

Commonly used attributes

The outputs.conf file provides a large number of configuration options that offerconsiderable control and flexibility in forwarding. Of the attributes available,several are of particular interest:

Attribute Default Whereconfigured Value

defaultGroup n/a global stanza

A comma-separated list of one or moretarget groups. Forwarder sends all events toall specified target groups. Don't set thisattribute if you don't want eventsautomatically forwarded to a target group.

indexAndForward false global stanza

If set to "true", the forwarder will index alldata locally, in addition to forwarding thedata to a receiving indexer.

Important: This attribute is onlyavailable for heavy forwarders. Auniversal forwarder cannot indexlocally.

server n/a target groupstanza

Required. Specifies the server(s) that willfunction as receivers for the forwarder. Thismust be set to a value using the format<ipaddress_or_servername>:<port>,where <port> is the receivingserver's receiving port.

48

Note: Starting with Splunk version4.3, you can use an IPv6 addresswhen specifying the receivingindexer. For more information, see"Configure Splunk for IPv6" in theAdmin manual.

disabled false any stanzalevel

Specifies whether the stanza is disabled. Ifset to "true", it is equivalent to the stanzanot being there.

sendCookedData true global or targetgroup stanza

Specifies whether data is cooked beforeforwarding.

compressed false global or targetgroup stanza

Specifies whether the forwarder sendscompressed data.

ssl.... n/a any stanzalevel

Set of attributes for configuring SSL. See"About securing data from forwarders" in theSecuring Splunk manual for information onhow to use these attributes.

useACK false global or targetgroup stanza

Specifies whether the forwarder waits forindexer acknowledgment confirming that thedata has been written to the file system.See "Protect against loss of in-flightdata".

dnsResolutionInterval 300 global or targetgroup stanza

Specifies base time interval in seconds atwhich indexer DNS names will be resolvedto IP address. See "DNS resolutioninterval".

The outputs.conf.spec file, which you can find here, along with severalexamples, provides details for these and all other configuration options. Inaddition, most of these settings are discussed in topics dealing with specificforwarding scenarios.

Note: In 4.2, the persistent queue capability was much improved. It is now afeature of data inputs and is therefore configured in inputs.conf. It is not related inany way to the previous, deprecated persistent queue capability, which wasconfigured through outputs.conf. See "Use persistent queues to prevent dataloss" for details.

DNS resolution interval

The dnsResolutionInterval attribute specifies the base time interval (inseconds) at which receiver DNS names will be resolved to IP addresses. Thisvalue is used to compute the run-time interval as follows:

49

run-time interval = dnsResolutionInterval + (number of receivers inserver attribute - 1) * 30

The run-time interval is extended by 30 seconds for each additional receiverspecified in the server attribute; that is, for each additional receiver across whichthe forwarder is load balancing. The dnsResolutionInterval attribute defaults to300 seconds.

For example, if you leave the attribute at the default setting of 300 seconds andthe forwarder is load-balancing across 20 indexers, DNS resolution will occurevery 14 minutes:

(300 + ((20 - 1) * 30)) = 870 seconds = 14 minutes

If you change dnsResolutionInterval to 600 seconds, and keep the number ofload-balanced indexers at 20, DNS resolution will occur every 19.5 minutes:

(600 + ((20 - 1) * 30)) = 1170 seconds = 19.5 minutes

Protect against loss of in-flight data

To guard against loss of data when forwarding to an indexer, you can useSplunk's indexer acknowledgment capability. With indexer acknowledgment,the forwarder will resend any data not acknowledged as "received" by theindexer.

This feature is disabled by default, because it can affect performance. Youenable it in outputs.conf, as described later.

Indexer acknowledgment is available for all varieties of forwarders: universal,light, or heavy.

Note: Both forwarders and indexers must be at version 4.2 or higher foracknowledgment to function. Otherwise, the transmission between forwarder andindexer will proceed without acknowledgment.

Indexer acknowledgment and clusters

When using forwarders to send data to peer nodes in a cluster, you must enable

50

indexer acknowledgment. To learn more about forwarders and clusters, read"Use forwarders to get your data" in the Managing Indexers and Clusters Manual.

How indexer acknowledgment works when everything goeswell

The forwarder sends data continuously to the indexer, in blocks of approximately64kB. The forwarder maintains a copy of each block in memory, in its wait queue,until it gets an acknowledgment from the indexer. While waiting, it continues tosend more data blocks.

If all goes well, the indexer:

1. Receives the block of data.

2. Parses the data.

3. Writes the data to the file system as events (raw data and index data).

4. Sends an acknowledgment to the forwarder.

The acknowledgment tells the forwarder that the indexer received the data andsuccessfully wrote it to the file system. Upon receiving the acknowledgment, theforwarder releases the block from memory.

If the wait queue is of sufficient size, it doesn't fill up while waiting foracknowledgments to arrive. But see this section for possible issues and ways toaddress them, including how to increase the wait queue size.

How indexer acknowledgment works when there's a failure

When there's a failure in the round-trip process, the forwarder does not receivean acknowledgment. It will then attempt to resend the block of data.

Why no acknowledgment?

These are the reasons that a forwarder might not receive acknowledgment:

Indexer goes down after receiving the data -- for instance, due to machinefailure.

•

Indexer is unable to write to the file system -- for instance, because thedisk is full.

•

51

Network goes down while acknowledgment is en route to the forwarder.•

How the forwarder deals with failure

After sending a data block, the forwarder maintains a copy of the data in its waitqueue until it receives an acknowledgment. In the meantime, it continues to sendadditional blocks as usual. If the forwarder doesn't get acknowledgment for ablock within 300 seconds (by default), it closes the connection. You can changethe wait time by setting the readTimeout attribute in outputs.conf.

If the forwarder is set up for auto load balancing, it then opens a connection tothe next indexer in the group (if one is available) and sends the data to it. If theforwarder is not set up for auto load balancing, it attempts to open a connectionto the same indexer as before and resend the data.

The forwarder maintains the data block in the wait queue until acknowledgmentis received. Once the wait queue fills up, the forwarder stops sending additionalblocks until it receives an acknowledgment for one of the blocks, at which point itcan free up space in the queue.

Other reasons the forwarder might close a connection

There are actually three conditions that can cause the forwarder to close thenetwork connection:

Read timeout. The forwarder doesn't receive acknowledgment within 300(default) seconds. This is the condition described above.

•

Write timeout. The forwarder is not able to finish a network write within300 (default) seconds. The value is configurable in outputs.conf bysetting writeTimeout.

•

Read/write failure. Typical causes include the indexer's machine crashingor the network going down.

•

In all these cases, the forwarder will then attempt to open a connection to thenext indexer in the load-balanced group, or to the same indexer again ifload-balancing is not enabled.

The possibility of duplicates

It's possible for the indexer to index the same data block twice. This can happenif there's a network problem that prevents an acknowledgment from reaching theforwarder. For instance, assume the indexer receives a data block, parses it, andwrites it to the file system. It then generates the acknowledgment. However, on

52

the round-trip to the forwarder, the network goes down, so the forwarder neverreceives the acknowledgment. When the network comes back up, the forwarderthen resends the data block, which the indexer will parse and write as if it werenew data.

To deal with such a possibility, every time the forwarder resends a data block, itwrites an event to its splunkd.log noting that it's a possible duplicate. The adminis responsible for using the log information to track down the duplicate data onthe indexer.

Here's an example of a duplicate warning:

10-18-2010 17:32:36.941 WARN TcpOutputProc - Possible duplication ofevents withchannel=source::/home/jkerai/splunk/current-install/etc/apps/sample_app/logs/maillog.1|host::MrT|sendmail|, streamId=5941229245963076846,offset=131072subOffset=219 on host=10.1.42.2:9992

Enable indexer acknowledgment

You enable indexer acknowledgment solely on the forwarder. You do not set anyattribute on the indexer side; it will send acknowledgments if the forwarder tells itto. (But remember, both the forwarder and the indexer must be at version 4.2 orgreater.)

To enable indexer acknowledgment, set the useACK attribute to true in theforwarder's outputs.conf:

[tcpout:<target_group>]server=<server1>, <server2>, ...useACK=true...

A value of useACK=true enables indexer acknowledgment.

By default, this feature is disabled: useACK=false

Note: You can set useACK either globally or by target group, at the [tcpout] or[tcpout:<target_group>] stanza levels. You cannot set it for individual serversat the [tcpout-server: ...] stanza level.

53

Indexer acknowledgment influence on forwarded datathroughput

Indexer acknowledgment can limit and/or reduce forwarder throughput in somescenarios. Here, we describe how this can occur and steps that you should taketo avoid this.

If you have enabled indexer acknowledgment on the forwarder through useACKand the receiving indexer is at version 4.2+, the forwarder will use a wait queueto manage the acknowledgment process. Otherwise, it won't have a wait queue.This section describes how to manage the wait queue for performance.

Because the forwarder sends data blocks continuously and does not wait foracknowledgment before sending the next block, its wait queue will typicallymaintain many blocks, each waiting for its acknowledgment. The forwarder willcontinue to send blocks until its wait queue is full, at which point it will stopforwarding. The forwarder then waits until it receives an acknowledgment, whichallows it to release a block from its queue and thus resume forwarding.

A wait queue can fill up when something is wrong with the network or indexer;however, it can also fill up even though the indexer is functioning normally. Thisis because the indexer only sends the acknowledgment after it has written thedata to the file system. Any delay in writing to the file system will slow the pace ofacknowledgment, leading to a full wait queue.

There are a few reasons that a normal functioning indexer might delay writingdata to the file system (and so delay its sending of acknowledgments):

The indexer is very busy. For example, at the time the data arrives, theindexer might be dealing with multiple search requests or with datacoming from a large number of forwarders.

•

The indexer is receiving too little data. For efficiency, an indexer onlywrites to the file system periodically -- either when a write queue fills up orafter a timeout of a few seconds. If a write queue is slow to fill up, theindexer will wait until the timeout to write. If data is coming from only a fewforwarders, the indexer can end up in the timeout condition, even if eachof those forwarders is sending a normal quantity of data. Since writequeues exist on a per hot bucket basis, the condition occurs when someparticular bucket is getting a small amount of data. Usually this means thata particular index is getting a small amount of data.

•

To ensure that throughput does not degrade because the forwarder is waiting onthe indexer for acknowledgment, you might need to use a larger wait queue size,

54

ensuring it has sufficient space to maintain all blocks in memory while waiting foracknowledgments to arrive. You'll need to experiment with the queue size that'sright for your forwarder's specific environment. On the other hand, if you havemany forwarders feeding a single indexer, and a moderate number of datasources per forwarder, you may be able to conserve a few megabytes of memoryby using a smaller size.

Note: You cannot configure the size of the wait queue directly. Its size is alwaysrelative to the size of the in-memory output queue, as described below.

Configure the wait queue size

Important: To optimize the wait queue size for indexer acknowledgment, Splunkrecommends that you increase the maxQueueSize attribute.

The maximum wait queue size is 3x the size of the in-memory output queue,which you set with the maxQueueSize attribute in outputs.conf:

maxQueueSize = [<integer>|<integer>[KB|MB|GB]]

For example, if you set maxQueueSize to 7MB, the maximum wait queue size willbe 21MB.

Note the following:

This attribute sets the maximum size of the forwarder's in-memory (RAM)output queue. It also determines the maximum size of the wait queue,which is 3x the setting for the output queue.

•

If specified as a lone integer (for example, maxQueueSize=100), itdetermines the maximum number of queued events (for parsed data) orblocks of data (for unparsed data). A block of data is approximately 64KB.For forwarders sending unparsed data (mainly universal forwarders),maxQueueSize is the maximum number of data blocks. For heavyforwarders sending parsed data, maxQueueSize is the maximum number ofevents. Since events are typically much shorter than data blocks, thememory consumed by the output and wait queues on a parsing forwarderwill likely be much smaller than on a non-parsing forwarder, if you use thisversion of the setting.

•

If specified as an integer followed by KB, MB, or GB (for example,maxQueueSize=100MB), it determines the maximum RAM allocated to theoutput queue and, indirectly, to the wait queue. If configured asmaxQueueSize=100MB, the maximum size of the output queue will be

•

55

100MB and the maximum size of the wait queue, if any, will be 300MB.maxQueueSize defaults to 500KB. The default wait queue size is 3x thatamount: 1500KB.

•

Although the wait queue and the output queues are configured by the sameattribute, they are separate queues.

Important: To ensure that the forwarder does not get blocked while waiting foracknowledgment from an indexer, you should set the maxQueueSize to 7MB:

[tcpout]maxQueueSize = 7MB

Note the following points regarding this recommendation:

This assumes that no thruput limit has been set in the forwarder'slimits.conf file.

•

This configuration will cause memory consumption to go up by about28MB.

•

Important: If you're enabling indexer acknowledgment, be careful to take intoaccount your system's available memory when setting maxQueueSize. You'll needto accommodate 4x the maxQueueSize setting (1x for the output queue + 3x forthe wait queue).

When the receiver is a forwarder, not an indexer

You can also use indexer acknowledgment when the receiving instance is anintermediate forwarder, instead of an indexer.

Assume you have an originating forwarder that sends data to an intermediateforwarder, which in turn forwards that data to an indexer. There are two mainpossibilities to consider:

The originating forwarder and the intermediate forwarder both haveacknowledgment enabled. In this case, the intermediate forwarder waitsuntil it receives acknowledgment from the indexer and then sendsacknowledgment back to the originating forwarder.

•

The originating forwarder has acknowledgment enabled; theintermediate forwarder does not. In this case, the intermediateforwarder sends acknowledgment back to the originating forwarder as

•

56

soon as it sends the data on to the indexer. It relies on TCP to safelydeliver the data to the indexer. Because it doesn't itself have useACKenabled, the intermediate forwarder cannot verify delivery of the data tothe indexer. This use case has limited value and is not recommended. Ifyou use indexer acknowledgment, you should generally enable it on allforwarding tiers. That is the only way to ensure that data gets delivered allthe way from the originating forwarder to the indexer.

57

Use the forwarder to create deploymenttopologies

Consolidate data from multiple machines

One of the most common forwarding use cases is to consolidate data originatingacross numerous machines. Forwarders located on the machines forward thedata to a central Splunk indexer. With their small footprint, universal forwardersordinarily have little impact on their machines' performance. This diagramillustrates a common scenario, where universal forwarders residing on machinesrunning diverse operating systems send data to a single Splunk instance, whichindexes and provides search capabilities across all the data:

The diagram illustrates a small deployment. In practice, the number of universalforwarders in a data consolidation use case could number upwards into thethousands.

This type of use case is simple to configure:

1. Determine what data, originating from which machines, you need to access.

2. Install a Splunk instance, typically on its own server. This instance will functionas the receiver. All indexing and searching will occur on it.

3. Enable the instance as a receiver through Splunk Web or the CLI. Using theCLI, enter this command from $SPLUNK_HOME/bin/:

./splunk enable listen <port> -auth <username>:<password>

For <port>, substitute the port you want the receiver to listen on. This also

58

known as the "receiver port".

4. If any of the universal forwarders will be running on a different operatingsystem from the receiver, install the app for the forwarder's OS on the receiver.For example, assume the receiver in the diagram above is running on a Linuxbox. In that case, you'll need to install the Windows app on the receiver. Youmight need to install the *nix app, as well. -- However, since the receiver is onLinux, you probably have already installed that app. Details and provisosregarding this can be found here.

After you have downloaded the relevant app, remove its inputs.conf file beforeenabling it, to ensure that its default inputs are not added to your indexer. For theWindows app, the location is:$SPLUNK_HOME/etc/apps/windows/default/inputs.conf.

5. Install universal forwarders on each machine that will be generating data.These will forward the data to the receiver.

6. Set up inputs for each forwarder. See "What Splunk can index".

7. Configure each forwarder to forward data to the receiver. For Windowsforwarders, you can do this at installation time, as described here. For *nixforwarders, you must do this through the CLI:

./splunk add forward-server <host>:<port> -auth <username>:<password>

For <host>:<port>, substitute the host and receiver port number of the receiver.For example, splunk_indexer.acme.com:9995.

Alternatively, if you have many forwarders, you can use an outputs.conf file tospecify the receiver. For example:

[tcpout:my_indexers]server= splunk_indexer.acme.com:9995

You can create this file once, then distribute copies of it to each forwarder.

Set up load balancing

With load balancing, a Splunk forwarder distributes data across several receiving

59

Splunk instances. Each receiver gets a portion of the total data, and together thereceivers hold all the data. To access the full set of forwarded data, you need toset up distributed searching across all the receivers. For information ondistributed search, see "About distributed search" in this manual.

Load balancing enables horizontal scaling for improved performance. In addition,its automatic switchover capability ensures resiliency in the face of machineoutages. If a machine goes down, the forwarder simply begins sending data tothe next available receiver.

Load balancing can also be of use when getting data from network devices likerouters. To handle syslog and other data generated across port 514, a singleheavy forwarder can monitor port 514 and distribute the incoming data acrossseveral Splunk indexers.

Note: When implementing load balancing between forwarders and receivers, youmust use the forwarder's inherent capability. Do not use an external loadbalancer. The use of external load balancers between forwarders and receiverswill not work properly.

How load balancing works

Splunk forwarders perform "automatic load balancing". The forwarder routes datato different indexers based on a specified time interval. For example, assume youhave a load-balanced group consisting of three indexers: A, B, and C. At somespecified interval, such as every 30 seconds, the forwarder switches the datastream to another indexer in the group, selected at random. So, the forwardermight switch from indexer B to indexer A to indexer C, and so on. If one indexeris down, the forwarder immediately switches to another.

To expand on this a bit, there is a data stream for each of the inputs that theforwarder is configured to monitor. The forwarder determines if it is safe for adata stream to switch to another indexer. Then, at the specified interval, itswitches the data stream to the newly selected indexer. If it cannot switch thedata stream to the new indexer safely, it keeps the connection to the previousindexer open and continues to send the data stream until it has been safely sent.

Important: Universal forwarders are not able to switch indexers when monitoringTCP network streams of data (including Syslog) unless an EOF is reached or anindexer goes down, at which point the forwarder will switch to the next indexer inthe list. Because the universal forwarder does not parse the data and identifyevent boundaries before forwarding the data to the indexer (unlike a heavyforwarder), it has no way of knowing when it's safe to switch to the next indexer

60

unless it receives an EOF.

Note: Round-robin load balancing, which was previously available as analternative to automatic load balancing, was deprecated in Splunk version 4.2.

This diagram shows a distributed search scenario, in which three forwarders areperforming load balancing across three receivers:

Targets for load balancing

When configuring the set of target receivers, you can employ either DNS or staticlists.

DNS lists provide greater flexibility and simplified scale-up, particularly for largedeployments. Through DNS, you can change the set of receivers without needingto re-edit each forwarder's outputs.conf file.

The main advantage of a static list is that it allows you to specify a different portfor each receiver. This is useful if you need to perform load balancing acrossmultiple receivers running on a single host. Each receiver can listen on aseparate port.

Static list target

To use a static list for the target, you simply specify each of the receivers in thetarget group's [tcpout] stanza in the forwarder's outputs.conf file. In thisexample, the target group consists of three receivers, specified by IP addressand receiver port number:

[tcpout: my_LB_indexers]server=10.10.10.1:9997,10.10.10.2:9996,10.10.10.3:9995

61

The universal forwarder will load balance between the three receivers listed. Ifone receiver goes down, the forwarder automatically switches to another one onthe list.

DNS list target

To use a DNS list, edit your forwarder's outputs.conf file to specify a single hostin the target group's [tcpout] stanza. For example:

[tcpout:my_LB_indexers]server=splunkreceiver.mycompany.com:9997

In your DNS server, create a DNS A record for each host's IP address,referencing the server name you specified in outputs.conf. For example:

splunkreceiver.mycompany.com A 10.10.10.1splunkreceiver.mycompany.com A 10.10.10.2splunkreceiver.mycompany.com A 10.10.10.3

The forwarder will use the DNS list to load balance, sending data in intervals,switching among the receivers specified. If a receiver is not available, theforwarder skips it and sends data to another one on the list.

If you have a topology with many forwarders, the DNS list method allows you toupdate the set of receivers by making changes in just a single location, withouttouching the forwarders' outputs.conf files.

Configure load balancing for horizontal scaling

To configure load balancing, first determine your needs, particularly yourhorizontal scaling and failover requirements. Then develop a topology based onthose needs, possibly including multiple forwarders, as well as receivers and asearch head to search across the receivers.

Assuming the topology of three universal forwarders and three receiversillustrated by the diagram at the start of this topic, set up load balancing withthese steps:

1. Install and enable a set of three Splunk instances as receivers. This exampleuses a DNS list to designate the receivers, so they must all listen on the sameport. For example, if the port is 9997, enable each receiver by going to its$SPLUNK_HOME/bin/ location and using this CLI command:

62

./splunk enable listen 9997 -auth <username>:<password>

2. Install the set of universal forwarders, as described here.

3. Set up a DNS list with an A record for each receiver's IP address:

splunkreceiver.mycompany.com A 10.10.10.1splunkreceiver.mycompany.com A 10.10.10.2splunkreceiver.mycompany.com A 10.10.10.3

4. Create a single outputs.conf file for use by all the forwarders. This onespecifies the DNS server name used in the DNS list and the port the receiversare listening on:

[tcpout]defaultGroup=my_LB_indexers

[tcpout:my_LB_indexers]disabled=falseautoLBFrequency=40server=splunkreceiver.mycompany.com:9997

This outputs.conf file uses the autoLBFrequency attribute to set a load-balancefrequency of 40 seconds. Every 40 seconds, the forwarders will switch to anotherreceiver. The default frequency, which rarely needs changing, is 30 seconds.

5. Distribute the outputs.conf file to all the forwarders. You can use thedeployment server to handle the distribution.

Specify load balancing from the CLI

You can also use the CLI to specify load balancing. You do this when you startforwarding activity to a set of receivers, using this syntax:

./splunk add forward-server <host>:<port> -method autobalance

where <host>:<port> is the host and receiver port of the receiver.

This example creates a load-balanced group of four receivers:

./splunk add forward-server indexer1:9997 -method autobalance


63



Route and filter data

Forwarders can filter and route data to specific receivers based on criteria suchas source, source type, or patterns in the events themselves. For example, aforwarder can send all data from one group of hosts to one Splunk server and alldata from a second group of hosts to a second Splunk server. Heavy forwarderscan also look inside the events and filter or route accordingly. For example, youcould use a heavy forwarder to inspect WMI event codes to filter or routeWindows events. This topic describes a number of typical routing scenarios.

Besides routing to receivers, forwarders can also filter and route data to specificqueues or discard the data altogether by routing to the null queue.

Important: Only heavy forwarders can route or filter data at the event level.Universal forwarders and light forwarders do not have the ability to inspectindividual events, but they can still forward data based on a data stream's host,source, or source type. They can also route based on the data's input stanza, asdescribed below, in the subtopic, "Route inputs to specific indexers based on thedata's input".

Here's a simple illustration of a forwarder routing data to three Splunk receivers:

This topic describes how to filter and route event data to Splunk instances. See"Forward data to third-party systems" in this manual for information on routing tonon-Splunk systems.

This topic also describes how to perform selective indexing and forwarding, inwhich you index some data locally on a heavy forwarder and forward thenon-indexed data to one or more sepearate indexers. See "Perform selectiveindexing and forwarding" later in this topic for details.

64

Configure routing

This is the basic pattern for defining most routing scenarios (using a heavyforwarder):

1. Determine what criteria to use for routing. How will you identify categories ofevents, and where will you route them?

2. Edit props.conf to add a TRANSFORMS-routing attribute to determine routingbased on event metadata:

[<spec>]TRANSFORMS-routing=<transforms_stanza_name>

<spec> can be:

<sourcetype>, the source type of an event• host::<host>, where <host> is the host for an event• source::<source>, where <source> is the source for an event•

Use the <transforms_stanza_name> specified here when creating an entry intransforms.conf (below).

Examples later in this topic show how to use this syntax.

3. Edit transforms.conf to specify target groups and to set additional criteria forrouting based on event patterns:

[<transforms_stanza_name>]REGEX=<routing_criteria>DEST_KEY=_TCP_ROUTINGFORMAT=<target_group>,<target_group>,....

Note:

<transforms_stanza_name> must match the name you defined inprops.conf.

•

In <routing_criteria>, enter the regex rules that determine which eventsget routed. This line is required. Use "REGEX = ." if you don't needadditional filtering beyond the metadata specified in props.conf.

•

DEST_KEY should be set to _TCP_ROUTING to send events via TCP. It canalso be set to _SYSLOG_ROUTING or _HTTPOUT_ROUTING for other output

•

65

processors.Set FORMAT to a <target_group> that matches a target group name youdefined in outputs.conf. A comma separated list will clone events tomultiple target groups.

•

Examples later in this topic show how to use this syntax.

4. Edit outputs.conf to define the target group(s) for the routed data:

[tcpout:<target_group>]server=<ip>:<port>

Note:

Set <target_group> to match the name you specified in transforms.conf.• Set the IP address and port to match the receiving server.•

The use cases described in this topic generally follow this pattern.

Filter and route event data to target groups

In this example, a heavy forwarder filters three types of events, routing them todifferent target groups. The forwarder filters and routes according to thesecriteria:

Events with a source type of "syslog" to a load-balanced target group• Events containing the word "error" to a second target group• All other events to a default target group•

Here's how you do it:

1. Edit props.conf in $SPLUNK_HOME/etc/system/local to set twoTRANSFORMS-routing attributes — one for syslog data and a default for allother data:

[default]TRANSFORMS-routing=errorRouting

[syslog]TRANSFORMS-routing=syslogRouting

2. Edit transforms.conf to set the routing rules for each routing transform:

66

[errorRouting]REGEX=errorDEST_KEY=_TCP_ROUTINGFORMAT=errorGroup

[syslogRouting]REGEX=.DEST_KEY=_TCP_ROUTINGFORMAT=syslogGroup

Note: In this example, if a syslog event contains the word "error", it will route tosyslogGroup, not errorGroup. This is due to the settings previously specified inprops.conf. Those settings dictated that all syslog events be filtered through thesyslogRouting transform, while all non-syslog (default) events be filtered throughthe errorRouting transform. Therefore, only non-syslog events get inspected forerrors.

3. Edit outputs.conf to define the target groups:

[tcpout]defaultGroup=everythingElseGroup

[tcpout:syslogGroup]server=10.1.1.197:9996, 10.1.1.198:9997

[tcpout:errorGroup]server=10.1.1.200:9999

[tcpout:everythingElseGroup]server=10.1.1.250:6666

syslogGroup and errorGroup receive events according to the rules specified intransforms.conf. All other events get routed to the default group,everythingElseGroup.

Replicate a subset of data to a third-party system

This example uses data filtering to route two data streams. It forwards:

All the data, in cooked form, to a Splunk indexer (10.1.12.1:9997)• A replicated subset of the data, in raw form, to a third-party server(10.1.12.2:1234)

•

The example sends both streams as TCP. To send the second stream as syslog

67

data, first route the data through an indexer.

1. Edit props.conf:

[syslog]TRANSFORMS-routing = routeAll, routeSubset

2. Edit transforms.conf:

[routeAll]REGEX=(.)DEST_KEY=_TCP_ROUTINGFORMAT=Everything

[routeSubset]REGEX=(SYSTEM|CONFIG|THREAT)DEST_KEY=_TCP_ROUTINGFORMAT=Subsidiary,Everything

3. Edit outputs.conf:

[tcpout]defaultGroup=nothing

[tcpout:Everything]disabled=falseserver=10.1.12.1:9997

[tcpout:Subsidiary]disabled=falsesendCookedData=falseserver=10.1.12.2:1234

For more information, see "Forward data to third party systems" in this manual.

Filter event data and send to queues

Although similar to forwarder-based routing, queue routing can be performed byan indexer, as well as a heavy forwarder. It does not use the outputs.conf file,just props.conf and transforms.conf.

You can eliminate unwanted data by routing it to nullQueue, Splunk's /dev/nullequivalent. When you filter out data in this way, the filtered data is not forwardedor added to the Splunk index at all, and doesn't count toward your indexing

68

volume.

Discard specific events and keep the rest

This example discards all sshd events in /var/log/messages by sending them tonullQueue:

1. In props.conf, set the TRANSFORMS-null attribute:

[source::/var/log/messages]TRANSFORMS-null= setnull

2. Create a corresponding stanza in transforms.conf. Set DEST_KEY to "queue"and FORMAT to "nullQueue":

[setnull]REGEX = \[sshd\]DEST_KEY = queueFORMAT = nullQueue

That does it.

Keep specific events and discard the rest

Here's the opposite scenario. In this example, you use two transforms to keeponly the sshd events. One transform routes sshd events to indexQueue, whileanother routes all other events to nullQueue.

Note: In this example, the order of the transforms in props.conf matters. The nullqueue transform must come first; if it comes later, it will invalidate the previoustransform and route all events to the null queue.

1. In props.conf:

[source::/var/log/messages]TRANSFORMS-set= setnull,setparsing

2. In transforms.conf:

[setnull]REGEX = .

69

DEST_KEY = queueFORMAT = nullQueue

[setparsing]REGEX = \[sshd\]DEST_KEY = queueFORMAT = indexQueue

Filter WMI events

To filter on WMI events, use [WMI:WinEventLog:Security] source type stanza inprops.conf. The following example uses regex to filter out two Windows eventcodes, 592 and 593:

In props.conf:

[WMI:WinEventLog:Security]TRANSFORMS-wmi=wminull

Note: In pre-4.2.x versions of Splunk, you must use [wmi] as the sourcetype inorder to send events to nullQueue.

In transforms.conf:

[wminull]REGEX=(?m)^EventCode=(592|593)DEST_KEY=queueFORMAT=nullQueue

Filter data by target index

Forwarders have a forwardedindex filter that allows you to specify whether datagets forwarded, based on the data's target index. For example, if you have onedata input targeted to "index1" and another targeted to "index2", you can use thefilter to forward only the data targeted to index1, while ignoring the index2 data.The forwardedindex filter uses whitelists and blacklists to specify the filtering.For information on setting up multiple indexes, see the topic "Set up multipleindexes".

Note: The forwardedindex filter is only applicable under the global [tcpout]stanza. This filter does not work if it is created under any other stanza inoutputs.conf.

70

Use the forwardedindex.<n>.whitelist|blacklist attributes in outputs.conf tospecify which data should get forwarded on an index-by-index basis. You set theattributes to regexes that filter the target indexes.

Default behavior

By default, Splunk forwards data targeted for all external indexes, including thedefault index and any user-created indexes. Regarding data for internal indexes,the default behavior varies according to who is doing the forwarding:

The universal forwarder forwards the data for the _audit internal indexonly. It does not forward data for other internal indexes. Its defaultoutputs.conf file located in$SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default specifiesthat behavior with these attributes:

•

[tcpout] forwardedindex.0.whitelist = .* forwardedindex.1.blacklist = _.* forwardedindex.2.whitelist = _audit

Heavy forwarder and full Splunk instances with forwarding enabled(for example, a search head with forwarding enabled) forward the data forthe _audit and _internal internal indexes. Their default outputs.conf file,located in $SPLUNK_HOME/etc/system/default, specifies that behavior withthese attributes:

•

[tcpout] forwardedindex.0.whitelist = .* forwardedindex.1.blacklist = _.* forwardedindex.2.whitelist = (_audit|_internal)

Note: The default behavior for heavy forwarders and full Splunk instanceschanged in Splunk version 5.0.2. In earlier versions, the _internal index was notforwarded by default. Those forwarder types had the same behavior as theuniversal forwarder: only data for the _audit internal index was forwarded.

In most deployments, you will not need to override the default settings. Seeoutputs.conf for more information on how to whitelist and blacklist indexes. Formore information on default and custom outputs.conf files and their locations,see "Types of outputs.conf files".

71

Forward all external and internal index data

If you want to forward all internal index data, as well as all external data, you canoverride the default forwardedindex filter attributes like this:

#Forward everything[tcpout]forwardedindex.0.whitelist = .*# disable theseforwardedindex.1.blacklist =forwardedindex.2.whitelist =

Forward data for a single index only

If you want to forward only the data targeted for a single index (for example, asspecified in inputs.conf), and drop any data that's not target for that index,here's how to do it:

[tcpout]#Disable the current filters from the defaults outputs.confforwardedindex.0.whitelist =forwardedindex.1.blacklist =forwardedindex.2.whitelist =

#Forward data for the "myindex" indexforwardedindex.0.whitelist = myindex

This first disables all filters from the default outputs.conf file. It then sets thefilter for your own index. Be sure to start the filter numbering with 0:forwardedindex.0.

Note: If you've set other filters in another copy of outputs.conf on your system,you must disable those as well.

You can use the CLI btools command to ensure that there aren't any other filterslocated in other outputs.conf files on your system:

splunk btools outputs list tcpout

This command returns the content of the tcpout stanza, after all versions of theconfiguration file have been combined.

72

Use the forwardedindex attributes with local indexing

On a heavy forwarder, you can index locally. To do that, you must setindexAndForward attribute to "true". Otherwise, the forwarder just forwards yourdata and does not save it on the forwarder. On the other hand, theforwardedindex attributes only filter forwarded data; they do not filter any datathat gets saved to the local index.

In a nutshell, local indexing and forwarder filtering are entirely separateoperations, which do not coordinate with each other. This can have unexpectedimplications when you're performing blacklist filtering:

If you set indexAndForward to "true" and then filter out some data throughforwardedindex blacklist attributes, the blacklisted data will not getforwarded, but it will still get locally indexed.

•

If you set indexAndForward to "false" (no local indexing) and then filter outsome data, the filtered data will get dropped entirely, since it doesn't getforwarded and it doesn't get saved (indexed) on the forwarder.

•

Route inputs to specific indexers based on the data's input

There is one type of routing that doesn't require a heavy forwarder. In thisscenario, you use inputs.conf and outputs.conf to route data to specificindexers, based on the data's input.

Here's an example that shows how this works.

1. In outputs.conf, you create stanzas for each receiving indexer:

[tcpout:systemGroup]server=server1:9997

[tcpout:applicationGroup]server=server2:9997

2. In inputs.conf, you use _TCP_ROUTING to specify the stanza in outputs.confthat each input should use for routing:

[monitor://.../file1.log]_TCP_ROUTING = systemGroup

[monitor://.../file2.log]_TCP_ROUTING = applicationGroup

73

The forwarder will route data from file1.log to server1 and data from file2.logto server2.

Perform selective indexing and forwarding

With a heavy forwarder only, you can index and store data locally, as well asforward the data onwards to a receiving indexer. There are two ways to do this:

Index all the data before forwarding it. To do this, just enable theindexAndForward attribute in outputs.conf.

•

Index a subset of the data before forwarding it or other data. This iscalled selective indexing. With selective indexing, you can index justsome of the data locally and then forward it on to a receiving indexer.Alternatively, you can choose to forward only the data that you don't indexlocally.

•

Important: Do not enable the indexAndForward attribute in the [tcpout] stanza ifyou're also enabling selective forwarding.

Configure selective indexing

To use selective indexing, you need to modify both your inputs.conf andoutputs.conf files:

1. In outputs.conf:

a. Add the [indexAndForward] stanza:

[indexAndForward]index=trueselectiveIndexing=true

The presence of this stanza, including the index and selectiveIndexingattributes, turns on selective indexing for the forwarder. It enables local indexingfor any input (specified in inputs.conf) that has the_INDEX_AND_FORWARD_ROUTING attribute. Use the entire [indexAndForward] stanzaexactly as shown here.

Note: This is a global stanza, which only needs to appear once in outputs.conf.

b. Include the usual target group stanzas for each set of receiving indexers:

74

[tcpout:<target_group>]server = <ip address>:<port>, <ip address>:<port>, ......

The named <target_group> is used in inputs.conf to route the inputs, asdescribed below.

2. In inputs.conf:

a. Add the _INDEX_AND_FORWARD_ROUTING attribute to the stanzas of each inputthat you want to index locally:

[input_stanza]_INDEX_AND_FORWARD_ROUTING=<any_string>...

The presence of the _INDEX_AND_FORWARD_ROUTING attribute tells the heavyforwarder to index that input locally. You can set the attribute to any string valueyou want. Splunk just looks for the attribute itself; the string value has no effect atall on behavior.

b. Add the _TCP_ROUTING attribute to the stanzas of each input that you want toforward:

[input_stanza]_TCP_ROUTING=<target_group>...

The <target_group> is the name used in outputs.conf to specify the targetgroup of receiving indexers.

The next several sections show how to use selective indexing in a variety ofscenarios.

Index one input locally and then forward the remaining inputs

In this example, the forwarder indexes data from one input locally but does notforward it. It also forwards data from two other inputs but does not index thoseinputs locally.

1. In outputs.conf, create these stanzas:

75

[tcpout]defaultGroup=noforwarddisabled=false


[tcpout:indexerB_9997]server = indexerB:9997

[tcpout:indexerC_9997]server = indexerC:9997

Since the defaultGroup is set to the non-existent group "noforward" (meaningthat there is no defaultGroup), the forwarder will only forward data that's beenrouted to explicit target groups in inputs.conf. All other data will get dropped.

2. In inputs.conf, create these stanzas:

[monitor:///mydata/source1.log]_INDEX_AND_FORWARD_ROUTING=local

[monitor:///mydata/source2.log]_TCP_ROUTING=indexerB_9997

[monitor:///mydata/source3.log]_TCP_ROUTING=indexerC_9997

The result:

Splunk indexes the source1.log data locally but does not forward it(because there's no explicit routing in its input stanza and there's nodefault group in outputs.conf).

•

Splunk forwards the source2.log data to indexerB but does not index itlocally.

•

Splunk forwards the source3.log data to indexerC but does not index itlocally.

•

Index one input locally and then forward all inputs

This example is nearly identical to the previous one. The difference is that here,you index just one input locally, but then you forward all inputs, including the oneyou've indexed locally.

76


[tcpout]defaultGroup=noforwarddisabled=false




This outputs.conf is identical to the previous example.


[monitor:///mydata/source1.log]_INDEX_AND_FORWARD_ROUTING=local_TCP_ROUTING=indexerB_9997



The only difference from the previous example is that here, you've specified the_TCP_ROUTING attribute for the input that you're indexing locally. Splunk will routeboth source1.log and source2.log to the indexerB_9997 target group, but it willonly locally index the data from source1.log.

Another way to index one input locally and then forward all inputs

You can achieve the same result as in the previous example by setting thedefaultGroup to a real target group.


[tcpout]defaultGroup=indexerB_9997disabled=false

77




This outputs.conf sets the defaultGroup to indexerB_9997.


[monitor:///mydata/source1.log]_INDEX_AND_FORWARD_ROUTING=local



Even though you haven't set up an explicit routing for source1.log, Splunk willstill forward it to the indexerB_9997 target group, since outputs.conf specifiesthat group as the defaultGroup.

Selective indexing and internal logs

Once you enable selective indexing in outputs.conf, the forwarder will only indexlocally those inputs with the _INDEX_AND_FORWARD_ROUTING attribute. This appliesto the internal logs in the /var/log/splunk directory (specified in the defaultetc/system/default/inputs.conf). By default, they will not be indexed. If youwant to index them, you must add their input stanza to your local inputs.conf file(which takes precedence over the default file) and include the_INDEX_AND_FORWARD_ROUTING attribute:

[monitor://$SPLUNK_HOME/var/log/splunk]index = _internal_INDEX_AND_FORWARD_ROUTING=local

78

Forward data to third-party systems

Splunk forwarders can forward raw data to non-Splunk systems. They can sendthe data over a plain TCP socket or packaged in standard syslog. Because theyare forwarding to a non-Splunk system, they can send only raw data.

By editing outputs.conf, props.conf, and transforms.conf, you can configure aheavy forwarder to route data conditionally to third-party systems, in the sameway that it routes data conditionally to other Splunk instances. You can filter thedata by host, source, or source type. You can also use regex to further qualifythe data.

TCP data

To forward TCP data to a third-party system, edit the forwarder's outputs.conf fileto specify the receiving server and port. You must also configure the receivingserver to expect the incoming data stream on that port. You can use any kind offorwarder, such as a universal forwarder, to perform this type of forwarding.

To route the data, you need to use a heavy forwarder, which has the ability toparse data. Edit the forwarder's props.conf and transforms.conf files as well asoutputs.conf.

Edit the configuration files

To simply forward data, edit outputs.conf:

Specify target groups for the receiving servers.• Specify the IP address and TCP port for each receiving server.• Set sendCookedData to false, so that the forwarder sends raw data.•

To route and filter the data (heavy forwarders only), also edit props.conf andtransforms.conf:

In props.conf, specify the host, source, or sourcetype of your data stream.Specify a transform to perform on the input.

•

In transforms.conf, define the transform and specify _TCP_ROUTING. Youcan also use regex to further filter the data.

•

79

Forward all data

This example shows how to send all the data from a universal forwarder to athird-party system. Since you are sending all the data, you only need to editoutputs.conf:

[tcpout]

[tcpout:fastlane]server = 10.1.1.35:6996sendCookedData = false

Forward a subset of data

This example shows how to use a heavy forwarder to filter a subset of data andsend the subset to a third-party system:

1. Edit props.conf and transforms.conf to specify the filtering criteria.

In props.conf, apply the bigmoney transform to all host names beginning withnyc:

[host::nyc*]TRANSFORMS-nyc = bigmoney

In transforms.conf, configure the bigmoney transform to specify TCP_ROUTING asthe DEST_KEY and the bigmoneyreader target group as the FORMAT:

[bigmoney]REGEX = .DEST_KEY=_TCP_ROUTINGFORMAT=bigmoneyreader

2. In outputs.conf, define both a bigmoneyreader target group for the non-Splunkserver and a default target group to receive any other data:

[tcpout]defaultGroup = default-clone-group-192_168_1_104_9997

[tcpout:default-clone-group-192_168_1_104_9997]server = 192.168.1.104:9997

[tcpout:bigmoneyreader]

80

server=10.1.1.197:7999sendCookedData=false

The forwarder will send all data from host names beginning with nyc to thenon-Splunk server specified in the bigmoneyreader target group. It will send datafrom all other hosts to the server specified in thedefault-clone-group-192_168_1_104_9997 target group.

Note: If you want to forward only the data specifically identified in props.confand transforms.conf, set defaultGroup=nothing.

Syslog data

You can configure a heavy forwarder to send data in standard syslog format. Theforwarder sends the data through a separate output processor. You can also filterthe data with props.conf and transforms.conf. You'll need to specify_SYSLOG_ROUTING as the DEST_KEY.

Note: The syslog output processor is not available for universal or lightforwarders.

The syslog output processor sends RFC 3164 compliant events to aTCP/UDP-based server and port, making the payload of any non-compliant dataRFC 3164 compliant. Yes, that means Windows event logs!

To forward syslog data, identify the third-party receiving server and specify it in asyslog target group in the forwarder's outputs.conf file.

Note: If you have defined multiple event types for syslog data, the event typenames must all include the string "syslog".

Forward syslog data

In outputs.conf, specify the syslog target group:

[syslog:<target_group>]<attribute1> = <val1><attribute2> = <val2>...

The target group stanza requires this attribute:

Default Value

81

RequiredAttribute

server n/a

This must be in the format<ipaddress_or_servername>:<port>. This is acombination of the IP address or servername of the syslogserver and the port on which the syslog server is listening.Note that syslog servers use port 514 by default.

These attributes are optional:

OptionalAttribute Default Value

type udp The transport protocol. Must be set to "tcp" or "udp".

priority

<13> -thissignifies afacility of1 ("user")and aseverityof 5("notice")

Syslog priority. This must be an integer 1 to 3 digits in length,surrounded by angle brackets; for example: <34>. This value willappear in the syslog header.

Mimics the number passed via syslog interface call;see outputs.conf for more information.

Compute the priority value as (<facility> * 8) +<severity>. If facility is 4 (security/authorizationmessages) and severity is 2 (critical conditions),priority value will be: (4 * 8) + 2 = 34, which youspecify in the conf file as <34>.

syslogSourceType n/aThis must be in the format sourcetype::syslog, thesource type for syslog messages.

timestampformat ""The format used when adding a timestamp to the header. Thismust be in the format: <%b %e %H:%M:%S>. See "Configuretimestamps" in the Getting Data In manual for details.

Send a subset of data to a syslog server

This example shows how to configure a heavy forwarder to forward data fromhosts whose names begin with "nyc" to a syslog server named"loghost.example.com" over port 514:

1. Edit props.conf and transforms.conf to specify the filtering criteria.

In props.conf, apply the send_to_syslog transform to all host names beginning

82

with nyc:

[host::nyc*]TRANSFORMS-nyc = send_to_syslog

In transforms.conf, configure the send_to_syslog transform to specify_SYSLOG_ROUTING as the DEST_KEY and the my_syslog_group target group as theFORMAT:

[send_to_syslog]REGEX = .DEST_KEY = _SYSLOG_ROUTINGFORMAT = my_syslog_group

2. In outputs.conf, define the my_syslog_group target group for the non-Splunkserver:

[syslog:my_syslog_group]server = loghost.example.com:514

83

Deploy the universal forwarder

Introducing the universal forwarder

The universal forwarder is Splunk's new lightweight forwarder. Use theuniversal forwarder to gather data from a variety of inputs and forward the data toa Splunk server for indexing and searching.

This section of the Distributed Deployment manual describes how to deploy theuniversal forwarder for a variety of systems and needs. For information on thedifferent kinds of forwarders and detailed information on configuring them for arange of topologies and use cases, see the "Forward data" chapter of thismanual.

The universal forwarder replaces the light forwarder.

Note: The universal forwarder is a separate executable from Splunk. Instancesof Splunk and the universal forwarder can co-exist on the same system.

For information on deploying the universal forwarder, see "Universal forwarderdeployment overview".

How universal forwarder compares to full Splunk

The universal forwarder's sole purpose is to forward data. Unlike a full Splunkinstance, you cannot use the universal forwarder to index or search data. Toachieve higher performance and a lighter footprint, it has several limitations:

The universal forwarder has no searching, indexing, or alerting capability.• The universal forwarder does not parse data.• The universal forwarder does not output data via syslog.• Unlike full Splunk, the universal forwarder does not include a bundledversion of Python.

•

Scripted inputs and Python

Full Splunk comes bundled with Python. The universal forwarder does not.Therefore, if you're currently using scripted inputs with Python and you want touse those scripts with the universal forwarder, you must first install your ownversion of Python. If you have been using calls specific to Splunk's Pythonlibraries, you cannot do so with the universal forwarder, since those libraries exist

84

only in full Splunk. You may use other scripting languages for scripted inputs withthe universal forwarder if they are otherwise supported on the target host (forexample, Powershell on Windows Server 2008.)

How universal forwarder compares to the light forwarder

The universal forwarder is a streamlined, self-contained forwarder that includesonly the essential components needed to forward data to Splunk indexers. Thelight forwarder, by contrast, is a full Splunk instance, with certain featuresdisabled to achieve a smaller footprint. In all respects, the universal forwarderrepresents a better tool for forwarding data to indexers. When you install theuniversal forwarder, you can migrate from an existing light forwarder, version 4.0or greater. See "Migrating from a light forwarder" for details.

Compared to the light forwarder, the universal forwarder provides a betterperforming and more streamlined solution to forwarding. These are the maintechnical differences between the universal forwarder and the light forwarder:

The universal forwarder puts less load on the CPU, uses less memory,and has a smaller disk footprint.

•

The universal forwarder has a default data transfer rate of 256Kbps• The universal forwarder does not come bundled with Python.• The universal forwarder is a forwarder only; it cannot be converted to a fullSplunk instance.

•

Read on!

For information on deploying the universal forwarder, see the topics that directlyfollow this one.

For information on using the universal forwarder to forward data and participatein various distributed topologies, see the topics in the "Forward data" section ofthis manual. Those topics also discuss light and heavy forwarders.

For information on third-party Windows binaries that the Windows version of theSplunk universal forwarder ships with, read "Information on Windows third-partybinaries distributed with Splunk" in the Installation Manual.

For information about running the Splunk universal forwarder in Windows SafeMode, read "Splunk Architecture and Processes" in the Installation Manual.

85

Universal forwarder deployment overview

The topics in this chapter describe how to install and deploy the universalforwarder. They include use cases that focus on installing and configuring theforwarder for a number of different scenarios.

Important: Before attempting to deploy the universal forwarder, you must befamiliar with how forwarding works and the full range of configuration issues.See:

the chapter "Forward data" for an overview of forwarding and forwarders.• the topics in the chapter "Configure forwarding" to learn how to configureforwarders.

•

the subtopic "Set up forwarding and receiving: universal forwarders" for aoverview of configuring Splunk forwarding and receiving.

•

Types of deployments

These are the main scenarios for deploying the universal forwarder:

Deploy a Windows universal forwarder manually, either with the installerGUI or from the commandline.

•

Deploy a nix universal forwarder manually, using the CLI to configure it.• Remotely deploy a universal forwarder (Windows or nix).• Make the universal forwarder part of a system image.•

Each scenario is described in its own topic. For most scenarios, there areseparate Windows and *nix topics.

Note: The universal forwarder is its own downloadable executable, separatefrom full Splunk. Unlike the light and heavy forwarders, you do not enable it froma full Splunk instance. To download the universal forwarder, go tohttp://www.splunk.com/download/universalforwarder .

Migrating from a light forwarder?

The universal forwarder provides all the functionality of the old light forwarderbut in a smaller footprint with better performance. Therefore, you might want tomigrate your existing light forwarder installations to universal forwarders. Splunkprovides tools that ease the migration process and ensure that the new universalforwarder does not send an indexer any data already sent by the old lightforwarder.

86

Note: You can only migrate from light forwarders of version 4.0 or later.

Migration is available as an option during the universal forwarder installationprocess. See "Migrate a Windows forwarder" or "Migrate a nix forwarder" fordetails. You will want to uninstall the old light forwarder instance once youruniversal forwarder is up and running (and once you've tested to ensuremigration worked correctly).

What migration does

Migration copies checkpoint data, including the fishbucket directory, from the oldforwarder to the new universal forwarder. This prevents the universal forwarderfrom re-forwarding data that the previous forwarder had already sent to anindexer. This in turn avoids unnecessary re-indexing, ensuring that you maintainyour statistics and keep your license usage under control. Specifically, migrationcopies:

the fishbucket directory (contains seek pointers for tailed files).• checkpoint files for WinEventLog (Windows only), WMI remote log(Windows only), and fschange.

•

What migration does not do

Migration does not copy any configuration files, such as inputs.conf oroutputs.conf. This is because it would not be possible to conclusively determinewhere all existing versions of configuration files reside on the old forwarder.Therefore, you still need to configure your data inputs and outputs, either duringinstallation or later. If you choose to configure later, you can copy over thenecessary configuration files manually or you can use the deployment server topush them out to all your universal forwarders. See this section below for moreinformation on configuration files.

If the data inputs for the universal forwarder differ from the old forwarder, you canstill migrate. Migrated checkpoint data pertaining to any inputs not configured forthe universal forwarder will just be ignored. If you decide to add those inputslater, the universal forwarder will use the migrated checkpoints to determinewhere in the data stream to start forwarding.

Migration also does not copy over any apps from the light forwarder. If you haveany apps that you want to migrate to the universal forwarder, you'll need to do somanually.

87

Before you start


When using forwarders to send data to peer nodes in a cluster, you deploy andconfigure them a bit differently from the description in this topic. To learn moreabout forwarders and clusters, read "Use forwarders to get your data" in theManaging Indexers and Clusters Manual.

Indexer and universal forwarder compatibility

See "Compatibility between forwarders and indexers" for details.

System requirements

See the Installation manual for specific hardware requirements and supportedoperating systems.

Licensing requirements

The universal forwarder ships with a pre-installed license. See "Types of Splunklicenses" in the Admin manual for details.

Other requirements

You must have admin or equivalent rights on the machine where you're installingthe universal forwarder.

Steps to deployment

The actual procedure varies depending on the type of deployment, but these arethe typical steps:

1. Plan your deployment.

2. Download the universal forwarder fromhttp://www.splunk.com/download/universalforwarder

3. Install the universal forwarder on a test machine.

4. Perform any post-installation configuration.

5. Test and tune the deployment.

88

6. Deploy the universal forwarder to machines across your environment (formulti-machine deployments).

These steps are described below in more detail.

Important: Deploying your forwarders is just one step in the overall process ofsetting up Splunk forwarding and receiving. For an overview of that process, read"Set up forwarding and receiving: universal forwarders".

Plan your deployment

Here are some of the issues to consider when planning your deployment:

How many (and what type of) machines will you be deploying to?• Will you be deploying across multiple OS's?• Do you need to migrate from any existing forwarders?• What, if any, deployment tools do you plan to use?• Will you be deploying via a system image or virtual machine?• Will you be deploying fully configured universal forwarders, or do you planto complete the configuration after the universal forwarders have beendeployed across your system?

•

What level of security does the communication between universalforwarder and indexer require?

•

Install, test, configure, deploy

For next steps, see the topic in this chapter that matches your deploymentrequirements most closely. Each topic contains one or more use cases that coverspecific deployment scenarios from installation through configuration anddeployment:

"Deploy a Windows universal forwarder via the installer GUI"•

"Deploy a Windows universal forwarder via the commandline"•

"Remotely deploy a Windows universal forwarder with a staticconfiguration"

•

"Deploy a nix universal forwarder manually"•

"Remotely deploy a nix universal forwarder with a static configuration"•

"Make a universal forwarder part of a system image"•

89

But first, read the next section to learn more about universal forwarderconfiguration.

Note: The universal forwarder's executable is named splunkd, the same as theexecutable for full Splunk. The service name is SplunkUniversalForwarder.

General configuration issues

Because the universal forwarder has no Splunk Web GUI, you must perform allconfiguration either during installation (Windows-only) or later, as a separatestep. To perform post-installation configuration, you can use the CLI, modify theconfiguration files directly, or use deployment server.

Where to configure

Key configuration files include inputs.conf (for data inputs) and outputs.conf (fordata outputs). Others include server.conf and deploymentclient.conf.

When you make configuration changes with the CLI, the universal forwarderwrites the changes to configuration files in the search app (except for changes tooutputs.conf, which it writes to a file in $SPLUNK_HOME/etc/system/local/). Thesearch app is the default app for the universal forwarder, even though you cannotactually use the universal forwarder to perform searches. If this seems odd, it is.

Important: The Windows installation process writes configuration changes to anapp called "MSICreated", not to the search app.

Note: The universal forwarder also ships with a SplunkUniversalForwarder app,which must be enabled. (This happens automatically.) This app includespreconfigured settings that enable the universal forwarder to run in a streamlinedmode. No configuration changes get written there. We recommend that you donot make any changes or additions to that app.

Learn more about configuration

Refer to these topics for some important information:

"About configuration files" and "Configuration file precedence" in theAdmin manual, for details on how configuration files work.

•

"Configure forwarders with outputs.conf", for information on outputs.confspecifically.

•

The topics in the "Use the forwarder to create deployment topologies"section, for information on configuring outputs with the CLI.

•

90

"Configure your inputs" in the Getting Data In manual, for details onconfiguring data inputs with inputs.conf or the CLI.

•

Deploy configuration updates

These are the main methods for deploying configuration updates across your setof universal forwarders:

Edit or copy the configuration files for each universal forwarder manually(for small deployments only).

•

Use the Splunk deployment server to push configured apps to your set ofuniversal forwarders.

•

Use your own deployment tools to push configuration changes.•

Restart the universal forwarder

Some configuration changes might require that you restart the forwarder. (Thetopics covering specific configuration changes will let you know if a change doesrequire a restart.)

To restart the universal forwarder, use the same CLI restart command that youuse to restart a full Splunk instance:

On Windows: Go to %SPLUNK_HOME%\bin and run this command:•

> splunk restart

On *nix systems: From a shell prompt on the host, run this command:•

# splunk restart

Deploy a Windows universal forwarder via theinstaller GUI

This topic describes how to manually install, configure, and deploy the universalforwarder in a Windows environment using the installer GUI. It assumes thatyou're installing directly onto the Windows machine, rather than using adeployment tool. This method of installation best suits these needs:

small deployments• proof-of-concept test deployments•

91

system image or virtual machine for eventual cloning•

If you are interested in a different deployment method or a different operatingsystem, look for another topic in this section that better fits your needs.

You can also install the universal forwarder from the command line, usingmsiexec. See "Deploy a Windows universal forwarder via the command line" formore information.

Important: If you do not want the universal forwarder to start immediately afterinstallation, you must install via the command line.

Before following the procedures in this topic, read "Deployment overview" tofurther understand the mechanics of a distributed Splunk deployment.

Steps to deployment

Once you have downloaded the universal forwarder and planned yourdeployment, as described in "Deployment overview", perform these steps:

1. Install the universal forwarder (with optional migration and configuration).



4. Deploy the universal forwarder across your environment.

Before you install

Choose where the universal forwarder will get its data from

When you install the universal forwarder, you can select where the forwarder willget its data. You have two choices:

Local Data Only• Remote Windows Data•

If you tell the installer to collect Local Data only, the universal forwarder cancollect any kind of data that is available on the local machine. It cannot, however,collect data from other machines.

92

You must tell the forwarder to collect Remote Windows Data if you intend to doany of the following:

Read Event Logs remotely• Collect performance counters remotely• Read network shares for log files• Enumerate the Active Directory schema, using Active Directory monitoring•

If you tell the installer to collect Remote Windows data, you must then specify auser which has access to this data. Read "Choose the Windows user Splunkshould run as" in the Installation Manual for concepts and procedures on the userrequirements that must be in place before you collect remote Windows data.

Important: You should choose - and configure - the user that Splunk will run asbefore attempting to install a universal forwarder for remote Windows datacollection.

Configure your Windows environment for remote data collection

If you do not need to install the universal forwarder to collect remote Windowsdata, you can continue to the installation instructions below.

If your monitoring needs require you to install the universal forwarder to collectremote Windows data, then you must configure your Windows environment forthe proper installation of the forwarder.

1. Create and configure security groups with the user you want the universalforwarder to run as.

2. Optionally, configure the universal forwarder account as a managed serviceaccount.

3. Create and configure Group Policy objects for security policy and user rightsassignment.

4. Assign appropriate user rights to the GPO.

5. Deploy the GPO(s) with the updated settings to the appropriate objects.

Note: These steps are high-level procedures only. For step-by-step instructions,read "Prepare your Windows network for a Splunk installation as a network ordomain user" in the Installation Manual.

93

Install the universal forwarder

The Windows installer guides you through the process of installing andconfiguring your universal forwarder. It also offers you the option of migratingyour checkpoint settings from an existing Splunk forwarder.

To install the universal forwarder, double-click the appropriate MSI file:

splunkuniversalforwarder-<...>-x86-release.msi (for 32-bit platforms)• splunkuniversalforwarder-<...>-x64-release.msi (for 64-bit platforms)•

The value of <...> varies according to the particular release; for example,splunkuniversalforwarder-4.2-86454-x64-release.msi.

Important: Running the 32-bit version of the universal forwarder on a 64-bitplatform is not recommended. If you can run 64-bit universal forwarder on 64-bithardware, we strongly recommend it. The performance is greatly improved overthe 32-bit version.

A series of dialogs guides you through the installation. When you're through witha dialog, click Next to move to the next in the series. Here are the dialogs, inorder:

1. "Welcome" dialog

To begin the installation, click Next.

2. "License Agreement" dialog

Read the license agreement and select "I accept the terms in the licenseagreement".

3. "Destination Folder" dialog

The installer puts the universal forwarder into the C:\ProgramFiles\SplunkUniversalForwarder directory by default.

Click Change... to specify a different installation directory.

Caution: Do not install the universal forwarder over an existing installation of fullSplunk. This is particularly important if you are upgrading from a light forwarder,as described in "Migrate a Windows light forwarder". The default installationdirectory for full Splunk is C:\Program Files\Splunk, so, if you stick with the

94

defaults, you're safe.

4. "Migration" pop-up

If the installer detects an existing version of Splunk, it will ask you whether or notyou want to migrate the existing Splunk's data checkpoint settings to theuniversal forwarder. If you click Yes, it will automatically perform the migration.

Important: This is the only time when you can migrate old settings to thisuniversal forwarder. You cannot perform the migration later.

See "Migrate a Windows forwarder" for more information on what migration does.

5. "Deployment Server" dialog

Enter the hostname or IP address and management port for your deploymentserver. The default management port is 8089.

You can use the deployment server to push configuration updates to theuniversal forwarder. See "About deployment server" for details.

Note: This step is optional, but if you skip it, you must enter a receiving indexerin step 6; otherwise, the universal forwarder does not do anything, as it does nothave any way of determining which indexer to forward data to.

6. "Receiving Indexer" dialog

Enter the hostname or IP address and receiving port of the receiving indexer(receiver). For information on setting up a receiver, see "Enable a receiver".

Note: This step is optional, but if you skip it, you must enter a deployment serverin step 5; otherwise, the universal forwarder does not do anything, as it does nothave any way of determining which indexer to forward to.

7. "Certificate Information" dialog

Select an SSL certificate for verifying the identity of this machine. This step isoptional.

Depending on your certificate requirements, you might need to specify apassword and a Root Certificate Authority (CA) certificate to verify the identity ofthe certificate. If not, these fields can be left blank.

95

Note: This dialog will only appear if you previously specified a receiving indexer(step 6).

8. "Where do you want to get data from?" dialogs

This step in the installer requires one or two dialogs, depending on whether theuniversal forwarder will be collecting local data exclusively.

In the first dialog, you specify the user context: whether you want the universalforwarder to collect only local data or also remote Windows data. The installeruses this information to determine the permissions the universal forwarderneeds.

Note: If you select Local data only, the universal forwarder installs as the LocalSystem user, and network resources will not be available to it. This isrecommended for improved security, unless this universal forwarder will becollecting event logs or metrics from remote machines.

For more help in determining what to select here, see "Before you install" earlierin this topic.

Once you've made your choice, click Next.

If you specified Local data only, the installer skips the second screen and takesyou directly to the "Enable Windows Inputs" dialog (step 8).

If you specified Remote Windows data, the installer now takes you to a seconddialog, where you need to enter domain and user information for this instance ofthe universal forwarder. The universal forwarder will run as the user you specifyin this dialog.

Important: The user you specify must have specific rights assigned to it prior tocompleting the installation. Failure to do so might result in a failed installation.Read "Before you install" earlier in this topic for specific information and links tostep-by-step instructions.

9. "Enable Windows Inputs" dialog

Select one or more Windows inputs from the list.

This step is optional. You can enable inputs later, by editing inputs.conf withinthe universal forwarder directory.

96

10. "Ready to Install the Program" dialog

Click Install to proceed.

The installer runs and displays the Installation Completed dialog.

Once the installation is complete, the universal forwarder automatically starts.SplunkForwarder is the name of the universal forwarder service. You shouldconfirm that it is running.

Test the deployment

Test your configured universal forwarder on a single machine, to make sure itfunctions correctly, before deploying the universal forwarder across yourenvironment. Confirm that the universal forwarder is getting the desired inputsand sending the right outputs to the indexer. You can use the deploymentmonitor to validate the universal forwarder.

If you migrated from an existing forwarder, make sure that the universalforwarder is forwarding data from where the old forwarder left off. If it isn't, youneed to modify or add data inputs, so that they conform to those on the oldforwarder.

Important: Migration does not automatically copy any configuration files. Youmust set those up yourself. The usual way to do this is to copy the files, includinginputs.conf, from the old forwarder to the universal forwarder. Compare theinputs.conf files on the universal forwarder and the old forwarder to ensure thatthe universal forwarder has all the inputs that you want to maintain.

If you migrated from an existing forwarder, you can delete that old instance onceyour universal forwarder has been thoroughly tested and you're comfortable withthe results.

Perform additional configuration

You can update your universal forwarder's configuration, post-installation, bydirectly editing its configuration files, such as inputs.conf and outputs.conf.You can also update the configuration using the CLI. See "Deployment overview"for information.

Note: When you use the CLI, you might need to authenticate into the Splunkforwarder to complete commands. The default credentials for a universalforwarder are:

97

Username: adminPassword: changeme

For information on distributing configuration changes across multiple universalforwarders, see "About deployment server".

Deploy the universal forwarder across your environment

If you need just a few universal forwarders, you might find it simpler just to repeatthe manual installation process, as documented in this topic. If you need to installa larger number of universal forwarders, it will probably be easier to deploy themremotely with a deployment tool or else as part of a system image or virtualmachine.

Uninstall the universal forwarder

To uninstall the universal forwarder, perform the following steps:

1. Use the Services MMC snap-in (Start > Administrative Tools > Services) tostop the SplunkForwarder service.

Note: You can also stop the service from the command line with the followingcommand:

NET STOP SplunkForwarder

2. Next, use the Add or Remove Programs control panel to uninstall theforwarder. On Windows 7, 8, Server 2008, and Server 2012, that option isavailable under Programs and Features.

Note: Under some circumstances, the Microsoft installer might present a rebootprompt during the uninstall process. You can safely ignore this request withoutrebooting.

Deploy a Windows universal forwarder via thecommand line

This topic describes how to install, configure, and deploy the universal forwarderin a Windows environment using the command line interface. If you prefer to usea GUI installer, see "Deploy a Windows universal forwarder via the installer GUI".

98

When to install from the command line?

You can manually install the universal forwarder on individual machines from acommand prompt or PowerShell window. Here are some scenarios whereinstalling from the command line is useful:

You want to install the forwarder, but don't want it to start right away.• You want to automate installation of the forwarder with a script.• You want to install the forwarder on a system that you will clone later.• You want to use a deployment tool such as Group Policy or SystemCenter Configuration Manager.

•

Read the following topics for additional information on installing universalforwarders:

"Deployment overview" for basics on universal forwarders.• "Remotely deploy a Windows universal forwarder with a staticconfiguration" for detailed information on using the command line interfacewith a deployment tool.

•

Steps to deployment

Once you have downloaded the universal forwarder and have planned yourdeployment, as described in "Deployment overview", perform these steps:

1. Install the universal forwarder (with optional migration and configuration).



4. Deploy the universal forwarder across your environment.

Before you install

Choose the Windows user the universal forwarder should run as

When you install the universal forwarder, you can select the user it should run as.By default, the user is Local System. To specify a domain account, use the flagsLOGON_USERNAME and LOGON_PASSWORD, described later in this topic.

If you install the forwarder as the Local System user, the forwarder can collect

99

any kind of data that is available on the local machine. It cannot, however, collectdata from other machines. This is by design.

You must give the universal forwarder a user account if you intend to do any ofthe following:

Read Event Logs remotely• Collect performance counters remotely• Read network shares for log files• Enumerate the Active Directory schema, using Active Directory monitoring•

Read "Choose the Windows user Splunk should run as" in the InstallationManual for concepts and procedures on the user requirements that must be inplace before you collect remote Windows data.

Important: You must choose - and configure - the user that Splunk will run asbefore attempting to install a universal forwarder for remote Windows datacollection. Failure to do so can result in a failed installation.

Configure your Windows environment prior to installation

To configure your Windows environment for the proper installation of theforwarder, follow these steps:

1. Create and configure security groups with the user you want the universalforwarder to run as.

2. Optionally, configure the universal forwarder account as a managed serviceaccount.

3. Create and configure Group Policy or Local Security Policy objects for userrights assignments.

4. Assign appropriate security settings.

5. If using Active Directory, deploy the Group Policy object(s) with the updatedsettings to the appropriate objects.

Note: These steps are high-level procedures only. For step-by-step instructions,read "Prepare your Windows network for a Splunk installation as a network ordomain user" in the Installation Manual.

100


You can install the universal forwarder from the command line by invokingmsiexec.exe, Microsoft's installer program.

For 32-bit platforms, use splunkuniversalforwarder-<...>-x86-release.msi:

msiexec.exe /i splunkuniversalforwarder-<...>-x86-release.msi[<flag>]... [/quiet]

For 64-bit platforms, use splunkuniversalforwarder-<...>-x64-release.msi:

msiexec.exe /i splunkuniversalforwarder-<...>-x64-release.msi[<flag>]... [/quiet]


Important: Running the 32-bit version of the universal forwarder on a 64-bitplatform is not recommended. If you can run 64-bit universal forwarder on 64-bithardware, we strongly recommend it. The performance is greatly improved overthe 32-bit version.

Command line flags allow you to configure your forwarder at installation time.Using command line flags, you can specify a number of settings, including:

The user the universal forwarder runs as. (Be sure the user you specifyhas the appropriate permissions to access the content you want toforward.)

•

The receiving Splunk instance that the universal forwarder will send datato.

•

A Splunk deployment server for updating the configuration.• The Windows event logs to index.• Whether the universal forwarder should start automatically when theinstallation is completed.

•

Whether to migrate checkpoint data from an existing light forwarder.•

The following sections list the flags available and provide a few examples ofvarious configurations.

101

List of supported flags

Important: The installer for the full version of Splunk is a separate executable,with its own installation flags. Review the installation flags for the full Splunkinstaller at "Install on Windows" in the Installation Manual.

Flag What it's for Default

AGREETOLICENSE=Yes|No

Use this flag to agreeto the EULA. This flagmust be set to Yes fora silent installation.

No

INSTALLDIR="<directory_path>"

Specifies theinstallation directory.

Important: Do notinstall theuniversalforwarder over anexisting installationof full Splunk. Thisis particularly vitalif you aremigrating from alight forwarder asdescribed in"Migrate aWindows lightforwarder". Thedefault installdirectory for fullSplunk isC:\Program

Files\Splunk, so,if you stick with thedefaults, you'resafe.

c:\ProgramFiles\SplunkUniversalForwarder

LOGON_USERNAME="<domain\username>"

LOGON_PASSWORD="<pass>"

Use these flags toprovidedomain\username andpassword informationfor the user to run theSplunkForwarder

service. You must

n/a

102

specify the domainwith the usernamein the format:domain\username.If you don't includethese flags, theuniversalforwarder installsas the LocalSystem user. See"Choose theWindows userSplunk should runas".

RECEIVING_INDEXER="<host:port>"

Use this flag to specifythe receiving indexerto which the universalforwarder will forwarddata. Enter the name(hostname or IPaddress) andreceiving port ofthe Splunkreceiver. Forinformation onsetting up areceiver, see"Enable areceiver".

Note: This flag isoptional, but if youdon't specify it andalso don't specifyDEPLOYMENT_SERVER,the universalforwarder will beunable to function,as it will not haveany way ofdetermining whichindexer to forwardto.

n/a

103

DEPLOYMENT_SERVER="<host:port>"

Use this flag to specifya deploymentserver for pushingconfigurationupdates to theuniversalforwarder. Enterthe deploymentserver's name(hostname or IPaddress) and port.

Note: This flag isoptional, but if youdon't specify it andalso don't specifyRECEIVING_INDEXER,the universalforwarder will beunable to function,as it will not haveany way ofdetermining whichindexer to forwardto.

n/a

LAUNCHSPLUNK=1|0

Use this flag to specifywhether the universalforwarder should beconfigured to launchautomatically when theinstallation finishes.

1 (yes)

SERVICESTARTTYPE=auto|manual Use this flag to specifywhether the universalforwarder should startautomatically when thesystem reboots.

Note: By settingLAUNCHSPLUNK to 0andSERVICESTARTTYPE

to auto, you willcause theuniversal

auto

104

forwarder to notstart forwardinguntil the nextsystem boot. Thisis useful whencloning a systemimage.

MONITOR_PATH="<directory_path>"Use this flag to specifya file or directory tomonitor.

n/a

WINEVENTLOG_APP_ENABLE=1|0

WINEVENTLOG_SEC_ENABLE=1|0

WINEVENTLOG_SYS_ENABLE=1|0

WINEVENTLOG_FWD_ENABLE=1|0

WINEVENTLOG_SET_ENABLE=1|0

Use these flags toenable these Windowsevent logs,respectively:

application

security

system

forwarders

setup

Note: You canspecify multipleflags.

0 (no)

PERFMON=<input_type>,<input_type>,...

Use this flag to enableperfmon inputs.<input_type> canbe any of these: cpumemorynetworkdiskspace

n/a

ENABLEADMON=1|0

Use this flag to enableActive Directorymonitoring for aremote deployment.

0 (not enabled)

CERTFILE=<c:\path\to\certfile.pem>

ROOTCACERTFILE=<c:\path\to\rootcacertfile.pem>

Use these flags tosupply SSLcertificates:

n/a

105

CERTPASSWORD=<password> Path to the cert filethat contains thepublic/private keypair.

Path to the file thatcontains the RootCA cert forverifyingCERTFILE islegitimate(optional).

Password forprivate key ofCERTFILE(optional).

Note: You mustalso setRECEIVING_INDEXER

for these flags tohave any effect.

MIGRATESPLUNK=1|0 Determines whethermigration from anexisting forwarder willoccur duringinstallation. IfMIGRATESPLUNK=1,the installer stopsthe existingforwarder andcopies itscheckpoint files tothe new universalforwarder. You areresponsible foruninstalling the oldforwarder. See"Deploymentoverview" and"Migrate aWindows light

0 (no migration)

106

forwarder" fordetails.

CLONEPREP=1|0

Tells Splunk to deleteany instance-specificdata in preparation forcreating a clone of amachine. This invokesthe splunkclone-prep

command from theCLI.

0 (do not prepare the instance forcloning.)

Silent installation

To run the installation silently, add /quiet to the end of your installationcommand string. You must also set the AGREETOLICENSE=Yes flag.

If your system is running UAC (which is sometimes on by default), you must runthe installation as Administrator. To do this, when opening a cmd prompt, rightclick and select "Run As Administrator". Then use the cmd window to run thesilent install command.

Examples

The following are some examples of using different flags.

Install the universal forwarder to run as the Local System user and requestconfiguration from deploymentserver1

You might do this for new deployments of the forwarder.

msiexec.exe /i splunkuniversalforwarder_x86.msiDEPLOYMENT_SERVER="deploymentserver1:8089" AGREETOLICENSE=Yes /quiet

Install the universal forwarder to run as a domain user, but do not launch itimmediately

You might do this when preparing a sample host for cloning.

msiexec.exe /i splunkuniversalforwarder_x86.msiLOGON_USERNAME="AD\splunk" LOGON_PASSWORD="splunk123"DEPLOYMENT_SERVER="deploymentserver1:8089" LAUNCHSPLUNK=0AGREETOLICENSE=Yes /quiet

107

Install the universal forwarder, enable indexing of the Windows securityand system event logs, and run the installer in silent mode

You might do this to collect just the Security and System event logs through a"fire-and-forget" installation.

msiexec.exe /i splunkuniversalforwarder_x86.msiRECEIVING_INDEXER="indexer1:9997" WINEVENTLOG_SEC_ENABLE=1WINEVENTLOG_SYS_ENABLE=1 AGREETOLICENSE=Yes /quiet

Install the universal forwarder, migrate from an existing forwarder, and runthe installer in silent mode

You might do this if you want to migrate now and redefine your inputs later,perhaps after a validation step.

msiexec.exe /i splunkuniversalforwarder_x86.msiRECEIVING_INDEXER="indexer1:9997" MIGRATESPLUNK=1 AGREETOLICENSE=Yes/quiet

Test the deployment


If you migrated from an existing forwarder, make sure that the universalforwarder is forwarding data from where the old forwarder left off. If it isn't, youprobably need to modify or add data inputs, so that they conform to those on theold forwarder.

Important: Migration does not automatically copy any configuration files; youmust set those up yourself. The usual way to do this is to copy the files, includinginputs.conf, from the old forwarder to the universal forwarder. Compare theinputs.conf files on the universal forwarder and the old forwarder to ensure thatthe universal forwarder has all the inputs that you want to maintain.


108


You can update your universal forwarder's configuration, post-installation, bydirectly editing its configuration files, such as inputs.conf and outputs.conf.You can also update the configuration using the CLI. See "Deployment overview"for information.

Note: When you use the CLI, you might need to authenticate into the Splunkforwarder to complete commands. The default credentials for a universalforwarder are:

Username: adminPassword: changeme



If you need just a few universal forwarders, you might find it simpler just to repeatthe command line installation process manually, as documented in this topic. Ifyou need to install a larger number of universal forwarders, it will probably beeasier to deploy them remotely with a deployment tool or else as part of a systemimage or virtual machine.

Uninstall the universal forwarder

To uninstall the universal forwarder, perform the following steps:

1. Stop the service from the command line with the following command:

NET STOP SplunkForwarder

Note: You can also use the Services MMC snap-in (Start > AdministrativeTools > Services) to stop the SplunkForwarder service.

2. Next, use the Add or Remove Programs control panel to uninstall theforwarder. On Windows 7, 8, Server 2008, and Server 2012, that option isavailable under Programs and Features.

Note: Under some circumstances, the Microsoft installer might present a rebootprompt during the uninstall process. You can safely ignore this request withoutrebooting.

109

Remotely deploy a Windows universal forwarderwith a static configuration

You typically deploy the universal forwarder with a static configuration for one ofthese reasons:

You don't need to change the configuration later - "fire-and-forget".• You'll be making any post-installation changes with a non-Splunkdeployment tool such as System Center Configuration Manager, Altris, orBigFix/Tivoli.

•

For this type of deployment, you install via the Windows commandline interface.During installation, you must specify all configuration options and use silent mode(/quiet). See "Deploy a Windows universal forwarder via the commandline" forinformation on the commandline interface, including a list of supported flags.

Steps to deployment


1. Install and configure the universal forwarder on a test machine, using thecommandline interface with the desired flags.


3. Load the universal forwarder MSI into your deployment tool, specifying thetested flags.

4. Execute deployment with your deployment tool.

5. Use the deployment monitor to verify that the universal forwarders arefunctioning.

Required installation flags

Besides specifying /quiet mode, you must include, at a minimum, thesecommandline flags:

AGREETOLICENSE=Yes• RECEIVING_INDEXER="<server:port>"•

110

At least one data input flag, such as WINEVENTLOG_APP_ENABLE=1. You canadd as many data input flags as you need.

•

See "Deploy a Windows universal forwarder via the commandline" for a list of allavailable commandline flags.

Example installation

This example sets the universal forwarder to run as Local System user, getinputs from Windows security and system event logs, send data to indexer1, andlaunch automatically:

msiexec.exe /i splunkuniversalforwarder_x86.msiRECEIVING_INDEXER="indexer1:9997" WINEVENTLOG_SEC_ENABLE=1WINEVENTLOG_SYS_ENABLE=1 AGREETOLICENSE=Yes /quiet

Deploy with a secure configuration

To deploy a secure configuration, you can specify an SSL certifcate. Use theseinstallation flags:

CERTFILE=<c:\path\to\certfile.pem>• ROOTCACERTFILE=<c:\path\to\rootcacertfile.pem>• CERTPASSWORD=<password>•

For more information, see this list of supported commandline flags.

Test the deployment


Deploy a *nix universal forwarder manually

This topic describes how to manually configure and deploy the universalforwarder in a *nix environment, such as Linux or Solaris. It assumes that you'reinstalling directly onto the *nix machine, rather than using a deployment tool. Thistype of deployment best suits these needs:

111

small deployments• proof-of-concept test deployments• system image or virtual machine for eventual cloning•

If you are interested in a different deployment scenario, look for another topic inthis section that better fits your needs.

Before following the procedures in this topic, read "Deployment overview".

Steps to deployment


1. Install the universal forwarder.

2. Configure (and optionally migrate) the universal forwarder.


4. Perform any additional post-installation configuration.


You can install the universal forwarder on a *nix machine using a package or atar file. To install the universal forwarder on any of the supported *nixenvironments, see the set of *nix install topics in the Installation manual:

Install on Linux• Install on Solaris• Install on Mac OS• Install on FreeBSD• Install on AIX• Install on HP-UX•

You install the universal forwarder the same way that you install a full Splunkinstance, as documented in these topics in the Installation manual. There areonly two differences:

The package name.• The default installation directory.•

112

The package name

When installing a package, substitute the name of the universal forwarderpackage for the full Splunk package name used in the commands in theInstallation manual.

For example, if installing the universal forwarder onto Red Hat Linux, use thiscommand:

rpm -i splunkforwarder_<package_name>.rpm

instead of this command for a full Splunk instance:

rpm -i splunk_<package_name>.rpm

The only difference is the prefix to the package name: "splunkforwarder", insteadof "splunk".

The default install directory

The universal forwarder installs by default in /opt/splunkforwarder. (The defaultinstall directory for full Splunk is /opt/splunk.)

Important: Do not install the universal forwarder over an existing installation offull Splunk. This is particuarly vital if you will be migrating from a light forwarderas described in "Migrate a nix light forwarder".

Configure the universal forwarder

The universal forwarder can run as any user on the local system. If you run theuniversal forwarder as a non-root user, make sure that it has the appropriatepermissions to read the inputs that you specify. Refer to the instructions forrunning Splunk as a non-root user for more information.

As part of configuration, you can migrate checkpoint settings from an existingforwarder to the universal forwarder. See "Deployment overview".

Use the Splunk CLI to start and configure your universal forwarders.

113

Start the universal forwarder

Important: If you want to migrate from an existing forwarder, you must perform aspecific set of actions before you start the universal forwarder for the first time.See "Migrate a nix forwarder" for details.

To start the universal forwarder, run the following command from$SPLUNK_HOME/bin directory (where $SPLUNK_HOME is the directory into which youinstalled the universal forwarder):

splunk start

Accept the license agreement automatically

The first time you start the universal forwarder after a new installation, you mustaccept the license agreement. To start the universal forwarder and accept thelicense in one step:

splunk start --accept-license

Note: There are two dashes before the accept-license option.

Configuration steps

After you start the universal forwarder and accept the license agreement, followthese steps to configure it:

1. Configure universal forwarder to auto-start:

splunk enable boot-start

2. Configure universal forwarder to act as a deployment client (optional). To dothis, just specify the deployment server:

splunk set deploy-poll <host>:<port>

where:

<host> is the deployment server's hostname or IP address and <port> isthe port it's listening on.

•

114

This step also automatically enables the deployment client functionality.

3. Configure the universal forwarder to forward to a specific receiving indexer,also known as the "receiver" (optional):

splunk add forward-server <host>:<port> -auth <username>:<password>

where:

<host> is the receiving indexer's hostname or IP address and <port> is theport it's listening on. By convention, the receiver listens for forwarders onport 9997, but it can be set to listen on any port, so you'll need to checkwith the receiver's administrator to obtain the port number. For informationon setting up a receiver, see "Enable a receiver".

•

<username>:<password> is the username and password for logging into theforwarder. By default, these are "admin:changeme" (To set a differentpassword than the default , issue the following command "splunk edit useradmin -password <new password> -role admin -auth admin:changeme").

•

During this step, you can also configure a certificate for secure intra-Splunkcommunications, using a set of optional ssl flags to specify a certificate, root CA,and password. For example:

splunk add forward-server <host>:<port> -ssl-cert-path /path/ssl.crt-ssl-root-ca-path /path/ca.crt -ssl-password <password>

Note: If you do not specify a receiving indexer, be sure to configure universalforwarder to act as a deployment client, as described in step 2, so that it can laterbe configured for a receiving indexer.

4. To configure the universal forwarder's inputs, use the CLI add command or editinputs.conf. See "About the CLI" and subsequent topics for details on using theCLI.

For a complete list of CLI commands supported in the universal forwarder, see"Supported CLI commands".

Test the deployment

Test your configured universal forwarder on a single machine, to make sure itfunctions correctly, before deploying the universal forwarder across your

115

environment. Confirm that the universal forwarder is getting the desired inputsand sending the right outputs to the indexer. You can use the deploymentmonitor to validate the universal forwarder.

If you migrated from an existing forwarder, make sure that the universalforwarder is forwarding data from where the old forwarder left off. If it isn't, youprobably need to modify or add data inputs, so that they conform to those on theold forwarder. Examine the two inputs.conf files to ensure that the newuniversal forwarder has all the inputs that you want to maintain.


See "Troubleshoot your deployment" for troubleshooting tips.


In addition to using the CLI, you can update the universal forwarder'sconfiguration by editing its configuration files, such as inputs.conf andoutputs.conf, directly. See "Deployment overview" for information.



If you need just a few universal forwarders, you might find it simpler just to repeatthe installation process manually, as documented in this topic. If you need toinstall a larger number of universal forwarders, however, it will probably be easierto deploy them remotely (using scripting or a deployment tool) or else as part of asystem image or virtual machine.

Troubleshoot your deployment

The universal forwarder forwards some internal logs to the receiving indexer.These are:

$SPLUNK_HOME/var/log/splunk/splunkd.log$SPLUNK_HOME/var/log/splunk/license_audit.log

116

The logs can be searched on the indexer for errors (index=_internalhost=<ua-machine>).

If the universal forwarder is malfunctioning such that it cannot forward the logs,use a text editor or grep to examine them on the universal forwarder machineitself.

Remotely deploy a *nix universal forwarder with astatic configuration

One of the main ways to deploy multiple universal forwarders remotely is throughscripting. You can also use deployment management tools such as yum andPuppet. This topic focuses on script deployment.

For information on how to install and configure a single universal forwarder, see"Deploy a nix universal forwarder manually". That topic explains how to installonto a wide variety of *nix platforms from a package or a tar file and how toconfigure (and optionally migrate) using the Splunk CLI.

Steps to deployment


1. Install and configure the universal forwarder on a test machine, as described in"Deploy a nix universal forwarder manually".

2. Test and tune the configuration.

3. Create a script wrapper for the installation and configuration commands.

4. Run the script on representative target machines to verify that it works with allrequired shells.

5. Execute the script against the desired set of hosts.

6. Use the deployment monitor to verify that the universal forwarders arefunctioning properly.

117

Create and execute the script

Once you've validated your installation and configuration process by testing afully configured universal forwarder, you're ready to incorporate the process intoa script.

Script requirements

You need to place the installation package or tar file in a network locationaccessible by the target machines. You can set this up so that the script pushesthe file over to each target host, or you can place the file in a generallyaccessible location, such as an NFS mount.

The script is responsible for error reporting. Logging to Splunk either directly orvia a flat file is recommended.

Sample script

Here's a sample script you can use as a starting point. Note that this is only anexample of the type of script you could create for your deployment. Thecomments in the script provide some guidance on how to modify it for yourneeds; however, the script will likely require further modification, beyond thatindicated by the comments.

Among other things, the script:

Deploys the forwarder's tar file to a list of hosts specified in a file that theHOST_FILE variable points to. You will need to provide this file, in theformat specified in the script comments.

•

Specifies the location on each destination host where the tar file will getunpacked.

•

Specifies a Splunk instance to serve as a deployment server that cansubsequently manage and update the forwarders. This is an optionalconfiguration step.

•

Starts the forwarder executable on each host.•

The script is well commented; be sure to study it carefully before modifying it foryour environment.

Here's the sample deployment script:

118

#!/bin/sh

# This script provides an example of how to deploy the Splunk universalforwarder# to many remote hosts via ssh and common Unix commands.## Note that this script will only work unattended if you have SSH hostkeys# setup & unlocked.# To learn more about this subject, do a web search for "openssh keymanagement".

# ----------- Adjust the variables below -----------

# Populate this file with a list of hosts that this script shouldinstall to,# with one host per line. You may use hostnames or IP addresses, as# applicable. You can also specify a user to login as, for example,"foo@host".## Example file contents:# server1# server2.foo.lan# you@server3# 10.2.3.4

HOSTS_FILE="/path/to/splunk.install.list"

# This is the path to the Splunk tar file that you wish to push out. You may# wish to make this a symlink to a versioned Splunk tar file, so as tominimize# updates to this script in the future.

SPLUNK_FILE="/path/to/splunk-latest.tar.gz"

# This is where the Splunk tar file will be stored on the remote hostduring# installation. The file will be removed after installation. Younormally will# not need to set this variable, as $NEW_PARENT will be used by default.## SCRATCH_DIR="/home/your_dir/temp"

# The location in which to unpack the new Splunk tar file on thedestination# host. This can be the same parent dir as for your existing Splunk# installation (if any). This directory will be created at runtime, ifit does# not exist.

119

NEW_PARENT="/opt"

# After installation, the forwarder will become a deployment client ofthis# host. Specify the host and management (not web) port of thedeployment server# that will be managing these forwarder instances. If you do not wishto use# a deployment server, you may leave this unset.## DEPLOY_SERV="splunkDeployMaster:8089"

# A directory on the current host in which the output of eachinstallation# attempt will be logged. This directory need not exist, but the userrunning# the script must be able to create it. The output will be stored as# $LOG_DIR/<[user@]destination host>. If installation on a host fails,a# corresponding file will also be created, as# $LOG_DIR/<[user@]destination host>.failed.

LOG_DIR="/tmp/splunkua.install"

# For conversion from normal Splunk installs to the Splunk UniversalAgent:# After installation, records of Splunk's progress in indexing files(monitor)# and filesystem change events (fschange) can be imported from anexisting# Splunk (non-forwarder) installation. Specify the path to thatinstallation here.# If there is no prior Splunk instance, you may leave this variableempty ("").## NOTE: THIS SCRIPT WILL STOP THE SPLUNK INSTANCE SPECIFIED HERE.## OLD_SPLUNK="/opt/splunk"

# If you use a non-standard SSH port on the remote hosts, you must setthis.# SSH_PORT=1234

# You must remove this line, or the script will refuse to run. This isto# ensure that all of the above has been read and set. :)

UNCONFIGURED=1

# ----------- End of user adjustable settings -----------

120

# helpers.

faillog() { echo "$1" >&2}

fail() { faillog "ERROR: $@" exit 1}

# error checks.

test "$UNCONFIGURED" -eq 1 && \ fail "This script has not been configured. Please see the notes inthe script."test -z "$HOSTS_FILE" && \ fail "No hosts configured! Please populate HOSTS_FILE."test -z "$NEW_PARENT" && \ fail "No installation destination provided! Please set NEW_PARENT."test -z "$SPLUNK_FILE" && \ fail "No splunk package path provided! Please populate SPLUNK_FILE."if [ ! -d "$LOG_DIR" ]; then mkdir -p "$LOG_DIR" || fail "Cannot create log dir at \"$LOG_DIR\"!"fi

# some setup.

if [ -z "$SCRATCH_DIR" ]; then SCRATCH_DIR="$NEW_PARENT"fiif [ -n "$SSH_PORT" ]; then SSH_PORT_ARG="-p${SSH_PORT}" SCP_PORT_ARG="-P${SSH_PORT}"fi

NEW_INSTANCE="$NEW_PARENT/splunkforwarder" # this would need to beedited for non-UA...DEST_FILE="${SCRATCH_DIR}/splunk.tar.gz"

### create script to run remotely.##

REMOTE_SCRIPT=" fail() { echo ERROR: \"\$@\" >&2 test -f \"$DEST_FILE\" && rm -f \"$DEST_FILE\" exit 1

121

}"

### try untarring tar file.REMOTE_SCRIPT="$REMOTE_SCRIPT (cd \"$NEW_PARENT\" && tar -zxf \"$DEST_FILE\") || fail \"could notuntar /$DEST_FILE to $NEW_PARENT.\""

### setup seed file to migrate input records from old instance, andstop old instance.if [ -n "$OLD_SPLUNK" ]; then REMOTE_SCRIPT="$REMOTE_SCRIPT echo \"$OLD_SPLUNK\" > \"$NEW_INSTANCE/old_splunk.seed\" || fail\"could not create seed file.\" \"$OLD_SPLUNK/bin/splunk\" stop || fail \"could not stop existingsplunk.\" "fi

### setup deployment client if requested.if [ -n "$DEPLOY_SERV" ]; then REMOTE_SCRIPT="$REMOTE_SCRIPT \"$NEW_INSTANCE/bin/splunk\" set deploy-poll \"$DEPLOY_SERV\"--accept-license --answer-yes \ --auto-ports --no-prompt || fail \"could not setup deploymentclient\" "fi

### start new instance.REMOTE_SCRIPT="$REMOTE_SCRIPT \"$NEW_INSTANCE/bin/splunk\" start --accept-license --answer-yes--auto-ports --no-prompt || \ fail \"could not start new splunk instance!\""

### remove downloaded file.REMOTE_SCRIPT="$REMOTE_SCRIPT rm -f "$DEST_FILE" || fail \"could not delete downloaded file$DEST_FILE!\""

### end of remote script.##

exec 5>&1 # save stdout.exec 6>&2 # save stderr.

122

echo "In 5 seconds, will copy install file and run the following scripton each"echo "remote host:"echoecho "===================="echo "$REMOTE_SCRIPT"echo "===================="echoecho "Press Ctrl-C to cancel..."test -z "$MORE_FASTER" && sleep 5echo "Starting."

# main loop. install on each host.

for DST in `cat "$HOSTS_FILE"`; do if [ -z "$DST" ]; then continue; fi

LOG="$LOG_DIR/$DST" FAILLOG="${LOG}.failed" echo "Installing on host $DST, logging to $LOG."

# redirect stdout/stderr to logfile. exec 1> "$LOG" exec 2> "$LOG"

if ! ssh $SSH_PORT_ARG "$DST" \ "if [ ! -d \"$NEW_PARENT\" ]; then mkdir -p \"$NEW_PARENT\"; fi";then touch "$FAILLOG" # restore stdout/stderr. exec 1>&5 exec 2>&6 continue fi

# copy tar file to remote host. if ! scp $SCP_PORT_ARG "$SPLUNK_FILE" "${DST}:${DEST_FILE}"; then touch "$FAILLOG" # restore stdout/stderr. exec 1>&5 exec 2>&6 continue fi

# run script on remote host and log appropriately. if ! ssh $SSH_PORT_ARG "$DST" "$REMOTE_SCRIPT"; then touch "$FAILLOG" # remote script failed. else test -e "$FAILLOG" && rm -f "$FAILLOG" # cleanup any past attemptlog.

123

fi

# restore stdout/stderr. exec 1>&5 exec 2>&6

if [ -e "$FAILLOG" ]; then echo " --> FAILED <--" else echo " SUCCEEDED" fidone

FAIL_COUNT=`ls "${LOG_DIR}" | grep -c '\.failed$'`if [ "$FAIL_COUNT" -gt 0 ]; then echo "There were $FAIL_COUNT remote installation failures." echo " ( see ${LOG_DIR}/*.failed )"else echo echo "Done."fi

# Voila.

Execute the script

Once you've executed the script, check any log files generated by yourinstallation script for errors. For example, the sample script saves to/tmp/splunkua.install/<destination hostname>.

Make a universal forwarder part of a system image

This topic describes how to deploy a universal forwarder as part of a systemimage or virtual machine. This is particularly useful if you have a large number ofuniversal forwarders to deploy. If you have just a few, you might find it simpler toinstall them manually, as described for Windows and nix machines.

Before following the procedures in this topic, read "Deployment overview".

Steps to deployment


124

1. Install the universal forwarder on a test machine. See below.

2. Perform any post-installation configuration, as described below, here.

3. Test and tune the deployment, as described below.

4. Install the universal forwarder with the tested configuration onto a sourcemachine.

5. Stop the universal forwarder.

6. Run this CLI command on the forwarder:

./splunk clone-prep-clear-config

This clears instance-specific information, such as the server name and GUID,from the forwarder. This information will then be configured on each clonedforwarder at initial start-up.

7. Prep your image or virtual machine, as necessary, for cloning.

8. On *nix systems, set the splunkd daemon to start on boot using cron or yourscheduling system of choice. On Windows, set the service to Automatic but donot start it.

9. Distribute system image or virtual machine clones to machines across yourenvironment and start them.

10. Use the deployment monitor to verify that the cloned universal forwardersare functioning.

Referenced procedures

Steps in the above deployment procedure reference these subtopics.


Install the universal forwarder using the procedure specific to your operatingsystem:

To install on a *nix machine, see "Deploy a nix universal forwardermanually".

•

125

For a Windows machine, you can use the installer GUI or thecommandline interface. To install with the GUI, see "Deploy a Windowsuniversal forwarder via the installer GUI". For information on thecommandline interface, see "Deploy a Windows universal forwarder viathe commandline".

•

Important: On a Windows machine, if you do not want the universal forwarder tostart immediately after installation, you must use the commandline interface.Using the proper commandline flags, you can configure the universal forwarderso that it does not start on the source machine when installed but does startautomatically on the clones, once they're activated.

At the time of installation, you can also configure the universal forwarder. See"General configuration issues" in the Deployment Overview.


You can update your universal forwarder's configuration, post-installation, bydirectly editing its configuration files, such as inputs.conf and outputs.conf.See "Deployment overview" for information.


Test the deployment


Migrate a Windows light forwarder

If you want to replace an existing light forwarder with a universal forwarder, youneed to first migrate its checkpoint data to the new forwarder. Checkpoint data isinternal data that the forwarder compiles to keep track of what data it has alreadyforwarded to an indexer. By migrating the checkpoint data, you prevent the newuniversal forwarder from forwarding any data already sent by the old lightforwarder. This ensures that the same data does not get indexed twice.

126

You can migrate checkpoint data from an existing Windows light forwarder(version 4.0 or later) to the universal forwarder. For an overview of migration, see"Migrating from a light forwarder" in the Deployment Overview.

If you want to migrate, you must do so during the installation process. Youcannot migrate post-installation.

You perform a Windows installation with either the installer GUI or thecommandline:

If you use the installer GUI, one of the screens will prompt you tomigrate. See "Deploy a Windows universal forwarder via the installer GUI"for a walkthrough of the GUI installation procedure.

•

If you install via the commandline, the flag MIGRATESPLUNK=1 specifiesmigration. See "Deploy a Windows universal forwarder via thecommandline" for a list of supported flags and how to use them toconfigure your installation.

•

Important: You must install the universal forwarder in a different directory fromthe existing light forwarder. Since the default install directory for the universalforwarder is C:\Program Files\SplunkUniversalForwarder and the default installdirectory for full Splunk (including the light forwarder) is C:\ProgramFiles\Splunk, you'll be safe if you just stick with the defaults.

What the installer does

Whichever installation method you use, the Windows installer performs thefollowing actions:

1. Searches for an existing heavy or light forwarder on the machine.

2. Determines whether the forwarder is eligible for migration (must be at version4.0 or above).

3. If it finds an eligible forwarder, the GUI offers the user the option of migrating.(The commandline installer looks to see whether the MIGRATESPLUNK=1 flagexists.)

4. If user specifies migration (or the MIGRATESPLUNK=1 flag exists), the installershuts down any running services (splunkd and, if running, splunkweb) for theexisting forwarder. It also sets the startup type of the services to manual, so thatthey don't start up again upon reboot.

127

5. Migrates the checkpoint files to the universal forwarder.

6. Completes installation and configuration of the universal forwarder.

What you need to do

At the end of this process, you might want to perform additional configuration onthe universal forwarder. Since the migration process only copies checkpoint files,you will probably want to manually copy over the old forwarder's inputs.confconfiguration file (or at least examine it, to determine what data inputs it wasmonitoring).

Once the universal forwarder is up and running (and after you've tested to ensuremigration worked correctly), you can uninstall the old forwarder.

Migrate a *nix light forwarder

If you want to replace an existing light forwarder with a universal forwarder, youneed to first migrate its checkpoint data to the new forwarder. Checkpoint data isinternal data that the forwarder compiles to keep track of what data it has alreadyforwarded to an indexer. By migrating the checkpoint data, you prevent the newuniversal forwarder from forwarding any data already sent by the old lightforwarder. This ensures that the same data does not get indexed twice.

You can migrate checkpoint data from an existing *nix light forwarder (version 4.0or later) to the universal forwarder. For an overview of migration, see "Migratingfrom a light forwarder" in the Deployment Overview.

Important: Migration can only occur the first time you start the universalforwarder, post-installation. You cannot migrate at any later point.

To migrate, do the following:

1. Stop any services (splunkd and splunkweb, if running) for the existingforwarder:

$SPLUNK_HOME/bin/splunk stop

2. Complete the basic installation of the universal forwarder, as described in"Deploy a nix universal forwarder manually". Do not yet start the universalforwarder.

128

Important: Make sure you install the universal forwarder into a different directoryfrom the existing light forwarder. Since the default install directory for theuniversal forwarder is /opt/splunkforwarder and the default install directory forfull Splunk (including the light forwarder) is /opt/splunk, you'll be safe if you juststick with the defaults.

3. In the universal forwarder's installation directory, (the new $SPLUNK_HOME),create a file named old_splunk.seed; in other words:$SPLUNK_HOME/old_splunk.seed. This file must contain a single line, consisting ofthe path of the old forwarder's $SPLUNK_HOME directory. For example:/opt/splunk.

4. Start the universal forwarder:

$SPLUNK_HOME/bin/splunk start

The universal forwarder will migrate the checkpoint files from the forwarderspecified in the $SPLUNK_HOME/old_splunk.seed file. Migration only occurs thefirst time you run the start command. You can leave the old_splunk.seed inplace; Splunk only looks at it the first time you start the forwarder after installingit.

5. Perform any additional configuration of the universal forwarder, as described in"Deploy a nix universal forwarder manually". Since the migration process onlycopies checkpoint files, you will probably want to manually copy over the oldforwarder's inputs.conf configuration file (or at least examine it, to determinewhat data inputs it was monitoring).

Once the universal forwarder is up and running (and after you've tested to ensuremigration worked correctly), you can uninstall the old forwarder.

Supported CLI commands

The universal forwarder supports a subset of objects for use in CLI commands.Certain objects valid in full Splunk, like index (as in add index), make no sensein the context of the universal forwarder.

Commands act upon objects. If you type an invalid command/object combination,the universal forwarder will return an error message.

129

Valid CLI objects

The universal forwarder supports all CLI commands for these objects:

ad app config datastore-dir default-hostname deploy-client deploy-poll eventlog exec forward-server monitor oneshot perfmon registry servername splunkd-port tcp udp user wmi

Note: A few commands, such as start and stop can be run without an object. Acommand with no object is also valid for the universal forwarder.

A brief introduction to CLI syntax

The general syntax for a CLI command is:

./splunk <command> [<object>] [[-<parameter>] <value>]...

As described above, it's the object that determines whether a command is validin the universal forwarder. For example, the above list includes the monitorobject. Therefore, the add monitor and edit monitor command/objectcombinations are both valid. For more information on the monitor object, see"Use the CLI to monitor files and directories" in the Getting Data In manual.

For more details on using the CLI in general, see the "Use Splunk's commandline interface" chapter in the Admin manual. In particular, the topic "CLI admincommands" provides details on CLI syntax, including a list of all commandssupported by full Splunk and the objects they can act upon.

130

Deploy heavy and light forwarders

Deploy a heavy or light forwarder


To enable forwarding and receiving, you configure both a receiver and aforwarder. The receiver is the Splunk instance receiving the data; the forwardersends data to the receiver.

You must first set up the receiver. You can then set up forwarders to send data tothat receiver.

Note: The receiver must be running the same (or later) version of Splunk as itsforwarder. A 4.0 receiver can receive data from a 3.4 forwarder, but a 3.4receiver cannot receive from a 4.0 forwarder.

Setting up a heavy or light forwarder is a two step process:

1. Install a full Splunk instance.

2. Enable forwarding on the Splunk instance.

The sections that follow describe these steps in detail.

Important: This topic describes deployment and configuration issues specific tolight and heavy forwarders. For information on how to deploy a universalforwarder, see "Universal forwarder deployment overview". For information onhow to enable receiver functionality on a Splunk instance, see "Enable areceiver".

Install a full Splunk instance

To deploy a light or heavy forwarder, you must first install a full Splunk instance.For detailed information about installing Splunk, including system requirementsand licensing issues, see the Installation manual.

Once the Splunk instance has been installed, you can enable forwarderfunctionality on it. You can also determine whether the forwarder should be a

131

light forwarder or a heavy forwarder. For information on the differences betweenthese types of forwarders, look here.

Set up forwarding

You can use Splunk Web or the Splunk CLI as a quick way to enable forwardingin a Splunk instance.

You can also enable, as well as configure, forwarding by creating anoutputs.conf file for the Splunk instance. Although setting up forwarders withoutputs.conf requires a bit more initial knowledge, there are obvious advantagesto performing all forwarder configurations in a single location. Most advancedconfiguration options are available only through outputs.conf. In addition, if youwill be enabling and configuring a number of forwarders, you can easilyaccomplish this by editing a single outputs.conf file and making a copy for eachforwarder. See the topic "Configure forwarders with outputs.conf" for moreinformation.

Note: When you install a Splunk instance to be used as a light forwarder, selectthe forwarder license. You can then enable the light forwarder, as describedbelow. For a heavy forwarder that performs indexing, you need an Enterpriselicense. For more information, see "Types of Splunk licenses".

Set up heavy forwarding with Splunk Web

Use Splunk Manager to set up a forwarder.

To set up a heavy forwarder:

1. Log into Splunk Web as admin on the server that will be forwarding data.



4. Click Add new in the Forward data section.

5. Enter the hostname or IP address for the receiving Splunk instance(s), alongwith the receiving port specified when the receiver was configured. Forexample, you might enter: receivingserver.com:9997. You can enter multiplehosts as a comma-separated list.

6. Click Save. You must restart Splunk to complete the process.

132

You can use Splunk Web to perform one other configuration (for heavyforwarders only). To store a copy of indexed data local to the forwarder:

1. From Forwarding and receiving, select Forwarding defaults.

2. Select Yes to store and maintain a local copy of the indexed data on theforwarder.

Important: A heavy forwarder has a key advantage over light and universalforwarders in that it can index your data locally, as well as forward the data toanother Splunk index. However, local indexing is turned off by default. If youwant to store data on the forwarder, you must enable that capability - either in themanner described above or by editing outputs.conf.

All other configuration must be done in outputs.conf.

Set up light forwarding with Splunk Web

Use Splunk Manager to set up a forwarder.

To enable light forwarding, you must first enable heavy forwarding on the Splunkinstance. Then you separately enable light forwarding. This procedure combinesthe two processes:

1. Log into Splunk Web as admin on the server that will be forwarding data.



4. Click Add new in the Forward data section.

5. Enter the hostname or IP address for the receiving Splunk instance, along withthe receiving port specified when the receiver was configured. For example, youmight enter: receivingserver.com:9997.

6. Click Save.

7. Return to Manager>Forwarding and receiving.

8. Click Enable lightweight forwarding in the Forward data section. You mustrestart Splunk to complete the process.

133

Important: When you enable a light forwarder, Splunk Web is immediatelydisabled. You will then need to use the Splunk CLI or outputs.conf to performany further configuration on the forwarder. Therefore, if you want to use SplunkWeb to configure your forwarder, do so before you enable light forwarding.

Set up forwarding with the Splunk CLI

With the CLI, setting up forwarding is a two step process. First you enableforwarding on the Splunk instance. Then you start forwarding to a specifiedreceiver.


To enable the forwarder mode, enter:

./splunk enable app [SplunkForwarder|SplunkLightForwarder] -auth<username>:<password>

Note: In the CLI enable command, SplunkForwarder represents the heavyforwarder.

Important: After this step, make sure you restart your Splunk instance asindicated! Attempting to start forwarding activity using the CLI before restartingsplunkd will not work!

To disable the forwarder mode, enter:

./splunk disable app [SplunkForwarder|SplunkLightForwarder] -auth<username>:<password>

By disabling forwarding, this command reverts the Splunk instance to a fullserver.

Start forwarding activity from the Splunk CLI


To start forwarding activity, enter:

134

./splunk add forward-server <host>:<port> -auth <username>:<password>

To end forwarding activity, enter:

./splunk remove forward-server <host>:<port> -auth<username>:<password>

Note: Although this command ends forwarding activity, the Splunk instanceremains configured as a forwarder. To revert the instance to a full Splunk server,use the disable command:

./splunk disable app [SplunkForwarder|SplunkLightForwarder] -auth<username>:<password>

Important: Make sure you restart your Splunk instance as indicated by the CLIto take these changes into account.

Upgrade a forwarder

To upgrade a forwarder to a new version, just upgrade the Splunk instance in theusual fashion. For details, read the upgrade section of the Installation manual.

Important: Before doing an upgrade, consider whether you really need to. Inmany cases, there's no compelling reason to upgrade a forwarder. Forwardersare always compatible with later version indexers, so you do not need to upgradethem just because you've upgraded the indexers they're sending data to.

Back up your files first

Before you perform the upgrade, we strongly recommend that you back up all ofyour files. Most importantly, back up your Splunk configuration files. Forinformation on backing up configurations, read "Back up configurationinformation" in the Admin manual.

If you're upgrading a heavy forwarder that's indexing data locally, you also needto back up the indexed data. For information on backing up data, read "Back upindexed data" in the Managing Indexers and Clusters manual.

Splunk does not provide a means of downgrading to previous versions; if youneed to revert to an older forwarder release, just reinstall it.

135

Heavy and light forwarder capabilities

Certain capabilities are disabled in heavy and light forwarders. This sectiondescribes forwarder capabilities in detail.


Splunk heavy forwarder details

The heavy forwarder has all Splunk functions and modules enabled by default,with the exception of the distributed search module. The file$SPLUNK_HOME/etc/apps/SplunkForwarder/default/default-mode.conf includesthis stanza:

[pipeline:distributedSearch]disabled = true

For a detailed view of the exact configuration, see the configuration files for theSplunkForwarder application in$SPLUNK_HOME/etc/apps/SplunkForwarder/default.

Splunk light forwarder details

Most features of Splunk are disabled in the Splunk light forwarder. Specifically,the Splunk light forwarder:

Disables event signing and checking whether the disk is full($SPLUNK_HOME/etc/apps/SplunkLightForwarder/default/default-mode.conf).

•

Limits internal data inputs to splunkd and metrics logs only, and makessure these are forwarded($SPLUNK_HOME/etc/apps/SplunkLightForwarder/default/inputs.conf).

•

Disables all indexing($SPLUNK_HOME/etc/apps/SplunkLightForwarder/default/indexes.conf).

•

Does not use transforms.conf and does not fully parse incoming data,but the CHARSET, CHECK_FOR_HEADER, NO_BINARY_CHECK,PREFIX_SOURCETYPE, and sourcetype properties from props.conf are used.

•

Disables the Splunk Web interface($SPLUNK_HOME/etc/apps/SplunkLightForwarder/default/web.conf ).

•

Limits throughput to 256KBps($SPLUNK_HOME/etc/apps/SplunkLightForwarder/default/limits.conf).

•

136

Disables the following modules in$SPLUNK_HOME/etc/apps/SplunkLightForwarder/default/default-mode.conf:

•

[pipeline:indexerPipe] disabled_processors= indexandforward, diskusage,signing,tcp-output-generic-processor, syslog-output-generic-processor,http-output-generic-processor, stream-output-processor

[pipeline:distributedDeployment] disabled = true

[pipeline:distributedSearch] disabled = true

[pipeline:fifo] disabled = true

[pipeline:merging] disabled = true

[pipeline:typing] disabled = true

[pipeline:udp] disabled = true

[pipeline:tcp] disabled = true

[pipeline:syslogfifo] disabled = true

[pipeline:syslogudp] disabled = true

[pipeline:parsing] disabled_processors=utf8, linebreaker, header, sendOut

[pipeline:scheduler] disabled_processors = LiveSplunks

These modules include the deployment server (not the deployment client),distributed search, named pipes/FIFOs, direct input from network ports, and thescheduler.

The defaults for the light forwarder can be tuned to meet your needs byoverriding the settings in$SPLUNK_HOME/etc/apps/SplunkLightForwarder/default/default-mode.conf on

137

a case-by-case basis.

Purge old indexes

When you convert a Splunk indexer instance to a light forwarder, among otherthings, you disable indexing. In addition, you no longer have access to any datapreviously indexed on that instance. However, the data still exists.

If you want to purge that data from your system, you must first disable theSplunkLightForwarder app, then run the CLI clean command, and then renablethe app. For information on the clean command, see "Remove indexed data fromSplunk" in the Managing Indexers and Clusters manual.

138

Search across multiple indexers

About distributed search

In distributed search, a Splunk instance called a search head sends searchrequests to a group of Splunk indexers, which perform the actual searches ontheir indexes. The search head then merges the results back to the user. In atypical scenario, one search head manages searches on several indexers.

These are some of the key use cases for distributed search:

Horizontal scaling for enhanced performance. Distributed searchfacilitates horizontal scaling by providing a way to distribute the indexingand searching loads across multiple Splunk components, making itpossible to search and index large quantities of data.

•

Access control. You can use distributed search to control access toindexed data. In a typical situation, some users, such as securitypersonnel, might need access to data across the enterprise, while othersneed access only to data in their functional area.

•

Managing geo-dispersed data. Distributed search allows local offices toaccess their own data, while maintaining centralized access at thecorporate level. Chicago and San Francisco can look just at their localdata; headquarters in New York can search its local data, as well as thedata in Chicago and San Francisco.

•

The Splunk instance that does the searching is referred to as the search head.The Splunk indexers that participate in a distributed search are called searchpeers or indexer nodes.

A search head can also index and serve as a search peer. However, inperformance-based use cases, such as horizontal scaling, it is recommendedthat the search head only search and not index. In that case, it is referred to as adedicated search head.

A search head by default runs its searches across all its search peers. You canlimit a search to one or more search peers by specifying the splunk_server fieldin your query. See "Search across one or more distributed servers" in the Searchmanual.

139

You can run multiple search heads across a set of search peers. To coordinatethe activity of multiple search heads (so that they share configuration settings,search artifacts, and job management), you need to enable search head pooling.

Some search scenarios

This diagram shows a simple distributed search scenario for horizontal scaling,with one search head searching across three peers:

In this diagram showing a distributed search scenario for access control, a"security" department search head has visibility into all the indexing searchpeers. Each search peer also has the ability to search its own data. In addition,the department A search peer has access to both its data and the data ofdepartment B:

Finally, this diagram shows load balancing with distributed search. There's adedicated search head and a search head on each indexer. All the search headscan search across the entire set of indexers:

140

For more information on load balancing, see "Set up load balancing" in thismanual.

For information on Splunk distributed searches and capacity planning, see"Dividing up indexing and searching" in the Installation manual.

Search heads and clusters

In index replication, clusters use search heads to search across the set ofindexers, or peer nodes. You deploy and configure search heads very differentlywhen they are part of a cluster. To learn more about search heads and clusters,read "Configure the search head" in the Managing Indexers and ClustersManual.

What search heads send to search peers

When initiating a distributed search, the search head replicates and distributes itsknowledge objects to its search peers. Knowledge objects include savedsearches, event types, and other entities used in searching across indexes. Thesearch head needs to distribute this material to its search peers so that they canproperly execute queries on its behalf. The set of data that the search headdistributes is called the knowledge bundle.

The indexers use the search head's knowledge bundle to execute queries on itsbehalf. When executing a distributed search, the indexers are ignorant of anylocal knowledge objects. They have access only to the objects in the searchhead's knowledge bundle.

The process of distributing knowledge bundles means that indexers by defaultreceive nearly the entire contents of all the search head's apps. If an appcontains large binaries that do not need to be shared with the indexers, you canreduce the size of the bundle by means of the [replicationWhitelist] or[replicationBlacklist] stanza in distsearch.conf. See "Limit knowledgebundle size" in this manual.

141

The knowledge bundle gets distributed to the$SPLUNK_HOME/var/run/searchpeers/ directory on each search peer. Becausethe search head distributes its knowledge, search scripts should not hardcodepaths to resources. The knowledge bundle will reside at a different location onthe search peer's file system, so hardcoded paths will not work properly.

By default, the search head replicates and distributes the knowledge bundle toeach search peer. For greater efficiency, you can instead tell the search peers tomount the knowledge bundle's directory location, eliminating the need for bundlereplication. When you mount a knowledge bundle, it's referred to as a mountedbundle. To learn how to mount bundles, read "Mount the knowledge bundle".

User authorization

All authorization for a distributed search originates from the search head. At thetime it sends the search request to its search peers, the search head alsodistributes the authorization information. It tells the search peers the name of theuser running the search, the user's role, and the location of the distributedauthorize.conf file containing the authorization information.

Licenses for distributed deployments

Each indexer node in a distributed deployment must have access to a licensepool.

Search heads performing no indexing or only summary indexing can use theforwarder license. If the search head performs any other type of indexing, it musthave access to a license pool.

See "Licenses for search heads" in the Installation manual for a detaileddiscussion of licensing issues.

Cross-version compatibility

It's recommended that you upgrade search heads and search peers to any newversion at the same time to take full advantage of the latest search capabilities.This section describes the consequences of deploying multi-version distributedsearch for specific scenarios.

142

All search nodes must be 4.x or later

All search nodes must be running Splunk 4.x or 5.x to participate in thedistributed search. Distributed search is not backwards compatible with Splunk3.x.

Search nodes and 5.0 features

You need to upgrade both search heads and search peers to version 5.0 to takeadvantage of search capabilities that are new to 5.0, such as report acceleration.

Compatibility between 4.3 search heads and 4.2 search peers

You can run 4.3 search heads across 4.2 search peers, but some 4.3search-related functionality will not be available. These are the main features thatrequire 4.3 search peers:

The spath search command.• Bloom filters.• Historical backfill for real-time data from the search peers.•

Also, note that 4.3-specific stats/chart/timechart functionality is less efficientwhen used against 4.2.x search peers because the search peers can't providemap/reduce capability for that functionality. The functionality affected includessparklines and the earliest() and latest() functions.

Compatibility between 4.2.5+ search heads and pre-4.2.5 search peers

Because of certain feature incompatibilities, pre-4.2.5 search peers can consume20-30% more CPU resources when deployed with a 4.2.5 or later search head.You might see error messages such as "ConfObjectManagerDB - Ignoring invaliddatabase setting" in splunkd.log on the search peers.

Bundle replication warning when running a 4.2 search head against a 4.1.xsearch peer

Bundle replication is the process by which the search head distributes knowledgebundles, containing the search-time configurations, to its search peers. Thisensures that all peers run searches using the same configurations, so that, forexample, all peers use the same definition of an event type.

Starting with 4.2, bundle replication occurs asynchronously. The search headperforms bundle replication in a non-blocking fashion that allows in-progress

143

searches to continue on the search peers. When issuing searches, the searchhead specifies the bundle version that the peers must use to run those searches.The peers will not start using the newly replicated bundles until the search headconfirms that all peers have the latest bundle version.

However, the new 4.2 search head behavior can cause pre-4.2 search peers toget out of sync and use different bundles when running their searches. If you runa 4.2 search head against 4.1.x search peers, you'll get this warning message:"Asynchronous bundle replication might cause (pre 4.2) search peers to runsearches with different bundle/config versions. Results might not be correct."

Note: This issue goes away in 4.2.1 search heads. Starting with 4.2.1, thesearch head will revert to synchronized bundled replication if any of the searchpeers is pre-4.2.

Install a dedicated search head

Distributed search is enabled by default on every Splunk instance, with theexception of forwarders. This means that every Splunk server can function as asearch head to a specified group of indexers, referred to as search peers.

In some cases, you might want a single Splunk instance to serve as both asearch head and a search peer. In other cases, however, you might want to setup a dedicated search head. A dedicated search head performs only searching;it does not do any indexing.

Note: If you do want to use a Splunk instance as both a search head and asearch peer, or otherwise perform indexing on the search head, just install thesearch head as a regular Splunk instance with a normal license, as described in"About Splunk licenses" in the Installation manual.

To install a dedicated search head, follow these steps:

1. Determine your hardware needs by reading this topic in the Installationmanual.

2. Install Splunk, as described in the topic in the Installation manual specific toyour operating system.

3. Add the search head to your Enterprise license group even if it's a dedicatedsearch head that's not expected to index any data. For more information, see"Types of Splunk licenses".

144

4. Establish distributed search from the search head to all the indexers, or"search peers", you want it to search. See "Configure distributed search" for howto do this.

5. Log in to your search head and do a search for *. Look at your results andcheck the splunk_server field. Verify that all your search peers are listed in thatfield.

6. Set up the authentication method you want to use on the search head, just asyou would for any other Splunk instance. Do not set up any indexing on yoursearch head, since that will violate its license.

Note: Clusters use search heads to search across the set of indexers, or peernodes. You deploy search heads very differently when they are part of a cluster.To learn more about deploying search heads in clusters, read "Enable the searchhead" in the Managing Indexers and Clusters Manual.

Configure distributed search

Distributed search is available on every Splunk server, with the exception offorwarders. This means that every Splunk server can function as a search headto a specified group of indexers, referred to as search peers. The distributedsearch capability is enabled by default.

To activate distributed search, a Splunk instance that you designate as a searchhead just needs to add search peers. Ordinarily, you do this by specifying eachsearch peer manually. A number of other configuration options are available, butordinarily you do not need to alter their default values.

When configuring Splunk instances as search heads or search peers, keep thiskey distinction in mind:

A search head must maintain a list of search peers, or it will have nothingto search on. A dedicated search head does not have additional datainputs, beyond the default ones.

•

A search peer must have specified data inputs, or it will have nothing toindex. A search peer does not maintain a list of other search peers.

•

These roles are not necessarily distinct. A Splunk instance can, and frequentlydoes, function simultaneously as both a search head and a search peer.

145

Important: Clusters also use search heads to search across the set of indexers,or peer nodes. You deploy and configure search heads very differently whenthey are part of a cluster. To learn more about configuring search heads inclusters, read "Configure the search head" in the Managing Indexers andClusters Manual.

Configuration overview

You set up distributed search on a search head using any of these methods:

Splunk Web• Splunk CLI• The distsearch.conf configuration file•

Splunk Web is the recommended method for most purposes.

The configuration happens on the designated search head. The main step is tospecify the search head's search peers. The distributed search capability itself isalready enabled by default.

Important: Before an indexer can function as a search peer, you must change itspassword from the default "changeme". Otherwise, the search head will not beable to authenticate against it.

Aside from changing the password, no configuration is generally necessary onthe search peers. Access to the peers is controllable through public keyauthentication. However, you do need to perform some configuration on thesearch peers if you mount the knowledge bundles, so that the peers can accessthem. See "Mount the knowledge bundle" for details.

Use Splunk Web

Specify the search peers

You specify the search peers through Manager on the search head:

1. Log into Splunk Web and click Manager.

2. Click Distributed search in the Distributed Environment area.

3. Click Search peers.

4. On the Search peers page, select New.

146

5. Specify the search peer, along with any authentication settings.

6. Click Save.

7. Repeat for each of the search head's search peers.

Configure miscellaneous distributed search settings

To configure other settings:

1. Log into Splunk Web and click Manager.

2. Click Distributed search in the Distributed Environment area.

3. Click Distributed search setup.

5. Change any settings as needed.

6. Click Save.

Use the CLI

Follow these instructions to enable distributed search via Splunk's CLI.

To use Splunk's CLI, navigate to the $SPLUNK_HOME/bin/ directory and use the./splunk command.

Enable distributed search

Distributed search is enabled by default, so this step is ordinarily not required:

splunk enable dist-search -auth admin:password

Add a search peer

Use the splunk add search-server command to add a search peer. When yourun this command, make sure you specify the splunkd management port of thepeer server. By default, this is 8089, although it might be different for yourdeployment.

Be sure to provide credentials for both the local and the remote machines. Usethe -auth flag for your local credentials and the -remoteUsername and

147

-remotePassword flags to specify the remote credentials (in this example, forsearch peer 10.10.10.10):

splunk add search-server -host 10.10.10.10:8089 -auth admin:password-remoteUsername admin -remotePassword passremote

A message indicates success, along with the need to restart the server.

Edit distsearch.conf

In most cases, the settings available through Splunk Web provide sufficientoptions for configuring distributed search environments. Some advancedconfiguration settings, however, are only available through distsearch.conf. Editthis file in $SPLUNK_HOME/etc/system/local/, or your own custom applicationdirectory in $SPLUNK_HOME/etc/apps/.

For the detailed specification and examples, see distsearch.conf.

For more information on configuration files in general, see "About configurationfiles".

Distribute the key files

If you add search peers via Splunk Web or the CLI, Splunk automatically handlesauthentication. However, if you add peers by editing distsearch.conf, you mustdistribute the key files manually.

After enabling distributed search on a Splunk instance (and restarting theinstance), you will find the keys in this location:$SPLUNK_HOME/etc/auth/distServerKeys/

Distribute the file $SPLUNK_HOME/etc/auth/distServerKeys/trusted.pem on thesearch head to$SPLUNK_HOME/etc/auth/distServerKeys/<searchhead_name>/trusted.pem onthe indexer nodes.

Support for keys from multiple Splunk instances

Any number of Splunk instances can have their certificates stored on otherinstances for authentication. The instances can store keys in$SPLUNK_HOME/etc/auth/distServerKeys/<peername>/trusted.pem

148

For example, if you have Splunk search heads A and B and they both need tosearch Splunk index node C, do the following:

1. On C, create $SPLUNK_HOME/etc/auth/distServerKeys/A/ andetc/auth/distServerKeys/B/.

2. Copy A's trusted.pem to $SPLUNK_HOME/etc/auth/distServerKeys/A/ and B'strusted.pem to $SPLUNK_HOME/etc/auth/distServerKeys/B/.

3. Restart C.

Limit the knowledge bundle size

The knowledge bundle is the data that the search head replicates anddistributes to each search peer to enable its searches. For information on thecontents and purpose of this bundle, see "What search heads send to searchpeers".

The knowledge bundle can grow quite large, as, by default, it includes nearly theentire contents of all the search head's apps. To limit the size of the bundle, youcan create a replication whitelist. To do this, edit distsearch.conf and specify a[replicationWhitelist] stanza:

[replicationWhitelist]<name> = <whitelist_regex>...

All files that satisfy the whitelist regex will be included in the bundle that thesearch head distributes to its search peers. If multiple regex's are specified, thebundle will include the union of those files.

In this example, the knowledge bundle will include all files with extensions ofeither ".conf" or ".spec":

[replicationWhitelist]allConf = *.confallSpec = *.spec

The names, such as allConf and allSpec, are used only for layering. That is, ifyou have both a global and a local copy of distsearch.conf, the local copy canbe configured so that it overrides only one of the regex's. For instance, assumethat the example shown above is the global copy. Assume you then specify a

149

whitelist in your local copy like this:

[replicationWhitelist]allConf = *.foo.conf

The two conf files will be layered, with the local copy taking precedence. Thus,the search head will distribute only files that satisfy these two regex's:

allConf = *.foo.confallSpec = *.spec

For more information on attribute layering in configuration files, see "Attributeprecedence" in the Admin manual.

You can also create a replication blacklist, using the [replicationBlacklist]stanza. The blacklist takes precedence over any whitelist. See distsearch.conf inthe Admin manual for details.

As an alternative to replicating and distributing a knowledge bundle, large orsmall, to search peers, you can mount the knowledge bundle on shared storage.For more information, read "Mount the knowledge bundle".

Manage distributed server names

The name of each search head and search peer is determined by its serverNameattribute, specified in server.conf. serverName defaults to the server's machinename.

In a distributed search cluster, all nodes must have unique names. TheserverName has three specific uses:

For authenticating search heads. When search peers are authenticating asearch head, they look for the search head's key file in/etc/auth/distServerKeys/<searchhead_name>/trusted.pem.

•

For identifying search peers in search queries. serverName is the value ofthe splunk_server field that you specify when you want to query a specificnode. See "Search across one or more distributed servers" in the Searchmanual.

•

For identifying search peers in search results. serverName gets reportedback in the splunk_server field.

•

150

Note: serverName is not used when adding search peers to a search head. Inthat case, you identify the search peers through their domain names or IPaddresses.

The only reason to change serverName is if you have multiple instances of Splunkresiding on a single machine, and they're participating in the same distributedsearch cluster. In that case, you'll need to change serverName to distinguishthem.

Remove a search peer

You can remove a search peer from a search head through the Distributedsearch page on Manager. However, this only removes the search peer entryfrom the search head; it does not remove the search head key from the searchpeer. In most cases, this is not a problem and no further action is needed.

If you need to disable the trust relationship between a search peer and a searchhead, you can delete the search-head-specific trusted.pem file from the directory$SPLUNK_HOME/etc/auth/distServerKeys/<searchhead_name>. It's unlikely thatyou'll need to do this.

Mount the knowledge bundle

The set of data that a search head distributes to its search peers is called theknowledge bundle. For information on the contents and purpose of this bundle,see "What search heads send to search peers".

The bundle contents reside in the search head's$SPLUNK_HOME/etc/{apps,users,system} subdirectories.

By default, the search head replicates and distributes the knowledge bundle toeach search peer. For greater efficiency, you can instead tell the search peers tomount the knowledge bundle's directory location, eliminating the need for bundlereplication. When you mount a knowledge bundle on shared storage, it's referredto as a mounted bundle.

Important: Most shared storage solutions don't work well across a WAN. Sincemounted bundles require shared storage, you generally should not implementthem across a WAN.

151

Why use mounted bundles

Mounted bundles are useful if you have large amounts of search-time data,which could otherwise slow the replication process. One common cause of slowbundle replication is large lookup tables.

Splunk writes a warning in splunkd.log if bundle replication is taking a long time.For example:

DistributedBundleReplicationManager - bundle replication to 2 peer(s)tooktoo long (12943ms), bundle file size=296150KB ...

Mounted bundle architectures

Depending on your search head configuration, there are a number of ways to setup mounted bundles. These are some of the typical ones:

For a single search head. Mount the knowledge bundle on sharedstorage. All the search peers then access the bundle to process searchrequests. This diagram illustrates a single search head with a mountedbundle on shared storage:

•

For multiple pooled search heads. For multiple search heads, you cancombine mounted bundles with search head pooling. The pooled searchheads maintain one bundle on the shared storage, and all search peersaccess that bundle. This diagram shows search head pooling with amounted bundle:

•

152

For multiple non-pooled search heads. Maintain the knowledgebundle(s) on each search head's local storage (no search head pooling).In this diagram, each search head maintains its own bundle, which eachsearch peer mounts and accesses individually:

•

There are numerous other architectures you can design with mounted bundles.You could, for example, use shared storage for multiple search heads, butwithout search head pooling. On the shared storage, you would maintainseparate bundles for each search head. The search peers would need to accesseach bundle individually.

In each case, the search peers need access to each search head's$SPLUNK_HOME/etc/{apps,users,system} subdirectories. In the case of searchhead pooling, the search peers need access to the pool's shared set ofsubdirectories.

Important: The search peers use the mounted directories only when fulfilling thesearch head's search requests. For indexing and other purposes not directlyrelated to distributed search, the search peers will use their own, local apps,users, and system directories, the same as any other indexer.

Configure mounted bundles

To set up mounted bundles, you need to configure both the search head and itssearch peers. The procedures described here assume the bundles are on shared

153

storage, but they do not need to be. They just need to be in some location thatboth the search head and its search peers can access.

Note: It's best not to locate mounted bundles in the search head's local$SPLUNK_HOME path.

These procedures also assume a single search head (no search head pooling).For details on how to configure mounted bundles with search head pooling, see"Use mounted bundles with search head pooling" below.

Configure the search head

Here are the steps you take on the search head:

1. Mount the bundle subdirectories ($SPLUNK_HOME/etc/{apps,users,system}) onshared storage. The simplest way to do this is to mount the search head's entire$SPLUNK_HOME/etc directory:

On *nix platforms, set up an NFS mount.•

On Windows, set up a CIFS (SMB) share.•

Important: The search head's Splunk user account needs read/write access tothe shared storage location. The search peers need read access to the bundlesubdirectories.

2. In the distsearch.conf file on the search head, set:

shareBundles=false

This stops the search head from replicating bundles to the search peers.

3. Restart the search head.

Configure the search peers

For each search peer, follow these steps to access the mounted bundle:

1. Mount the bundle directory on the search peer.

2. Create a distsearch.conf file in $SPLUNK_HOME/etc/system/local/ on thesearch peer. For each search head that the peer is connected to, create a

154

[searchhead:<searchhead-splunk-server-name>] stanza, with these attributes:

[searchhead:<searchhead-splunk-server-name>]mounted_bundles=truebundles_location=<path_to_bundles>

Note the following:

The search peer's configuration file must contain only the[searchhead:<searchhead-splunk-server-name>] stanza(s). The otherstanzas in distsearch.conf are for search heads only.

•

To identify the <searchhead-splunk-server-name>, run this command onthe search head:

•

splunk show servername

The <path_to_bundles> needs to specify the mountpoint on the searchpeer, not on the search head. For example, say $SPLUNK_HOME on yoursearch head is /opt/splunk, and you export /opt/splunk/etc via NFS.Then, on the search peer, you mount that NFS share at/mnt/splunk-head. The value of <path_to_bundles> should be/mnt/splunk-head, not /opt/splunk.

•

Important: If multiple search heads will be distributing searches to this searchpeer, you must create a separate stanza on the search peer for each of them.This is necessary even if you're using search head pooling.

3. Restart the search peer.

Note: You can optionally set up symbolic links to the bundle subdirectories(apps,users,system) to ensure that the search peer has access only to thenecessary subdirectories in the search head's /etc directory. See the followingexample for details on how to do this.

Example configuration

Here's an example of how to set up mounted bundles on shared storage:

155

Search head

On a search head whose Splunk server name is "searcher01":

1. Mount the search head's $SPLUNK_HOME/etc directory to shared storage withread/write access.

2. In the distsearch.conf file on the search head, set:

[distributedSearch]...shareBundles = false

3. Restart the search head.

Search peers

For each search peer:

1. Mount the search head's $SPLUNK_HOME/etc directory on the search peer to:

/mnt/searcher01

2. (Optional.) Create a directory that consists of symbolic links to the bundlesubdirectories:

/opt/shared_bundles/searcher01 /opt/shared_bundles/searcher01/system -> /mnt/searcher01/system /opt/shared_bundles/searcher01/users -> /mnt/searcher01/users /opt/shared_bundles/searcher01/apps -> /mnt/searcher01/apps

Note: This optional step is useful for ensuring that the peer has access only tothe necessary subdirectories.

3. Create a distsearch.conf file in $SPLUNK_HOME/etc/system/local/ on thesearch peer, with this stanza:

[searchhead:searcher01]mounted_bundles = truebundles_location = /opt/shared_bundles/searcher01

4. Restart the search peer.

156

5. Repeat the process for each search peer.

Use mounted bundles with search head pooling

The process for configuring mounted bundles is basically no different if you'reusing search head pooling to manage multiple search heads. A few things tokeep in mind:

Use the same shared storage location for both the search head pool andthe mounted bundles. Search head pooling uses a subset of thedirectories required for mounted bundles.

•

Search head pooling itself only requires that you mount the$SPLUNK_HOME/etc/{apps,users} directories. However, when usingmounted bundles, you must also provide a mounted$SPLUNK_HOME/etc/system directory. This doesn't create any conflictamong the search heads, as they will always use their own versions of thesystem directory and ignore the mounted version.

•

The search peers must create separate stanzas in distsearch.conf foreach search head in the pool. The bundles_location in each of thosestanzas must be identical.

•

See "Configure search head pooling" for information on setting up a search headpool.

Example configuration: Search head pooling with mounted bundles

This example shows how to combine search head pooling and mounted bundlesin one system. There are two main sections to the example:

1. Set up a search head pool consisting of two search heads. In this part, youalso mount the bundles.

2. Set up the search peers so that they can access bundles from the search headpool.

The example assumes you're using an NFS mount for the shared storagelocation.

Part 1: Set up the search head pool

Before configuring the pool, perform these preliminary steps:

157

1. Enable two Splunk instances as search heads. This example assumes that theinstances are named "searcher01" and "searcher02".

2. Set up a shared storage location accessible to each search head. Thisexample assumes that you set up an NFS mountpoint, specified on the searchheads as /mnt/search-head-pooling.

For detailed information on these steps, see "Create a pool of search heads".

Now, configure the search head pool:

1. On each search head, stop splunkd:

splunk stop splunkd

2. On each search head, enable search head pooling. In this example, you'reusing an NFS mount of /mnt/search-head-pooling as your shared storagelocation:

splunk pooling enable /mnt/search-head-pooling [--debug]

Among other things, this step creates empty /etc/apps and /etc/usersdirectories under /mnt/search-head-pooling. Step 3 uses those directories.

3. Copy the contents of the $SPLUNK_HOME/etc/apps and $SPLUNK_HOME/etc/usersdirectories on one of the search heads into the /etc/apps and /etc/userssubdirectories under /mnt/search-head-pooling:

cp -r $SPLUNK_HOME/etc/apps/* /mnt/search-head-pooling/etc/apps

cp -r $SPLUNK_HOME/etc/users/* /mnt/search-head-pooling/etc/users

4. Copy one search head's $SPLUNK_HOME/etc/system directory to/mnt/search-head-pooling/etc/system.

cp -r $SPLUNK_HOME/etc/system /mnt/search-head-pooling/etc/

5. On each search head, edit the distsearch.conf file to set shareBundles =false:

158

[distributedSearch]...shareBundles = false

6. On each search head, start splunkd:

splunk start splunkd

Your search head pool should now be up and running.

Part 2: Mount bundles on the search peers

Now, mount the bundles on the search peers.

On each search peer, perform these steps:

1. Mount the shared storage location (the same location that was earlier set to/mnt/search-head-pooling on the search heads) so that it appears as/mnt/bundles on the peer.

2. Create a directory that consists of symbolic links to the bundle subdirectories:

/opt/shared_bundles/bundles/system -> /mnt/bundles/etc/system /opt/shared_bundles/bundles/users -> /mnt/bundles/etc/users /opt/shared_bundles/bundles/apps -> /mnt/bundles/etc/apps

3. Create a distsearch.conf file in $SPLUNK_HOME/etc/system/local/ on thesearch peer, with stanzas for each of the two search heads:

[searchhead:searcher01]mounted_bundles = truebundles_location = /opt/shared_bundles/bundles

[searchhead:searcher02]mounted_bundles = truebundles_location = /opt/shared_bundles/bundles

4. Restart the search peer:

splunk restart splunkd

159

Repeat the process for each search peer.

Configure search head pooling

Important: Search head pooling is an advanced feature. It's recommended thatyou contact the Splunk sales team to discuss your deployment beforeattempting to implement it.

Search head pooling feature overview

You can set up multiple search heads so that they share configuration and userdata. This is known as search head pooling. The main reason for havingmultiple search heads is to facilitate horizontal scaling when you have largenumbers of users searching across the same data. Search head pooling can alsoreduce the impact if a search head becomes unavailable.

You must enable search head pooling on each search head, so that they canshare configuration and user data. Once search head pooling has been enabled,these categories of objects will be available as common resources across allsearch heads in the pool:

configuration data -- configuration files containing settings for savedsearches and other knowledge objects.

•

search artifacts, records of specific search runs.• scheduler state, so that only one search head in the pool runs aparticular scheduled search.

•

For example, if you create and save a search on one search head, all the othersearch heads in the pool will automatically have access to it.

160

Search head pooling makes all files in $SPLUNK_HOME/etc/{apps,users} availablefor sharing. This includes *.conf files, *.meta files, view files, search scripts,lookup tables, etc.

Important: Be aware of these key issues:

Most shared storage solutions don't perform well across a WAN. Sincesearch head pooling requires low-latency shared storage capable ofserving a high number of operations per second, implementing searchhead pooling across a WAN is not supported.

•

All search heads in a pool should be running the same version of Splunk.Be sure to upgrade all of them at once. See "Upgrade your distributeddeployment" for details.

•

Search head pooling and knowledge bundles

The set of data that a search head distributes to its search peers is known as theknowledge bundle. For details, see "What search heads send to search peers".

By default, only one search head in a search head pool sends the knowledgebundle to the set of search peers. Also, if search heads in a pool are also searchpeers of each other, they will not send bundles to each other, since they canaccess the bundles in the pool.This is an optimization introduced in version 4.3.2but made the default in version 5.0. It is controllable by means of theuseSHPBundleReplication attribute in distsearch.conf.

As a further optimization, you can mount knowledge bundles on shared storage,as described in "Mount the knowledge bundle". By doing so, you eliminate theneed to distribute the bundle to the search peers. For information on how tocombine search head pooling with mounted knowledge bundles, read the sectionin that topic called "Use mounted bundles with search head pooling".

Create a pool of search heads

To create a pool of search heads, follow these steps:

1. Set up a shared storage location accessible to each search head.

2. Configure each individual search head.

3. Stop the search heads.

161

4. Enable pooling on each search head.

5. Copy user and app directories to the shared storage location.

6. Restart the search heads.

The steps are described below in detail:

1. Set up a shared storage location accessible to each search head

So that each search head in a pool can share configurations and artifacts, theyneed to access a common set of files via shared storage:

On *nix platforms, set up an NFS mount.•

On Windows, set up a CIFS (SMB) share.•

Important: The Splunk user account needs read/write access to the sharedstorage location. When installing a search head on Windows, be sure toinstall it as a user with read/write access to shared storage. The LocalSystem user does not have this access. For more information, see "Choosethe user Splunk should run as" in the Installation manual.

2. Configure each search head

a. Set up each search head individually, specifying the search peers in the usualfashion. See "Configure distributed search".

b. Make sure that each search head has a unique serverName attribute,configured in server.conf. See "Manage distributed server names" for detailedinformation on this requirement. If the search head does not have a uniqueserverName, Splunk will generate a warning at start-up. See "Warning aboutunique serverName attribute" for details.

c. Specify the necessary authentication. You have two choices:

Specify user authentication on each search head separately. A valid useron one search head is not automatically a user on another search head inthe pool. You can use LDAP to centrally manage user authentication, asdescribed in "Set up user authentication with LDAP".

•

Place a common authentication configuration on shared storage, to beused by all pool members. You must restart the pool members after any

•

162

change to the authentication.

Note: Any authentication change made on an individual pool member (forexample, via its Splunk Manager) overrides for that pool member only anyconfiguration on shared storage. You should, therefore, generally avoid makingauthentication changes through Splunk Manager if a common configurationalready exists on shared storage.

3. Stop the search heads

Before enabling pooling, you must stop splunkd. Do this for each search head inthe pool.

4. Enable pooling on each search head

You use the pooling enable CLI command to enable pooling on a search head.The command sets certain values in server.conf. It also creates subdirectorieswithin the shared storage location and validates that Splunk can create and movefiles within them.

Here's the command syntax:

splunk pooling enable <path_to_shared_storage> [--debug]

Note:

On NFS, <path_to_shared_storage> should be the NFS's sharemountpoint.

•

On Windows, <path_to_shared_storage> should be the UNC path of theCIFS/SMB share.

•

The --debug parameter causes the command to log additional informationto btool.log.

•

Execute this command on each search head in the pool.

The command sets values in the [pooling] stanza of the server.conf file in$SPLUNK_HOME/etc/system/local. For detailed information on server.conf, lookhere.

5. Copy user and app directories to the shared storage location

Copy the contents of the $SPLUNK_HOME/etc/apps and $SPLUNK_HOME/etc/usersdirectories on an existing search head into the empty /etc/apps and /etc/users

163

directories in the shared storage location. Those directories were created in step4 and reside under the <path_to_shared_storage> that you specified at that time.

For example, if your NFS mount is at /tmp/nfs, copy the apps subdirectories thatmatch this pattern:

$SPLUNK_HOME/etc/apps/*

into

/tmp/nfs/etc/apps

This results in a set of subdirectories like:

/tmp/nfs/etc/apps/search/tmp/nfs/etc/apps/launcher/tmp/nfs/etc/apps/unix[...]

Similarly, copy the user subdirectories:

$SPLUNK_HOME/etc/users/*

into

/tmp/nfs/etc/users

Important: You can choose to copy over just a subset of apps and usersubdirectories; however, be sure to move them to the precise locations describedabove.

6. Restart the search heads

After running the pooling enable command, restart splunkd. Do this for eachsearch head in the pool.

Use a load balancer

You will probably want to run a load balancer in front of your search heads. Thatway, users can access the pool of search heads through a single interface,

164

without needing to specify a particular one.

Another reason for using a load balancer is to ensure access to search artifactsand results if one of the search heads goes down. Ordinarily, RSS and emailalerts provide links to the search head where the search originated. If that searchhead goes down (and there's no load balancer), the artifacts and results becomeinaccessible. However, if you've got a load balancer in front, you can set thealerts so that they reference the load balancer instead of a particular searchhead.

Configure the load balancer

There are a couple issues to note when selecting and configuring the loadbalancer:

The load balancer must employ layer-7 (application-level) processing.•

Configure the load balancer so that user sessions are "sticky" or"persistent". This ensures that the user remains on a single search headthroughout their session.

•

Generate alert links to the load balancer

To generate alert links to the load balancer, you must edit alert_actions.conf:

1. Copy alert_actions.conf from a search head to the appropriate app directoryin the shared storage location. In most cases, this will be/<path_to_shared_storage>/etc/apps/search/local.

2. Edit the hostname attribute to point to the load balancer:

hostname = <proxy host>:<port>

For details, see alert_actions.conf in the Admin manual.

The alert links should now point to the load balancer, not the individual searchheads.

Other pooling operations

Besides the pooling enable CLI command, there are several other commandsthat are important for managing search head pooling:

165

pooling validate• pooling disable• pooling display•

You must stop splunkd before running pooling enable or pooling disable.However, you can run pooling validate and pooling display while splunkd iseither stopped or running.

Validate that each search head has access to shared resources

The pooling enable command validates search head access when you initiallyset up search head pooling. If you ever need to revalidate the search head'saccess to shared resources (for example, if you change the NFS configuration),you can run the pooling validate CLI command:

splunk pooling validate [--debug]

Disable search head pooling

You can disable search head pooling with this CLI command:

splunk pooling disable [--debug]

Run this command for each search head that you need to disable.

Important: Before running the pooling disable command, you must stopsplunkd. After running the command, you should restart splunkd.

Display pooling status

You can use the pooling display CLI command to determine whether pooling isenabled on a search head:

splunk pooling display

This example shows how the system response varies depending on whetherpooling is enabled:

$ splunk pooling enable /foo/bar$ splunk pooling displaySearch head pooling is enabled with shared storage at: /foo/bar$ splunk pooling disable$ splunk pooling displaySearch head pooling is disabled

166

Manage configuration changes

Important: Once pooling is enabled on a search head, you must notify thesearch head whenever you directly edit a configuration file.

Specifically, if you add a stanza to any configuration file in a local directory, youmust run the following command:

splunk btool fix-dangling

Note: This is not necessary if you make changes by means of Splunk WebManager or the CLI.

Deployment server and search head pooling

With search head pooling, all search heads access a single set of configurations,so you don't need to use a deployment server or a third party deploymentmanagement tool like Puppet to push updates to multiple search heads.However, you might still want to use a deployment tool with search head pooling,in order to consolidate configuration operations across all Splunk instances.

If you want to use the deployment server to manage your search headconfiguration, note the following:

Designate one of the search heads as a deployment client by creating adeploymentclient.conf file in $SPLUNK_HOME/etc/system/local. You onlyneed to designate one search head as a deployment client.

•

In serverclass.conf on the deployment server, define a server class forthe search head. Set its repositoryLocation attribute to the sharedstorage mountpoint on the search head. You can also specify the value indeploymentclient.conf on the search head, but in either case, the valuemust point to the shared storage mountpoint.

•

For detailed information on the deployment server, see "About deploymentserver" and the topics that follow it.

Select timing for configuration refresh

You might want to configure search head pooling to synchronize from the storagelocation less frequently than the default. Particularly in deployments with largenumbers of users (in the hundreds or thousands), Splunk can spend excessive

167

time reading configuration changes from the pool.

In the server.conf file (typically $SPLUNK_HOME/etc/system/local/server.conf),the following settings affect configuration refresh timing:

# current defaults[pooling]poll.interval.rebuild = 2spoll.interval.check = 5s

With these default values, a change made on one search head would becomeavailable on another search head at most seven seconds later. There is usuallyno need for updates to be propagated that quickly. By changing the settings touse values in the scale of minutes, the load on the shared storage system isgreatly reduced.

Answers

Have questions? Visit Splunk Answers and see what questions and answers theSplunk community has about search head pooling.

How authorization works in distributed searches

The authorization settings that a search peer uses when processing distributedsearches are different from those which it uses for its local activities, such asadministration and local search requests:

When processing a distributed search, the search peer uses the settingscontained in the knowledge bundle that the search head distributes to allthe search peers when it sends them a search request. These settings arecreated and managed on the search head.

•

When performing local activities, the search peer uses the authorizationsettings created and stored locally on the search peer itself.

•

When managing distributed searches, it is therefore important that youdistinguish between these two types of authorization. You need to be particularlyaware of how authorization settings get distributed through the knowledge bundlewhen you're managing a system with search head pooling or mountedbundles.

For background information on key concepts, read these topics:

168

Splunk authorization: "About role-based user access" in the SecuringSplunk manual

•

Mounted bundles: "Mount the knowledge bundle" in this chapter• Search head pooling: "Configure search head pooling" in this chapter•

Manage authorization for distributed searches

All authorization settings are stored in one or more authorize.conf files. Thisincludes settings configured through Splunk Manager or the CLI. It is theseauthorize.conf files that get distributed from the search head to the searchpeers. On the knowledge bundle, the files are usually located in either/etc/system/{local,default} and/or /etc/apps/<app-name>/{local,default}.

Since search peers automatically use the settings in the knowledge bundle,things normally work fine. You configure roles for your users on the search head,and the search head automatically distributes those configurations to the searchpeers when it distributes the search itself.

With search head pooling, however, you must take care to ensure that the searchheads and the search peers all use the same set of authorize.conf file(s). Forthis to happen, you must make sure:

All search heads in the pool use the same set of authorize.conf files•

The set of authorize.conf files that the search heads use goes into theknowledge bundle so that they get distributed to the search peers.

•

This topic describes the four main scenarios, based on whether or not you'reusing search head pooling or mounted bundles. It describes the scenarios inorder from simple to complex.

Four scenarios

What you need to do with the distributed search authorize.conf files depends onwhether your deployment implements search head pooling or mounted bundles.The four scenarios are:

No search head pooling, no mounted bundles• No search head pooling, mounted bundles• Search head pooling, no mounted bundles• Search head pooling, mounted bundles•

169

The first two scenarios "just work" but the last two scenarios require carefulplanning. For the sake of completeness, this section describes all four scenarios.

Note: These scenarios address authorization settings for distributed search only.Local authorization settings function the same independent of your distributedsearch deployment.

No search head pooling, no mounted bundles

Whatever authorization settings you have on the search head get automaticallydistributed to its search peers as part of the replicated knowledge bundle thatthey receive with distributed search requests.

No search head pooling, mounted bundles

Whatever authorization settings you have on the search head get automaticallyplaced in the mounted bundle and used by the search peers during distributedsearch processing.

Search head pooling, no mounted bundles

The search heads in the pool share their /apps and /users directories but nottheir /etc/system/local directories. Any authorize.conf file in an /appssubdirectory will be automatically shared by all search heads and included in theknowledge bundle when any of the search heads distributes a search request tothe search peers.

The problem arises because authorization changes can also wind up in anauthorize.conf file in a search head's /etc/system/local directory (for example,if you update the search head's authorization settings via Manager). Thisdirectory does not get shared among the search heads in the pool, but it still getsdistributed to the search peers as part of the knowledge bundle. Unless youaccount for this situation, the search peers can end up using differentauthorization settings for different searches, depending on which search headdistributed the search to them. For most situations, this is likely not what youwant to occur.

Therefore, you need to make sure that any changes made to a search head's/etc/system/local/authorize.conf file get propagated to all search heads in thepool. One way to handle this is to move any changed/etc/system/local/authorize.conf file into an app subdirectory, since all searchheads in the pool share the /apps directory.

170

Search head pooling, mounted bundles

This is similar to the previous scenario. The search heads in the pool share their/apps and /users directories but not their /etc/system/local directories. Anyauthorize.conf file in an /apps subdirectory will be automatically shared by allsearch heads. It will also be included in the mounted bundle that the searchpeers use when processing a search request from any of the search heads.

However, authorization changes can also wind up in an authorize.conf file in asearch head's /etc/system/local directory (for example, if you update thesearch head's authorization settings via Manager). This directory does not getautomatically shared among the search heads in the pool. It also does not getautomatically distributed to the mounted bundle that the search peers use.Therefore, you must provide some mechanism that ensures that all the searchheads and all the search peers have access to that version of authorize.conf.

The simplest way to handle this is to move any changed/etc/system/local/authorize.conf file into an app subdirectory, since both thepooled search heads and all the search peers share the /apps directory.

Use distributed search

From the user standpoint, specifying and running a distributed search isessentially the same as running any other search. Behind the scenes, the searchhead distributes the query to its search peers and consolidates the results whenpresenting them to the user.

Users can limit the search peers that participate in a search. They also need tobe aware of the distributed search configuration to troubleshoot.

Perform distributed searches

In general, you specify a distributed search through the same set of commandsas for a local search. However, Splunk provides several additional commandsand options to assist with controlling and limiting a distributed search.

A search head by default runs its searches across all search peers in its cluster.You can limit a search to one or more search peers by specifying thesplunk_server field in your query. See "Search across one or more distributedservers" in the User manual.

171

The search command localop is also of use in defining distributed searches. Itenables you to limit the execution of subsequent commands to the search head.See the description of localop in the Search Reference for details and anexample.

In addition, the lookup command provides a local argument for use withdistributed searches. If set to true, the lookup occurs only on the search head; iffalse, the lookup occurs on the search peers as well. This is particularly usefulfor scripted lookups, which replicate lookup tables. See the description of lookupin the Search Reference for details and an example.

Troubleshoot distributed search

This topic describes issues to be aware of when configuring or using distributedsearch.

General configuration issues

Clock skew between search heads and search peers can affect searchbehavior

It's important to keep the clocks on your search heads and search peers in sync,via NTP (network time protocol) or some similar means. If the clocks areout-of-sync by more than a few seconds, you can end up with search failures orpremature expiration of search artifacts.

Search head pooling configuration issues

When implementing search head pooling, there are a few potential issues youshould be aware of, mainly having to do with coordination among search heads.

Authentication and authorization changes made in Manager apply only to asingle search head

Authentication and authorization changes made through a search head'sManager apply only to that search head and not to other search heads in thatpool. Each member of the pool maintains its local configurations in$SPLUNK_HOME/etc/system/local. To share configurations across the pool, setthem up in shared storage, as described in "Configure search head pooling".

172

Clock skew between search heads and shared storage can affect searchbehavior

It's important to keep the clocks on your search heads and shared storage serverin sync, via NTP (network time protocol) or some similar means. If the clocks areout-of-sync by more than a few seconds, you can end up with search failures orpremature expiration of search artifacts.

Permission problems on the shared storage server can cause poolingfailure

On each search head, the user account Splunk runs as must have read/writepermissions to the files on the shared storage server.

NFS client concurrency limits can cause search timeouts or slow searchbehavior

The search performance in a search head pool is a function of the throughput ofthe shared storage and the search workload. The combined effect of concurrentsearch users and concurrent scheduled searches running will yield a total IOPsthat the shared volume needs to support. IOP requirements will also vary by thekind of searches run. To adequately provision a device to be shared betweensearch heads, you need to know the number of concurrent users submittingsearches and the number of jobs/apps that will be executed simultaneously.

If searches are timing out or running slowly, you might be exhausting themaximum number of concurrent requests supported by the NFS client. To solvethis problem, increase your client concurrency limit. For example, on a Linux NFSclient, adjust the tcp_slot_table_entries setting.

NFS latency for large user count can incur splunk configuration accesslatency or slow dispatch reaping

Splunk synchronizes the search head pool storage configuration state with thein-memory state when it detects changes. Essentially, it reads the configurationinto memory when it detects updates. When dealing either with overloadedsearch pool storage or with large numbers of users, apps, and configuration files,this synchronization process can reduce performance. To mitigate this, theminimum frequency of reading can be increased, as discussed in "Select timingfor configuration refresh".

173

Warning about unique serverName attribute

Each search head in the pool must have a unique serverName attribute. Splunkvalidates this condition when each search head starts. If it finds a problem, itgenerates this error message:

serverName "<xxx>" has already been claimed by a member of this searchhead poolin <full path to pooling.ini on shared storage>There was an error validating your search head pooling configuration.For moreinformation, run 'splunk pooling validate'

The most common cause of this error is that another search head in the pool isalready using the current search head's serverName. To fix the problem, changethe current search head's serverName attribute in .system/local/server.conf.

There are a few other conditions that also can generate this error:

The current search head's serverName has been changed.• The current search head's GUID has been changed. This is usually due to/etc/instance.cfg being deleted.

•

To fix these problems, run

splunk pooling replace-member

This updates the pooling.ini file with the current search head'sserverName->GUID mapping, overwriting any previous mapping.

Artifacts and incorrectly-displayed items in Manager UI after upgrade

When upgrading pooled search heads, you must copy all updated apps - eventhose that ship with Splunk (such as the Search app and the data previewfeature, which is implemented as an app) - to the search head pool's sharedstorage after the upgrade is complete. If you do not, you might see artifacts orother incorrectly-displayed items in Manager.

To fix the problem, copy all updated apps from an upgraded search head to theshared storage for the search head pool, taking care to exclude the localsub-directory of each app.

174

Important: Excluding the local sub-directory of each app from the copy processprevents the overwriting of configuration files on the shared storage with localcopies of configuration files.

Once the apps have been copied, restart Splunk on all search heads in the pool.

Distributed search error messages

This table lists some of the more common search-time error messagesassociated with distributed search:

Error message Meaningstatus=down The specified remote peer is not available.

status=not a splunk server The specified remote peer is not a Splunk server.

duplicate license The specified remote peer is using a duplicate license.

certificate mismatch Authentication with the specified remote peer failed.

175

Monitor your deployment

About Splunk Deployment Monitor App

In pre-5.0 versions of Splunk, the Splunk Deployment Monitor App was includedas part of the core distribution. This is no longer the case. You now need todownload it from Splunkbase. However, if you are migrating from a pre-5.0version of Splunk with an enabled Deployment Monitor App, the app will remainon your system.

In addition, documentation for the Deployment Monitor App is no longer locatedhere in the Distributed Deployment Manual. Instead, to learn about the latestversion of the app, read the Deploy and Use Splunk Deployment Monitor Appmanual.

176

Deploy configuration updates and appsacross your environment

About deployment server

Important: The deployment server handles configuration and contentupdates to existing Splunk installations. You cannot use it to install orupgrade Splunk software components.

To learn how to install and deploy Splunk, see "Step-by-step installationprocedures" for full Splunk and "Universal forwarder deployment overview" forthe Splunk universal forwarder.

To learn how to upgrade your deployment to a new version of Splunk, see"Upgrade your deployment".The deployment server is Splunk's tool for pushing out configurations, apps,and content updates to distributed Splunk instances. You can use it to pushupdates to any Splunk component: forwarder, indexer, or search head.

A key use case is to manage configuration for groups of forwarders. Forexample, if you have several sets of forwarders, each set residing on a differentmachine type, you can use the deployment server to push out different contentaccording to machine type. Similarly, in a distributed search environment, youcan use a deployment server to push out content to sets of indexers.

Important: Do not use deployment server to manage configuration files acrossnodes in a cluster. Instead, use the configuration bundle method discussed in"Update common peer configurations" in the Managing Indexers and Clustersmanual.

The first several topics in this section explain how to configure a deploymentserver and its clients. Topics then follow that show how to employ this technologyfor specific use cases.

The big picture (in words and diagram)

In a Splunk deployment, you use a deployment server to push out content andconfigurations (collectively called deployment apps) to deployment clients,grouped into server classes.

177

A deployment server is a Splunk instance that acts as a centralizedconfiguration manager, collectively managing any number of Splunk instances,called "deployment clients". Any full, enterprise Splunk instance -- even oneindexing data locally -- can act as a deployment server.

A deployment client is a Splunk instance remotely configured by a deploymentserver. A Splunk instance can be both a deployment server and client at thesame time. Each deployment client belongs to one or more server classes.

A server class is a set of deployment clients, grouped by configurationcharacteristics, managed as a unit. You can group clients by application, OS,type of data, or any other feature of your Splunk deployment. To update theconfiguration for a set of clients, the deployment server pushes configuration filesto all or some members of a server class. Besides configuration files, you canpush any sort of content. You configure server classes on the deployment server.

This diagram provides a conceptual overview of the relationship between adeployment server and its set of deployment clients and server classes:

In this example, each deployment client is a Splunk forwarder that belongs to twoserver classes, one for its OS and the other for its geographical location. Thedeployment server maintains the list of server classes and uses those serverclasses to determine what content to push to each client. For an example of howto implement this type of arrangement to govern the flow of content to clients,see "Deploy several forwarders".

A deployment app is a set of deployment content (including configuration files)deployed as a unit to clients of a server class. A deployment app might consist ofjust a single configuration file, or it can consist of many files. Depending onfiltering criteria, an app might get deployed to all clients in a server class or to asubset of clients. Over time, an app can be updated with new content and thenredeployed to its designated clients. The deployment app can be an existing

178

Splunk app, or one developed solely to group some content for deploymentpurposes.

Note: The term "app" has a somewhat different meaning in the context of thedeployment server from its meaning in the general Splunk context. For moreinformation on Splunk apps in general, see "What are apps and add-ons?".

For more information on deployment servers, server classes, and deploymentapps, see "Define server classes". For more information on deployment clients,see "Configure deployment clients".

A multi-tenant environment means that you have more than one deploymentserver running on the same Splunk instance, and each deployment server isserving content to its own set of deployment clients. For information aboutmulti-tenant environments, see "Deploy in multi-tenant environments".

Key terms

Here's a recap of the key definitions:

Term Meaning

deployment server A Splunk instance that acts as a centralized configuration manager. Itpushes configuration updates to other Splunk instances.

deployment client A remotely configured Splunk instance. It receives updates from thedeployment server.

server class A deployment configuration category shared by a group of deploymentclients. A deployment client can belong to multiple server classes.

deployment app A unit of content deployed to one or more members of a server class orclasses.

multi-tenantenvironment A deployment environment involving multiple deployment servers.

Communication between deployment server and clients

Each deployment client periodically polls the deployment server, identifying itself.The deployment server then determines whether it has new or updated contentto push to that particular client. If there is content, the deployment server tells theclient, which then retrieves the content and treats it according to the instructionsfor the server class it belongs to. Depending on those instructions, the clientmight restart, run a script, or wait for further instructions.

179

Lookup tables and deployment server

In some cases, your indexers or search heads might be running apps that saveinformation in lookup tables. Be careful about using the deployment server tomanage such instances. When the deployment server pushes an updated appconfiguration, it overwrites the existing app. At that point, you'll lose those lookuptables.

Plan a deployment

If you've got Splunk instances serving a variety of different groups within yourorganization, chances are their configurations vary depending on who uses themand for what purpose. You might have some Splunk instances serving the helpdesk team, configured with a specific app to accelerate troubleshooting ofWindows desktop issues. You might have another group of Splunk instances inuse by your operations staff, set up with a few different apps designed toemphasize tracking of network issues, security incidents, and email trafficmanagement. A third group of Splunk instances might serve the Web hostinggroup within the operations team.

Rather than trying to manage and maintain these divergent Splunk instances oneat a time, you can group them based on their use, identify the configurations andapps needed by each group, and then use the deployment server to updatetheir apps and configurations as needed.

In addition to grouping Splunk instances by use, there are other useful types ofgroupings you can specify. For example, you might group Splunk instances byOS or hardware type, by version, or by geographical location or timezone.

Configuration overview

For the great majority of deployment server configurations, perform these steps:

1. Designate one of your Splunk instances as the deployment server.

Note: While in small environments (fewer than 30 deployment clients), it may beperfectly viable to provide the deployment server service from an indexer orsearch head node, Splunk strongly recommends putting the deployment serveron its own Splunk instance when using it with larger numbers of clients. Anotherthing to consider is the need to restart the deployment server when makingcertain configuration changes, which may affect user searches if it shares a

180

system with a search head. For additional information about deployment serversizing, refer to this topic about the deployment server on the Splunk CommunityWiki.

2. Group the deployment clients into server classes. A server class defines theclients that belong to it and what content gets pushed out to them. Eachdeployment client can belong to multiple server classes.

3. Create a serverclass.conf file on the deployment server. It specifies theserver classes and the location of the deployment apps. Refer to "Define serverclasses" in this manual for details.

Note: You can also add server classes and perform simple configuration throughSplunk Manager, as described in "Define server classes".

4. Create the directories for your deployment apps, and put the content to bedeployed into those directories. Refer to "Deploy apps and configurations" in thismanual for details.

5. On each deployment client, create a deploymentclient.conf file. It specifieswhat deployment server the client should communicate with, the specific locationon that server from which it should pick up content, and where it should put itlocally. Refer to "Configure deployment clients" in this manual for details.

6. For more complex deployments with multiple deployment servers, create atenants.conf file on one of the deployment servers. This allows you to definemultiple deployment servers on a single Splunk instance and redirect incomingclient requests to a specific server according to rules you specify. Refer to"Deploy in multi-tenant environments" in this manual for more information aboutconfiguring tenants.conf. Most deployment server topologies don't needtenants.conf, however.

For an example of an end-to-end configuration, see "Deploy several forwarders".

Note: The deployment server and its deployment clients must agree in the SSLsetting for their splunkd management ports. They must all have SSL enabled, orthey must all have SSL disabled. To configure SSL on a Splunk instance, set theenableSplunkdSSL attribute in server.conf to "true" or "false". For detailedinformation on using SSL with deployment server, see "Securing deploymentserver and clients".

181

Restart or reload?

The first time you configure the deployment server and its clients, you'll need torestart all instances of Splunk. When you restart the deployment server, itautomatically deploys any new content to its clients. Later on, to deploy new orupdated content without restarting, you can use the CLI reload command, asdescribed in "Deploy apps and configurations" in this manual.

Enable and disable deployment server using the CLI

To enable a deployment server, run the following command from the SplunkCLI:

./splunk enable deploy-server

Now restart the deployment server to make the change take effect.

To disable a deployment server, run the following command:

./splunk disable deploy-server

Configure deployment clients

This topic explains how to set up deployment clients to receive content from adeployment server. Each deployment client belongs to one or more serverclasses. Server classes define what content a deployment client shoulddownload, based on qualifying criteria.

Two configuration files control this process:

The deployment server has a serverclass.conf file, which specifies its setof server classes. See "Define server classes" for details on how toconfigure this file.

•

Each deployment client has a deploymentclient.conf file, which specifieswhich deployment server it contacts for content, along with someassociated information. The current topic describes how to configure thisfile.

•

You can view deployment client settings in Splunk Manager. Go to SplunkManager > Deployment and select the Deployment client row.

182

For information about configuration files, including an explanation of their basicstructure, see About configuration files in the Admin manual.

The next section provides a reference for the deployment client configurationsettings. You might want to read it while referring to the set of simple exampleconfigurations presented later in this topic. In addition, there are several longerand more complete examples in later topics in this manual, including "Deployseveral forwarders".

Important: Although a deployment server can be a deployment client of anotherdeployment server, it cannot be a deployment client of itself.

Enable a deployment client

To enable a Splunk instance as a deployment client, create adeploymentclient.conf file in $SPLUNK_HOME/etc/system/local.

You can also enable a deployment client through the CLI. See "Enable anddisable deployment clients using the CLI", later in this topic.

The deploymentclient.conf file provides two stanzas:

Stanza Meaning

[deployment-client]

Includes a number of configuration attributes,including where to find new or updated content.The default values for this stanza usually workfine.

[target-broker:deploymentServer]

Specifies the location of this client's deploymentserver. deploymentServer is the defaultname for a deployment server. This is themost important configuration for this file.

These are the main attributes available in the [deployment-client] stanza:

Attribute What it's for Default

disabledSet to true or false. If true, itdisables the deployment client.

false

clientNameA name the deployment server can useto filter on. It takes precedence overhostnames.

deploymentClient

workingDirA temporary folder used by thedeployment client to download serverclasses and applications.

$SPLUNK_HOME/var/run/deploy-client

183

repositoryLocation

The repository location wheredeployment apps are installed afterbeing downloaded from a deploymentserver.

Apps must be installed in thedefault location($SPLUNK_HOME/etc/apps) orSplunk will be unable torecognize them.

Settings defined inserverclass.conf on thedeployment server can alsospecify a repository location for adeployment client. Thedeployment client uses itsserverRepositoryLocationPolicy

attribute to determine whichsetting to use.

$SPLUNK_HOME/etc/apps

serverRepositoryLocationPolicy

Set to one of these values:

acceptSplunkHome - Acceptthe repository locationsupplied by thedeployment server, if andonly if it is rooted by$SPLUNK_HOME.

•

acceptAlways - Alwaysaccept the repositorylocation supplied by thedeployment server.

•

rejectAlways - Alwaysreject the server-suppliedrepository location.Instead, use therepositoryLocation

specified in thisconfiguration file.

•

acceptSplunkHome

endpoint The HTTP endpoint from which contentshould be downloaded.

$deploymentServerUri$/services/streams/deployment?name=$serverClassName$:$appName$

184

Note: The deployment server canspecify a different endpoint fromwhich to download each set ofcontent (individual apps, etc).The deployment client uses theserverEndpointPolicy attributeto determine which value to use.

$deploymentServerUri$ resolvesto targetUri, defined in the[target-broker] stanza.

$serviceClassName$ and$appName$ mean what they say.

serverEndpointPolicy

Set to one of these values:

acceptAlways - Alwaysaccept the endpointsupplied by thedeployment server.

•

rejectAlways - Alwaysreject the endpointsupplied by thedeployment server.Instead, use the endpointdefined by the endpointattribute.

• acceptAlways

phoneHomeIntervalInSecsA number that determines howfrequently the deployment client shouldcheck for new content.

60

In the [target-broker:deploymentServer] stanza,you specify the deploymentserver for this client:


targetUriSet to <Deployment_server_URI>:<Mgmt_port>. Specifiesthe deployment server connection information. Themanagement port is typically 8089.

n/a

Note: For a complete list of attribute values, see the deploymentclient.confspecification file in the Admin manual.

185

Examples

Here are several examples of defining deployment clients through thedeploymentclient.conf file:

# Example 1# Deployment client receives apps, placing them into the samerepositoryLocation locally,# relative to $SPLUNK_HOME, that it picked them up from - typically$SPLUNK_HOME/etc/apps.# There is nothing in [deployment-client], because the deploymentclient is not overriding# the location value set on the deployment server side.

[deployment-client]

[target-broker:deploymentServer]targetUri= deploymentserver.splunk.mycompany.com:8089

# Example 2# Deployment server keeps apps to be deployed in a non-standard locationon the server side# (perhaps for organizational purposes).# Deployment client receives apps and places them in the standardlocation.# This configuration rejects any location specified by the deploymentserver and replaces# it with the standard client-side location.

[deployment-client]serverRepositoryLocationPolicy = rejectAlwaysrepositoryLocation = $SPLUNK_HOME/etc/apps


# Example 3# Deployment client should get apps from an HTTP server that isdifferent from the one# specified by the deployment server.

[deployment-client]serverEndpointPolicy = rejectAlwaysendpoint =http://apache.mycompany.server:8080/$serviceClassName$/$appName$.tar

[target-broker:deploymentServer]

186

targetUri= deploymentserver.splunk.mycompany.com:8089

# Example 4# Deployment client should get apps from a location on the file systemand not from a# location specified by the deployment server.

[deployment-client]serverEndpointPolicy = rejectAlwaysendpoint = file:/<some_mount_point>/$serviceClassName$/$appName$.tar


Enable and disable deployment clients using the CLI

To enable a Splunk instance as a deployment client, run this command onthe client:

splunk set deploy-poll <IP address/hostname>:<port>

You specify the IP address/hostname and management port of the deploymentserver you want the client to connect with.

You must restart the deployment client for the change to take effect.

To disable a deployment client, run this command on the deployment client:

splunk disable deploy-client

Define server classes

A server class defines a deployment configuration shared by a group ofdeployment clients. It defines both the criteria for being a member of the classand the set of content to deploy to members of the class. This content(encapsulated as "deployment apps") can consist of Splunk apps, Splunkconfigurations, and other related content, such as scripts, images, andsupporting material. You can define different server classes to reflect the differentrequirements, OSes, machine types, or functions of your deployment clients.

You define server classes in serverclass.conf on the deployment server. Youcan also define server classes and perform basic configuration through the

187

Splunk Manager. To perform advanced configuration tasks, however, you'll needto edit serverclass.conf.

Use serverclass.conf

You can define server classes in serverclass.conf on the deployment server.Create one in $SPLUNK_HOME/etc/system/local. For information aboutconfiguration files, including an explanation of their basic structure, see "Aboutconfiguration files" in the Admin manual.

If you have multiple server classes, you might want to define one server classthat applies to all deployment clients by default. You can then override variousaspects of it as needed by defining more specific server classes. For example, ifyou have a mix of Windows and Linux universal forwarders sending data to thesame indexer, you might want to make sure that all forwarders get a commonoutputs.conf file, but that Windows forwarders get one inputs.conf file whileLinux forwarders get a different one. In that case, you could define an "allforwarder" server class that distributes a deployment app containing theoutputs.conf file to all forwarders, while also defining Windows and Linux serverclasses that distribute separate apps containing the different inputs.conf files tothe appropriate subsets of forwarders.

In addition to defining attributes and content for specific server classes, you canalso define attributes that pertain just to a single app within a server class.

A deployment client has its own configuration, defined in deploymentclient.conf.The information in deploymentclient.conf tells the deployment client where togo to get the content that the server class it belongs to says it should have.

The next section provides a reference for the server class configuration settings.You might want to read it while referring to the set of simple exampleconfigurations presented later in this topic. In addition, there are several longerand more complete examples presented later in this manual, including "Deployseveral forwarders".

What you can configure for a server class

You can specify settings for a global server class, as well as for individual serverclasses or apps within server classes. There are three levels of stanzas toenable this:

Stanza Meaning Scope

188

[global]The globalserver class.

Attributes defined herepertain to all serverclasses.

[serverClass:<serverClassName>]

Individualserver class. AserverClass

is acollection ofapps.

Attributes defined herepertain to just the serverclass<serverClassName>.

Note:<serverClassName>

cannot containspaces.

[serverClass:<serverClassName>:app:<appName>]App withinserver class.

Attributes defined herepertain to just thespecified deploymentapp <appName> withinthe specified<serverClassName>.To indicate all appswithin<serverClassName>,<appName> can bethe wildcardcharacter: *, in whichcase it will cause allcontent in therepositoryLocation tobe added to thisserverClass.

Important: When defining app names, you should be aware of the rules ofconfiguration file precedence, as described in the topic "Configuration fileprecedence" in the Admin manual. In particular, note that app directories areevaluated by ASCII sort order. For example, if you set an attribute/value pairwhatever=1 in the file x.conf in an app directory named "A", the setting in app Aoverrides the setting whatever=0 in x.conf in an app named "B", etc. For details,see the subtopic "How app names affect precedence".

Attributes in more specific stanzas override less specific stanzas. Therefore, anattribute defined in a [serverClass:<serverClassName>] stanza will override thesame attribute defined in [global].

The attributes are definable for each stanza level, unless otherwise indicated.Here are the most common ones:

189


repositoryLocationThe location on the deployment serverwhere the content to be deployed for thisserver class is stored.

$SPLUNK_HOME/etc/deployment-apps

targetRepositoryLocation

The location on the deployment clientwhere the content to be deployed for thisserver class should be installed. You canoverride this indeploymentclient.conf on thedeployment client.

$SPLUNK_HOME/etc/apps

continueMatching

If set to false, the deployment server willlook through the list of server classes inthis configuration file and stop when itmatches the first one to a client. If set totrue, the deployment server will continueto look and match. This option is availablebecause you can define multiple, layeredsets of server classes. A serverClass

can override this property and stopthe matching.

true

endpoint

The HTTP location from which contentcan be downloaded by a deploymentclient. The deployment server fills in thevariable substitutions itself, based oninformation received from the client. Youcan provide any URI here, as long as ituses the same variables. In most cases,this attribute does not need to bespecified.

$deploymentServerUri$/services/streams/deployment?name=$serverClassName$:$appName$

filterType Set to "whitelist" or "blacklist".

This determines the order ofexecution of filters. If filterType iswhitelist, all whitelist filters areapplied first, followed by blacklistfilters. If filterType is blacklist,all blacklist filters are applied first,followed by whitelist filters.

The whitelist setting indicates afiltering strategy that pulls in asubset:

Items are not considered to•

whitelist

190

match the stanza by default.Items that match anywhitelist entry, and do notmatch any blacklist entry,are considered to match thestanza.

•

Items that match anyblacklist entry are notconsidered to match thestanza, regardless ofwhitelist.

•

The blacklist setting indicates afiltering strategy that rules out asubset:

Items are considered tomatch the stanza by default.

•

Items that match anyblacklist entry, and do notmatch any whitelist entry,are considered to not matchthe stanza.

•

Items that match anywhitelist entry areconsidered to match thestanza.

•

More briefly:

whitelist: default no-match-> whitelists enable ->blacklists disable

•

blacklist: default match ->blacklists disable->whitelists enable

•

You can override this value at theserverClass and serverClass:applevels. If you specify whitelist atthe global level, and then specifyblacklist for an individual serverclass, the setting becomes

191

blacklist for that server class,and you have to provide anotherfilter in that server class definitionto replace the one you overrode.

whitelist.<n>

blacklist.<n>

<n> is a number starting at 0, andincrementing by 1.

Set the attribute to ipAddress,hostname, or clientName:

ipAddress is the IP addressof the deployment client.Can use wildcards, such as10.1.1.*

•

hostname is the host nameof deployment client. Canuse wildcards, such as*.splunk.com.

•

clientName is a logical ortag name that can beassigned to a deploymentclient indeploymentclient.conf.clientName takesprecedence over ipAddressor hostname when matchinga client to a filter.

•

Here are some examples: WhenfilterType is whitelist:

whitelist.0=*.fflanda.comblacklist.0=printer.fflanda.comblacklist.1=scanner.fflanda.com

This will cause all hosts infflanda.com, except 'printer' and'scanner' to match this serverclass.

When filterType is blacklist:

n/a

192

blacklist.0=*whitelist.0=*.web.fflanda.comwhitelist.1=*.linux.fflanda.com

This will cause only the 'web' and'linux' hosts to match the serverclass. No other hosts will match.

You can override this value at theserverClass and serverClass:applevels.

Important: Overriding one type offilter (whitelist/blacklist) causes theother to be overridden too. If, forexample, you override thewhitelist, the blacklist will not beinherited from the parent; you mustprovide one in the stanza.

stateOnClient

Set to "enabled", "disabled", or "noop".This setting specifies whether thedeployment client receiving an app shouldenable or disable the app once it isinstalled. The "noop" value is for apps thatdo not require enablement; for example,apps containing only Splunk knowledge,such as event or source types.

enabled

machineTypesFilter Matches any of the machine types in acomma-separated list.

This setting lets you use thehardware type of the deploymentclient as a filter. This filter will beused only if a client matches thewhitelist/blacklist filters.

The value for machineTypesFilteris a comma-separated list ofmachine types; for example,linux-i686, linux-x86_64, etc. Eachmachine type is a specific stringdesignated by the hardwareplatform itself.

n/a

193

Note: A machineTypesFilter valuecan contain wildcards; forexample: linux-*, windows-*, oraix-*

The method for finding this stringon the client varies by platform, butif the deployment client is alreadyconnected to the deploymentserver, you can determine thestring's value by using this SplunkCLI command on the deploymentserver:

./splunk list deploy-clients

This will return a value for utsnamethat you can use to specifymachineTypesFilter.

This setting will match any of themachine types in acomma-delimited list.Commonly-used machine typesare linux-x86_64, windows-x64,linux-i686, freebsd-i386,darwin-i386, sunos-sun4u,linux-x86_64, sunos-i86pc,freebsd-amd64.

restartSplunkWeb

Set to "true" or "false". Determineswhether the client's Splunk Web restartsafter the installation of a server class orapp.

false

restartSplunkd

Set to "true" or "false". Determineswhether the client's splunkd restartsafter the installation of a serverclass or app.

false

Note: The most accurate and up-to-date list of settings available for a givenconfiguration file is in the .spec file for that configuration file. You can find thelatest version of the .spec and .example files for serverclass.conf inserverclass.conf in the Configuration file reference in the Admin manual, or in$SPLUNK_HOME/etc/system/README.

194

Examples

Here are several examples of defining server classes in the serverclass.conffile:

# Example 1# Matches all clients and includes all apps in the server class

[global]whitelist.0=*# whitelist matches all clients.[serverClass:AllApps][serverClass:AllApps:app:*]# a server class that encapsulates all apps in the repositoryLocation

# Example 2# Assign server classes based on hostnames.

[global]

[serverClass:AppsForOps]whitelist.0=*.ops.yourcompany.com[serverClass:AppsForOps:app:unix][serverClass:AppsForOps:app:SplunkLightForwarder]

[serverClass:AppsForDesktops]filterType=blacklist# blacklist everybody except the Windows desktop machines.blacklist.0=*whitelist.0=*.desktops.yourcompany.com[serverClass:AppsForDesktops:app:SplunkDesktop]

# Example 3# Deploy server class based on machine types

[global]# whitelist.0=* at the global level ensures that the machineTypesFilterattribute# invoked later will apply.whitelist.0=*

[serverClass:AppsByMachineType]machineTypesFilter=windows-*, linux-i686, linux-x86_64

[serverClass:AppsByMachineType:app:SplunkDesktop]# Deploy this app only to Windows boxes.machineTypesFilter=windows-*

195

[serverClass:AppsByMachineType:app:unix]# Deploy this app only to unix boxes - 32/64 bit.machineTypesFilter=linux-i686, linux-x86_64

Use Splunk Manager

To define deployment server classes in Splunk Manager:

1. Go to Splunk Manager > Deployment.

2. Select Add new from the Deployment server row.

3. Enter values for the Server class and Repository location fields.

4. Optionally, enter values for whitelists or blacklists.

For more information on what to enter in these fields, see the descriptions for thecorresponding attributes in "What you can configure for a server class", locatedabove.

You can only configure the most basic options using Splunk Manager. Foradvanced configurations, edit serverclass.conf directly.

Deploy apps and configurations

After configuring the deployment server and clients, you're ready to distributecontent to the deployment clients. This involves two steps:

1. Put the new or updated content into deployment directories on the deploymentserver.

2. Inform the clients that it's time to download new content.

Put the content into directories on the deployment server

You place the deployment apps into individual subdirectories in a special locationon the deployment server. From there, the deployment server can push thecontent to its deployment clients. The default location is$SPLUNK_HOME/etc/deployment-apps, but this is configurable through therepositoryLocation attribute in serverclass.conf. Underneath this location, each

196

app must have its own subdirectory, with the same name as the app itself. Theapp name is specified in serverclass.conf.

Note: On the deployment clients, the downloaded apps reside in a differentlocation, which defaults to $SPLUNK_HOME/etc/apps. This location is configurablein deploymentclient.conf, but, for most purposes, you should not change it fromthe default.

This example creates a deployment directory in the default repository location onthe deployment server, for an app named "fwd_to_splunk1":

mkdir ?p $SPLUNK_HOME/etc/deployment-apps/fwd_to_splunk1/default

Place the content for each app into the app's subdirectory. To update the appwith new or changed content, just add or overwrite the files in the directory.

Inform the clients of new content

When you first configure the deployment server, and whenever you update itsconfiguration by editing serverclass.conf, you'll need to restart or reload it forthe changes to take effect. The clients will then pick up any new or changedcontent. Instead of restarting the server, you can use the CLI reloaddeploy-server command.

This version of the command checks all server classes for change and notifiesthe relevant clients:

splunk reload deploy-server

This version of the command notifies and updates only the server class youspecify:

splunk reload deploy-server -class <server class>

For example:

splunk reload deploy-server -class www

In this example, the command notifies and updates only the clients that aremembers of the "www" server class.

197

Confirm the deployment update

To confirm that all clients received the configuration correctly, run this commandfrom the deployment server:

splunk list deploy-clients

This lists all the deployment clients that have contacted the deployment serverwithin the last two minutes. It specifies the last time that they successfullysynced.

App management issues

Once you start using the deployment server to manage an app, you cannot laterstop using the deployment server to manage the app. It is important tounderstand the implications of this.

If you remove an app from the deployment server's repositoryLocation (definedin serverclass.conf), the deployment client will delete its copy of the app. Thereis no way instead to tell the deployment client to start managing the app on itsown.

For example, say you are using the deployment server to manage updates to"appA". To do this, you have created a directory called "appA" in therepositoryLocation on the deployment server and you have placed the app'scontents there. From now on, whenever the deployment clients poll the server tocheck for updates, they compare their checksum for appA with the server'schecksum for appA. If the checksums differ, the clients download the latestversion of the app from the server. However, if appA has been deleted from theserver's app repository, the client behavior is to delete their own instances of theapp.

Therefore, by deleting an app from the deployment server's repositoryLocation,you are not telling the clients to stop using the deployment server to manage theapp and to start managing it on their own. Instead, you're actually telling them todelete the app. Once the deployment server manages an app, it alwaysmanages that app.

Warning: Because of this behavior, you should be extremely cautious beforedeciding to use the deployment server to manage the Splunk search app. Bymanaging the search app through the deployment server, you are preventingusers from saving any unique searches on their search heads. And since there's

198

no way to tell the deployment server to stop managing an app, you're effectivelystuck with that decision.

Extended example: deploy configurations toseveral forwarders

What we're aiming for

A common use for deployment servers is to manage configuration files forforwarders. In some distributed environments, forwarders can number into thethousands, and the deployment server greatly eases the work of configuring andupdating them. This example shows how to use the deployment server to initiallyconfigure a set of dissimilar universal forwarders. A follow-up example, in"Example: add an input to forwarders", shows how to use the deployment serverto update the forwarders' configurations with new inputs.

Before reading this example, you should already be familiar with how forwardersand receivers communicate, as described in "About forwarding and receiving"and the topics that follow it.

The example sets up the following distributed environment, in which adeployment server deploys configurations for three universal forwarders sendingdata to two indexers:

The deployment server Fflanda-SPLUNK3 (10.1.2.4) managesdeployment for these universal forwarders:

Fflanda-WIN1♦ Fflanda-LINUX1♦ Fflanda-LINUX2♦

•

Fflanda-WIN1 forwards Windows event logs to the receiving indexerFflanda-SPLUNK1 (10.1.2.2).

•

Fflanda-LINUX1 and Fflanda-LINUX2 forward Linux messages to thereceiving indexer Fflanda-SPLUNK2 (10.1.2.3).

•

The forwarders are deployment clients, receiving their configuration files fromthe deployment server.

Here's the basic set up:

199

For information on forwarders (including the universal forwarder), see "Aboutforwarding and receiving" in this manual.

For information on monitoring Windows event log data, see "Monitor Windowsevent log data".

For information on monitoring files, such as message logs, see "Monitor files anddirectories".

Overview of the set up

Here's an overview of the set up process (the detailed steps follow in the nextsection):

On the deployment server:

1. Create the set of server classes and apps for the deployment clients(forwarders) in serverclass.conf. You'll create two server classes to representthe two OS types (Windows, Linux). For each server class, you'll also create twodeployment apps, for a total of four apps. The apps encapsulate:

The type of input -- the data that the universal forwarder will monitor(Windows event logs or Linux messages).

•

The type of output -- the indexer the forwarder will send data to(SPLUNK1 or SPLUNK2).

•

This configuration results in each universal forwarder belonging to a server classand receiving two apps: one for its inputs and one for its outputs.

2. Create directories to hold the deployment apps.

200

3. Create configuration files (outputs.conf and inputs.conf) to deploy to theforwarders. These files constitute the deployment apps and reside in the appdirectories.

4. Restart the deployment server.

On each Splunk indexer that will be receiving data from universalforwarders:

1. Enable receiving through the Splunk CLI.

2. Restart the indexer.

On each forwarder/deployment client:

1. Create a deploymentclient.conf file that points to the deployment server.

2. Restart the forwarder.

The rest is Splunk magic. After a short delay (while the forwarders receive andact upon their deployed content), Windows event logs begin flowing fromFflanda-WIN1 to Fflanda-SPLUNK1, and /var/log/messages begin flowing fromFflanda-LINUX1 and Fflanda-LINUX2 to Fflanda-SPLUNK2.

Detailed configuration steps

On the deployment server, Fflanda-SPLUNK3:

1. Install Splunk, if you haven't already done so.

2. Set up your server classes. Create$SPLUNK_HOME/etc/system/local/serverclass.conf with the following settings:

# Global server class [global] # Filter (whitelist) all clients whitelist.0=*

# Server class for Windows [serverClass:Fflanda-WIN] # Filter (whitelist) all Windows clients whitelist.0=Fflanda-WIN*

# App for inputting Windows event logs

201

# This app is only for clients in the server class Fflanda-WIN [serverClass:Fflanda-WIN:app:winevt] #Enable the app and restart Splunk, after the client receives theapp stateOnClient=enabled restartSplunkd=true

# App for forwarding to SPLUNK1 # This app is only for clients in the server class Fflanda-WIN [serverClass:Fflanda-WIN:app:fwd_to_splunk1] stateOnClient=enabled restartSplunkd=true

# Server class for Linux [serverClass:Fflanda-LINUX] # Filter (whitelist) all Linux clients whitelist.0=Fflanda-LINUX*

# App for inputting Linux messages # This app is only for clients in the server class Fflanda-LINUX [serverClass:Fflanda-LINUX:app:linmess] stateOnClient=enabled restartSplunkd=true

# App for forwarding to SPLUNK2 # This app is only for clients in the server class Fflanda-LINUX [serverClass:Fflanda-LINUX:app:fwd_to_splunk2] stateOnClient=enabled restartSplunkd=true

See "Define server classes" for details on how to configure this file.

3. Create the app deployment directories with the following commands:

mkdir ?p $SPLUNK_HOME/etc/deployment-apps/fwd_to_splunk1/default mkdir ?p $SPLUNK_HOME/etc/deployment-apps/fwd_to_splunk2/default mkdir ?p $SPLUNK_HOME/etc/deployment-apps/winevt/default mkdir ?p $SPLUNK_HOME/etc/deployment-apps/linmess/default

Each app gets its own directory, so that they can be deployed individually. Inaddition, the directory name determines the name of the app.

4. Create$SPLUNK_HOME/etc/deployment-apps/fwd_to_splunk1/default/outputs.conf

with the following settings:

[tcpout]

202

defaultGroup=splunk1

[tcpout:splunk1] # Specifies the server that receives data from the forwarder. server=10.1.2.2:9997

For information on outputs.conf, see "Configure forwarders with outputs.conf" inthis manual.

5. Create$SPLUNK_HOME/etc/deployment-apps/fwd_to_splunk2/default/outputs.conf

with the following settings:

[tcpout] defaultGroup=splunk2

[tcpout:splunk2] server=10.1.2.3:9997

6. Create $SPLUNK_HOME/etc/deployment-apps/winevt/default/inputs.conf withthe following settings:

[WinEventLog:Application] disabled=0

[WinEventLog:Security] disabled=0

[WinEventLog:System] disabled=0

For information on monitoring Windows event log data, see "Monitor Windowsevent log data" in the Getting Data In manual.

7. Create $SPLUNK_HOME/etc/deployment-apps/linmess/default/inputs.confwith the following settings:

[monitor:///var/log/messages] disabled=false sourcetype=syslog

For information on monitoring files, such as message logs, see "Monitor files anddirectories" in the Getting Data In manual.

203

8. Restart Splunk on the deployment server.

Note: Because the deployment server in this example is newly configured, itrequires a restart for its configuration to take effect. When clients poll the serverfor the first time, they'll get all the content designated for them. To deploysubsequent content, you generally do not need to restart the server. Instead, youjust invoke the Splunk CLI reload command on the server, as described in"Deploy apps and configurations". By doing so, you ensure that the server willinform its clients of content changes. However, whenever you editserverclass.conf, you must always restart the deployment server for theconfiguration changes to take effect.

To set up the receiving indexers, Fflanda-SPLUNK1 and Fflanda-SPLUNK2,perform these steps on the machines where you want them to reside:

1. Install Splunk, if you haven't already done so.

2. Run the following CLI command:

./splunk enable listen 9997 -auth <username>:<password>

This specifies that the indexer will serve as a receiver, listening for data on port9997. With proper authorization, any forwarder can now send data to the receiverby designating its IP address and port number. You must enable the indexers asreceivers before you enable the forwarders that will be sending data to them.

For more information on enabling receivers, see "Enable a receiver" in thismanual.

3. Restart Splunk.

On each universal forwarder, Fflanda-WIN1, Fflanda-LINUX1, andFflanda-LINUX2:

1. Install the forwarder, if you haven't already done so.

2. You can specify the deployment server as part of the installation process, oryou can specify it in $SPLUNK_HOME/etc/system/local/deploymentclient.confafter the installation, using these settings:

[deployment-client]

204

[target-broker:deploymentServer] # Specify the deployment server that the client will poll. targetUri= 10.1.2.4:8089

See "Configure deployment clients" for details on how to configure this file. Tolearn how to specify the deployment server at installation time, see "Universalforwarder deployment overview" and the topics that follow it.

3. Restart the universal forwarder.

Each forwarder will now poll the deployment server, download its configurationfiles, restart, and begin forwarding data to its receiving indexer.

For a follow-up example showing how to use the deployment server to updateforwarder configurations, see "Example: Add an input to forwarders".

What the communication between the deployment server andits clients looks like

Using the above example, the communication from Fflanda-WIN1 toFflanda-SPLUNK3 on port 8089 would look like this:

Fflanda-WIN1: Hello, I am Fflanda-WIN1.

Fflanda-SPLUNK3: Hello, Fflanda-WIN1. I have been expecting to hear fromyou. I have you down as a member of the Fflanda-WIN server class, and youshould have the fwd_to_splunk1 (checksum=12345) and winevt(checksum=12378) apps.

Fflanda-WIN1: Hmmm, I don?t have those configs. Using this connection I justopened up to you, can I grab the configs from you?

Fflanda-SPLUNK3: Sure! I have them ready for you.

Fflanda-WIN1: Thanks! I am going to back off a random number of secondsbetween 1 and 60 (in case you have a lot of clients that are polling you at themoment) ... OK, now send me the files.

Fflanda-SPLUNK3: Done! You now have fwd_to_splunk1-timestamp.bundle andwinevt-timestamp.bundle.

Fflanda-WIN1: Awesome! I am going to store them in my$SPLUNK_HOME/etc/apps directory. Now I am going to restart myself, and

205

when I come back up I am going to read the configurations that you sent medirectly out of the .bundle files, which I know are just tar balls with a differentextension.

A couple of minutes go by....


Fflanda-SPLUNK3: Hello, Fflanda-WIN1. I have been expecting to hear fromyou. I have you down as a member of the Fflanda-WIN server class, and youshould have the fwd_to_splunk1 (checksum=12345) and winevt(checksum=12378) Apps.

Fflanda-WIN1: Hmmm, I already have both of those, but thanks anyway!

Later on, an admin modifies the winevt/inputs.conf file on Fflanda-SPLUNK3 todisable the collection of system event logs, and then runs the CLI commandsplunk reload deploy-server to force the deployment server to rescanserverclass.conf and the app directories. The next time Fflanda-WIN1 talks toFflanda-SPLUNK3, it goes like this:


Fflanda-SPLUNK3: Hello, Fflanda-WIN1. I have been expecting to hear fromyou. I have you down as a member of the Fflanda-WIN server class, and youshould have the fwd_to_splunk1 (checksum=12345) and winevt(checksum=13299) Apps.

Fflanda-WIN1: Hmmm, I know I have those configs, but the checksum I have forthe winevt configs is different than the one you just told me about. Using thisconnection I just opened up to you, can I grab the updated winevt config fromyou?

Fflanda-SPLUNK3: Sure! I have it ready for you.

Fflanda-WIN1: Thanks! I am going to back off a random number of secondsbetween 1 and 60 (in case you have a lot of clients that are polling you at themoment) ... Ok, now send me the updated config.

Fflanda-SPLUNK3: Done! You now have winevt-newer_timestamp.bundle.

Fflanda-WIN1: Awesome! I am going to store it my $SPLUNK_HOME/etc/appsdirectory and move the old winevt.bundle I had out of the way. Now I am going to

206

restart myself, and when I come back up, I'll have the most up-to-date config.

Example: add an input to forwarders

The previous topic, "Extended example: deploy several forwarders", describedsetting up a deployment environment to manage a set of universal forwarders. Itshowed how to configure a new deployment server to deploy content to a newset of deployment clients. The current example follows on directly from there,using the configurations created in that topic. It shows how to update a forwarderconfiguration file and deploy the updated file to a subset of forwarders, definedby a server class.

Overview of the update process

This example starts with the set of configurations and Splunk instances createdin the topic "Extended example: deploy several forwarders". The Linux universalforwarders now need to start monitoring data from a second source. Toaccomplish this, perform these steps on the deployment server:

1. Edit the inputs.conf file for the Linux server class to add the new source,overwriting the previous version in its apps directory.

2. Use CLI to reload the deployment server, so that it becomes aware of thechange and can deploy it to the appropriate set of clients (forwarders).

You need make changes only on the deployment server. When the deploymentclients in the Linux server class next poll the server, they'll be notified of the newinputs.conf file. They'll download the file, enable it, restart Splunk, andimmediately begin monitoring the second data source.

Detailed configuration steps

On the deployment server:

1. Edit $SPLUNK_HOME/etc/deployment-apps/linmess/default/inputs.conf toadd a new input:

[monitor:///var/log/messages] disabled=false sourcetype=syslog

207

[monitor:///var/log/httpd] disabled=false sourcetype = access_common

2. Use Splunk CLI to reload the deployment server:

./splunk reload deploy-server -class Fflanda-LINUX

Once this command has been run, the deployment server notifies the clients thatare members of the Fflanda-LINUX server class of the changed file. Since thechange doesn't affect the Fflanda-WIN server class, its members don't need toknow about it.

Example: deploy an app

This example walks through the configuration needed to deploy an app, in thiscase, the Splunk light forwarder.

On the deployment server

1. Copy the SplunkLightForwarder app from $SPLUNK_HOME/etc/apps to thedeployment directory, $SPLUNK_HOME/etc/deployment-apps on the deploymentserver.

2. Edit serverclass.conf in /system/local on the deployment server. Add aserver class named "lightforwarders" that includes the light forwarder app:

[global]whitelist.0=*

[serverClass:lightforwarders]whitelist.0=*

[serverClass:lightforwarders:app:SplunkLightForwarder]stateOnClient=enabledrestartSplunkd=true

Note the following:

The [global] stanza is required. It contains any settings that should beglobally applied.

•

208

In the [global] stanza, whitelist.0=* signifies that all of thedeployment server's clients match all server classes defined in thisconfiguration file. In this example, there is just a single server class.

♦

The server class name is "lightforwarders". You can call your serverclasses anything you want.

In the [serverClass:lightforwarders] stanza, whitelist.0=*signifies that all clients match the lightforwarders server class.

♦

•

The [serverClass:lightforwarders:app:SplunkLightForwarder] stanzacontains settings specific to the SplunkLightForwarder app on thelightforwarders server class.

stateOnClient specifies that this app should be enabled on theclient when it is deployed.

♦

restartSplunkd specifies that when this app is deployed, splunkdshould be restarted.

♦

•

See "Define server classes" for details on how to configure this file.

On the deployment client

Edit deploymentclient.conf in /system/local on the deployment client to tellthe client how to contact the deployment server:

[deployment-client][target-broker:deploymentServer]targetUri=<IP:port>

Note the following:

deploymentServer is the default name for a deployment server.• <IP:port> is the IP address and port number for this client's deploymentserver.

•

The file points the client to the deployment server located at IP:port. There, itwill pick up the Splunk light forwarder app, enable it, and restart. See "Configuredeployment clients" for details on how to configure this file.

Deploy in multi-tenant environments

Important: It is recommended that you work with Splunk Professional Serviceswhen designing a multi-tenant deployment.

209

A multi-tenant deployment server topology means that you have more thanone deployment server running on the same Splunk instance, and eachdeployment server is serving content to its own set of deployment clients. (Youcan also achieve the same effect by using two Splunk instances, each with itsown configuration.)

Use tenants.conf to redirect incoming requests from deployment clients toanother deployment server or servers. The typical reason for doing this is tooffload splunkd's HTTP server -- if too many deployment clients aresimultaneously hitting the splunkd HTTP server to download apps andconfigurations, it can overload the deployment server. Over 400 connections atone time has been shown to bog down splunkd's HTTP server, but this does nottake into account hardware or the size of the package the client is downloading.

To set up multiple deployment servers on a single Splunk instance, you:

Create a tenants.conf containing a whitelist or blacklist that tellsdeployment clients which deployment server instance to use.

•

Create a separate instance of serverclass.conf for each deploymentserver, named for that deployment server, like so:<tenantName>-serverclass.conf.

•

For each deployment client, configure deploymentclient.conf the wayyou would if there were just one deployment server.

•

What you can define in tenants.conf

You identify the different deployment servers as "tenants" in tenants.conf on theSplunk instance that will host these deployment servers. There isn't atenants.conf file by default, so you must create one in$SPLUNK_HOME/etc/system/local and define the tenants in it.

For each tenant, create a stanza with the heading [tenant:<tenantName>] withthese attributes:


filterType

Set to whitelist or blacklist. Determines thetype of filter to use. Deployment clients use thefilter to determine which deployment server toaccess.

whitelist

whitelist.<n>

blacklist.<n>

<n> is a number starting at 0, and incrementingby 1. The client stops looking at the filter when<n> breaks.

n/a

210

Set the attribute to one of these valuecategories:

ipAddress: The IP address of thedeployment client. You can usewildcards, for example, 10.1.1.*

•

hostname: The host name ofdeployment client. You can usewildcards, for example, *.splunk.com

•

clientName: A logical, or tag, name thatcan be assigned to a deployment clientin deploymentclient.conf. A clientNametakes precedence over ipAddress orhostname when matching a client to afilter.

•

Example

Here is an example of defining two tenants in the tenants.conf file:

# Define two tenants - dept1 and dept2.# Deployment server configuration for dept1 will be in a matchingdept1-serverclass.conf# Deployment server configuration for dept2 will be in a matchingdept2-serverclass.conf

[tenant:dept1]whitelist.0=*.dept1.splunk.com

[tenant:dept2]whitelist.0=*.dept2.splunk.com

211

Upgrade your deployment

Upgrade your distributed environment

This topic discusses the process of upgrading components of a distributedSplunk deployment.

Upgrading a distributed Splunk environment presents challenges over upgradingan indexer-only Splunk installation. For the purposes of reducing downtime andensuring no data is lost, we strongly recommend that you upgrade your Splunkcomponents in a specific order. This order is depicted in the instructions below.

Note: This is a high-level guidance on upgrading Splunk in a distributedenvironment. We realize that every distributed Splunk environment is different,and therefore do not offer detailed step-by-step procedures. If you haveadditional questions about upgrading your distributed Splunk environment afterreading this topic, you can log a case via the Splunk Support Portal.

Cross-version compatibility between distributed components

For information on compatibility between differerent versions of search headsand search peers (indexers), see "Cross-version compatibility for search heads".

For information on compatibility between indexers and forwarders, see "Indexerand universal forwarder compatibility".

Test your apps prior to the upgrade

Before upgrading your distributed environment, make sure that all of your Splunkapps work on the version of Splunk that you plan to upgrade to.

Important: This procedure is required if you are upgrading a distributedenvironment with a search head pool, because pooled search heads useshared storage space for apps and configurations.

To ensure that your apps work on the desired upgraded version of Splunk:

1. On a reference machine, install the full version of Splunk that you currentlyrun.

212

Note: You can also use an existing Splunk instance, provided that it is notindexing relevant data and is at the same version level as the other instances inyour environment.

2. Install the apps on this Splunk instance.

3. Confirm that the apps work as expected.

4. Upgrade the Splunk instance to the desired version.

5. Test the apps again to make sure they work as desired in the new version.

If the apps work as expected, you can move them to the appropriate locationduring the upgrade of your distributed Splunk environment:

If you use non-pooled search heads, move the apps to$SPLUNK_HOME/etc/apps on each search head during the search headupgrade process.

•

If you use pooled search heads, move the apps to the shared storagelocation where the pooled search heads expect to find the apps.

•

Caution: The migration utility warns you of apps that need to be copied toshared storage for pooled search heads when you upgrade them. It does not,however, copy them for you. You must manually copy all updated apps -including apps that ship with Splunk (such as the Search app and the datapreview feature, which is implemented as an app) - to shared storage duringthe upgrade process. Failure to do so can cause problems with Splunk's userinterface after the upgrade is complete.

Upgrade a distributed environment with multiple indexers andnon-pooled search heads

To maintain availability, Splunk recommends that, when upgrading yourdistributed Splunk environment with multiple indexers and non-pooled searchheads, that you upgrade the search heads first, then upgrade the indexinginfrastructure that supports the search heads. If you have deployment servers inthe environment, be sure to disable those prior to upgrading your search heads.

To upgrade a distributed Splunk environment with multiple indexers andnon-pooled search heads:

213

Prepare the upgrade

1. Confirm that any apps that the pooled search heads use will work on theupgraded version of Splunk, as described in "Test your apps prior to theupgrade" in this topic.

2. If you use a deployment server in your environment, disable it temporarily.This prevents the server from distributing invalid configurations to your otherSplunk components.

3. Upgrade your deployment server, but do not restart it.

Upgrade the search heads

4. Disable and upgrade one of the search heads. Do not allow it to restart.

5. After you upgrade the search head, place the confirmed working apps into the$SPLUNK_HOME/etc/apps directory of the search head.

6. Restart this search head and test for operation and functionality.

7. If there are no problems with the search head, then disable and upgrade theremaining search heads, one by one. Repeat this step until you have reached thelast search head in your environment. Optionally, you can test each search headfor operation and functionality after you bring it up.

8. Once you have upgraded the last search head, test all of the search heads foroperation and functionality.

Upgrade the indexers

9. Disable and upgrade your indexers, one by one. You can restart the indexersimmediately after you upgrade them.

10. Test your search heads to ensure that they find data across all your indexers.

11. After all indexers have been upgraded, restart your deployment server.

Upgrade a distributed environment with multiple indexers andpooled search heads

If your distributed Splunk environment has pooled search heads, the process toupgrade the environment becomes significantly more complex. If your

214

organization has restrictions on downtime, this type of upgrade is best donewithin a maintenance window.

The key concepts to understand about upgrading this kind of environment are:

Pooled search heads must be enabled and disabled as a group.• The version of Splunk on all pooled search heads must be the same.• Apps and configurations that the search heads use must be tested prior toupgrading the search head pool.

•

If you have additional concerns about the guidance shown here, you can log acase via the Splunk Support Portal.

To upgrade a distributed Splunk environment with multiple indexers and pooledsearch heads:

Prepare the upgrade

1. Confirm that any apps that the pooled search heads use will work on theupgraded version of Splunk, as described in "Test your apps prior to theupgrade" in this topic.

2. If you use a deployment server in your environment, disable it temporarily.This prevents the server from distributing invalid configurations to your otherSplunk components.

3. Upgrade your deployment server, but do not restart it.

Upgrade the search head pool

4. Designate a search head (Search Head #1) in your search head pool toupgrade as a test for functionality and operation.

Note: Search heads must be removed from the search head pool temporarilybefore you upgrade them. This must be done for several reasons:

To prevent changes to the apps and/or user objects hosted on the searchhead pool shared storage.

•

To stop the inadvertent migration of local apps and system settings toshared storage during the upgrade.

•

To ensure that you have a valid local configuration to use as a fallback,should a problem occur during the upgrade.

•

215

If problems occur as a result of the upgrade, search heads can be temporarilyused in a non-pooled configuration as a backup.

5. Bring down all of the search heads in your environment.

Note: Search capability will be unavailable at this time, and will remainunavailable until you restart all of the search heads after upgrading.

6. Place the confirmed working apps (as tested in Step 1) in the search headpool shared storage area.

7. Remove Search Head #1 from the search head pool.

Note: Review "Configure search head pooling" for instructions on how to enableand disable search head pooling on each search head.

8. Upgrade Search Head #1.

9. Restart Search Head #1 and test for operation and functionality.

Note: In this case, 'operation and functionality' means that the Splunk instancestarts and that you can log into it. It does not mean that you can use apps orobjects hosted on shared storage. It also does not mean distributed searches willrun correctly.

10. If the upgraded Search Head #1 functions as desired, bring it down and add itback to the search head pool.

11. Upgrade the remaining search heads in the pool, one by one, following Steps7 through 10 above.

Caution: Remove each search head from the search head pool before youupgrade, and add them back to the pool after you upgrade. While it is notnecessary to confirm operation and functionality of each search head, only onesearch head at a time can be up during the upgrade phase. Do not start any ofthe other search heads until you have upgraded all of them.

12. Once you have upgraded the last search head in the pool, then restart all ofthem.

13. Test all search heads for operation and functionality across all of the appsand user objects that are hosted on the search head pool.

216

14. Test distributed search across all of your indexers.

Upgrade the indexers

15. Once you have confirmed that your search heads are functioning as desired,choose an indexer to keep the environment running (Indexer #1), and another toupgrade initially (Indexer #2).

Note: If you do not have downtime concerns, you do not need to perform thisstep.

16. Bring down all of the indexers except Indexer #1.

Note: If you do not have downtime concerns, you can bring down all of theindexers.

17. Upgrade Indexer #2.

18. Bring up Indexer #2 and test for operation and functionality.

Note: Search heads running the latest version of Splunk can communicate withindexers running earlier versions of Splunk.

19. Once you have confirmed proper operation on Indexer #2, bring downIndexer #1.

20. Upgrade Indexer #1 and all of the remaining indexers, one by one. You canrestart the indexers immediately after you upgrade them.

21. Confirm operation and functionality across all of your indexers.

22. Restart your deployment server, and confirm its operation and functionality.

Upgrade forwarders

When upgrading your distributed Splunk environment, you can also upgrade anyuniversal forwarders in that environment. This is not required, however, and youmight want to consider whether or not you need to. Forwarders are alwayscompatible with later version indexers, so you do not need to upgrade them justbecause you've upgraded the indexers they're sending data to.

To upgrade universal forwarders, review the following topics in this manual:

217

Upgrade the Windows universal forwarder• Upgrade the universal forwarder on *nix systems•

Upgrade the Windows universal forwarder

This topic describes the procedure for upgrading your Splunk universal forwarderfrom version 4.2.x or 4.3.x to 5.0.

The upgrade process is much simpler than the original installation. The MSI doesa straight-forward upgrade with no configuration changes. If you need to changeany configuration settings on your forwarders, you can do so after the upgrade,preferably through the deployment server.

Important: Before doing an upgrade, consider whether you really need to. Inmost cases, there's no compelling reason to upgrade a forwarder. Forwardersare always compatible with later version indexers, so you do not need to upgradethem just because you've upgraded the indexers they're sending data to.

This topic describes three upgrade scenarios:

Upgrade a single forwarder with the GUI installer• Upgrade a single forwarder with the command line installer• Perform a remote upgrade of a group of forwarders•

For deployments of any size, you will most likely want to use this last scenario.

Before you upgrade

Be sure to read this section before performing an upgrade.

Back your files up

Before you perform the upgrade, we strongly recommend that you back up yourSplunk configuration files. For information on backing up configurations, read"Back up configuration information" in the Admin manual.

Configure auto load balancing for clustering

If you plan to do load-balanced forwarding to a Splunk cluster, you mustconfigure your existing forwarders to use auto-load balancing. To learn how to dothis, read "Set up load balancing" in this manual.

218

Splunk does not provide a means of downgrading to a previous version; if youneed to revert to an older forwarder release, just uninstall the current version andreinstall the older release.

Upgrade using the GUI installer

You can upgrade a single forwarder with the GUI installer:

1. Download the new MSI file from the Splunk universal forwarder downloadpage.

2. Double-click the MSI file. The Welcome panel is displayed. Follow theonscreen instructions to upgrade the forwarder.

Note: You do not need to stop the forwarder before upgrading. The MSI will dothis automatically as part of the upgrade process.

3. The forwarder will start automatically when you complete the installation.

The installer puts a log of upgrade changes in %TEMP%. It also reports any errors inthe Application Event log.

Upgrade using the command line

You can upgrade a single forwarder by running the command line installer. Toupgrade a group of forwarders, you can load the command line installer into adeployment tool, as described below.

Here are the steps for using the command line installer to upgrade a singleforwarder:

1. Download the new MSI file from the Splunk universal forwarder downloadpage.

2. Install the universal forwarder from the command line by invoking msiexec.exe.

For 32-bit platforms, usesplunkuniversalforwarder-<...>-x86-release.msi:

•

msiexec.exe /i splunkuniversalforwarder-<...>-x86-release.msi[AGREETOLICENSE=Yes /quiet]

219

For 64-bit platforms, usesplunkuniversalforwarder-<...>-x64-release.msi:

•

msiexec.exe /i splunkuniversalforwarder-<...>-x64-release.msi[AGREETOLICENSE=Yes /quiet]


Important: You cannot make configuration changes during upgrade. If youspecify any command line flags besides "AGREETOLICENSE", the MSI justignores them.

Note: You do not need to stop the forwarder before upgrading. The MSI will dothis automatically as part of the upgrade process.

3. The forwarder will start automatically when you complete the installation.

The installer puts a log of upgrade changes in %TEMP%. It also reports any errors inthe Application Event log.

Perform a remote upgrade

To upgrade a group of forwarders across your environment:

1. Load the universal forwarder MSI into your deployment tool. In most cases,you will want to run the command like this:

msiexec.exe /i splunkuniversalforwarder-<...>.msi AGREETOLICENSE=Yes/quiet

See the previous section, "Upgrade using the command line", for details on theMSI command.

2. Execute deployment with your deployment tool.


You might want to test the upgrade locally on one machine before performing aremote upgrade across all your forwarders.

220

Upgrade the universal forwarder for *nix systems

This topic describes the procedure for upgrading your Splunk universal forwarderfrom version 4.2.x or 4.3.x to 5.0.

Important: Before doing an upgrade, consider whether you really need to. Inmost cases, there's no compelling reason to upgrade a forwarder. Forwardersare always compatible with later version indexers, so you do not need to upgradethem just because you've upgraded the indexers they're sending data to.

This topic describes two upgrade scenarios:

Upgrade a single forwarder manually• Perform a remote upgrade of a group of forwarders•

For deployments of any size, you will most likely want to use this secondscenario.

Before you upgrade

Be sure to read this section before performing an upgrade.

Back your files up

Before you perform the upgrade, we strongly recommend that you back up yourSplunk configuration files. For information on backing up configurations, read"Back up configuration information" in the Admin manual.

Splunk does not provide a means of downgrading to a previous version; if youneed to revert to an older forwarder release, just reinstall it.

Configure auto load balancing for clustering

If you plan to do load-balanced forwarding to a Splunk cluster, you mustconfigure your existing forwarders to use auto-load balancing. To learn how to dothis, read "Set up load balancing" in this manual.

How upgrading works

After performing the installation of the new version, your configuration changesare not actually made until you start Splunk.. You can run the migration previewutility at that time to see what will be changed before the files are updated. If you

221

choose to view the changes before proceeding, a file containing the changes thatthe upgrade script proposes to make is written to$SPLUNK_HOME/var/log/splunk/migration.log.<timestamp>

Upgrade a single forwarder

1. Execute the stop command:

$SPLUNK_HOME/bin/splunk stop

Important: Make sure no other processes will start the forwarder automatically(such as Solaris SMF).

2. Install the Splunk package over your existing Splunk deployment:

If you are using a .tar file, expand it into the same directory with the sameownership as your existing universal forwarder instance. This overwritesand replaces matching files but does not remove unique files.

•

If you are using a package manager, such as an RPM, type rpm -U<splunk_package_name>.rpm

•

If you are using a .dmg file (on MacOS), double-click it and follow theinstructions. Be sure to specify the same installation directory as yourexisting installation.

•

If you use init scripts, be sure to include the following so the EULA getsaccepted:

•

./splunk start --accept-license

3. Execute the start command:

$SPLUNK_HOME/bin/splunk start

The following output is displayed:

This appears to be an upgrade of Splunk.--------------------------------------------------------------------------------Splunk has detected an older version of Splunk installed on thismachine. Tofinish upgrading to the new version, Splunk's installer willautomaticallyupdate and alter your current configuration files. Deprecated

222

configurationfiles will be renamed with a .deprecated extension.You can choose to preview the changes that will be made to yourconfigurationfiles before proceeding with the migration and upgrade:If you want to migrate and upgrade without previewing the changes thatwill bemade to your existing configuration files, choose 'y'.If you want to see what changes will be made before you proceed withtheupgrade, choose 'n'.Perform migration and upgrade without previewing configuration changes?[y/n]

4. Choose whether you want to run the migration preview script to see whatchanges will be made to your existing configuration files, or proceed with themigration and upgrade right away.

5. If you choose to view the expected changes, the script provides a list.

6. Once you've reviewed these changes and are ready to proceed with migrationand upgrade, run $SPLUNK_HOME/bin/splunk start again.

Note: You can complete Steps 3 to 5 in one line:

To accept the license and view the expected changes (answer 'n') beforecontinuing the upgrade:

•

$SPLUNK_HOME/bin/splunk start --accept-license --answer-no

To accept the license and begin the upgrade without viewing the changes(answer 'y'):

•

$SPLUNK_HOME/bin/splunk start --accept-license --answer-yes

Perform a remote upgrade

To upgrade a group of forwarders across your environment:

1. Upgrade the universal forwarder on a test machine, as described above.

2. Create a script wrapper for the upgrade commands, as described in "Remotelydeploy a nix universal forwarder with a static configuration". You will need tomodify the sample script to meet the needs of an upgrade.

223

3. Run the script on representative target machines to verify that it works with allrequired shells.

4. Execute the script against the desired set of hosts.


224

Documents

Splunk 5.0.2 Deploy Forwarders