Click here to load reader

Islands*of*Splunk* - .conf2017 | The 8th Annual Splunk ... · PDF fileBackground*! Splunk*administrators*are*increasingly*required*to*provision*Splunk* as*aservice*offering*to*mulJple*customers*

  • View

  • Download

Embed Size (px)

Text of Islands*of*Splunk* - .conf2017 | The 8th Annual Splunk ... · PDF fileBackground*!...

  • Copyright 2014 Splunk Inc.

    Michael de Buin, Schuberg Philis [email protected] Gert Kremer, Schuberg Philis [email protected] Dani Flexer, Splunk [email protected]

    Islands of Splunk MulJple Splunk as a Service Architecture and ImplementaJon

  • Disclaimer


    During the course of this presentaJon, we may make forward looking statements regarding future events or the expected performance of the company. We cauJon you that such statements reflect our current expectaJons and

    esJmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements,

    please review our filings with the SEC. The forward-looking statements made in the this presentaJon are being made as of the Jme and date of its live presentaJon. If reviewed aTer its live presentaJon, this presentaJon may not contain current or accurate informaJon. We do not assume any obligaJon to update any forward looking statements we may make. In addiJon, any informaJon about our roadmap outlines our general product direcJon and is subject to change at any Jme without noJce. It is for informaJonal purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligaJon either to develop the features or funcJonality described or to

    include any such feature or funcJonality in a future release.

  • Agenda

    ! The MulJ-Splunk-as-a-Service (MSaaS) framework ! MSaaS implementaJon @ Schuberg Philis ! DemonstraJon


  • MSaaS Architecture

  • Background ! Splunk administrators are increasingly required to provision Splunk as a service offering to mulJple customers Commonly requires provisioning a Splunk instance to each customer

    ! MSaaS is a conceptual framework designed to help deliver such an offering

    ! Schuberg-Philis have implemented this framework using Chef on Apache CloudStack


  • FuncJonal Requirements


    ! AutomaJc deployment and update of mulJple Splunk instances ! Packaged binaries, scripts and configuraJons ! Modular ! Each instance can scale from very small to as large as necessary ! Each instance is customized as needed ! System must funcJon independently without external resources

    Except for authenJcaJon, a datacentre automaJon (DCA) tool and an opJonal license manager to fulfill its purpose

    ! Archiving, backup and resilience requirements are defined per customer

  • FuncJonal Requirements ! Resilience

    Disaster recovery (DR), no single point of failure, indexing Jer resilience, storage resilience

    ! Strict data segregaJon ! Different network jurisdicJons isolated from each other with no shared resources nor shared informaJon

    ! Cross-jurisdicJon access possible when explicitly enabled ! JurisdicJon hierarchy supported

    A jurisdicJon can include other jurisdicJons


  • FuncJonal Requirements ! Indexed data can be shared, subject to jurisdicJon, and indexes copied between instances

    ! MulJple licensing models supported central license manager and license pools per-Island license manager and mulJple keys assigned on deployment global license

    ! Roles defined independently for each instance ! All credenJals maintained in the enterprise idenJty management system and allocated at deployment


  • Architecture Concepts: Island ! A complete Splunk deployment ! No informaJon sharing between Islands ! Own set of users and roles ! Manages a set of Forwarders ! Forwarders can send data to many Islands but are managed by a single Island


  • Architecture Concepts ! Bridges

    Islands without indexing capabiliJes that enable search on mulJple Islands

    ! Deployment unit An independently deployed collecJon of Splunk

    components ! Customer

    An independent user of Splunk a business unit or customer

    ! Island service agributes ReplicaJon factor, search factor, DR requirements,

    security, backup, storage Jer, performance, retenJon plan, daily volume

    ! AdministraJon Island monitors the other Islands


  • MSaaS Deployment Server ! A centralized system responsible for installing, configuring, and updaJng the Islands

    ! Maintains all binaries, applicaJons, configuraJons, apps and rouJng informaJon in a version control system (VCS)

    ! Updates the Islands binaries when necessary

    ! Each Splunk DS maintains its Islands components ConfiguraJon files supplied by the MSaaS

    DS and propagated by Island DS


  • ApplicaJons Each Island has any number of Splunk applicaJons a.k.a. apps Apps are

    Managed centrally and deployed with the Island Versions are maintained in the VCS Customized for the MSaaS as a whole or for a subset of the Islands

    ! MSaaS administraJon apps reside on a dedicated Island Monitors the other Islands for usage, security, charge-back, health and need

    for maintenance

    ! Standard apps deployed with each Island S.o.S Splunk on Splunk App for Unix or for Windows Infrastructure as appropriate


  • MSaaS ImplementaJon @SBP

  • My Company and I Our customers:

    Gert Kremer Mission CriJcal Engineer since 2007

  • Engineering the MSaaS Architecture


    ! Dual datacenter setup, no data backups (RF=SF=2) ! Maximum 100 GB/day per Island ! Centralized license server ! AcJve-standby search heads (rsync-ed)

    ! Not implemented: Splunk Deployment Server AdministraJon Island Dedicated Job Servers

  • Ingredients


    Requirements & use cases M-SaaS Meta Architecture DescripJon


    Splunk Chef Cloud(Stack) Github enterprise

  • Will it blend?


    Configura)on Indexing: Avg KBPS

    Indexing: Avg EPS

    Search: Avg First Event (sec)

    Search: Avg Search (sec)

    HP DL380G7; CPU: 26 Xeon 2.67GHz; Memory: 12GB; OS: Linux 64-bit, Fedora 14 (*)

    22,400 79,057 2.48 20.18

    Linux on EC2: c1.xlarge 800 pIOPS (*)

    12,410 43,639 2.12 27.37

    SBP: 4 CPU/16GB 12,449 43,865 2.82 18.24

    SBP: 8 CPU/32GB 14,715 51,959 1.37 17.24

    (*) hgp://

  • Prior art on Splunk, Chef and Cloud


    ! Best Buy Splunk cookbook hgps://

    ! OpsCode Splunk cookbook hgps://

    ! Splunk Storm (Splunk as a Service) hgp://

  • Scope of AutomaJon


    ! Island creaJon ! Server instanJaJon ! Splunk soTware installaJon

    ! Cluster configuraJon ! Data disk and indexes: creaJon and management

    ! Data replicaJon between Search Heads ! Security (firewall rules, SSL setup) ! Monitoring (process, connecJvity, cluster health, Splunk alerts)

    ! Single-sign-on (SSO) ! Splunk applicaJons

  • Splunk Enterprise at Schuberg Philis

    Search head Search head

    Indexer Indexer Cluster master License server

    Datacenter 1

    Datacenter 2

  • SupporJng Systems

    Splunk Enterprise at Schuberg Philis

    Bridge (SIEM)

    Search head Search head AcJve Directory

    License server

    SupporJng Cloud


    AuthenJcaJng proxy

    Island Customer

    Search head Search head

    Indexer Indexer Cluster master

    Island Customer

    Search head Search head

    Indexer Indexer Cluster master

    Island Customer

    Search head Search head

    Indexer Indexer Cluster master

  • License Server

    Island InstanJaJon

    Search Head

    Indexer Cluster Master

    Proxy Servers


    Search Head

    Datacenter 1

    Datacenter 2

    1. Configure island 2. Deploy island 3. Integrate island

    1. Configure island 2. Deploy island 1. Configure island

  • License Server

    Island InstanJaJon

    Search Head

    Indexer Cluster Master

    Proxy Servers


    Search Head

    Datacenter 1

    Datacenter 2

    1. Configure island 2. Deploy island 3. Integrate island

    1. Configure island 2. Deploy island 1. Configure island

  • Returning the favor


    ! Generalize available Chef Cookbooks ! Splunk monitoring Nagios plugin available in Nagios Exchange ! Splunk deployment best pracJces and tools

  • DemonstraJon by Michael


    [email protected], [email protected]

  • Process Requirements


    ! Service Request When customers request the service, a process is triggered that results in a

    deployed instance of Splunk that implements the customers use-cases and the other agributes of the service requested.

    ! Charge Back The cost of the service to its operators can be charged back to its customers

    based on the actual cost of provisioning the service.

    ! Easy to Onboard The process of incorporaJng data sources into the system is well defined

    and simple

  • Data RouJng ! Islands use-cases can overlap requiring them to share data and data-sources

    ! Data rouJng is maintained in a global rouJng table ! On update, the rouJng table is converted into Splunk configuraJon elements suitable for inserJon into the transforms.conf file that is then propagated to the data collecJon Jer by the Master Deployment Server and Isla

Search related