Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
FUNCTIONALDEVOPSEVENT PROCESSING IN CLOJURE
Andy Marks @andee_marks
Background
we all code!
(prn “Hello World”)
(ns hello.core)
(defn -main [] (prn “Hello World”))
Jetty
(ns hello.core)
(defn -main [] (prn “Hello World”))
✓access requests ✓queues ✓sessions
AWS EC2
Jetty
(ns hello.core)
(defn -main [] (prn “Hello World”))
✓disk ✓network interfaces ✓memory ✓cores/cpus
AWS EC2
Jetty
AWS EC2
Jetty
AWS EC2
Jetty
DNS (Master)
DNS (Slave)
✓records ✓replication ✓resolution
AWS EC2
Jetty
AWS EC2
Jetty
AWS EC2
Jetty
DNS (Master)
DNS (Slave)
DB
✓performance ✓indexing ✓locking ✓connections
AWS EC2
Jetty
AWS EC2
Jetty
AWS EC2
Jetty
DNS (Master)
DNS (Slave)
DB
Load Balancer
AWS EC2
Jetty
AWS EC2
Jetty
✓connections ✓DoS ✓Clients
AWS EC2
Jetty
AWS EC2
Jetty
AWS EC2
Jetty
DNS (Master)
DNS (Slave)
DB
Load Balancer
AWS EC2
Jetty
AWS EC2
Jetty
Message Q
ueue
✓queues ✓timeouts
that is a lot of information…
from a lot of different places
this is a problem
this is the problem Riemann helps
solve
Riemann is written in
Clojure
52 source files7256 lines
377 functions30 macros
Riemann is written and configured
in Clojure
And so… to Riemann
kyl
e ki
ngsb
ury (
@ap
hyr)
OVERVIEW
Correlate events from disparate sources
What is the latency of the web servers when the DB servers are generating timeouts?
Filter events based on key criteria Ignore all network events with a normal status
Aggregate events to perform statistical analysis (e.g., percentiles) What is the average response rate for 95% of requests?
Monitor status of sources by availability of events
Which services haven’t sent any events for the last 5 minutes?
Route event streams to multiple consumers
Send all error events from DB servers to the DBA team as well as the main operational dashboards
SAMPLE RIEMANN USE CASES
i = f(e)e: thundering herd of events i: information ready for downstream processing
Hypothesis
Event processing is function application at large scale… using
Clojure to build/configure Riemann is a logical choice.
(defrecord Event [ service state description metric tags time ttl
KEY CONCEPTS: EVENTS
“reg1.foo.bar” “reg-service http” “ok” “apache-log” 201 [“reg1” “http”] 104534892 60
riem
ann
sour
ce
KEY CONCEPTS: STREAMS
(logging/init {:file "riemann.log" :console true})
(instrumentation {:interval 5}) ;; self monitor every 5 seconds (periodically-expire 1) ;; check for expired events every second
(tcp-server) (repl-server)
(streams ;; insert magic here!!! prn)
riem
ann
confi
g
KEY CONCEPTS: INDEX
flow of events
index
riemann process
General FP goodness
HOMOICONICITY
“[It] is where a program's source code is written as a basic data structure that the
programming language knows how to access.” Wikipedia
(defn validate-config [file]
(try
(read-strings (slurp file)) (catch clojure.lang.LispReader$ReaderException e (throw (logging/nice-syntax-error e file))))) ri
eman
n so
urce
HIGHER ORDER FUNCTIONS
“A stream is just a function that takes a variable number of child streams and returns a function
that takes an event and responds to the event it is passed when it is invoked.”
http://riemann.io/quickstart.html
“a higher-order function… does at least one of the following:
[1] takes one or more functions as an input,
[2] outputs a function” Wikipedia
HIGHER ORDER FUNCTIONS
“A stream is just a function that takes a variable number of child streams and returns a function
that takes an event and responds to the event it is passed when it is invoked.”
http://riemann.io/quickstart.html
(streams
(rate 5 prn) )
riem
ann
confi
g
IMMUTABILITY
“Streams ‘change’ events by sending altered, shared-structure *copies* of events
downstream.” http://riemann.io/howto.html#split-streams
(streams
(where (host "db04") (with :service "foo" prn) (with :service "bar" prn)) )
riem
ann
confi
g
Specific Clojure goodness
“clojure's remarkably fast for what it is, sexprs make the tree structure of the streams visually apparent, it makes writing the concurrent algos much simpler, and macros allow us to express things like 'by and 'pipe in ways that would be awkward in other languages without building our own AST & interpreter or transpiler, etc.”
“clojure's remarkably fast for what it is, sexprs make the tree structure of the streams visually apparent, it makes writing the concurrent algos much simpler, and macros allow us to express things like 'by and 'pipe in ways that would be awkward in other languages without building our own AST & interpreter or transpiler, etc.”
@APHYR SAID…
S-EXPRESSIONS
“a notation for nested list (tree-structured) data… popularised by the programming
language Lisp” Wikipedia
S-EXPRESSIONS
(streams
(where (and (service #"^riak") (state “critical”)) (email “[email protected]")) )
1 Filter certain events…
2 Let people know…
1
2
riem
ann
confi
g
“make the tree structure of the streams visually apparent”
3
1
2
S-EXPRESSIONS
1 Split on event fields…
2 Detect state changes…
3 Collate and email(streams
(by [:host :service] (changed :state (rollup 5 3600 (email “[email protected]")))) ) ri
eman
n co
nfig
“make the tree structure of the streams visually apparent”
MACROS
(defmacro pipe [marker & stages] `(let [~@(->> stages reverse (interpose marker) (cons marker))] ~marker))
(streams
(pipe - (splitp = service "http req rate" - "0mq req rate" (scale 2 -)) (rate 5
(graphite {:host “127.0.0.1” :port 2003}))) )
riem
ann
confi
g
riem
ann
sour
ce
MACROS: PIPE
(streams
(let [aggregate (rate 5 (graphite {:host “127.0.0.1” :port 2003}))] (splitp = service "http req rate" aggregate "0mq req rate" (scale 2 aggregate))) )
or
riem
ann
confi
gri
eman
n co
nfig
(streams
(pipe - (splitp = service "http req rate" - "0mq req rate" (scale 2 -)) (rate 5
(graphite {:host “127.0.0.1” :port 2003}))) )
Conclusion
Hypothesis
Event processing is function application at large scale: using Clojure to build/configure Riemann is a logical
choice.
homoiconicityhigher order functions
s-expressionsmacros
immutability
CLOJURE AD ABSURDUM
(defn do-you-trust-me? [event] (prn (load-string (:description event))))
(streams
(where (service "event-gen") do-you-trust-me?) )
(ns event-gen.core (:require [riemann.client :refer :all]) (:gen-class :main true))
(defn -main [& args] (let [c (tcp-client {:host "127.0.0.1"})]
(send-event c {:service "event-gen" :description "(+ 1 1)"}) (close-client c)))
riem
ann
clie
ntri
eman
n co
nfig
CLOJURE AD ABSURDUM
(defrecord Event [service description ])
“event-gen” “(+ 1 1)”
Riemann (Clojure)
riemann.config (Clojure)
Client (Clojure)
Clojure
=> “2”