3 years with Clojure

Life of an immutant3 years with Clojure

About me

• @michaelklishin

• github.com/michaelklishin

Setting expectations


• This is an experience report



• Not a language primer




• Assumes no familiarity except basic facts




• Assumes no familiarity except basic facts

• ~200 slides. This is gonna be fast.

Background

Background

• Data processing

Background

• Data processing

• Scheduling systems

Background

• Data processing


• Specialized Web crawlers

Background

• Data processing



• Billing, accounting

Background

• Data processing



• Billing, accounting

• Infrastructure automation

Wanted

Wanted

• An environment that does not suck

Wanted

• An environment that does not suck is operationally sane

Wanted


• A runtime that has decent tooling (profilers, concurrency-related metrics, GC metrics)

Wanted



• A runtime that yields good baseline performance

Wanted



• A runtime that yields good baseline performance

• Painless upgrade paths

Wanted

• Community

Wanted

• Community

• Benefit from existing libraries and tools

Wanted

• Community


• Concurrency and parallelism

Wanted

• Community


• Concurrency and parallelism

• A productive, concise language

Wanted

JVM

JVM

JVM

• One of the most mature runtimes you can find

JVM


• Excellent tooling

JVM


• Excellent tooling (Visual VM, YourKit, jstack, jhat, VM/+XX flags, BTrace, …)

JVM



• Excellent baseline performance

JVM



• Excellent baseline performance

• Multilingual: Clojure, Java, JRuby, Scala, JS implementations

Clojure

Clojure

• Very high level, concise, expressive

Clojure


• Excellent concurrency/parallelism story

Clojure



• Very good baseline performance

Clojure




• Consistent, well designed

Clojure





• Decent tooling

Clojure





• Decent tooling

• Fun to work with (motivation)

Clojure

• No bored academics, built by a practicioner

Clojure: concurrency/parallelism story


• Immutable data structures by default



• Identity/values separation




• Concurrency-aware reference types





• java.util.concurrent


When I find myself in times of trouble Prof. Doug Lea comes to me Coding lines of wisdom j.u.c.





• java.util.concurrent

• Reactor, Akka, Disruptor, RxJava…


• Async I/O: NIO, Netty, Vert.x…


• You have choice


• You have choice

• Runtime parallelism (on JVM, CLR)


• You have choice

• Runtime parallelism (on JVM, CLR)

• Hundreds, thousands of threads at the same time

Clojure: baseline performance


• What matters the most is what bytecode the compiler produces

contextneeded.com



• Java interop bytecode is typically the same as produced by javac



• Java interop bytecode is typically the same as produced by javac

• Watch out for reflective calls, primitive boxing


• Optional type hints



• Type inference in simple cases


(let [d (java.util.Date.)] d)


(let [d (java.util.Date.)] d)







• If JVM can optimize your bytecode, things run screaming fast




• If JVM can optimize your bytecode, things run screaming fast

• From 2% (Java lib wrappers) to 200% (naïve numerics heavy code, tight loops) penalty compared to Java


• Immutable data structures have O(log32 n) access complexity



• Immutable data structures have associated GC tax



• Immutable data structures have associated GC tax

• Focus on exploiting parallelism over squeezing µs from a single thread


• Poor startup time, excellent execution performance


• Poor startup time, excellent execution performance

• Anecdotal evidence: 10 — 50 times throughput improvements compared to Ruby in real apps

Clojure: stability

Clojure: stability

• Have been using 1.3 since alpha1

Clojure: stability


• Hit by 4 Clojure bugs in ~3 years

Clojure: stability



• 1 bug is an edge case that is not worth fixing (I did something really stupid)

Clojure: stability




• Alphas more solid than GA releases of almost all other languages I’ve worked with

Clojure: stability



• Alphas more solid than GA releases of almost all other languages I’ve worked with

• Others (e.g. ThoughtWorks) report the same experience

Clojure: stability

• Upgraded 28 apps & libraries from 1.3 to 1.4. Just 1 change due to a tricky 3rd party library. The rest was a drop-in replacement.

Clojure: stability


• When upgrading is this easy, people do it happily

Clojure: stability


• When upgrading is this easy, people do it happily

• No library binary compatibility hell some other JVM languages suffer from

Clojure: stability

• Many active developers are conservative with versioning & calling things “done”

Clojure: stability

• Many active developers are conservative with versioning & calling things “done”

• Fits JVM ecosystem maturity really well

Clojure: stability

• The real reason: Clojure is small and was designed to be simple

Clojure: stability

• No monkey patching (there are features that give you all the good parts of teh)

Clojure: consistency


• Collections/sequences library is very unified



• Easy to implement your own data types that work just like core Clojure types




• Almost no “bad baggage” in the core




• Almost no “bad baggage” in the core

• The language was actually designed

Clojure: deployment

Clojure: deployment

• Pretty standard JVM deployment options

Clojure: deployment


• Compile everything AOT into a single jar (“überjar”), including dependencies, assets, etc

Clojure: deployment



• A tiny shell script + start-stop-daemon

Clojure: deployment



• A tiny shell script + start-stop-daemon

• Chef or Capistrano

Clojure: deployment

• CloudFoundry

Clojure: deployment

• CloudFoundry

• Heroku

Clojure: deployment

• CloudFoundry

• Heroku

• Immutant/JBoss AS

Clojure: deployment

• CloudFoundry

• Heroku

• Immutant/JBoss AS

• OpenShift

Clojure: operations

Clojure: operations

• VisualVM, jstack, jheap, …

Clojure: operations


• nREPL

Clojure: operations


• nREPL

• Remote REPL over HTTP(S)

Clojure: operations


• nREPL

• Remote REPL over HTTP(S)

• Not quite Erlang but still impressive

Clojure: operations

• VisualGC gives real insight into what is going on with object allocation, tenuring, death rate

Clojure: operations


• JVM DTrace support (Solaris, SmartOS, FreeBSD, OS X)

Clojure: operations


• JVM DTrace support (Solaris, SmartOS, FreeBSD, OS X)

• BTrace

Clojure: community

Clojure: community

• Full of intelligent engineers who don’t give two shits about Hacker News

Clojure: community


• Over 10 conferences to date

Clojure: community



• ~10K people on the mailing list

Clojure: community



• ~10K people on the mailing list

• Users from small and not-so-small startups to Fortune 50 corporations in finance, biotech, retail

Clojure: fun to work with


• Profoundly changes the way you think

Clojure: changes the way you think


• About managing your program state



• About the value of experimenting (REPL driven development)




• About metaprogramming




• About metaprogramming

• About isolating side effects



• Program = pure core + parts that communicate with the outside world (network, disk I/O)



• Pure parts are ridiculously easy to reason about, test, experiment with and reuse



• Pure parts are ridiculously easy to reason about, test, experiment with and reuse

• Typical Rails app: 95% of methods have side effects


• Typical Clojure app: 40% of functions have side effects



• Anecdotal evidence: 3-4 times less time spent on tests



• Anecdotal evidence: 3-4 times less time spent on tests

• Moral of the story: make more of your code not have side effects





• Motivates you to learn




• Easy things are easy, hard things are possible




• Easy things are easy, hard things are possible

• Wheels are already invented in clojure.core or JDK


• Stable: fewer distractions, get in the flow



• REPL driven development keeps you in the flow for longer periods



• REPL driven development keeps you in the flow for longer periods

• Polymorphism done right <3


“Clojure demands that you raise your game and rewards you greatly for it…”


• The community is full of smart and down to earth people.

Clojure: use cases

Clojure: use cases

• Data processing

Clojure: use cases

• Data processing

• Stream processing

Clojure: use cases

• Data processing


• Micro-services

Clojure: use cases

• Data processing


• Micro-services

• Machine learning

Clojure: use cases

• Data processing


• Micro-services

• Machine learning

• Web development

Clojure: misc

Clojure: misc

• Leiningen (leiningen.org) is excellent

Clojure: misc


• Deps management, build tool, custom tasks are just Clojure functions

Clojure: misc



• Packages your code + all deps into a single jar for deployment

Clojure: misc



• Packages your code + all deps into a single jar for deployment

• Makes it trivial to develop against multiple Clojure versions

Clojure: misc

• clojars.org

Clojure: misc

• clojars.org

• Deploy via SSH (SCP) or HTTPS

Clojure: misc

• clojars.org


• Very easy to deploy snapshot releases

Clojure: misc

• clojars.org


• Very easy to deploy snapshot releases

• Clojars and Leiningen are maintained by the same core team

Clojure: misc

• travis-ci.org support

Clojure: misc

• travis-ci.org support

• Test against multiple JDKs

Clojure: ecosystem

• Feels a bit like Ruby circa 2009

Clojure: ecosystem


• But with the entire JVM ecosystem at your reach

Clojure: ecosystem



• And Leiningen + clojars.org

http://clojars.org

Clojure: ecosystem




• Community ~ doubles every year in size

Clojure: ecosystem




• Community ~ doubles every year in size

• Google for “The State of Clojure” survey

Clojure: ecosystem

• ~35% of people come from Java, ~35% from Ruby or Python

Clojure: ecosystem


• Libraries, not frameworks

Clojure: ecosystem



• Decent libraries just for almost every problem

Clojure: ecosystem




• 15 books, 2 outdated (cover 1.2)

Clojure: ecosystem




• 15 books, 2 outdated (cover 1.2)

• clojure-doc.org, tryclj.com, …

http://clojure-doc.org

http://tryclj.com

Clojure: ecosystem

Clojure: what sucks

Clojure: what sucks

• clojure.org

Clojure: what sucks

• clojure.org

• Documentation (guides, the ref is mostly OK, books are great)

Clojure: what sucks

• clojure.org


• Compiler messages can be cryptic

Clojure: what sucks

• clojure.org


• Compiler messages can be cryptic

• JVM startup time + code compilation on each run (without REPL or tools such as Drip)

Clojure: what sucks

• Paper contributor agreement

ClojureWerkz

ClojureWerkz

“A growing collection of open source Clojure libraries…”

ClojureWerkz

“It just werkz…”

ClojureWerkz

• Modern targets (Clojure 1.5+, JDK 6)

ClojureWerkz


• Feature rich

ClojureWerkz


• Feature rich

• Well documented

ClojureWerkz


• Feature rich

• Well documented

• Friendly license (EPL, the same as Clojure)

ClojureWerkz


• Feature rich

• Well documented

• Friendly license (EPL, the same as Clojure)

• Tested against multiple Clojure versions, 3 JDKs on travis-ci.org

http://travis-ci.org

ClojureWerkz

clojurewerkz.org

ClojureWerkz

• Projects reuse a lot of Java libraries

ClojureWerkz

• Projects reuse a lot of Java libraries

• Which are often officially supported (e.g. RabbitMQ, ElasticSearch, MongoDB, Riak, Reactor)

“ClojureWerkz stuff is really improving my Clojure experience, which is rapidly becoming my language of choice…”

Clojure: the takeaway


• A lot of merit of its own



• Great stability



• Great stability

• Hosted (symbiotic) language is a great thing



• Great stability


• Don’t fear the JVM



• Great stability


• Don’t fear the JVM

• Immutability is essential for sane concurrent programming


• Clojure greatly rewards those who choose to use it

The real reason?

The real reason?

???

clojure.core/lazy-cat

Thank you

• @michaelklishin

• github.com/michaelklishin

• clojurewerkz.org

• [email protected]

mailto:[email protected]

Software

3 years with Clojure