3 years with Clojure

Life of an immutant3 years with Clojure

About me

• @michaelklishin

• github.com/michaelklishin

Setting expectations

Setting expectations

• This is an experience report

Setting expectations

• This is an experience report

• Not a language primer

Setting expectations

• This is an experience report

• Not a language primer

• Assumes no familiarity except basic facts

Setting expectations

• This is an experience report

• Not a language primer

• Assumes no familiarity except basic facts

• ~200 slides. This is gonna be fast.

Background

Background

• Data processing

Background

• Data processing

• Scheduling systems

Background

• Data processing

• Scheduling systems

• Specialized Web crawlers

Background

• Data processing

• Scheduling systems

• Specialized Web crawlers

• Billing, accounting

Background

• Data processing

• Scheduling systems

• Specialized Web crawlers

• Billing, accounting

• Infrastructure automation

Wanted

Wanted

• An environment that does not suck

Wanted

• An environment that does not suck is operationally sane

Wanted

• An environment that does not suck is operationally sane

• A runtime that has decent tooling (profilers, concurrency-related metrics, GC metrics)

Wanted

• An environment that does not suck is operationally sane

• A runtime that has decent tooling (profilers, concurrency-related metrics, GC metrics)

• A runtime that yields good baseline performance

Wanted

• An environment that does not suck is operationally sane

• A runtime that has decent tooling (profilers, concurrency-related metrics, GC metrics)

• A runtime that yields good baseline performance

• Painless upgrade paths

Wanted

• Community

Wanted

• Community

• Benefit from existing libraries and tools

Wanted

• Community

• Benefit from existing libraries and tools

• Concurrency and parallelism

Wanted

• Community

• Benefit from existing libraries and tools

• Concurrency and parallelism

• A productive, concise language

Wanted

JVM

JVM

JVM

• One of the most mature runtimes you can find

JVM

• One of the most mature runtimes you can find

• Excellent tooling

JVM

• One of the most mature runtimes you can find

• Excellent tooling (Visual VM, YourKit, jstack, jhat, VM/+XX flags, BTrace, …)

JVM

• One of the most mature runtimes you can find

• Excellent tooling

• Excellent baseline performance

JVM

• One of the most mature runtimes you can find

• Excellent tooling

• Excellent baseline performance

• Multilingual: Clojure, Java, JRuby, Scala, JS implementations

Clojure

Clojure

• Very high level, concise, expressive

Clojure

• Very high level, concise, expressive

• Excellent concurrency/parallelism story

Clojure

• Very high level, concise, expressive

• Excellent concurrency/parallelism story

• Very good baseline performance

Clojure

• Very high level, concise, expressive

• Excellent concurrency/parallelism story

• Very good baseline performance

• Consistent, well designed

Clojure

• Very high level, concise, expressive

• Excellent concurrency/parallelism story

• Very good baseline performance

• Consistent, well designed

• Decent tooling

Clojure

• Very high level, concise, expressive

• Excellent concurrency/parallelism story

• Very good baseline performance

• Consistent, well designed

• Decent tooling

• Fun to work with (motivation)

Clojure

• No bored academics, built by a practicioner

Clojure: concurrency/parallelism story

Clojure: concurrency/parallelism story

• Immutable data structures by default

Clojure: concurrency/parallelism story

• Immutable data structures by default

• Identity/values separation

Clojure: concurrency/parallelism story

• Immutable data structures by default

• Identity/values separation

• Concurrency-aware reference types

Clojure: concurrency/parallelism story

• Immutable data structures by default

• Identity/values separation

• Concurrency-aware reference types

• java.util.concurrent

Clojure: concurrency/parallelism story

When I find myself in times of trouble Prof. Doug Lea comes to me Coding lines of wisdom j.u.c.

Clojure: concurrency/parallelism story

• Immutable data structures by default

• Identity/values separation

• Concurrency-aware reference types

• java.util.concurrent

• Reactor, Akka, Disruptor, RxJava…

Clojure: concurrency/parallelism story

• Async I/O: NIO, Netty, Vert.x…

Clojure: concurrency/parallelism story

• You have choice

Clojure: concurrency/parallelism story

• You have choice

• Runtime parallelism (on JVM, CLR)

Clojure: concurrency/parallelism story

• You have choice

• Runtime parallelism (on JVM, CLR)

• Hundreds, thousands of threads at the same time

Clojure: baseline performance

Clojure: baseline performance

• What matters the most is what bytecode the compiler produces

contextneeded.com

Clojure: baseline performance

• What matters the most is what bytecode the compiler produces

• Java interop bytecode is typically the same as produced by javac

Clojure: baseline performance

• What matters the most is what bytecode the compiler produces

• Java interop bytecode is typically the same as produced by javac

• Watch out for reflective calls, primitive boxing

Clojure: baseline performance

• Optional type hints

Clojure: baseline performance

• Optional type hints

• Type inference in simple cases

Clojure: baseline performance

(let [d (java.util.Date.)] d)

Clojure: baseline performance

(let [d (java.util.Date.)] d)

Clojure: baseline performance

• Optional type hints

• Type inference in simple cases

Clojure: baseline performance

• Optional type hints

• Type inference in simple cases

• If JVM can optimize your bytecode, things run screaming fast

Clojure: baseline performance

• Optional type hints

• Type inference in simple cases

• If JVM can optimize your bytecode, things run screaming fast

• From 2% (Java lib wrappers) to 200% (naïve numerics heavy code, tight loops) penalty compared to Java

Clojure: baseline performance

• Immutable data structures have O(log32 n) access complexity

Clojure: baseline performance

• Immutable data structures have O(log32 n) access complexity

• Immutable data structures have associated GC tax

Clojure: baseline performance

• Immutable data structures have O(log32 n) access complexity

• Immutable data structures have associated GC tax

• Focus on exploiting parallelism over squeezing µs from a single thread

Clojure: baseline performance

• Poor startup time, excellent execution performance

Clojure: baseline performance

• Poor startup time, excellent execution performance

• Anecdotal evidence: 10 — 50 times throughput improvements compared to Ruby in real apps

Clojure: stability

Clojure: stability

• Have been using 1.3 since alpha1

Clojure: stability

• Have been using 1.3 since alpha1

• Hit by 4 Clojure bugs in ~3 years

Clojure: stability

• Have been using 1.3 since alpha1

• Hit by 4 Clojure bugs in ~3 years

• 1 bug is an edge case that is not worth fixing (I did something really stupid)

Clojure: stability

• Have been using 1.3 since alpha1

• Hit by 4 Clojure bugs in ~3 years

• 1 bug is an edge case that is not worth fixing (I did something really stupid)

• Alphas more solid than GA releases of almost all other languages I’ve worked with

Clojure: stability

• Hit by 4 Clojure bugs in ~3 years

• 1 bug is an edge case that is not worth fixing (I did something really stupid)

• Alphas more solid than GA releases of almost all other languages I’ve worked with

• Others (e.g. ThoughtWorks) report the same experience

Clojure: stability

• Upgraded 28 apps & libraries from 1.3 to 1.4. Just 1 change due to a tricky 3rd party library. The rest was a drop-in replacement.

Clojure: stability

• Upgraded 28 apps & libraries from 1.3 to 1.4. Just 1 change due to a tricky 3rd party library. The rest was a drop-in replacement.

• When upgrading is this easy, people do it happily

Clojure: stability

• Upgraded 28 apps & libraries from 1.3 to 1.4. Just 1 change due to a tricky 3rd party library. The rest was a drop-in replacement.

• When upgrading is this easy, people do it happily

• No library binary compatibility hell some other JVM languages suffer from

Clojure: stability

• Many active developers are conservative with versioning & calling things “done”

Clojure: stability

• Many active developers are conservative with versioning & calling things “done”

• Fits JVM ecosystem maturity really well

Clojure: stability

• The real reason: Clojure is small and was designed to be simple

Clojure: stability

• No monkey patching (there are features that give you all the good parts of teh)

Clojure: consistency

Clojure: consistency

• Collections/sequences library is very unified

Clojure: consistency

• Collections/sequences library is very unified

• Easy to implement your own data types that work just like core Clojure types

Clojure: consistency

• Collections/sequences library is very unified

• Easy to implement your own data types that work just like core Clojure types

• Almost no “bad baggage” in the core

Clojure: consistency

• Collections/sequences library is very unified

• Easy to implement your own data types that work just like core Clojure types

• Almost no “bad baggage” in the core

• The language was actually designed

Clojure: deployment

Clojure: deployment

• Pretty standard JVM deployment options

Clojure: deployment

• Pretty standard JVM deployment options

• Compile everything AOT into a single jar (“überjar”), including dependencies, assets, etc

Clojure: deployment

• Pretty standard JVM deployment options

• Compile everything AOT into a single jar (“überjar”), including dependencies, assets, etc

• A tiny shell script + start-stop-daemon

Clojure: deployment

• Pretty standard JVM deployment options

• Compile everything AOT into a single jar (“überjar”), including dependencies, assets, etc

• A tiny shell script + start-stop-daemon

• Chef or Capistrano

Clojure: deployment

• CloudFoundry

Clojure: deployment

• CloudFoundry

• Heroku

Clojure: deployment

• CloudFoundry

• Heroku

• Immutant/JBoss AS

Clojure: deployment

• CloudFoundry

• Heroku

• Immutant/JBoss AS

• OpenShift

Clojure: operations

Clojure: operations

• VisualVM, jstack, jheap, …

Clojure: operations

• VisualVM, jstack, jheap, …

• nREPL

Clojure: operations

• VisualVM, jstack, jheap, …

• nREPL

• Remote REPL over HTTP(S)

Clojure: operations

• VisualVM, jstack, jheap, …

• nREPL

• Remote REPL over HTTP(S)

• Not quite Erlang but still impressive

Clojure: operations

• VisualGC gives real insight into what is going on with object allocation, tenuring, death rate

Clojure: operations

• VisualGC gives real insight into what is going on with object allocation, tenuring, death rate

• JVM DTrace support (Solaris, SmartOS, FreeBSD, OS X)

Clojure: operations

• VisualGC gives real insight into what is going on with object allocation, tenuring, death rate

• JVM DTrace support (Solaris, SmartOS, FreeBSD, OS X)

• BTrace

Clojure: community

Clojure: community

• Full of intelligent engineers who don’t give two shits about Hacker News

Clojure: community

• Full of intelligent engineers who don’t give two shits about Hacker News

• Over 10 conferences to date

Clojure: community

• Full of intelligent engineers who don’t give two shits about Hacker News

• Over 10 conferences to date

• ~10K people on the mailing list

Clojure: community

• Full of intelligent engineers who don’t give two shits about Hacker News

• Over 10 conferences to date

• ~10K people on the mailing list

• Users from small and not-so-small startups to Fortune 50 corporations in finance, biotech, retail

Clojure: fun to work with

Clojure: fun to work with

• Profoundly changes the way you think

Clojure: changes the way you think

Clojure: changes the way you think

• About managing your program state

Clojure: changes the way you think

• About managing your program state

• About the value of experimenting (REPL driven development)

Clojure: changes the way you think

• About managing your program state

• About the value of experimenting (REPL driven development)

• About metaprogramming

Clojure: changes the way you think

• About managing your program state

• About the value of experimenting (REPL driven development)

• About metaprogramming

• About isolating side effects

Clojure: changes the way you think

Clojure: changes the way you think

• Program = pure core + parts that communicate with the outside world (network, disk I/O)

Clojure: changes the way you think

• Program = pure core + parts that communicate with the outside world (network, disk I/O)

• Pure parts are ridiculously easy to reason about, test, experiment with and reuse

Clojure: changes the way you think

• Program = pure core + parts that communicate with the outside world (network, disk I/O)

• Pure parts are ridiculously easy to reason about, test, experiment with and reuse

• Typical Rails app: 95% of methods have side effects

Clojure: changes the way you think

• Typical Clojure app: 40% of functions have side effects

Clojure: changes the way you think

• Typical Clojure app: 40% of functions have side effects

• Anecdotal evidence: 3-4 times less time spent on tests

Clojure: changes the way you think

• Typical Clojure app: 40% of functions have side effects

• Anecdotal evidence: 3-4 times less time spent on tests

• Moral of the story: make more of your code not have side effects

Clojure: fun to work with

• Profoundly changes the way you think

Clojure: fun to work with

• Profoundly changes the way you think

• Motivates you to learn

Clojure: fun to work with

• Profoundly changes the way you think

• Motivates you to learn

• Easy things are easy, hard things are possible

Clojure: fun to work with

• Profoundly changes the way you think

• Motivates you to learn

• Easy things are easy, hard things are possible

• Wheels are already invented in clojure.core or JDK

Clojure: fun to work with

• Stable: fewer distractions, get in the flow

Clojure: fun to work with

• Stable: fewer distractions, get in the flow

• REPL driven development keeps you in the flow for longer periods

Clojure: fun to work with

• Stable: fewer distractions, get in the flow

• REPL driven development keeps you in the flow for longer periods

• Polymorphism done right <3

Clojure: fun to work with

“Clojure demands that you raise your game and rewards you greatly for it…”

Clojure: fun to work with

• The community is full of smart and down to earth people.

Clojure: use cases

Clojure: use cases

• Data processing

Clojure: use cases

• Data processing

• Stream processing

Clojure: use cases

• Data processing

• Stream processing

• Micro-services

Clojure: use cases

• Data processing

• Stream processing

• Micro-services

• Machine learning

Clojure: use cases

• Data processing

• Stream processing

• Micro-services

• Machine learning

• Web development

Clojure: misc

Clojure: misc

• Leiningen (leiningen.org) is excellent

Clojure: misc

• Leiningen (leiningen.org) is excellent

• Deps management, build tool, custom tasks are just Clojure functions

Clojure: misc

• Leiningen (leiningen.org) is excellent

• Deps management, build tool, custom tasks are just Clojure functions

• Packages your code + all deps into a single jar for deployment

Clojure: misc

• Leiningen (leiningen.org) is excellent

• Deps management, build tool, custom tasks are just Clojure functions

• Packages your code + all deps into a single jar for deployment

• Makes it trivial to develop against multiple Clojure versions

Clojure: misc

• clojars.org

Clojure: misc

• clojars.org

• Deploy via SSH (SCP) or HTTPS

Clojure: misc

• clojars.org

• Deploy via SSH (SCP) or HTTPS

• Very easy to deploy snapshot releases

Clojure: misc

• clojars.org

• Deploy via SSH (SCP) or HTTPS

• Very easy to deploy snapshot releases

• Clojars and Leiningen are maintained by the same core team

Clojure: misc

• travis-ci.org support

Clojure: misc

• travis-ci.org support

• Test against multiple JDKs

Clojure: ecosystem

• Feels a bit like Ruby circa 2009

Clojure: ecosystem

• Feels a bit like Ruby circa 2009

• But with the entire JVM ecosystem at your reach

Clojure: ecosystem

• Feels a bit like Ruby circa 2009

• But with the entire JVM ecosystem at your reach

• And Leiningen + clojars.org

Clojure: ecosystem

• Feels a bit like Ruby circa 2009

• But with the entire JVM ecosystem at your reach

• And Leiningen + clojars.org

• Community ~ doubles every year in size

Clojure: ecosystem

• Feels a bit like Ruby circa 2009

• But with the entire JVM ecosystem at your reach

• And Leiningen + clojars.org

• Community ~ doubles every year in size

• Google for “The State of Clojure” survey

Clojure: ecosystem

• ~35% of people come from Java, ~35% from Ruby or Python

Clojure: ecosystem

• ~35% of people come from Java, ~35% from Ruby or Python

• Libraries, not frameworks

Clojure: ecosystem

• ~35% of people come from Java, ~35% from Ruby or Python

• Libraries, not frameworks

• Decent libraries just for almost every problem

Clojure: ecosystem

• ~35% of people come from Java, ~35% from Ruby or Python

• Libraries, not frameworks

• Decent libraries just for almost every problem

• 15 books, 2 outdated (cover 1.2)

Clojure: ecosystem

• ~35% of people come from Java, ~35% from Ruby or Python

• Libraries, not frameworks

• Decent libraries just for almost every problem

• 15 books, 2 outdated (cover 1.2)

• clojure-doc.org, tryclj.com, …

Clojure: ecosystem

Clojure: what sucks

Clojure: what sucks

• clojure.org

Clojure: what sucks

• clojure.org

• Documentation (guides, the ref is mostly OK, books are great)

Clojure: what sucks

• clojure.org

• Documentation (guides, the ref is mostly OK, books are great)

• Compiler messages can be cryptic

Clojure: what sucks

• clojure.org

• Documentation (guides, the ref is mostly OK, books are great)

• Compiler messages can be cryptic

• JVM startup time + code compilation on each run (without REPL or tools such as Drip)

Clojure: what sucks

• Paper contributor agreement

ClojureWerkz

ClojureWerkz

“A growing collection of open source Clojure libraries…”

ClojureWerkz

“It just werkz…”

ClojureWerkz

• Modern targets (Clojure 1.5+, JDK 6)

ClojureWerkz

• Modern targets (Clojure 1.5+, JDK 6)

• Feature rich

ClojureWerkz

• Modern targets (Clojure 1.5+, JDK 6)

• Feature rich

• Well documented

ClojureWerkz

• Modern targets (Clojure 1.5+, JDK 6)

• Feature rich

• Well documented

• Friendly license (EPL, the same as Clojure)

ClojureWerkz

• Modern targets (Clojure 1.5+, JDK 6)

• Feature rich

• Well documented

• Friendly license (EPL, the same as Clojure)

• Tested against multiple Clojure versions, 3 JDKs on travis-ci.org

ClojureWerkz

clojurewerkz.org

ClojureWerkz

• Projects reuse a lot of Java libraries

ClojureWerkz

• Projects reuse a lot of Java libraries

• Which are often officially supported (e.g. RabbitMQ, ElasticSearch, MongoDB, Riak, Reactor)

“ClojureWerkz stuff is really improving my Clojure experience, which is rapidly becoming my language of choice…”

Clojure: the takeaway

Clojure: the takeaway

• A lot of merit of its own

Clojure: the takeaway

• A lot of merit of its own

• Great stability

Clojure: the takeaway

• A lot of merit of its own

• Great stability

• Hosted (symbiotic) language is a great thing

Clojure: the takeaway

• A lot of merit of its own

• Great stability

• Hosted (symbiotic) language is a great thing

• Don’t fear the JVM

Clojure: the takeaway

• A lot of merit of its own

• Great stability

• Hosted (symbiotic) language is a great thing

• Don’t fear the JVM

• Immutability is essential for sane concurrent programming

Clojure: the takeaway

• Clojure greatly rewards those who choose to use it

The real reason?

The real reason?

???

clojure.core/lazy-cat

Thank you

• @michaelklishin

• github.com/michaelklishin

• clojurewerkz.org

• michael@clojurewerkz.org