Microservices Tracing with Spring Cloud and Zipkin

Microservices tracing with Spring Cloud and Zipkin

Marcin Grzejszczak

Marcin Grzejszczak @mgrzejszczak, 11-13 May 2016

About meDeveloper at Pivotal

Part of Spring Cloud Team

Working with OSS:● Accurest - Consumer Driven Contracts verifier for Java● JSON Assert - fluent JSON assertions● Spock Subjects Collaborators Extension● Gradle Test Profiler● Up To Date Gradle Plugin

TWITTER: @MGrzejszczakBLOG: http://TOOMUCHCODING.COM

Marcin Grzejszczak @mgrzejszczak, Kraków, 11-13 May 2016

AgendaWhat is distributed tracing?

How to correlate logs with Spring Cloud Sleuth?

How to visualize latency with Spring Cloud Sleuth and Zipkin?

An ordinary system...

UI calls backend

UI -> BACKEND

Everything is awesome

CLICK 200

Until it’s not

CLICK 500

Time to debug

https://tonysbologna.files.wordpress.com/2015/09/mario-and-luigi.jpg?w=468&h=578&crop=1

It doesn’t look like this

More like this

On which server / instance was the exception thrown?

SSH and grep for ERROR to find it?

Distributed tracing - terminology

Logs (annotations)

Tags (binary annotations)

Logs (annotations)

The basic unit of work (e.g. sending RPC)

● Spans are started and stopped

● They keep track of their timing information

● Once you create a span, you must stop it at some point in the future

● Has a parent and can have multiple children

A set of spans forming a tree-like structure.

● For example, if you are running a book store then

○ Trace could be retriving a list of available books

○ Assuming that to retrive the books you have to send 3 requests to 3 services then you could have at least 3 spans (1 for each hop) forming 1 trace

SERVICE 1

REQUEST

No Trace IdNo Span Id

RESPONSE

SERVICE 2

SERVICE 3

Trace Id = XSpan Id = A

REQUEST

RESPONSE

Trace Id = XSpan Id = BClient Sent

Trace Id = XSpan Id = B

Client Received

Server Received

Trace Id = XSpan Id = C

Trace Id = XSpan Id = BServer Sent

REQUEST

RESPONSE

Trace Id = XSpan Id = DClient Sent

Trace Id = XSpan Id = D

Client Received

Server Received

Trace Id = XSpan Id = E

Trace Id = XSpan Id = DServer Sent

SERVICE 4

REQUEST

RESPONSE

Trace Id = XSpan Id = FClient Sent

Trace Id = XSpan Id = F

Client Received

Server Received

Trace Id = XSpan Id = G

Trace Id = XSpan Id = FServer Sent

Span Id = AParent Id = null

Span Id = BParent Id = A

Span Id = CParent Id = B

Span Id = DParent Id = C

Span Id = EParent Id = D

Span Id = FParent Id = C

Span Id = GParent Id = F

Is it that simple?

How do you pass tracing information (incl. Trace ID) between:

● different libraries?

● thread pools?

● asynchronous communication?

● …?

Log correlation with Spring Cloud Sleuth

We take care of passing tracing information between threads / libraries / contexts for● Hystrix● RxJava● Rest Template● Feign● Messaging with Spring Integration● Zuul● ...

If you don’t do anything unexpected there’s nothing you need to do to make Sleuth work. Check the docs for more info.

Now let’s aggregate the logs!

Instead of SSHing to the machines aggregate the logs!

● With Cloud Foundry’s (CF) Loggergator the logs from different instances are streamed into a single place

● You can harvest your logs with Logstash Forwarder / FileBeat

● You can use ELK stack to stream and visualize the logs

Spring Cloud Sleuth with Maven

<groupId>org.springframework.cloud</groupId>

<artifactId>spring-cloud-dependencies</artifactId>

<version>Brixton.RELEASE</version>

<scope>import</scope>

</dependency>

</dependencies>

</dependencyManagement>

<artifactId>spring-cloud-starter-sleuth</artifactId>

</dependency>

Spring Cloud Sleuth with Gradle

dependencies {

compile "org.springframework.cloud:spring-cloud-starter-sleuth"

dependencyManagement {

imports {

mavenBom "org.springframework.cloud:spring-cloud-dependencies:Brixton.

RELEASE"

Log correlation with Spring Cloud SleuthDEMO

Great! We’ve found the exception!But meanwhile....

The system is slow...

CLICK 200

One of the services is slow?

Which one?How to measure that?

● Client Sent (CS) - The client has made a request

● Server Received (SR) - The server side got the request and will start processing it

● Server Send (SS) - Annotated upon completion of request processing

● Client Received (CR) - The client has successfully received the response from the server side

Let’s log events!

CS 0 ms SR 100 ms

SS 200 msCR 300 ms

● The request started at T=0ms

● It took 300 ms for the client to receive a response

● Server side received the request at T=100 ms

● The request got processed on the server side in 100 ms

● Why is there a delay between sending and receiving messages?

Conclusions

CS 0 ms SR 100 ms

SS 200 msCR 300 ms

https://blogs.oracle.com/jag/resource/Fallacies.html

Logs (annotations)

Represents an event in time associated with a span

● Every span has zero or more logs

● Each log is a timestamped event name

● Event should be the stable name of some notable moment in the lifetime of a span

● For instance, a span representing a browser page load might add an event for each of the Performance.timing moments (check https://developer.mozilla.org/en-US/docs/Web/API/PerformanceTiming)

Main logs

● Client Send (CS)○ The client has made a request - the span was started

● Server Received (SR)○ The server side got the request and will start processing it

○ SR timestamp - CS timestamp = NETWORK LATENCY

Main logs

● Server Send (SS)○ Annotated upon completion of request processing

○ SS timestamp - SR timestamp = SERVER SIDE PROCESSING TIME

● Client Received (CR)○ The client has successfully received the response from the server side

○ CR timestamp - CS timestamp = TIME NEEDED TO RECEIVE RESPONSE

○ SS timestamp - CR timestamp = NETWORK LATENCY

Key-value pair

● Every span may also have zero or more key/value Tags

● They do not have timestamps and simply annotate the spans.

● Example of default tags in Sleuth○ message/payload-size○ http.method○ commandKey for Hystrix

How to visualise latency in a distributed system?

● Zipkin is a distributed tracing system

● It runs as a separate process (you can run it as a Spring Boot application)

● It helps gather timing data needed to troubleshoot latency problems in microservice architectures

● The front end is a "waterfall" style graph of service calls showing call durations as horizontal bars

The answer is: Zipkin

How does Zipkin work?

SPANS SENT TO COLLECTORS

STORE IN DB

UI QUERIES FOR TRACE INFO VIA API

Spring Cloud Sleuth and Zipkin integration

● We take care of passing tracing information between threads / libraries / contexts

● Upon closing of a Span we will send it to Zipkin○ either via HTTP (spring-cloud-sleuth-zipkin)○ or via Spring Cloud Stream (spring-cloud-sleuth-stream)

● You can run Zipkin Sping Cloud Stream Collector as a Spring Boot app (spring-cloud-sleuth-zipkin-stream)○ you can add the dependency to Zipkin UI!

Spring Cloud Sleuth Zipkin with Maven

<artifactId>spring-cloud-dependencies</artifactId>

<version>Brixton.RELEASE</version>

<scope>import</scope>

</dependency>

</dependencies>

</dependencyManagement>

<artifactId>spring-cloud-starter-zipkin</artifactId>

</dependency>

Spring Cloud Sleuth Zipkin with Gradle

dependencies {

compile "org.springframework.cloud:spring-cloud-starter-zipkin"

dependencyManagement {

imports {

mavenBom "org.springframework.cloud:spring-cloud-dependencies:Brixton.

RELEASE"

SERVICE 1/start

REQUEST

No Trace IdNo Span Id

RESPONSE

SERVICE 2/foo

SERVICE 3/bar

Trace Id = XSpan Id = A

REQUEST

RESPONSE

Trace Id = XSpan Id = BClient Sent

Client Received

Server Received

Trace Id = XSpan Id = BServer Sent

REQUEST

RESPONSE

Trace Id = XSpan Id = DClient Sent

Client Received

Server Received

Trace Id = XSpan Id = DServer Sent

SERVICE 4/baz

REQUEST

RESPONSE

Trace Id = XSpan Id = FClient Sent

Client Received

Server Received

Trace Id = XSpan Id = FServer Sent

Zipkin for Brewery

● A test app for Spring Cloud end to end tests

● Source code: https://github.com/spring-cloud-samples/brewery

● Around 10 applications involved

Summary

● Log correlation allows you to match logs for a given trace

● Distributed tracing allows you to quickly see latency issues in your system

● Zipkin is a great tool to visualize the latency graph and system dependencies

● Spring Cloud Sleuth integrates with Zipkin and grants you log correlation

THANK YOU● https://github.com/marcingrzejszczak/vagrant-elk-box/tree/presentation - code for this presentation (clone

and run getReadyForConference.sh - NOTE: you need Vagrant!)

● https://github.com/spring-cloud/spring-cloud-sleuth - Spring Cloud Sleuth repository

● http://cloud.spring.io/spring-cloud-sleuth/spring-cloud-sleuth.html - Sleuth’s documentation

● http://toomuchcoding.com/blog/2016/03/25/spring-cloud-sleuth-rc1-deployed/ - article about RC1 release

● https://github.com/openzipkin/zipkin-java - Repo with Spring Boot Zipkin server

● http://docssleuth-service1.cfapps.io/start - The service1 app from this presentation deployed to Pivotal Cloud Foundry - point of entry to the app

● http://docssleuth-zipkin-server.cfapps.io/ - Zipkin deployed to Pivotal Cloud Foundry

● http://brewery-zipkin-web.cfapps.io - Zipkin deployed to PCF for Brewery Sample app

Marcin Grzejszczak, @mgrzejszczak Kraków, 11-13 May 2016

Microservices Tracing with Spring Cloud and Zipkin

Technology

Porazdeljeno spremljanje izvajanja v arhitekturi mikrostoriteveprints.fri.uni-lj.si/4364/1/63140094-DOMEN_JERIČ... · 2019-02-18 · JAX-RS microservices with distributed tracing,

Advances in Cassandra Tracing with Zipkin (Michael Semb Wever, The Last Pickle) | C* Summit 2016

Tracing 1. Contents Why Tracing Why Tracing Tracing in ASP.NET Tracing in ASP.NET Page Level tracing Page Level tracing Application

Oracle's Microservices Strategy · Microservices. • Operations is not integrated in the platform making it an expensive ... (Zipkin/Jaeger, Vizceral) • Istio (Service mesh) •

Microservices Tracing With Spring Cloud and Zipkin @Szczecin JUG

Implementing Microservices Tracing with Spring Cloud and Zipkin

Microservices At Gilt - NYC Microservices Meetup

Microservices Tracing with Spring Cloud and Zipkin (devoxx)

Microservices Application Tracing Standards and Simulators - Adrians at OSCON

MICROSERVICES. MICROSERVICES CON WSo2. · PDF file1 Microservices. icroservices on So2 Sunqu 2016 WhitePaper MICROSERVICES. MICROSERVICES CON WSo2. por Julio Cejas - IT Advisor SUNQU

Microservices on GKE at Mercari - crash.academy · 2018-06-01 · Microservices is a software development technique that structures an ... Common logging & Tracing Request buffering

Microservices & Containers - Pneuron · Microservices & Containers The Pneuron Perspective. ... The Rise of Microservices Microservices is an architectural style for application

Microservices Tracing With Spring Cloud and Zipkin @CybercomDEV

LINE Shop Architecture - static.line-scdn.net fileLINE Shop Asynchronous Thrift RPC with Armeria Architecture æ«¤µÄ: TIPQ TFSWFS Distributed Tracing - Zipkin Thrift RPC (Remote

Building microservices .net - Dublin Microservices UG 20160928

Loki: An Opensource Zipkin/Prometheus Mashup written in Go

Serverless Platforms - QCon New York...Microservices Batch Processing LBaaS Identity Management Tracing Services Applications Release Management Build Infrastructure Conﬁguration

Tracing 2000+ polyglot microservices at Uber with Jaeger and OpenTracing

Microservices - NYS Forum Home cb..1.pdf · Agenda Technology What are Microservices? Characteristics of Microservices Microservices and SOA Challenges Strategy Adoption –Microservices

2014 Marin Market Report - Roger Zipkin