Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Implementing Microservices Tracing with Spring Cloud and

ZipkinBy Reshmi Krishna

@reshmi9kBy Marcin Grzejszczak

@mgrzejszczak

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

About us

• Software Engineer

• Platform Architect, Pivotal

• Women In Tech Community Member

Twitter : @reshmi9kMeetUp : Cloud Native New Yorkhttp://www.meetup.com/Cloud-Native-New-York/

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

About us

• Spring Cloud developer

• Working mostly on Spring Cloud Sleuth and Spring Cloud Contract

• Automation freak

Twitter: @mgrzejszczakBlog: http://toomuchcoding.com

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Agenda• How to correlate logs with Spring Cloud Sleuth?

• How to visualize latency with Spring Cloud Sleuth and Zipkin?

• How to use Pivotal Cloud Foundry to deploy and integrate apps with Spring Cloud Services and Zipkin?

• How to integrate Spring Cloud Sleuth and Zipkin in an existing app?

• How to Investigate issues using PCF Metrics and Zipkin?

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

An ordinary system...

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

CLICK 200

Everything is awesome

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

CLICK 500

Until it’s not

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

https://tonysbologna.files.wordpress.com/2015/09/mario-and-luigi.jpg?w=468&h=578&crop=1

Time to debug

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

It doesn’t look like this

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

More like this

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

On which server / instancewas the exception thrown?

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

The answer: distributed tracing• Span

• Trace

• Logs (annotations)

• Tags (binary annotations)

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Is context propagation simple?How do you pass tracing information (incl. Trace ID) between:

• different libraries?

• thread pools?

• asynchronous communication?

• …?

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Log correlation with Spring Cloud SleuthWe take care of passing tracing information between threads / libraries / contexts for:• Hystrix• RxJava• Rest Template• Feign• Messaging with Spring Integration• Zuul• ...

If you don’t do anything unexpected there’s nothing you need to do to make Sleuth work. Check the docs for more info.

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Now let’s aggregate the logs!Instead of SSHing to the machines to grep logs lets aggregate them!

• With Cloud Foundry’s (CF) Loggregator the logs from different instances are streamed into a single place

• You can harvest your logs with Logstash Forwarder / FileBeat

• You can use ELK stack to stream and visualize the logs

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

<dependencyManagement>

<dependencies>

<dependency>

<groupId>org.springframework.cloud</groupId>

<artifactId>spring-cloud-dependencies</artifactId>

<version>Brixton.SR4</version>

<type>pom</type>

<scope>import</scope>

</dependency>

</dependencies>

</dependencyManagement>

<dependency>

<groupId>org.springframework.cloud</groupId>

<artifactId>spring-cloud-starter-sleuth</artifactId>

</dependency>

Spring Cloud Sleuth with Maven

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

SERVICE 1/readtimeout

REQUEST

BOOM!

SERVICE 2

REQUEST

BOOM!

REQUEST

BOOM!

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Log correlation with Spring Cloud SleuthDEMO

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Great! We’ve found the exception!But meanwhile....

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

CLICK 200

The system is slow...

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

One of the services is slow...

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Which one?How to measure that?

How to visualise latency in a distributed system?

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

The answer is:ZIPKIN

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

SPANS SENT TO COLLECTORS

SPANS SENT TO COLLECTORS

STORE IN DB

APP

APP

UI QUERIES FOR TRACE INFO VIA API

How does Zipkin work?

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Spring Cloud Sleuth and Zipkin integration

● We take care of passing tracing information between threads / libraries / contexts

● Upon closing of a Span we will send it to Zipkin○ either via HTTP (spring-cloud-sleuth-zipkin)○ or via Spring Cloud Stream (spring-cloud-sleuth-stream)

● You can run Zipkin Spring Cloud Stream Collector as a Spring Boot app (spring-cloud-sleuth-zipkin-stream)○ you can add the dependency to Zipkin UI!

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Spring Cloud Sleuth Zipkin with Maven

<dependencyManagement>

<dependencies>

<dependency>

<groupId>org.springframework.cloud</groupId>

<artifactId>spring-cloud-dependencies</artifactId>

<version>Brixton.SR4</version>

<type>pom</type>

<scope>import</scope>

</dependency>

</dependencies>

</dependencyManagement>

<dependency>

<groupId>org.springframework.cloud</groupId>

<artifactId>spring-cloud-starter-zipkin</artifactId>

</dependency>

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Hold it!

● If I have billion services that emit gazillion spans - won’t I kill Zipkin?

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Sampling to the rescue!

● By default Spring Cloud Sleuth sends only 10% of requests to Zipkin

● You can change that by changing the property spring.sleuth.sampler.percentage (for 100% pass 1.0)

● Or register a custom org.springframework.cloud.sleuth.Sampler implementation

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

SERVICE 1/start

REQUEST

RESPONSE

SERVICE 2/foo

SERVICE 3/bar

REQUEST

RESPONSE

REQUEST

RESPONSE

SERVICE 4/baz

REQUEST

RESPONSE

SPRINGONESERVICE/springone

REQUEST

RESPONSE

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

DEMO

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

AGENDA

• How to use Pivotal Cloud Foundry to deploy and integrate apps with Spring Cloud Service and Zipkin?

• How to integrate Spring Cloud Sleuth and Zipkin in an existing app?

• How to Investigate issues using PCF Metrics and Zipkin?

34

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

PIvotal Cloud Foundry & Spring Cloud Services

Applications

Data

Runtime

Middleware

O/S

Virtualization

Servers

Storage

Networking

PCF

IAAS

SPRING CLOUD SERVICESV 1.1.1

CONFIG SERVER

SERVICE REGISTRY

CIRCUIT BREAKER

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Trader App Architecture

Web-UI

Portfolio Accounts

Quotes

DB DB

CONFIG SERVER

SERVICE REGISTRY

CIRCUIT BREAKER

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Integrating Sleuth Stream and Zipkin

SleuthWeb-UI

Quotes

Portofolio

Account

Sleuth

Sleuth

Sleuth

Transport(MQ,HTTP)

Collector

Span Store

Query Server

Zipkin UI

ZIPKIN

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Let’s look at some code & Demo !!!!

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Changes Made

- Add sleuth stream and rabbit binder to build.gradle/pom.xml - Add Sleuth Sampler percentage and logging level for sleuth in config file

(Config Server)

- Deploy Zipkin as a service in PCF- Connect/Bind apps to the same Rabbit MQ instance as zipkin

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

● Log correlation allows you to match logs for a given trace

● Distributed tracing allows you to quickly see latency issues in your system

● Zipkin is a great tool to visualize the latency graph and system dependencies

● Spring Cloud Sleuth integrates with Zipkin and grants you log correlation

● PCF makes integration of your apps and Spring Cloud Sleuth and Zipkin easier

Summary

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

• Code for this presentation (clone and run getReadyForConference.sh - NOTE: you need Vagrant!) : https://github.com/marcingrzejszczak/vagrant-elk-box/tree/presentation

• Sleuth’s documentation:http://cloud.spring.io/spring-cloud-sleuth/spring-cloud-sleuth.html• Repo with Spring Boot Zipkin server: https://github.com/openzipkin/zipkin-java• Zipkin deployed to PCF for Brewery Sample app: http://docsbrewing-zipkin-server.cfapps.io • Pivotal Web Services trial : https://run.pivotal.io/• Trader Application : https://github.com/reshmik/cf-SpringBootTrader.git• Zipkin deployed as PCF

https://github.com/reshmik/Zipkin/tree/master/spring-cloud-sleuth-samples/spring-cloud-sleuth-sample-zipkin-stream

• PCF on your laptop : https://docs.pivotal.io/pcf-dev/

Links

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

Learn More. Stay Connected.

https://gitter.im/spring-cloud/spring-cloud-sleuth

How to Properly Blame Things for Causing Latency: An Introduction to Distributed Tracing and Zipkin by Adrian Cole

@springcentralspring.io/blog

@pivotalpivotal.io/blog

@pivotalcfhttp://engineering.pivotal.io