Practicalnon-blocking microservices

in Java 8

Michał BalińskiSystem Architect

Oleksandr GoldobinSystem Architect

Cloud of microservices for secure IoT

gateway backingservice

coreservice

gateway

gatewaycore

servicebackingservice

The demand and characteristics of our domain

Crowded IoT environment

Slow, long lived connections

Big amount of concurrent connections

Scalability and resilience

Rather I/O intensive than CPU intensive

OTA Gateway – Use Case

TSM

TrustedService

Manager

OTA Gateway – Use Case

TSM

TrustedService

Manager

Security Module

OTA Gateway – Use Case

OTA(Over-the-Air)

Gateway

TSM

TrustedService

Manager

Security Module

DB

OTA Gateway – Use Case

OTA(Over-the-Air)

Gateway

TSM

TrustedService

Manager

Security Module

DB

HTTP

1. submit scripts

OTA Gateway – Use Case

OTA(Over-the-Air)

Gateway

TSM

TrustedService

Manager

Security Module

DB

HTTP

1. submit scripts

HTTP2. encrypt

scripts

OTA Gateway – Use Case

OTA(Over-the-Air)

Gateway

TSM

TrustedService

Manager

Security Module

DB

HTTP

1. submit scripts

HTTP2. encrypt

scripts

TCP

3. store scripts

OTA Gateway – Use Case

OTA(Over-the-Air)

Gateway

TSM

TrustedService

Manager

Security Module

DB

HTTP

1. submit scripts

HTTP2. encrypt

scripts

TCP

4. submition response

3. store scripts

OTA Gateway – Use Case

OTA(Over-the-Air)

Gateway

TSM

TrustedService

Manager

Security Module

DB

HTTP

1. submit scripts

HTTP2. encrypt

scripts

TCP

4. submition response

HTTP5. poll scripts

3. store scripts

OTA Gateway – Use Case

OTA(Over-the-Air)

Gateway

TSM

TrustedService

Manager

Security Module

DB

HTTP

1. submit scripts

HTTP2. encrypt

scripts

TCP

3. store scripts

4. submition response

HTTP5. poll scripts

6. search scripts

OTA Gateway – Use Case

OTA(Over-the-Air)

Gateway

TSM

TrustedService

Manager

Security Module

DB

HTTP

1. submit scripts

HTTP2. encrypt

scripts

TCP

3. store scripts

4. submition response

HTTP5. poll scripts

6. search scripts

7. response with scripts

OTA Gateway – Use Case

OTA(Over-the-Air)

Gateway

TSM

TrustedService

Manager

Security Module

DB

HTTP HTTP

HTTP

TCP

OTA Gateway – Use Case

OTA(Over-the-Air)

Gateway

TSM

TrustedService

Manager

Security Module

DB LogStorage

HTTP HTTP

HTTP

TCP File I/OMonitoringSystem

HTTP

Let’s test blocking approach!

OTA Gateway Blocking - technologies

May 3, 2023 17

TSM

DB LogStorage

HTTP

TCP STDOUT

OTAGateway

JAX-RS

Logback appender

Security Module

JAX-RS

Jedis

JAX-RS Client

OTA Gateway – Blocking - test

OTA(Over-the-Air)

Gateway

send 2 scripts per

request

Security Module

DB LogStorage

HTTP HTTP

HTTP

TCP File I/O

emulated latency200 ms

expected latency

< 450 ms

verified with throughput

over 7k req/s

Blocking – 1k threads

OTA(Over-the-Air)

Gateway

send 2 scripts per

request

Security Module

DB LogStorage

HTTP HTTP

HTTP

TCP File I/O

emulated latency200 ms

max 1000connections

max 1000 threads

expected latency

< 450 ms

Blocking – 1k threads500 req/s 1000 req/s 1500 req/s 2000 req/s

The drawbacks of classic synchronous I/O

One thread per connection

Threads waiting instead of running

Context switches

Resource wasting (~1 MB per thread - 64bit)

Let’s switch from blocking to non-blocking

OTA Gateway Non-blocking - technologies

May 3, 2023 23

TSM

DB LogStorage

HTTP

TCP STDOUT

OTAGateway

JAX-RS 2.0

Logback asyncappender

Async Http Client 2 (Netty)

Security Module

JAX-RS 2.0

Lettuce(Netty)

Non-blocking – 16 threads

OTA(Over-the-Air)

Gateway

send 2 scripts per

request

Security Module

DB LogStorage

HTTP HTTP

HTTP

TCP File I/O

emulated latency200 ms

no limit for connections

max 16 threads

expected latency

< 450 ms

Non-blocking – 16 threads1k req/s 2k req/s 2.5k req/s 3k req/s

Blocking Non-blocking

1000 threads 56 threads

1.2 GB 0.5 GB

~1.2k r/s 2.5k r/s

Let’s talk about challenges

OTA Gateway Blocking – sequence diagram

OTAGatewayTSM Security

Module DB Logs

loop

1. submit scripts

2. encrypt script

3a. store script

4. submition response

3b. count scripts

OTA Gateway Non-blocking – sequence diagram

OTAGatewayTSM Security

Module DB Logs

loop

1. submit scripts

2. encrypt script

3a. store script

4. submition response

3b. count scripts

OTA Non-blocking – realityOTAGateway

TSM Security Module DB Logs

„loop

ed p

roce

ssin

g ch

ain”

1. submit scripts

4. submition response

3b. count scripts

HTT

PSe

rver

2. encrypt script

3a. store script

Logg

ing

DB

Clie

nt

Secu

rity

Cl

ient

Code. Bird view

31May 3, 2023

package org.demo.ota.blocking.rest;

@Path("se")public class ScriptSubmissionResource extends Application {

private static final Logger log = LoggerFactory.getLogger(ScriptSubmissionResource.class); private static final ResourceMetrics METRICS = new ResourceMetrics("ota_submission");

private SecureModuleClient secureModuleClient = SecureModuleClient.instance(); private ScriptStorageClient scriptStorageClient = ScriptStorageClient.instance();

@POST @Path("/{seId}/scripts") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.TEXT_PLAIN) public long submitScripts(@PathParam("seId") String seId, List<Script> scripts) { MDC.put("flow", "submission"); MDC.put("se", seId);

return METRICS.instrument(() -> { log.debug("Processing {} scripts submission", scripts.size(), seId);

for (int i = 0; i < scripts.size(); i++) { final Script script = scripts.get(i);

log.debug("Encrypting {} script", i); final String encryptedPayload = secureModuleClient.encrypt(seId, script.getPayload()); script.setPayload(encryptedPayload);

log.debug("Storing encrypted script {}", i); scriptStorageClient.storeScript(seId, script); }

long numberOfScripts = scriptStorageClient.numberOfScriptsForSe(seId); log.debug("Request processed", seId); return numberOfScripts; }); }

@Override public Set<Object> getSingletons() { return Collections.singleton(this); }}

package org.demo.ota.nonblocking.rest;

@Path("se")public class ScriptSubmissionResource extends Application {

private static final Logger log = LoggerFactory.getLogger(ScriptSubmissionResource.class); private static final ResourceMetrics METRICS = new ResourceMetrics("ota_submission");

private SecureModuleClient secureModuleClient = SecureModuleClient.instance(); private ScriptStorageClient scriptStorageClient = ScriptStorageClient.instance();

@POST @Path("/{seId}/scripts") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.TEXT_PLAIN) public void submitScripts( @PathParam("seId") String seId, List<Script> scripts, @Suspended final AsyncResponse asyncResponse ) { final DiagnosticContext diagnosticContext = new DiagnosticContext("submission", seId);

METRICS.instrumentStage(() -> { log.debug("{} Processing {} scripts submission", diagnosticContext, scripts.size());

return encryptAndStoreAllScripts(diagnosticContext, seId, scripts) .thenCompose( ignore -> scriptStorageClient.numberOfScriptsForSe(seId) ); }) .whenComplete((numberOfScripts, e) -> { if (e != null) { asyncResponse.resume(e); } else { log.debug("{} Request processed", diagnosticContext); asyncResponse.resume(numberOfScripts); } }); }

private CompletionStage<Void> encryptAndStoreAllScripts( final DiagnosticContext diagnosticContext, final String seId, final List<Script> scripts ) { CompletionStage<Void> stage = null; // <- non final field, potential concurrent access bug!

for (int i = 0; i < scripts.size(); i++) { final int scriptIndex = i; final Script script = scripts.get(scriptIndex);

if (stage == null) { stage = encryptAndStoreSingleScript(diagnosticContext, seId, scriptIndex, script); } else { stage = stage.thenCompose(ignore -> encryptAndStoreSingleScript(diagnosticContext, seId, scriptIndex, script)); } }

return stage; }

private CompletionStage<Void> encryptAndStoreSingleScript( final DiagnosticContext diagnosticContext, final String seId, final int scriptIndex, final Script script ) { log.debug("{} Encrypting script {}", diagnosticContext, scriptIndex);

return secureModuleClient .encrypt(seId, script.getPayload()) .thenCompose( encryptedPayload -> { log.debug("{} Storing encrypted script {}", diagnosticContext, scriptIndex); return scriptStorageClient.storeScript(seId, new Script(encryptedPayload)); } ); }

@Override public Set<Object> getSingletons() { return new HashSet<>(Collections.singletonList(this)); }}

Code. Blocking. Submission 1

May 3, 2023 32

@POST@Path("/{seId}/scripts")@Consumes(MediaType.APPLICATION_JSON)@Produces(MediaType.TEXT_PLAIN)public long submitScripts(@PathParam("seId") String seId, List<Script> scripts) {

MDC.put("flow", "submission"); // Setting diagnostic context MDC.put("se", seId);

return METRICS.instrument(() -> { // Instrumenting with metrics //...

Code. Blocking. Submission 2

May 3, 2023 33

log.debug("Processing {} scripts submission", scripts.size(), seId);

for (int i = 0; i < scripts.size(); i++) { Cycle through the scripts

final Script script = scripts.get(i);

log.debug("Encrypting {} script", i); final String encryptedPayload = secureModuleClient .encrypt(seId, script.getPayload()); Encrypting the script

script.setPayload(encryptedPayload);

log.debug("Storing encrypted script {}", i); scriptStorageClient.storeScript(seId, script); Saving the script into

DB}

long numberOfScripts = scriptStorageClient.numberOfScriptsForSe(seId); Getting current number

of scripts in DB

log.debug("Request processed", seId);return numberOfScripts;

Code. Non-blocking. Submission 1

May 3, 2023 34

@POST@Path("/{seId}/scripts")@Consumes(MediaType.APPLICATION_JSON)@Produces(MediaType.TEXT_PLAIN)public void submitScripts( @PathParam("seId") String seId, List<Script> scripts, @Suspended final AsyncResponse asyncResponse) { final DiagnosticContext diagnosticContext = new DiagnosticContext("submission", seId); Creating diagnostic

context

METRICS.instrumentStage(() -> { Instrumenting with metrics

Code. Non-blocking. Submission 1

May 3, 2023 35

@POST@Path("/{seId}/scripts")@Consumes(MediaType.APPLICATION_JSON)@Produces(MediaType.TEXT_PLAIN)public void submitScripts( @PathParam("seId") String seId, List<Script> scripts, @Suspended final AsyncResponse asyncResponse) { final DiagnosticContext diagnosticContext = new DiagnosticContext("submission", seId); Creating diagnostic

context

METRICS.instrumentStage(() -> { Instrumenting with metrics

Code. Non-blocking. Submission 2

May 3, 2023 36

METRICS.instrumentStage(() -> { log.debug(

"{} Processing {} scripts submission", diagnosticContext, scripts.size());

return encryptAndStoreAllScripts(diagnosticContext, seId, scripts) .thenCompose( ignore -> scriptStorageClient.numberOfScriptsForSe(seId) );}).whenComplete((numberOfScripts, e) -> { if (e != null) { asyncResponse.resume(e); } else { log.debug("{} Request processed", diagnosticContext); asyncResponse.resume(numberOfScripts); }});

Code. Non-blocking. Submission 3

May 3, 2023 37

private CompletionStage<Void> encryptAndStoreAllScripts( final DiagnosticContext diagnosticContext, final String seId, final List<Script> scripts) { CompletionStage<Void> stage = null; // <- non final field, potential // concurrent access bug!

for (int i = 0; i < scripts.size(); i++) { Cycle through the scripts

final int scriptIndex = i; final Script script = scripts.get(scriptIndex);

if (stage == null) { stage = encryptAndStoreSingleScript(

diagnosticContext, seId, scriptIndex, script); } else { stage = stage.thenCompose(ignore -> encryptAndStoreSingleScript(

diagnosticContext, seId, scriptIndex, script)); } } return stage;}

Code. Non-blocking. Submission 4

May 3, 2023 38

private CompletionStage<Void> encryptAndStoreSingleScript( final DiagnosticContext diagnosticContext, final String seId, final int scriptIndex, final Script script) { log.debug("{} Encrypting script {}", diagnosticContext, scriptIndex);

return secureModuleClient .encrypt(seId, script.getPayload()) Encrypting the script .thenCompose( encryptedPayload -> { log.debug(

"{} Storing encrypted script {}", diagnosticContext, scriptIndex);

return scriptStorageClient.storeScript( Saving the script into seId,

the DB new Script(encryptedPayload));

} );}

Code. Blocking. Integration

May 3, 2023 39

private final JedisPool pool;

public void storeScript(String seId, Script script) { try (Jedis jedis = pool.getResource()) { jedis.rpush(seId, script.getPayload()); }}

public long numberOfScriptsForSe(String seId) { try (Jedis jedis = pool.getResource()) { return jedis.llen(seId); }}

public Optional<Script> nextScript(String seId) { try (Jedis jedis = pool.getResource()) { return Optional.ofNullable(jedis.lpop(seId)).map(Script::new); }}

public String encrypt(String keyDiversifier, String payload) {// ...}

Code. Non-Blocking. Integration

May 3, 2023 40

private final RedisAsyncCommands<String, String> commands;

public CompletionStage<Void> storeScript(String seId, Script script) { return commands .rpush(seId, script.getPayload()) .thenApply(ignore -> null);}

public CompletionStage<Long> numberOfScriptsForSe(String seId) { return commands.llen(seId);}

public CompletionStage<Optional<String>> nextScript(String seId) { return commands.lpop(seId).thenApply(Optional::ofNullable);}

public CompletionStage<String> encrypt(String keyDiversifier, String payload) {// ...}

Diagnostics in non-blocking systems

No clear stack traces, need for good logs

Name your threads properly

MDC becomes useless (thread locals)

Explicitly pass debug context to trace flows

Be prepared for debuging non-obvious errors

NIO technology landscapeJDK 1.4

NIOJDK 1.7NIO.2

JAX-RS 2.x

Servlet API 3.x

Lessons learned, part 1

Vanila Java 8 for NIO µ-services

Netty best for custom protocols in NIO

Unit tests should be synchronous

Load/stress testing is a must

Make bulkheading and plan your resources

Lessons learned, part 2

Functional programming patterns for readability

Immutability as 1-st class citizen

Scala may be good choice ;-)

Conclusion

Non-blocking processing can really save your resources (== money)

But!

Weight all pros and cons and use non-blocking processing only if you really need it.

Thank you.

Michał Balińskim.balinski@oberthur.com

Oleksandr Goldobino.goldobin@oberthur.com

goldobin

@goldobin

balonus

@MichalBalinski

Readings:• https://github.com/balonus/blocking-vs-nonblocking-demo

• C10k Problem, C10M Problem, Asynchronous I/O

• Boost application performance using asynchronous I/O (M. Tim Jones, 2006)

• Zuul 2 : The Netflix Journey to Asynchronous, Non-Blocking Systems (Netflix, 2016)

• Thousands of Threads and Blocking I/O: The Old Way to Write Java Servers Is New Again (and Way Better) (Paul Tyma, 2008)

• Why Non-Blocking? (Bozhidar Bozhanov, 2011)