Armada Technical Aspects of Highly Available Microservices.

Agenda.

• Armada overview.

• Service Discovery.

• Clustering & Recovery.

• Both combined.

What is Armada?

Open-source DevOps tool for microservices.

• Development.

• Deployment.

• Configuration.

• Management.

Main components.

• HashiCorp Consul – Distributed Service Catalog. • Stores information about running services. • Every container registers its services.

• HAProxy – High Performance Load Balancer. • Runs inside container. • Maps external service’s address to localhost.

• Supervisord. • Executes service discovery scripts with user specified

parameters.

Service discovery Inside of containers

Service discovery.

• Containers register their services in Consul automatically.

• Other containers can discover it by: • Asking consul for a service with specified name.

• Using HAProxy to bind service’s host address to container’s localhost address.

Service discovery example.

• ‚ leaderboard-server’ service. • Independent backend service.

• ‚leaderboard-panel’ service. • Displays leaderboards.

• Requires ‚ leaderboard-server’ service to work.

Service discovery example.

• ‚leaderboard-server’ service registers itself in consul.

• ‚leaderboard-panel’ service runs a service discovery script through Supervisor. • Service discovery script asks consul for specified

service’s host address.

• Haproxy binds service’s host address to container’s localhost specified port.

Simple supervisor config.

[program:leaderboard_panel] directory=/opt/my-frontend/src command=python3 server.py

[program:require_leaderboard_server] directory=/opt/microservice/src/local_magellan command=python require_service.py 2000 leaderboard-server

Under the hood.

Consul

Supervisor

HAProxy Supervisor

Discovery

require_service.py

Host

Registers leaderboard- server

leaderboard- server?

Registers leaderboard- server

Update config

Creates config

Executes

leaderboard- server leaderboard- panel

Automatic Load Balancing

• Discovered services are load balanced using round-robin method.

• Service is automatically dropped from load balancer when: • It’s stopped. • It enters `critical` state.

• Service is automatically added to load balancer when: • It’s launched. • It enters `passing` or `warning` state.

Service states.

• Armada services use health checks to set states: • Passing

• service works.

• Warning • service works but e.g.: logs too many errors.

• Critical • service doesn’t work.

Services states.

• Each service runs health checks every ~10 seconds.

• Final state equals to the ‚worst’ result of all health checks.

• Service informs Consul about its state after every check.

• If no information is provided ‚critical’ state is assumed.

Custom health-checks.

• By default every service only checks whether port 80 is open. • Yes – passing.

• No – critical.

• Custom health checks can be added to `health-checks` directory anywhere inside the project.

• Default health check can be overridden.

Clustering & Recovery. Making sure everything runs. Always.

Architecture Overview

Armada Agent :8500

Docker containters

Armada Agent :8500

Docker containters

Armada Agent :8500

Docker containters

10.0.0.1

10.0.0.2

10.0.0.3

Clustering

• Joining hosts is a one-command operation. • `armada join 10.0.0.1`.

• `armada join baffled-buffalo`.

• We simply join Consul agents together.

Cluster’s health.

• Raft Consensus Algorithm. • Quorum based.

• Guards data consistency in distributed catalog.

• Requires at least 3 hosts to form recoverable cluster. • Quorum = 2 hosts.

• Single-host development configuration.

Disaster recovery.

• Armada stores runtime information in two places:

• Consul’s key-value store (cluster info).

• Physical file (local services info).

• We’re able to recover from:

• Armada failure.

• Docker failure.

• Host failure.

• Manual screw up.

Armada failure.

• Services keep running (Static configuration).

• Once Armada recovers, services automatically re-register themselves into it.

Docker / Host failure.

• Armada on startup: • Automatically rejoins cluster.

• Restores services from runtime info file.

• In case of disk failure, we’re able to recover services from Consul key-value store.

Manual screw up.

• Armada stores hourly-granulated services’ archive.

• We can manually specify from which file we want to perform recovery.

• Only missing services will be recovered.

• Running services remain intact.

Combining clustering and service discovery. For maximum high availability.

leaderboard-server

leaderboard-server

Armada Agent

Armada Agent

leaderboard-panel

leaderboard-panel

?

2

?

2

sync

10.0.0.1 10.0.0.2

leaderboard-server

Armada Agent

Armada Agent

leaderboard-panel

leaderboard-panel

SERIOUS ERROR

sync

?

1

?

1

10.0.0.1 10.0.0.2

leaderboard-server

leaderboard-server

Armada Agent

Armada Agent

leaderboard-panel

leaderboard-panel

recover

?

sync

2

?

2

10.0.0.1 10.0.0.2

Try it out!

• Vagrant up.

• Development-ready.

• You don’t need 3 VMs.

• `armada create my-service`

That’s all!

Visit our website & github:

• http://armada.sh

• https://github.com/armadaplatform/armada