Andrew Brown - Senior Platform EngineerJan 18th 2016

HashiCorp at JUST EAT Part II

Overview

JUST EAT

Consul

Vault

Summary

JUST EAT“the world’s leading marketplace for online food delivery”

- Amazon Web Services

- Run hundreds of EC2 Instances at peak

- Customers Worldwide

- Microservices > 100

- Multiple accounts, environments

- Using HashiCorp tools in since 2015

Consul - Initial Rollout

- Cluster Per Environment

- Seed key/values from GitHub (git2consul)

- Expand keys : true

- Every EC2 Instance runs Consul Agent

- Each microservice registers a service and has custom ACL

- Used Consul-Template to generate .config files

Consul

eu-west-1a eu-west-1b eu-west-1c eu-west-1x

GIT2CONSUL

Consul Key/ValuesFeature Configuration

/config/features/<featureName>/….

Environment Configuration

/config/environment/monitoring/….

Settings

/config/settings/messageRegion/….

Consul-TemplateSetup

- Consul-Template pre-baked on AMIs (Packer)

- Service installed during deployment

- Consul-Template files bundled with feature package

- Generated *.config files (.Net)

Lessons Learnt

- Replacing static files meant more effort to run locally

- Worked well

- However... web.config and connectionstrings.config cannot be changed without cycling AppPool !

Consul Libraries.NET Libraries

- using https://github.com/PlayFab/consuldotnet

JustEatConsul

- Setup Client

- Setup default endpoints - key prefixes

- Create configuration object recursively checking all key/values (remove prefix)

- Return configuration object:

var defaultTimeout = configurationRoot[“timeout”]

Consul Libraries

AppSettings

- Add AppSettings as a configuration source

Consul WatchesHttpReload

- Endpoint to trigger rebuild of the ConfigurationRoot

Consul Watch

- Windows (NSSM) Service setup during deployment

- “Watches” key prefixes for the feature deployed

- Invokes a WebRequest to local endpoint

- Only rebuild when a value used by the feature is changed

Consul Watches

Lessons Learnt

- Building functionality into the Code vs Static files

- Watch per key prefix…..

- Reliable

Consul ACLs

ACL per Feature

- Generated during Deployment

Lessons Learnt

- Adding new endpoints, meant a deployment to get new ACL

- Use events and triggers

- Avoid different rules across types of environment

- Create Ad-hoc ACLs...and tidy them afterwards!

- Clear browser local storage

Source of Truth

- Launched a Consul Cluster for Just Eat Source of Truth

- Accounts, Environments, Features, Teams

- UI has transformed JSON files to readable content

- Reliable

Use Case : Centralised Data

Use Case : Consul Locks“Unique” Instance Value

- Require unique node name, used for :

- Consul Node

- Monitoring Stats

- Log Files

- Each EC2 needs an instance position

- Use AWS service SimpleDB - being retired….

Deployment

- Instance Locks key/value

- Finds free position,

- Allocates values

- Releases Lock

Lessons Learnt

- Very easy to setup

- Configurable - retries, timeouts, number of locks

- Reliable

Consul Locks001

003

002

Vault“Why aren’t you using Vault?”

- “We plan to….”

Approach

- Launch separate Vault Clusters

- Keep “config” distinct

- Not held in source control

- Backed by Consul

- Manually added passwords

Vault

eu-west-1a

eu-west-1b

eu-west-1c

eu-west-1a

eu-west-1b

eu-west-1c

Vault

Challenges

- Mechanism to create “tokens” for EC2 instances

- How to access the secrets

- Keeping it simple vs making it secure

- Backing up the data

- Mechanism to add secrets

- Alerting, Auditing, Logging

Vault - Token

Launch Stack

i-12345i-12345

i-12345

ASG

CloudWatch Event

{ “acl” : a1a1a1-b2b2b2 }

.NET Library

- using https://github.com/rajanadar/VaultSharp

JustEatVault

- Use token generated during deployment

var password = configuration[“my_password”]

Vault - Accessing Secrets

Lessons Learnt

- Vault instances didn’t forward requests…..

- Look after your keys

- Read the docs slowly

Vault

- Consul, Packer, Vault, Nomad….

- Use available open source tools

- Newer versions solving workarounds

Summary

Summary

- Some roll-out/migration pain - ports, stats

- Read the docs

- Use the features that suit your needs

- Still making the case for Terraform

Questions?