SERVICE DISCOVERY AND CONFIGURATION MANAGEMENT AT
TRANSFERWISE
FIRST INTRODUCTIONS
EVANGELICAL CUSTOMERS
$1 BILLION IN TRANSFERS PER MONTH
SAVING OUR CUSTOMERS $50MPER MONTH
ENGINEERING CULTURE
WEAK CODE OWNERSHIP
EVERY PART OF THE CODE IS OWNED
ANY TEAM CAN CHANGE ANY PART OF THE CODE
ENGINEERING FOR CUSTOMER IMPACT
ENGINEERS GET CLOSE TO CUSTOMERS AND MAKE DECISIONS
BASED ON DATA
SOME OF OUR TEAMS SERVICES NEED TO BE HIGHLY RESPONSIVE,
SYNCHRONOUS AND SCALE REALLY WELL
OUR PLATFORM TEAM IS 8 PEOPLE STRONG
WITH A HUGE BACKLOG
HOW DID WE HANDLE IT BEFORE?
AN ENGINEER ASKS FOR A PRODUCTION SERVICE
NEEDS TO CHANGE PRODUCTION CONFIG
THIS NEEDS TO STOP
TOO MUCH CHATTERTOO LONG WAITS
STRESS FOR EVERYONE
WE NEED:AUTOMATIC ALLOCATION
PROVISIONINGLOGGING AND MONITORING
WITHOUT ENGINEERS
SERVICE DISCOVERY CAN HELP
WHICH DISCOVERY APPROACH TO CHOOSE?
CLIENT SIDE DISCOVERY
SELF REGISTRATION
NETFLIX OSS IS OUR WAY TO GO
MAKE OUR MONOLITH CONSUME 1 SERVICE IN PRODUCTION
UNDERSTANDING THE SCOPE
Problem: making friends between spring cloud and anything else except spring boot can be tricky. There is no good
adoption mechanism
Solution: grails and spring are close. Let’s read the source, find out what hides beyond spring boot’s netflix-specific
annotations, mimic the approach it was designed for
Lets create a shared bean in the discovery space (used by our monolith parts that consumes the config and starts
communication with Eureka)
Step 2: Let’s start a Ribbon LoadBalancer in that bean, connect it with Eureka client and let him start listening for
apps
Step 3: Most of our communication is through restTemplate. How can we make restTemplate awesome
and @LoadBalanced? Interceptors! Let’s build one
Step 4: Let’s add the interceptor to the needed RestTemplate
Step 5: What if discovery completely fails? Fallbacks!
IS THIS A PERFECT SOLUTION? OF COURSE NOT!
NEED TO MAINTAINTIED TO RESTTEMPLATES
NEEDS ENGINEERS TO THINK ABOUT IT
BUILD CLIENTS AND MAKE EVERY CASE WORK AS A BLACK BOX
IF YOUR SERVICE WANTS TO BE DISCOVERED - MAKE YOUR CLIENT
PROVIDE THE TOOLSET
SOME SERVICES HAVE A EUREKA CLIENT OF THEIR OWN
EUREKA IS NOT USED TO IT (CONFIGURATIONS ARE STORED IN A CONTEXT BEAN)
RESULTING IN OVERRIDES UNLESS CONTROLLED
TESTING IS PROBLEMATICAS REQUIRES A FULL COPY OF PRODUCTION
TESTING THE PIECES IS COMPLETELY ON THE ADOPTING ENGINEER
VARIATION IN HTTP COMMS AND IMPLEMENTATIONS
FEIGN AS A CLIENT BUILDERDIRECT FALLBACKS
HYSTRIX CIRCUIT BREAKINGSPLITTING BALANCING AND DISCOVERY
BETWEEN SERVICES
SOLUTION COMPATIBILITYSPRING BOOT
EUREKARIBBON
SUPPORTING TOOLSET
AROUND 75% ADOPTION PROBLEMS
3 MONTHS30 SERVICES IN PRODUCTION
20 MORE IN THE MAKING
GREATNOW WE HAVE LOTS OF SERVICE
CONFIG TO TRACK
LET’S MAKE THINGS AS BORING AS POSSIBLE
LET’S PUT EVERYTHING IN (PUPPET | CHEF | SALT | ANSIBLE)
SECRET MANAGEMENT IS TRICKY
WE CAN MAKE IT WORK. BUT IS THERE A LESS TEDIOUS WAY?
SPRING CLOUD CONFIG SERVER
NETFLIX ARCHAIUS
HASHICORP VAULT
NICE DOCUMENTATION: HTTPS://CLOUD.SPRING.IO/SPRING-CLOUD-
CONFIG/SPRING-CLOUD-CONFIG.HTML
NAME/VALUE PAIRS
/ENCRYPT & /DECRYPT ENDPOINTS
EASY TO EMBED IN SPRING BOOT APPLICATIONS
SIMPLE REST API, SO PLAYS WELL WITH NON-SPRING TOO!
SUPPORTS TEMPLATE FILES… BUT WE’VE NOT USED THEM
SECRETS DON’T NEED TO BE STORED IN PLAIN TEXT:
/ENCRYPT/DECRYPT
VAULT BACKEND: REQUIRES A TOKEN FROM CLIENT
FAILFAST VS RETRY
GETTING IT INTO PRODUCTION:
“VOLUNTEERED” A FEW SERVICE OWNERS
EVERYTHING WORKED SMOOTHLY. VERY SUSPICIOUS.
GITHUB DOWN: CONFIG SERVER GOES MENTAL
CONFIG SERVER CACHES LOCALLY, BUT ALSO WILL ALWAYS CHECK FOR
NEW CONFIG IN THE REPO
VERY INFREQUENTLY… IT JUST DIES
AND WE DON’T KNOW WHY (YET)
(NEVER HAD >1 NODE DIE AT ONCE, SO NOT AWFUL IMPACT, BUT CONCERNING)
RESULT: SERVICES ARE GETTING INTO PRODUCTION FASTER
Let us change money transfer forever. Together.
https://transferwise.com/jobs/