Human-friendly DNSpowered by Golang

We love to work in a casual environment that energizes us to revolutionize the fitness industry.

Site Reliability Engineering @ eGym

● Scale

Site Reliability Engineering @ eGym

Site Reliability Engineering @ eGym

Site Reliability Engineering @ eGym

Room for more SREs :)

Site Reliability Engineering @ eGym

● Scale● Automate● Own reliability

○ 24/7 Monitoring○ On-call

● Consult teams

DNS Basics

DNS Basics

Names

www egym de $NULL. . .

separator

label

DNS Hosts

www egym de. .

domain

host

Top-level domain

DNS Name Spacing

int egym de. .

zone

Sub zone

DNS Name Spacing

http

s://e

n.w

ikip

edia

.org

/wik

i/Dom

ain_

Nam

e_S

yste

m

DNS @ eGym● 20+ domains

○ and counting...

● 18 delegations to sub zones○ “team spaces”

● 700+ resource records○ without sub zone records

Registrar

HosterCloud DNS

Delegation (NS RR)

ZoneFile

Deployment

Change

Scripts

Scripts

Scripts

Read/Parse

Rethinking DNS @ eGym

Single SourceOf Truth

Production DNS(Cloud DNS)Tools

Single Source of Truth● Version Controlled● Human Readable● Non-repetitive

Cloud DNS● Scales● API

Tools● Rollbacks● Replays● Automated (non-interactive mode)● Safeguards

Rethinking DNS @ eGym

Single SourceOf Truth

Production DNS(Cloud DNS)Tools

Single Source of Truth● Version Controlled● Human Readable● Non-repetitive

Cloud DNS● Scales● API

Tools● Rollbacks● Replays● Automated (non-interactive mode)● Safeguards

YAML

Templates

DNS Tooling● PaPuDNS

○ Parses YAML-formatted zone information○ Parses YAML-formatted templates○ Applies templates○ In-Memory database with all resource records○ Fetches current zone information from Cloud DNS via API○ Calculates difference○ Pushes the changes (atomically)

● dns-check○ Parses YAML-formatted “expectations”○ Checks via live DNS if expectations are meet○ Does not (yet) use the same format/database

Zones

github.com/egymgmbh/papudns

zones: - zone: egym.coffee description: Test zone. ttl: 300 templates: - gmail - website names: - name: '@' texts: data: - foobar-site-verification-123456 - name: paloalto forwarding: ttl: 60 target: flaky.cloud.example.com. - name: losangeles addresses: literals: - 192.0.2.99 - 2001:db8:200::99

- Sets zone TTL- Pulls in “names” from templates

- TXT resource record

- CNAME resource record(custom TTL)

- A and AAAA resource records

Templatestemplates: - template: gmail description: > This template adds Google mailservers to a zone. names: - name: '@' mail: ttl: 604800 # 1 week = 604800 seconds mailservers: - mailserver: aspmx.l.google.com. priority: 10 - mailserver: alt1.aspmx.l.google.com. priority: 20 - name: google._domainkey texts: data: - > v=DKIM1; k=rsa; p=foobar123456

www egym de $NULL. . .

github.com/egymgmbh/papudns

Go and YAML● Package yaml.v2

○ gopkg.in/yaml.v2

● func Unmarshal(in []byte, out interface{}) (err error)○ Byte stream to custom struct type○ Struct fields are only unmarshalled if they are exported

(have an upper case first letter), and are unmarshalledusing the field name lowercased as the default key.

● See demo!

Go and DNS● Pure Go resolver

○ export GODEBUG=netdns=go # force pure Go resolver

● Cgo resolver○ export GODEBUG=netdns=cgo # force cgo resolver

● Raw DNS queries○ github.com/miekg/dns/

By default the pure Go resolver is used, because a blocked DNS request consumes only a goroutine, while a blocked C call consumes an operating system thread.

https://golang.org/pkg/net/#Resolver

On the wire: Pure Go resolver vs. Cgo resolver16:33:52.097709 IP (tos 0x0, ttl 64, id 53695, offset 0, flags [DF], proto UDP (17), length 88)

force.59722 > google-public-dns-a.google.com.domain: [bad udp cksum 0x7757 -> 0x35c8!] 18325+ [1au] AAAA? danrl.com. ar: . OPT UDPsize=4096 DO (60)

16:36:26.279509 IP (tos 0x0, ttl 64, id 15282, offset 0, flags [DF], proto UDP (17), length 88)

force.55245 > google-public-dns-a.google.com.domain: [bad udp cksum 0x7757 -> 0x6724!] 10166+ [1au] AAAA? danrl.com. ar: . OPT UDPsize=4096 DO (60)

Resolvers● See demo!

CNAME

www.egym.de.

dualstack.egym-server-1779992439.eu-west-1.elb.amazonaws.com.

2a01:578:3::36d9:cf8b

2a01:578:3::2e89:6e8a

2a01:578:3::36f7:bfb5

CNAME

AAAA

Putting it all together● DNS check

○ Install dns-check○ Define expectations○ Reality check

● PaPuDNS○ Install PaPuDNS○ Define zone information○ Deploy!

Summary● We started building tools in Go

○ SRE ❤� Go

● We intentionally NOT use goroutines for critical deployments○ We want humans (slow) to be able to veto the tool’s actions (fast)

● TODO: Use goroutines/channels for dns-check● TODO: Use single source of truth for monitoring, too

○ And see if that works well (debatable)

- career.egym.com (we are growing!)- code.egym.de (dev blog)- github.com/egymgmbh/papudns (deploy tool)- github.com/egymgmbh/dns-tools (monitoring tool)

- @danrl_com (twitter)- danrl.com (my homepage)- github.com/danrl/playground-2017-08-gopher-meetup

(today’s code)

What time is it?

Time to socialize!