Embed Size (px)
Citation preview
Netflix and FreeBSD:Using Open Sourceto Deliver StreamingVideo
Jonathan LooneyFOSDEM 2019
NETFLIX AND FREEBSDFOSDEM 2019
Open Connect
Open Connect is Netflix’s CDN.It is global, efficient, and purpose-built for distributing Netflix’s content.
NETFLIX AND FREEBSDFOSDEM 2019
The Open Connect Appliance
The OCA is the “backbone” of the Open Connect network.
The OCA almost exclusively runs open-source software.
NETFLIX AND FREEBSDFOSDEM 2019
40Gb/s Storage Appliance with 248TB storage(2RU form factor)
Open Connect Traffic
NETFLIX AND FREEBSDFOSDEM 2019
Video Apps
Video Apps
Video Apps
Video Apps
Open Connect
Netflix OCA Workload
NETFLIX AND FREEBSDFOSDEM 2019
CPU
RAM
Disks
Plain-text Data
Encrypted Data
NICs
Using FreeBSD and commodity parts, we achieve 90 Gb/s serving TLS-encrypted connections with ~55% CPU on a 16-core 2.6-GHz CPU.
NETFLIX AND FREEBSDFOSDEM 2019
NETFLIX AND FREEBSDFOSDEM 2019
NETFLIX AND FREEBSDFOSDEM 2019
OCA Operating System
(Abridged) BSD “Family Tree”
NETFLIX AND FREEBSDFOSDEM 2019
BSD
AT&T Unix
NetBSD FreeBSD
FreeBSD Release Cycle
NETFLIX AND FREEBSDFOSDEM 2019
head
stable/11
stable/12
12.0 12.1
11.0 11.1 11.2
?
We Track FreeBSD “Head”
NETFLIX AND FREEBSDFOSDEM 2019
FreeBSD head
Netflix master
Netflix release branches
Typical Release Cycle
NETFLIX AND FREEBSDFOSDEM 2019
Five Weeks of Development Five Weeks of Testing/Deployment
FreeBSD Merge
Feature Development/Integration
Testing
Dev Testing
Canary Testing
Phased Rollout
● NUMA enhancements● Asynchronous sendfile● Kernel TLS● Pbuf allocation enhancements● “Unmapped” mbufs● I/O scheduling● TCP algorithms● TCP logging infrastructure
Examples of Features
NETFLIX AND FREEBSDFOSDEM 2019
Tracking “head” lets us stayforward looking andfocused on innovation.
NETFLIX AND FREEBSDFOSDEM 2019
Downstream users of open-source projects can be stuck in “vicious” or “virtuous” cycles.
NETFLIX AND FREEBSDFOSDEM 2019
Vicious Cycle
NETFLIX AND FREEBSDFOSDEM 2019
Infrequent Merges
ManyConflicts/Regressions
SlowerFeatureVelocity
Virtuous Cycle
NETFLIX AND FREEBSDFOSDEM 2019
Frequent Merges
FewConflicts/Regressions
FasterFeature
Velocity/Collaboration
● Information covered under NDA● Feature which is still in development/testing● Feature which needs to be generalized
Reasons We Keep Local Diffs
NETFLIX AND FREEBSDFOSDEM 2019
It is our intention to upstream any code which we can.
NETFLIX AND FREEBSDFOSDEM 2019
● Quicker feature iteration● Quicker access to new FreeBSD features● Quicker bug fixes● Enables collaboration● Minimizes merge conflicts● Amortizes merge “cost”
Benefits to Netflix of Tracking FreeBSD “Head”
NETFLIX AND FREEBSDFOSDEM 2019
● Wide deployment of “head” branch code (albeit in a narrow use case)● Early intensive testing● Incentive for Netflix to upstream code
Benefits to FreeBSD
NETFLIX AND FREEBSDFOSDEM 2019
● It isn’t stable● Why should you pay to find the bugs others will find while testing head?● Aren’t there more security bugs?● No one runs development branches● Pay monthly “cost” to do merges● You get new bugs each month
Objections to Running “Development” Code
NETFLIX AND FREEBSDFOSDEM 2019
Running FreeBSD “head” lets us deliver large amounts of data to our users very efficiently, while maintaining a high velocity of feature development.
NETFLIX AND FREEBSDFOSDEM 2019
Thank you
