Embed Size (px)
Citation preview
Data Science DevOps
yhat
Blog
Hi, my name is Eric… and I’m a software engineer.
{ "pred_class" : 0, "prob": 0.87}
Data Science DevOps
Today’s Agenda1. Data science or
machine learning?2. A story about protein3. What makes DevOps
hard4. Attempting solutions5. Q & A
First, let’s get a couple things out of the way.
Next.ML
Next.ML
Data Science or Machine Learning?!?
Data Science“Applying machine learning to organizational problems.”
Different ProblemsSoftware engineering, not strictly engineering.
Different ProblemsHow do I work with others?
Different ProblemsHow do I version this?
Different ProblemsHow do I not break things?
Data ScienceIn the end still machine learning.
Bro, do you even big data?
Data mining vs. model building
Data mining vs. model building
Okay, story time
Let’s talk about protein
(Crystallography Team)
(Crystallography Team)
●Uses “brute force” approach to crystallize proteins
(Crystallography Team)
●Uses “brute force” approach to crystallize proteins
●Manually scores images one at a time
Crystal
$$$$$
Murky stuff
Lighting difference
s
Important murky stuff
What the hell is this line?
R & D
R & D Production
Porting this was hard
Will it make my job easier?
:(
yhat
R & D Production
Difficult to encapsulate
How to make them production ready?
“Production” - Reliable- Reproducible- Scalable
Some reasons about why this is hard to achieve
Model != Service(bare with me on this one)
Software stacks are complicated
Software stacks are complicated
All technologies can be connected to over a network
Where does machine learning fit into this stack?
What stops us from doing this?
Machine learning is stateful
import alg…model = alg.train(data)…model.predict(newdata)
import alg…model = alg.train(data)…model.predict(newdata)
- Source code doesn’t encapsulate program
- Source code doesn’t encapsulate program- Training is expensive (don’t want to do it every time)
Serialization
Serialization$python>>> import pickle>>> x = 3>>> p = pickle.dumps(x)>>> y = pickle.loads(p)>>> y3
Sterilization has it’s own set of problems
Traceback (most recent call last): File "diabetes.py", line 33, in <module>
R2 = cross_val_score(clf, X, y=y, cv=KFold(y.size, K), n_jobs=1).mean() File "/home/eric/programming/python/env/local/lib/python2.7/site-packages/sklearn/cross_validation.py", line 1361, in cross_val_score
for train, test in cv) File "/home/eric/programming/python/env/local/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 659, in __call__
self.dispatch(function, args, kwargs) File "/home/eric/programming/python/env/local/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 406, in dispatch
job = ImmediateApply(func, args, kwargs) File "/home/eric/programming/python/env/local/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 140, in __init__
self.results = func(*args, **kwargs) File "/home/eric/programming/python/env/local/lib/python2.7/site-packages/sklearn/cross_validation.py", line 1459, in _fit_and_score
estimator.fit(X_train, y_train, **fit_params) File "/home/eric/programming/python/env/local/lib/python2.7/site-packages/sklearn/ensemble/forest.py", line 273, in fit
for i, t in enumerate(trees)) File "/home/eric/programming/python/env/local/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 659, in __call__
self.dispatch(function, args, kwargs) File "/home/eric/programming/python/env/local/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 406, in dispatch
job = ImmediateApply(func, args, kwargs) File "/home/eric/programming/python/env/local/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 140, in __init__
self.results = func(*args, **kwargs) File "/home/eric/programming/python/env/local/lib/python2.7/site-packages/sklearn/ensemble/forest.py", line 94, in _parallel_build_trees
tree.fit(X, y, sample_weight=curr_sample_weight, check_input=False) File "/home/eric/programming/python/env/local/lib/python2.7/site-packages/sklearn/tree/tree.py", line 227, in fit
raise ValueError("min_weight_fraction_leaf must in [0, 0.5]")ValueError: min_weight_fraction_leaf must in [0, 0.5]
Customer - “My model isn’t working”
What’s the problem?…R2 = cross_val_score(clf, X, y=y, cv=KFold(y.size, K), n_jobs=1).mean()…tree.fit(X, y, sample_weight=curr_sample_weight, check_input=False)
raise ValueError("min_weight_fraction_leaf must in [0, 0.5]")
ValueError: min_weight_fraction_leaf must in [0, 0.5]
What’s the problem? - Pickle a scikit-learn 0.16.1 model- Unpickle it in 0.15.1
Interpreted languages have a lot of run time dependencies
Reproducing dependencies is critical
Reproducing dependencies is criticalDependency detection can be hard to automate
Reproducing dependencies is criticalPackage managersaren’t perfect
Example: pip
Example: pip- Not standard
Example: pip- Not standard- Can do a poor job of installing dependencies
Example: pip- Not standard- Can do a poor job of installing dependencies- Only recently precompiled
Example: r
Example: r- Can’t install specific version of package
Example: r- Can’t install specific version of package- No, seriously
Solution
Solution- Use a better package manager
Solution- Use a better package manager- Ship your dependencies
Complied languages are easier- Matlab (MCC), Scala
Complied languages are easier- Matlab (MCC), Scala- Linking still an issue
Data transforms can be critical to the model
PMML
PMML ?def tokenize(s): s = s.lower() s = s.split(" ") return s
$ python>>> def tokenize(s):... return s.lower().split(" ")...>>> import pickle>>> pickle.dumps(tokenize)'c__main__\nclean_sentence\np0\n.'
$ python>>> def tokenize(s):... return s.lower().split(" ")...>>> import pickle>>> pickle.dumps(tokenize)'c__main__\nclean_sentence\np0\n.'
Trying to get models onto a network
Databases are great
1) Compute regression
2) Shove coefficients in database
3) …4) Profit?!?
Simple Web Servers
Simple Web ServersYou’re still stuck with environment management problems
Simple Web ServersSome modeling languages are not languages you want to write a server in…
Simple Web Servers- Division of roles- NPR uses flask for visualization dev but not production website
A solution we (yhat) have decided on
Containers FTW
Containers FTWContainers address a lot of previous concerns
Containers FTWReproducibility, managing environments, etc.
Containers FTWCheap
Containers FTWWord of warning:
Containers FTWWord of warning:If you choose this route you will be manage models and Docker
{ "pred_class" : 0, "prob": 0.87}
Take a model
“Deploy” to our platform
Defer to Docker
$ pip install foo==0.2.4$ pip install bar==1.4.9Attempt to
recreate env
$ pip install foo==0.2.4$ pip install bar==1.4.9Replicate as
necessary
The dev team is always trying to learn of better ways of doing this
Thanks!
And remember to be nice to your DevOps.
