Building Automated REST APIs with Python

JeffKnupp@jeffknuppjeff@jeffknupp.comWhartonWebConference2014

Authorof“WritingIdiomaticPython”Full-timePythondeveloper@AppNexusBloggeratjeffknupp.comCreatorofthe“sandman”Pythonlibrary

We'regoingtousePythonto generateaRESTAPI.

Andwe'regoingtodoitwithoutwritingasinglelineofcode.

We'llgooverwhataRESTAPIis,howitworks,andwhyit'susefulWe'llreviewtheHTTPprotocolandhowthewebworksWe'llseealotofPythoncode

Sevenletters.Twoacronyms.Buywhatdoesitmean?

Programmaticwayofinteractingwithathird-partysystem.

WaytointeractwithAPIsoverHTTP(thecommunicationprotocoltheInternetisbuilton).

"REST"wascoinedbyRoyFieldinginhis2000doctoraldissertation.Includessetofdesignprinciplesandbestpracticesfordesigningsystemsmeanttobe"RESTful".

InRESTfulsystems,applicationstateismanipulatedbytheclientinteractingwithhyperlinks.Arootlink(e.g.

)describeswhatactionscanbetakenbylistingresourcesandstateashyperlinks.

http://example.com/api/

HTTPisjustamessagingprotocol.HappenstobetheonetheInternetisbasedon.

RESTfulsystemsusethisprotocoltotheiradvantagee.g.cachingresources

GETPOSTPUTPATCHDELETE

TounderstandhowRESTAPIswork,wehavetounderstandhowthewebworks.

EverythingyouseeonthewebistransferredtoyourcomputerusingHTTP.

Whathappenswhenwetypehttp://www.jeffknupp.comintoourbrowser?

Let'stracethelifecycleofabrowser'srequest.

AprotocolcalledtheDomainNameService(DNS)isusedtofindthe"real"(IP)addressofjeffknupp.com.

GET

ThebrowsersendsaGETrequestto192.168.1.1forthepageataddress/(thehomeor"root"page).

The (aprogramusedtoserviceHTTPrequeststoawebsite)receivestherequest,findstheassociatedHTMLfile,andsendsitasanHTTPResponse.

Ifthereareanyimages,videos,orscriptsthattheHTMLmakesreferenceto,separateHTTPGETrequestsaremade

forthoseaswell.

Programs,likecurl,canalsoissueHTTPrequests

CURL

curltalkstothewebserver,usingapublicAPI(viaHTTP)

ARESTAPIexposesyourinternalsystemtotheoutsideworld

It'salsoafantasticwaytomakeasystemavailabletoother,internalsystemswithinanorganization.

ExamplesofpopularRESTAPIs:TwitterGitHubGoogle(foralmostallservices)

Ifyou'reaSaaSprovider,youareexpectedtohaveaRESTAPIforpeopletowriteprogramstointeractwithyour

service.

FourcoreconceptsarefundamentaltoallRESTservices(courtesyWikipedia)

WhenusingHTTP,thisisdoneusingaURI.Importantly,aresourceand arecompletelyorthogonal.Theserverdoesn'treturndatabaseresultsbutratherthe

JSONorXMLorHTMLrepresentationoftheresource.

Whentheservertransmitstherepresentationoftheresourcetotheclient,itincludesenoughinformationforthe

clienttoknowhowtomodifyordeletetheresource.

Eachrepresentationreturnedbytheserverincludesinformationonhowtoprocessthemessage(e.g.usingMIME

types

Clientsare .Theyknownothingabouthowtheserviceislaidouttobeginwith.Theydiscoverwhatactionstheycan

takefromtherootlink.Followingalinkgivesfurtherlinks,definingexactlywhatmaybedonefromthatresource.

Clientsaren'tassumedtoknow exceptwhatthemessagecontainsandwhattheserveralreadytoldthem.

ARESTAPIallows tosend tomanipulate .

...SoweneedtowriteaservercapableofacceptingHTTPrequests,actingonthem,andreturningHTTPresponses.

Yep.ARESTfulAPIServiceisjustawebapplicationand,assuch,isbuiltusingthesamesetoftools.We'llbuildours

usingPython,Flask,andSQLAlchemy

EarlierwesaidaRESTAPIallowsclientstomanipulateviaHTTP.

Prettymuch.Ifyou'resystemisbuiltusingORMmodels,yourresourcesarealmostcertainlygoingtobeyourmodels.

Webframeworksreducetheboilerplaterequiredtocreateawebapplicationbyproviding:

ofHTTPrequeststohandlerfunctionsorclassesExample:/foo=>defprocess_foo()

ofHTTPresponsestoinjectdynamicdatainpre-definedstructure

Example:<h1>Hello{{user_name}}</h1>

ThemoretimeyouspendbuildingRESTAPIswithwebframeworks,themoreyou'llnoticethesubtle(andattimes,

glaring)impedancemismatch.

URLsas toprocessingfunctions;RESTAPIstreatURLsastheaddressofaresourceorcollectionHTMLtemplating,whileRESTAPIsrarely.JSON-relatedfunctionalityfeelsbolted-on.

Imaginewe'reTwitterweeksafterlaunch.AshtonKutcherseemstobeabletouseourservice,butwhatabout

?

That'sright,we'llneedtocreateanAPI.Beinganinternetcompany,we'llbuildaRESTAPIservice.Fornow,we'llfocus

ontworesources:usertweet

Allresourcesmustbeidentifiedbyauniqueaddressatwhichtheycanbereached,theirURI.Thisrequireseachresource

containauniqueID,usuallyamonotonicallyincreasingintegerorUUID(likeaprimarykeyinadatabasetable).

OurpatternforbuildingURLswillbe/resource_name[/resource_id[/resource_attribute]]

Herewedefineourresourcesisafilecalledmodels.py:

classUser(db.Model,SerializableModel):__tablename__='user'

id=db.Column(db.Integer,primary_key=True)username=db.Column(db.String)

classTweet(db.Model,SerializableModel):__tablename__='tweet'

id=db.Column(db.Integer,primary_key=True)content=db.Column(db.String)posted_at=db.Column(db.DateTime)user_id=db.Column(db.Integer,db.ForeignKey('user.id'))user=db.relationship(User)

classSerializableModel(object):"""ASQLAlchemymodelmixinclassthatcanserializeitselfasJSON."""

defto_dict(self):"""Returndictrepresentationofclassbyiteratingoverdatabasecolumns."""value={}forcolumninself.__table__.columns:attribute=getattr(self,column.name)ifisinstance(attribute,datetime.datetime):attribute=str(attribute)value[column.name]=attributereturnvalue

Here'sthecodethathandlesretrievingasingletweetandreturningitasJSON:

frommodelsimportTweet,User

@app.route('/tweets/<int:tweet_id>',methods=['GET'])defget_tweet(tweet_id):tweet=Tweet.query.get(tweet_id)iftweetisNone:response=jsonify({'result':'error'})response.status_code=404returnresponseelse:returnjsonify({'tweet':tweet.to_dict()})

Let'scurlournewAPI(preloadedwithasingletweetanduser):

$curllocalhost:5000/tweets/1{"tweet":{"content":"Thisisawesome","id":1,"posted_at":"2014-07-0512:00:00","user_id":1}}

@app.route('/tweets/',methods=['POST'])defcreate_tweet():"""CreateanewtweetobjectbasedontheJSONdatasentintherequest."""ifnotall(('content','posted_at','user_id'inrequest.json)):response=jsonify({'result':'ERROR'})response.status_code=400#HTTP400:BADREQUESTreturnresponseelse:tweet=Tweet(content=request.json['content'],posted_at=datetime.datetime.strptime(request.json['posted_at'],'%Y-%m-%d%H:%M:%S'),user_id=request.json['user_id'])db.session.add(tweet)db.session.commit()returnjsonify(tweet.to_dict())

InRESTAPIs,agroupofresourcesiscalleda .RESTAPIsareheavilybuiltonthenotionofresourcesand

collections.Inourcase,the oftweetsisalistofalltweetsinthesystem.

ThetweetcollectionisaccessedbythefollowingURL(accordingtoourrules,describedearlier):/tweets.

@app.route('/tweets',methods=['GET'])defget_tweet_collection():"""ReturnalltweetsasJSON."""all_tweets=[]fortweetinTweet.query.all():all_tweets.append({'content':tweet.content,'posted_at':tweet.posted_at,'posted_by':tweet.user.username})

Allthecodethusfarhasbeenprettymuchboilerplate.EveryRESTAPIyouwriteinFlask(modulobusinesslogic)willlook

identical.Howcanweusethattoouradvantage?

Wehaveself-drivingcarsanddeliverydrones,whycan'twebuildRESTAPIsautomatically?

Thisallowsonetoworkatahigherlevelofabstraction.Solvetheproblemonceinageneralwayandletcodegeneration

solveeachindividualinstanceoftheproblem.

Partof

SANDBOY

ThirdpartyFlaskextensionwrittenbythedashingJeffKnupp.Defineyourmodels.Hitabutton.BAM!RESTfulAPI

servicethat .

(Thenamewillmakemoresenseinafewminutes)

GeneralizesRESTresourcehandlingintonotionofa(e.g.the"TweetService"handlesalltweet-relatedactions).classService(MethodView):"""Baseclassforallresources."""

__model__=None__db__=None

defget(self,resource_id=None):"""ReturnresponsetoHTTPGETrequest."""ifresource_idisNone:returnself._all_resources()else:resource=self._resource(resource_id)ifnotresource:raiseNotFoundExceptionreturnjsonify(resource.to_dict())

def_all_resources(self):"""ReturnallresourcesofthistypeasaJSONlist."""ifnot'page'inrequest.args:resources=self.__db__.session.query(self.__model__).all()else:resources=self.__model__.query.paginate(int(request.args['page'])).itemsreturnjsonify({'resources':[resource.to_dict()forresourceinresources]})

Here'showPOSTworks.Noticetheverify_fieldsdecoratoranduseof**request.jsonmagic...

@verify_fieldsdefpost(self):"""ReturnresponsetoHTTPPOSTrequest."""resource=self.__model__.query.filter_by(

**request.json).first()ifresource:returnself._no_content_response()instance=self.__model__(**request.json)self.__db__.session.add(instance)self.__db__.session.commit()returnself._created_response(instance.to_dict())

Wehaveourmodelsdefined.HowdowetakeadvantageofthegenericServiceclassandcreateservicesfromour

models?defregister(self,cls_list):"""RegisteraclasstobegivenaRESTAPI."""forclsincls_list:serializable_model=type(cls.__name__+'Serializable',(cls,SerializableModel),{})new_endpoint=type(cls.__name__+'Endpoint',(Service,),{'__model__':serializable_model,'__db__':self.db})view_func=new_endpoint.as_view(new_endpoint.__model__.__tablename__)self.blueprint.add_url_rule('/'+new_endpoint.__model__.__tablename__,view_func=view_func)self.blueprint.add_url_rule('/{resource}/<resource_id>'.format(resource=new_endpoint.__model__.__tablename__),view_func=view_func,methods=[

'GET','PUT','DELETE','PATCH','OPTIONS'])

TYPE

InPython,typewithoneargumentreturnsavariable'stype.Withthreearguments,

.

TYPE

serializable_model=type(cls.__name__+'Serializable',(cls,SerializableModel),{})

new_endpoint=type(cls.__name__+'Endpoint',(Service,),{'__model__':serializable_model,'__db__':self.db})

Let'splaypretendagain.Nowwe'reaIaaScompanythatletsusersbuildprivateclouds.We'llfocusontworesources:

cloudandmachine

classCloud(db.Model):__tablename__='cloud'

id=db.Column(db.Integer,primary_key=True)name=db.Column(db.String,nullable=False)description=db.Column(db.String,nullable=False)

classMachine(db.Model):__tablename__='machine'

id=db.Column(db.Integer,primary_key=True)hostname=db.Column(db.String)operating_system=db.Column(db.String)description=db.Column(db.String)cloud_id=db.Column(db.Integer,db.ForeignKey('cloud.id'))cloud=db.relationship('Cloud')is_running=db.Column(db.Boolean,default=False)

fromflaskimportFlaskfromflask.ext.sandboyimportSandboyfrommodelsimportMachine,Cloud,db

app=Flask(__name__)app.config['SQLALCHEMY_DATABASE_URI']='sqlite:///db.sqlite3'db.init_app(app)withapp.app_context():db.create_all()sandboy=Sandboy(app,db,[Machine,Cloud])app.run(debug=True)

Incaseswherewe'rebuildingaRESTAPIfromscratch,thisisprettyeasy.Butwhatif:

WehaveanexistingdatabaseWewanttocreateaRESTfulAPIforitIthas200tables

OnlydownsideofFlask-Sandboyisyouhavetodefineyourmodelclassesexplicitly.Ifyouhavealotofmodels,this

wouldbetedious.

...Idon'tdotedious

Wehaveprivatecompaniesbuildingrocketshipsandelectriccars.Whycan'twehaveatoolthatyoupointatanexistingdatabaseandhitabutton,then,BLAM!RESTfulAPIservice.

SANDMAN

,alibrarybyteenheartthrobJeffKnupp,createsaRESTfulAPIservicefor with

.

Here'showyourunsandmanagainstamysqldatabase:

$sandmanctlmysql+mysqlconnector://localhost/Chinook*Runningonhttp://0.0.0.0:8080/*Restartingwithreloader

$curl-vlocalhost:8080/artists?Name=AC/DCHTTP/1.0200OKContent-Type:application/jsonDate:Sun,06Jul201415:55:21GMTETag:"cea5dfbb05362bd56c14d0701cedb5a7"Link:</artists/1>;rel="self"

{"ArtistId":1,"Name":"AC/DC","links":[{"rel":"self","uri":"/artists/1"}],"self":"/artists/1"}

ETagsetcorrectly,allowingforcachingresponsesLinkHeadersettoletclientsdiscoverlinkstootherresourcesSearchenabledbysendinginanattributenameandvalue

Wildcardsearchingsupported

Wecancurl/andgetalistofallavailableservicesandtheirURLs.Wecanhit/<resource>/metatogetmeta-infoaboutthe

service.Example(the"artist"service):

$curl-vlocalhost:8080/artists/metaHTTP/1.0200OKContent-Length:80Content-Type:application/jsonDate:Sun,06Jul201416:04:25GMTETag:"872ea9f2c6635aa3775dc45aa6bc4975"Server:Werkzeug/0.9.6Python/2.7.6

{"Artist":{"ArtistId":"integer(11)","Name":"varchar(120)"}}

Andnowfora(probablybroken)live-demo!

"Real"RESTAPIsenableclientstousetheAPIusingonlytheinformationreturnedfromHTTPrequests.sandmantriestobeas"RESTful"aspossiblewithoutrequiringanycodefrom

theuser.

WouldbenicetobeabletovisualizeyourdatainadditiontointeractingwithitviaRESTAPI.

1. Codegeneration2. Databaseintrospection3. Lotsofmagic

sandmancamefirst.HasbeennumberonePythonprojectonGitHubmultipletimesandisdownloaded25,000timesa

month.Flask-Sandboyissandman'slittlebrother...

ThefactthattheendresultisaRESTAPIisnotespeciallyinterestingMoreimportantaretheconceptsunderpinningsandmanandFlask-Sandboy

WorkathigherlevelofabstractionSolveaproblemonceinagenericmannerReduceserrors,improvesperformance

Ingeneral:

Speakingofautomation,here'showmybookis"built"...

sandman=Flask+SQLAlchemy+LotsofGlueRequiresyouknowthecapabilitiesofyourtoolsPartoftheUNIXPhilosophy

ThebestprogrammingadviceIevergotwasto"belazy"

SandmanexistsbecauseIwastoolazytowriteboilerplateORMcodeforanexistingdatabaseFlask-SandboyexistsbecauseIwastoolazytowritethesameAPIservicesoverandoverBeinglazyforcesyoutolearnyourtoolsandmakeheavyuseofthem

Contactmeat:jeff@jeffknupp.com@jeffknupponTwitter

onthetubeshttp://www.jeffknupp.com