Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
PREFACE
Commandlineclientsareeverywhere.Almosteveryone,atleastintech,isusingthem.
Therearealotofsuccessfulcommandlineclientsoutthere:theLinuxprojecthasgitandthe Node.js project has npm. We use some of them multiple times per day. ApacheCouchDBrecentlygotnmo(speak:nemo),atooltomanagethedatabasecluster.Wecanlearnalotfromsuccessfulcommandlineinterfacesinordertowritebettercommandlineclients.
WhenIstartedtoget interestedincommandlineclientsIrealisedthat therearealotofdiscussionsandinformationsonthewebaboutwritingAPIs.Thewebisfulloftutorialstoteach you how to build APIs, especiallyREST-APIs, but almost nothing can be foundaboutwritinggoodCLIs.ThisbooktriestoexplainwhatmakesagoodCLI.InthesecondpartofthebookwewillbuildasmallcommandlineclienttolearnhowtouseNode.jstocreategreatcommandlineclientsthatpeoplelove.
Thegoalofthebookistoshowtheprinciplestobuildasuccessfulcommandlineclient.The provided code should give you a good understanding what is important to buildsuccessfulcommandlineclientsandhowyoucouldimplementthem.
Everysectionhasitsowncodeexamples.Beforeyourunthecode,youhavetorunnpm
installinthefolderthatbelongstothesection.
Youcandownloadthecodesamplesathttp://theclibook.com/material/sourcecode.zip.
I trust you and published this book without DRM. Please buy a copy athttp://theclibook.comincaseyoudidnotbuythebook.
I am very happy about feedback. Please send and feedback or corrections [email protected]:@robinson_k
Ihopeyouenjoythebook–pleaserecommenditincaseyoulikeit. �
▪
▪
▪
WHATMAKESAGOODCLI?
Inthischapterwewilltakealookatsuccessfulcommandlineclientsandwhattheyaredoing pretty well, which will help us to understand the problems users face using theTerminal.UnderstandingtheproblemsofouruserswillhelpustobuildbettercommandlineclientswithNodelaterinthebook.
Let’stakealookathowpeopleusuallyuseaCLI:mostofthetimeahumansitsinfrontof a keyboard and interacts with a terminal. We want to use simple and recognisablecommandsforourCLI.Sadlyjusteasyrecognisablecommandsdon’tgetusveryfarrightnow.
Maybetheproblemiseasier tounderstandifwetakea lookatsomethingwhatIwouldcallabadCLI:
$mycli-A-a16rfoo.html
error:undefinedisnotafunction
Inmy example I have to enter cryptic commandswhich is answered by a very crypticerrormessage.Whatdoes -A -a16 and rmean?Why I amgetting an error back, am Iusingitwrong?WhatdoestheerrormeanandhowcanIgetmytaskdone?
SowhatmakesagoodCLI?Let’stryitwiththefollowingthreeprinciples:
younevergetstuck
itissimpleandsupportspowerusers
youcanuseitforallthethings!
Inshort:AsuccessfulCLIissuccessfulbecauseitsusersaresuccessfulandhappy.
YounevergetstuckNobodylikestobeinatrafficjam,stuck,justmakingafewmetersperminute.Wewantto reachour targetdestination, that’sallwewant!Thesameapplies forourusers.Bothdevelopersandusersareextremelyunhappywhenthetoolstheyusearestandingintheirway.Theyjustwanttogettheirtaskdone.
Sowhatdoes,„Younevergetstuck“mean,exactly?Itmeansthatweshouldalwaysofferourusersawaytosolvetheirtask,acommandshouldneverbeadeadend.Additionally
thedevelopersoftheCLIshouldavoideverysourceoffrictionintheirtool.
Let’stakealookatme,tryingtousegit:
$gitpoll
git:'poll'isnotagitcommand.See'git--help'.
Didyoumeanthis?
pull
InthisexampleIenteredawrongcommand.gitanswersfriendly:„HeyRobert, it lookslikeyouenteredawrongcommand,butifyoutypeingit--help,youcanlistallthe
existingcommands.Andhey, it just looks likeyoumistypedgitpull, didyoumeangitpull?“
gitoffersusawaytocontinueourworkandfinishthetask.
Andifwetakealookatnpm,anothersuccessfulCLIclient,we’llseethesameconcept:
$npmragrragr
Usage:npm<command>
where<command>isoneof:
access,add-user,adduser,apihelp,author,bin,bugs,c,
cache,completion,config,ddp,dedupe,deprecate,dist-tag,
dist-tags,docs,edit,explore,faq,find,find-dupes,get,
help,help-search,home,i,info,init,install,issues,la,
link,list,ll,ln,login,logout,ls,outdated,owner,
pack,prefix,prune,publish,r,rb,rebuild,remove,repo,
restart,rm,root,run-script,s,se,search,set,show,
shrinkwrap,star,stars,start,stop,t,tag,test,tst,un,
uninstall,unlink,unpublish,unstar,up,update,upgrade,
v,verison,version,view,whoami
npm<cmd>-hquickhelpon<cmd>
npm-ldisplayfullusageinfo
npmfaqcommonlyaskedquestions
npmhelp<term>searchforhelpon<term>
npmhelpnpminvolvedoverview
Specifyconfigsintheini-formattedfile:
/Users/robert/.npmrc
oronthecommandlinevia:npm<command>—keyvalue
Configinfocanbeviewedvia:npmhelpconfig
[email protected]/Users/robert/.nvm/versions/node/v0.12.2/lib/node_modules/npm
In thisexample I try toputgarbage intonpm,sonpmanswers friendly:„HeyRobert, Idon’tknowthatcommand,buthereareallthecommandsthatwouldbepossible.Youcanusethemlikethisandgethelpaboutthembytypinginnpmhelp<command>.“
Likegit,npmimmediatelyoffershelp toenablemetofinishmytask,evenif Ihavenoideahowtousenpmatall.
Stilllost?WhatifIstillneedhelp?MaybeIwanttogetsomehelpbeforeIjusttryoutcommands.Turns out there is a quite reliableway to ship documentation onUnix or Linux,man-pages!
Figure1.Theman-pageforgitpull
Man-pagesarequitenice,asyoudon’tneedtheinternettoopenthem.Youcanalsostayin thesameterminalwindowtoreadthemanddon’thavetoswitchtoanotherwindow,e.g.abrowser.
Butsomeusersdon’tknowaboutman-pagesortheydon’tliketousethem.AdditionallymanyofthemwillbeonWindowswhichcan’thandleman-pagesnatively,sogitandnpmoffertheirdocumentationaswebpages,too:
Figure2.Thedocumentationwebsiteofthegitproject
Bothgitandnpmaremakinguseofatrick:theywritetheirdocumentationonce(e.g.inMarkdownorAsciidoc)andusetheinititalsourceasthebaseforthedifferentformatsoftheirdocs.Latertheyconvertthemtodifferentformats,e.g.tohtml.
Ifyoutakealookattheman-pagesofgitandnpm,youwillnoticethattheirwebsitesarebasicallyframingthecontentfromtheman-pagewithaheaderandasidebar.
Figure3.Themanpagefornpmpublish
Figure4.Thedocumentationwebsiteofnpm
ErrorhandlingSometimesthingsgostillhorriblywrong…Let’stakealookatmyexampleforabadCLIagain:
$mycli-A-a16rfoo.py
events.js:85
thrower;//Unhandled'error'event
^
Error:ENOENT,open'cli.js'
atError(native)
Inthiscasewearegettingbackastacktracewithoutmuchcontext.Formostpeoplethesestacktraces look quite cryptic, especially for people that don’twriteNode.js on a dailybasis.
Anditisevenworse:Ireallycan’ttellifIjusthitabuginthecommandlineclientorifIamjustusingtheCLIinawrongway.Lookingatthatsmallterminal,withnoideawhattodo,Igetextremelyunhappyandsoouruserswillgetunhappy.
Onethingnmosupportsis„usageerrors“—hereiswhattheylooklike:
$nmoclusterdsf
ERR!Usage:
nmoclusterget[<clustername>],[<nodename>]
nmoclusteradd<nodename>,<url>,<clustername>
nmoclusterjoin<clustername>
Ifausertriestouseacommandinawrongway,nmowilltellthemimmediatelyhowtheycanusethecommandtogettheirjobdone.Noneedtoopenthedocumentation.
nmoalsoshowsstacktracestoauser,ifnmocrashesforseriousreasons:
$nmoclusterjoinanemone
ERR!dfisnotdefined
ERR!ReferenceError:dfisnotdefined
ERR!at/Users/robert/apache/nmo/lib/cluster.js:84:5
ERR!atcli(/Users/robert/apache/nmo/lib/cluster.js:68:27)
ERR!at/Users/robert/apache/nmo/bin/nmo-cli.js:32:6
ERR!
ERR!nmo:1.0.1node:v0.12.2
ERR!pleaseopenanissueincludingthislogon
https://github.com/robertkowalski/nmo/issues
nmoaddsthecurrentnmoandnodeversiontothestacktrace,likenpmdoes.Wealsoasktheusertocopythestacktraceandtoopenanissuecontainingthestacktrace.
The reports make it easy for the team to identify the bug, solve it, and release a newversionofnmobyseeingthestacktrace.
Andagaintheuserisnotstuck.Theusergetshelptosolvetheirtask,intheworstcasewehelptheminourissuetracker.
ItsupportspowerusersPowerusersareimportantforyourCLI.TheyareuserswhowilltalkaboutyourCLIandraisetheoveralladoptionbyspreadingtheword.
ShortcutsMostpowerusersareusingyourCLImultipletimeseveryday.Aneasywaytosupportthemisbyprovidingshortcuts.
npmhaslotsofshortcuts.Forinstance,npmiistheshortformfornpminstall.git
letsyoudefineyourownshortcutsinthe.gitconfigfile.Iusegitcoasashortcut
forgitcheckout,forexample.
Scripting
Atsomepointyourcommandlineclientwillgetquitesuccessful,peoplearelovingitandstart usingyourCLI in creativeways.The command line clientwill suddenly runon aJenkins,aspartof theirdeployment inacheforPuppet run,oryouruserswilluse it inwaysyounevercouldhaveimagined!
SoonerorlaternotonlyhumanswilluseyourCLI,butalsoautomatedprocesses.TomakeyourCLIevenmoresuccessfulit’sagoodideatosupportscripting.
exitcodesOperatingsystemshavedifferentexitcodestosignalifacommandwassuccessful.Youwillgetbacka0ifyourrecentcommandwassuccessful.1wouldbeageneralerror.
Exitcodesareveryusefulforusersthatwanttowrapyourcommandlineclientinabashscript.
Hereisanexample:
$gitpoll
git:'poll'isnotagitcommand.See'git--help'.
Didyoumeanthis?
pull
$echo$?
1
git notifiesme that something went wrong - I am getting back a1 as exit code.With
properexitcodeseverywriterofabashscriptcanhandlethecaseswhereacommandisnotsuccessful.
JSONoutputInnmoeverycommandthatgivesbackinformationsupportsjson-formattedoutput:
$nmoclusterget--json
{anemone:
{node1:'http://node1.local',
node2:'http://node2.local',
node3:'http://node3.local'}}
JSONsupport enablesuser toprocessdata easily in theprogramming languageof theirchoice,asmostlanguagessupportJSON.Theyspawnachildprocessinlanguagexand
listentostdoutfortheoutput.Theycanalsodirectlypipetheoutputintoaconsumeron
theshell:
$nmoclusterget--json|consumer.py
JSONoutputgivesalotofflexibilitytotheusers.
TheAPIintheCommandLineClientThereisanotherconcepttomakescriptingeasier:Icallitthe„TheAPIintheCommandLineClient“:
constnmo=require('nmo');
nmo.load({}).then(()=>{
nmo.commands.cluster
.get('testcluster','[email protected]')
.then((res)=>{
console.log(res);
});
});
Innmoeverycommand is exposedonnmo.commands. If auserwants tousenmoas
partof theirnode scripts, theyare able to require it.The JavaScriptAPI isdocumentedliketheCLI.
TheJavaScriptAPIenablestheuserstoembednmointheirNode.jsscriptsforcomplexprocesses.Theycouldevenforknmoandembeditintotheirowncommandlineclient.
ConfigurationPowerusers love configuration. Given they use a command line client a lot, maybemultipletimesaday,itisnosurprisethattheywouldliketohavesomefeaturesenabledperdefault.Butinrarecases,thereisanexceptionandtheydon’tneedthedefaultsetting.
npmsupportsoptionargumentsonthecommandline:
$npmihapi--registry=https://reg.kowalski.gd
/Users/robert
This command tries to download the packagehapi from a private registry at
https://reg.kowalski.gd.
ButIcanalsosetthisprivateregistryasthenewdefaultregistry:
$npmconfigsetregistryhttps://reg.kowalski.gd
npmwritesthenewregistryintotheconfig:
$cat~/.npmrc
loglevel=http
registry=https://reg.kowalski.gd
The next time I try to install a package, npm will use my new default registry,https://reg.kowalski.gd:
$npmihapi
IfIdon’twanttousethisnewdefaultregistryIcanpassanargumenttotheCLIanditwillusethealternateregistryjustforthiscall:
$npmihapi--registry=https://registry.npmjs.org
/Users/robert
Thatmeanswehavedifferentprioritiesbetweendefaultconfigurationsandcommandlineargumentsinnpmandthiscombinationisextremelypowerful.
Youcanuseitforallthethings!Let’s take a look at the last principle, and the solution to it sounds very easy at first.WheneverIhavetodoataskmultipletimesanditfitsintothedomainofmycommandlineclient, I’ll justadd it asanewcommand.Thishabit turns intoawin-winsituation:Youhavetodolessboringtasksandyourusersgethappybecausetheygetanewfeatureandtheyalsohavetodolessmonkeytasks-makingyourcommandlineclientevenmoresuccessful.Sadlyitcanbequitehardtospotcommonpainpoints,especiallyifyouworkwithmultiple teams and/or a lot different people.Additionallymost of us are sufferingorganisationalblindnessworkingonthesametopicafteracertaintime.Butifyouidentifyatasktoautomateforyouandyourusers,youwillbehugelyrewarded! �
▪
▪
▪
WRITINGADATABASEADMINISTRATIONTOOLWITHNODE.JS
In thispartof thebookwewillwriteadatabaseadministration toolnamed loungerandfollowtheprinciplesthatmakeagoodCLI.Thecodeforeverysectioncanbefoundinthezipfile thatcomeswiththebook.Ifyouwanttoplaywiththecode,don’tforget torunnpminstallinthefolderofthesectionyouwanttotest.Thecodealsohasatestsuite,
youcanrunitwithnpmtest.
Afterwewrote the code that bootstraps the client, you can run the client directlywithnodebin/lounger-cli.Ifyouprefertoruntheclientlikeaninstallationfromthe
registry,typenpmlinkinthedirectoryofthesectionyouwanttotest.Afterwardsyou
canruntheversionthatiscurrentlylinkedwithlounger.
WhyuseNode.js?IsometimesgetaskedwhyIwritecommandlineclientsusingNode.js.Formethemainreasonsare:
ahugeecosystemwithmodulesineveryflavour
veryfastdevelopmentspeed
writingJavaScriptisfun!
For me these three reasons make Node.js the perfect platform to write command lineclients.
SetupWewillwriteourcommandlineclientinES2015(alsoknownasES6),whichisthemostrecent version of JavaScript. In order to use it, we have to install Node v4 fromhttps://nodejs.org. If youwant to support olderNode.js versions, I can recommend theBabel transpiler to transpile ES6 code to ES5 compatible code. You can get Babel athttps://babeljs.io/.
The tool that we’ll write will be a small database administration tool for CouchDB /PouchDB.Therearemultiplewaystogetadevelopmentdatabaseserverupandrunning.
OnewayistoinstallErlangandCouchDBforyourOperatingSystem.Youcandownloadofficial packages athttp://couchdb.apache.org and many Linux distributions haveCouchDBintheirpackagerepository,too.
I think the easiest way is to use the PouchDB Server that is available in the attachedsourcecode for thebookor togetaCouchDBinstanceathttps://cloudant.comwhich isfreeuntilyouhitalimit.
UsingthePouchDBdatabaseserverThedatabaseserverislocatedinsourcecode/database,inordertouseitwehaveto
installtheneededdependencies:
$cdsourcecode/database
$npminstall
Tobootthedatabasewejustrun:
$npmrunstart
WecannowinteractwiththedatabaseserverviaHTTP,asCouchDBandPouchDBaredatabaseswithanHTTPAPI:
$curl-XGEThttp://127.0.0.1:5984/
{"express-pouchdb":"Welcome!","version":"1.0.1","vendor":{"name":"PouchDB
authors","version":"1.0.1"},"uuid":"4fad2c01-ba32-4249-8278-8786e877c397"}
Let’screateadatabasecalledpeople:
$curl-XPUThttp://127.0.0.1:5984/people
{"ok":true}
Wecannowinsertdocumentsintoourdatabasepeople:
$curl-XPOSThttp://127.0.0.1:5984/people-d'{"name":"RockoArtischocko",
\
"likes":["Burritos","Node.js","Music"]}'-H'Content-Type:
application/json'
{"ok":true,"id":"21b5ad83-0ad6-47c7-86f8d9636113160a","rev":"1-
411894affa038a6fd7a164e1bfd84146"}
Usingtheidwecanretrievethedocumentsfromthedatabase:
$curl-XGEThttp://127.0.0.1:5984/people/21b5ad83-0ad6-47c7-86f8-
d9636113160a
{"name":"RockoArtischocko","likes":
["Burritos","Node.js","Music"],"_id":"21b5ad83-0ad6-47c7-86f8-
d9636113160a","_rev":"1-411894affa038a6fd7a164e1bfd84146"}
Great!Wehaveadatabaseupandrunning!
TroubleshootingGettingcurlcurl is a command lineclient forHTTP requests. It is available forallmajorOperatingSystems.OSXuserscaninstallitusingbrewandforWindowsthereareWindowsbuilds
availableathttp://curl.haxx.se/download.html.
FilewatchersOnLinuxIgotanerrorbecausemyuseralreadywatchedtoomuchfiles:
$npmrunstart
>[email protected]/home/rocko/clibook/sourcecode/database
>pouchdb-server--in-memory
fs.js:1236
throwerror;
^
Error:watch./log.txtENOSPC
atexports._errnoException(util.js:874:11)
atFSWatcher.start(fs.js:1234:19)
atObject.fs.watch(fs.js:1262:11)
atTail.watch
(/home/rocko/clibook/sourcecode/database/node_modules/pouchdb-
server/node_modules/tail/tail.js:83:32)
atnewTail
(/home/rocko/clibook/sourcecode/database/node_modules/pouchdb-
server/node_modules/tail/tail.js:72:10)
at/home/rocko/clibook/sourcecode/database/node_modules/pouchdb-
server/lib/logging.js:69:20
atFSReqWrap.cb[asoncomplete](fs.js:212:19)
npmERR!Linux3.13.0-71-generic
npmERR!argv"/home/rocko/.nvm/versions/node/v4.2.3/bin/node"
"/home/rocko/.nvm/versions/node/v4.2.3/bin/npm""run""start"
npmERR!nodev4.2.3
npmERR!npmv3.5.1
npmERR!codeELIFECYCLE
[email protected]:`pouchdb-server--in-memory`
npmERR!Exitstatus1
npmERR!
[email protected]'pouchdb-server
--in-memory'.
npmERR!Makesureyouhavethelatestversionofnode.jsandnpminstalled.
npmERR!Ifyoudo,thisismostlikelyaproblemwiththetheclibook-
databasepackage,
npmERR!notwithnpmitself.
npmERR!Telltheauthorthatthisfailsonyoursystem:
npmERR!pouchdb-server--in-memory
npmERR!Youcangetinformationonhowtoopenanissueforthisproject
with:
npmERR!npmbugstheclibook-database
npmERR!Orifthatisn'tavailable,youcangettheirinfovia:
npmERR!npmownerlstheclibook-database
npmERR!Thereislikelyadditionalloggingoutputabove.
npmERR!Pleaseincludethefollowingfilewithanysupportrequest:
npmERR!/home/rocko/clibook/sourcecode/database/npm-debug.log
Ifixeditwithbyraisingthelimitusingthiscommand:
$echofs.inotify.max_user_watches=524288|sudotee-a/etc/sysctl.conf&&
sudosysctl-p
AsimplestatuscheckOurfirstcommandwillcheckifthedatabaseisupandrunning.Ouruserscantakealookifthedatabaseserverisrunningandwecanusethecommandinternallyforthecommandswhichrequirearunningdatabase.
Thecommandtocheckifadatabaseserverisonlinewilllooklikethis:
$loungerisonlinehttp://192.168.0.1:5984
http://192.168.0.1:5984isupandrunning
TheAPIwouldlooklikethis:
$lounger.commands.isonline('http://example.com')
GettingstartedfromscratchTo get startedwe have to create apackage.json file. Luckily npm provides a nice
assistanttocreatethose:
$npminit
Wethenjustanswerthequestionsnpmasksus.
Figure1.Theassistantfromnpminittocreateapackage.json
Additionallywehavetocreatethreefolders:test,libandbin.testwillcontainour
unitandintegrationtests,libwillcontainthecoreofourcommandlineclient.Thebin
folderwillcontainasmallwrapperthatwillbootupthecoreofourclient.
CouchDBandPouchDBbothreturnawelcomemessagewhenweaccess therooturlathttp://localhost:5984
$curllocalhost:5984
CouchDBreturns:
{"couchdb":"Welcome","uuid":"17ed4b2d8923975cf658e20e219faf95","version":"1.5
.0","vendor":{"version":"14.04","name":"Ubuntu"}}
PouchDBreturns:
{"express-pouchdb":"Welcome!","version":"1.0.1","vendor":{"name":"PouchDB
authors","version":"1.0.1"},"uuid":"4fad2c01-ba32-4249-8278-8786e877c397"}
Chooseyourownflavours
Wewillmakeuseofthisbehaviourtocheckifthedatabaseisonline.
As already mentioned inWhy use Node.js? Node.js has a great ecosystem. There aremanybattleprovenmodulesthathelpustosolveourtasks.
For our status check we will use the modulerequest to handle our HTTP requests.
mocha will run our testsuite andnock helps us tomockHTTP responseswithout the
needtobootadatabaseinstanceforthetestsuite.
The arguments--save and--save-dev will add the packages to the
dependencies anddevDependencies section of ourpackage.json.
devDependencies are needed just for development, not for running the package in
production:
$npmi--saverequest
$npmi--save-devmochanock
Afterrunningthecommandsweshouldhaveeverythingwewillneedfornow.
There aremany good test runners for Node.js, some alternatives tomocha are the npm modulestap,
tapeorlab
Mypackage.jsonlookslikethisnow:
{
"name":"lounger",
"version":"1.0.0",
"description":"atoolforcouchdb/pouchdbadministration",
"main":"lib/lounger.js",
"directories":{
"test":"test"
},
"dependencies":{
"request":"^2.67.0"
},
"devDependencies":{
"mocha":"^2.3.4",
"nock":"^5.2.1"
},
"scripts":{
"test":"mocha-Rspec"
},
"keywords":[
"couchdb",
"pouchdb"
],
"author":"RobertKowalski<[email protected]>"
}
Thisbookisnotfocussedondifferentdevelopment techniques likeTDD,but ifyouarereally intoTest-Driven-Development, you canwrite failing testswithmocha beforeweimplementtheactualcode.Afewsuggestions:
1. itdetectsifthedatabaseisonline
2. itdetectsofflinedatabases
3. itdetectsifsomethingisonline,butnotaCouchDB/PouchDB
4. itonlyacceptsvalidurls
WritteninmochaandES6wegetafewfailingtestsintest/isonline.js:
'usestrict';
constassert=require('assert');
constnock=require('nock');
describe('isonline',()=>{
it('detectsifthedatabaseisonline',()=>{
assert.equal('foo','toimplement');
});
it('detectsofflinedatabases',()=>{
assert.equal('foo','toimplement');
});
it('detectsifsomethingisonline,butnotaCouchDB/PouchDB',()=>{
assert.equal('foo','toimplement');
});
it('justacceptsvalidurls',()=>{
assert.equal('foo','toimplement');
});
});
Torun the testsuite,wehave to typeeithernpmtest ornpmton the terminal.The
codeofthissectioncanbefoundinsourcecode/client-boilerplate.
TheinternalsofthecommandLet’screateandeditthefilelib/isonline.js.Thefilenameisimportant,aswewill
usethenameofthefilelaterduringthebootstrapoftheclient.Asafirststep,wehavetorequireourdependencyrequest:
'usestrict';
constrequest=require('request');
TomakearequestwecreatethefunctionisOnlinewhichwilltakeanurlandsendthe
request:
functionisOnline(url){
returnnewPromise((resolve,reject)=>{
request({
uri:url,
json:true
},(err,res,body)=>{
IfthereisnoHTTPserviceatalllisteningonthespecifiedurlweresolvethePromisewithanobjectwithcontainstheurlasakey,andfalseasavalue:
if(err&&(err.code==='ECONNREFUSED'||err.code==='ENOTFOUND')){
returnresolve({[url]:false});
}
ForallothererrorswerejectthePromise:
//anyothererror
if(err){
returnreject(err);
}
If we get aWelcome from CouchDB or PouchDB, we can safely assume that the
databaseserverisonline:
//maybewegotawelcomefromCouchDB/PouchDB
constisDatabase=(body.couchdb==='Welcome'||
body['express-pouchdb']==='Welcome!');
returnresolve({[url]:isDatabase});
Asalaststepwehavetoexportthefunction:
exports.api=isOnline;
Hereisthewholefunction:
functionisOnline(url){
returnnewPromise((resolve,reject)=>{
request({
uri:url,
json:true
},(err,res,body)=>{
//dbisdown
if(err&&(err.code==='ECONNREFUSED'||err.code==='ENOTFOUND')){
returnresolve({[url]:false});
}
//anyothererror
if(err){
returnreject(err);
}
//maybewegotawelcomefromCouchDB/PouchDB
constisDatabase=(body.couchdb==='Welcome'||
body['express-pouchdb']==='Welcome!');
returnresolve({[url]:isDatabase});
});
});
}
exports.api=isOnline;
WecantryourfunctionontheNode.jsREPL:
$node
>constisonline=require('./lib/isonline.js');
undefined
>isonline('http://example.com').then(console.log);
Promise{<pending>}
>{'http://example.com':false}
>isonline('http://doesnotexist.example.com').then(console.log);
Promise{<pending>}
>{'http://doesnotexist.example.com':false}
>isonline('http://localhost:5984').then(console.log);
Promise{<pending>}
>{'http://localhost:5984':true}
Congratulations!Wejustfinishedthefirstpart,theAPIpartofournewcommand!
TheCLIpartItwouldbefrustratingfortheenduserifthatwasthecommandlineinterfacetheyhavetouse.Theoutput is not easily readable and the functionalitynot easy tounderstand.TheAPIfunctiondoesn’tprinttotheconsole,whichisperfectforanAPI,butnotdesirableforaCLI.ItevenrequiressomeNode.jsknowledgetorunit.Solet’saddaniceCLIfunctiontoourfileisonline.jsandexportitascli:
functioncli(url){
returnnewPromise((resolve,reject)=>{
});
}
exports.cli=cli;
Thefunctionreturnsapromiselikeallourotherfunctions.WethencallisOnlineand
printtheresultonstdoutfortheusersthatusethecommandlineclientontheterminal:
isOnline(url).then((res)=>{
Object.keys(results).forEach((entry)=>{
letmsg='seemstobeoffline';
if(results[entry]){
msg='seemstobeonline';
}
//printonstdoutforterminalusers
console.log(entry,msg);
resolve(results);
});
});
functioncli(url){
returnnewPromise((resolve,reject)=>{
isOnline(url).then((res)=>{
Object.keys(results).forEach((entry)=>{
letmsg='seemstobeoffline';
if(results[entry]){
msg='seemstobeonline';
}
//printonstdoutforterminalusers
console.log(entry,msg);
resolve(results);
});
});
});
}
exports.cli=cli;
Itisimportanttonote:weexportthecommandfortheAPIasexports.api,whilewe
exporttheCLIcommandunderthecliproperty.
Thecodeforthissectioncanbefoundatsourcecode/the-status-check.
BootingthetoolLounger still needs the code thatmakes it usable on the command line. It also lacks acomfortablewaytorunourAPIcommands.Rightnowwejusthaveonecommand,butwe’lladdmoresoon(see[_you_can_use_it_for_all_the_things]).Tomakeallcommandseasy to use, we have to load all available commands into a namespace. The filelib/lounger.jswilltakecareofit.
Thefilelounger.jsistheheartofourcommandlineclient.Werequirethefsmodule
aswehavetolistthefilesthatcouldcontaincommmands:
'usestrict';
constfs=require('fs');
We require thepackage.json and expose the current version of the module as a
property on the lounger object, which will come in handy later. We also setlounger.loadedtofalse:
constpkg=require('../package.json');
constlounger={loaded:false};
lounger.version=pkg.version;
Wewill need a place to store theAPI and theCLI commands.As the bootstrapping isasync,wewillthrowanerrorifsomeonetriestoaccesstheexposedfunctionsbeforethe
bootstrapisfinished:
constapi={},cli={};
Object.defineProperty(lounger,'commands',{
get:()=>{
if(lounger.loaded===false){
thrownewError('runlounger.loadbefore');
}
returnapi;
}
});
Object.defineProperty(lounger,'cli',{
get:()=>{
if(lounger.loaded===false){
thrownewError('runlounger.loadbefore');
}
returncli;
}
});
ThecustomgetterforObject.definePropertywillthrowiflounger.loadedis
falseandwetrytoaccessapropertyonlounger.cliandlounger.api.
It also works for the autocomplete in the Node.js REPL when you try to hit tab forcompletion.Justtryit!
The last part of the file is the actual bootstrapping, where we get all files inlib and
requirethemiftheyareJavaScriptfilesandnotlounger.js, thefilewearecurrently
workingwith.
Thefunctiontobootstraploungeriscalledlounger.load.Inthiscaseweareusinga
named function.A named function can be very helpful in a stacktrace in case the appcrashes:
lounger.load=functionload(){
returnnewPromise((resolve,reject)=>{
});
};
Thefunctionfs.readdirwilllistallfilesinadirectory.Weiterateoverthelistoffiles
fs.readdirreturns:
lounger.load=functionload(){
returnnewPromise((resolve,reject)=>{
fs.readdir(__dirname,(err,files)=>{
files.forEach((file)=>{
});
});
});
};
IfthefileisnotaJSfileorislounger.jsitself,weignoreitbyreturningearly:
if(!/\.js$/.test(file)||file==='lounger.js'){
return;
}
Inallothercasesweassumethatwefoundacommandforlounger.Wetakeeverythingfromthefilenamebeforethe.jsandsaveitascmd:
constcmd=file.match(/(.*)\.js$/)[1];
Werequirethefile:
constmod=require('./'+file);
If a file exports an API command as theapi property, we expose it on
lounger.commands.AllCLIcommandsareavailableonlounger.cli:
if(mod.cli){
cli[cmd]=mod.cli;
}
if(mod.api){
api[cmd]=mod.api;
}
After theforEach loop is finished and all commands are loaded we can set
lounger.loaded totrue. This will prevent the checks we added previously from
throwing:
lounger.loaded=true;
AslaststepweresolvethePromise:
resolve(lounger);
Thewholefunctionlounger.load:
lounger.load=functionload(){
returnnewPromise((resolve,reject)=>{
fs.readdir(__dirname,(err,files)=>{
files.forEach((file)=>{
if(!/\.js$/.test(file)||file==='lounger.js'){
return;
}
constcmd=file.match(/(.*)\.js$/)[1];
constmod=require('./'+file);
if(mod.cli){
cli[cmd]=mod.cli;
}
if(mod.api){
api[cmd]=mod.api;
}
});
lounger.loaded=true;
resolve(lounger);
});
});
};
Wealmostforgottoexportlounger,soweaddamodule.exportsattheendofthe
file:
module.exports=lounger;
Wecanalreadyusethecode:
$node
>constlounger=require('./lib/lounger.js');
lounger.load().then(console.log);
Promise{<pending>}
>{loaded:true,version:'1.0.0',load:[Function:load]}
>lounger.commands
{isonline:[Function:isOnline]}
>lounger.cli
{isonline:[Function:cli]}
>lounger.commands.isonline('http://localhost:5984').then(console.log);
Promise{<pending>}
>{'http://localhost:5984':true}
Thelaststepinthissectionistomakeloungerusableontheterminalitself.Forthistaskwe’vecreated thebinfolderalready.Timetoputafilecalledlounger-cli.jsinto
bin!
npm has a nice feature: if we add a JavaScript file to a property calledbin in the
package.jsonofamodule,npmwilladd it toourPATH ifweare installing thepackage
globally(e.g.withnpminstall-glounger).
Ifwehavethebin-propertydefinedandtheloungerisgloballyinstalled,itgetsavailableaslounger(whichisourpackagename)ontheterminal,quitecomfortable!Toinform
the user that a package is intended to get installed globally, we can addpreferGlobal: true to thepackage.json (see also:
https://docs.npmjs.com/files/package.json#preferglobal).
Toenablethetwofeaturesweaddthesetwolinestoourpackage.json:
"bin":"./bin/lounger-cli",
"preferGlobal":true,
Additionally we have tomakebin/lounger-cli executable, ifwe are onLinuxor
OSX:
$chmod+xbin/lounger-cli
The first line of ourlounger-cli filewill be a “shebang line”, it tellsLinxux/Unix
shellusersthatofLinux/UnixusersthatitmustrunourfilewithNode.js:
#!/usr/bin/envnode
Afterwardsweloadlib/lounger.js,thecoreofourcommandlinetool:
constlounger=require('../lib/lounger.js');
The next step is to parse our command line arguments. In this case we are using themodulenopttoparsethecommandlinearguments(installitwith$npmi--save
nopt).Itwouldbepossibletotrytoparsetheargumentsonourown,butabattleproven
module likenopt offers a lot more features and is easier to use. We get the passed
commandbyaccessingparsed.argv.remain:
constnopt=require('nopt');
constparsed=nopt({},{},process.argv,2);
constcmd=parsed.argv.remain.shift();
Thenextstep is toboot theclientbycallinglounger.load,whichwillbootstrapthe
client and populatelounger.commands andlounger.cli. After the promise got
resolved,wecallthecommandthatwaspassedonthecommandline:
lounger.load().then(()=>{
lounger.cli[cmd]
.apply(null,parsed.argv.remain)
.catch((err)=>{
console.error(err);
});
}).catch((err)=>{
console.error(err);
});
AseverycommandreturnsaPromise,wecatcherrorswithcatchandprintthemtothe
console.
Wecannowtestourminimalisticcommandlineclientonthecommandline:
$npminstall-g.
$loungerisonlinehttp://couchdb.example.com
http://couchdb.example.comseemstobeofflineornodatabase
AndgivenourPouchDB/CouchDBserverisrunning:
$loungerisonlinehttp://localhost:5984
http://localhost:5984seemstobeonline
Insteadofrunningnpminstall-g.aftereachcodechange,youcanalsorunnpm
linkinyourmoduledirectory.Itwilllinktheglobalinstallationtothecurrentdirectory,
whichmeansthateverychangeisimmediatelyavailable,aslongasthedirectorydoesnotchange.
ChooseyourownflavoursTherearecountlessgoodargumentparsinglibrariesonnpm,somealternativestonoptare:commander,
optimistandyargs.
Thatwas the firstbasicbuildingblock forourcommand lineclient,butweare still faraway from a product that our users would really love and promote.Wewill fix thoseissues(errorhandling,help,documentation)inthenextsections.Thecodeforthecurrentsectionisavailableatsourcecode/client-bootstrap.
ErrorhandlingWealreadylearnedin[_you_never_get_stuck]thatnouserenjoyscrypticerrormessagesandstacktraces.Sadlythatisstillthecaseforourloungerapplicationandmakingthe
applicationmoreaccessibleshouldbeourfirstpriority.
Rightnowloungerdoesnotdoanythingaboutwronginputfortheisonlinecommand:
$loungerisonlineragrragr
$
Another caveat to consider is, are we handling errors correctly so people can use thecommandlineclientintheirbashscripts?
$loungerisonlineragrragr
$echo$?
0
Wouldn’titbeniceriftheuserswouldgetahintaboutthecorrectusageoftheprogramrightaway,withoutopeninganydocumentation?Nobodyenjoyssittinginfrontofablackterminalhavingno ideawhat todo(see[_you_never_get_stuck]).Whileweareatitwecan also fix the wrong exit code which is currently signalling a successful executedprogram(seealso[_it_supports_powerusers_exit_codes]).
Whyisn’tourconsole.errorcall inbin/lounger-cliprintinganything?Turns
outweintroducedasubtlebug:weforgotthe.catchforourPromisereturningcallin
lib/isonline.js.GiventheAPIfunctionisOnlinerejectsthePromise,wehave
nohandlerinthefunctionclitotakecareofit.Noproblem,we’lladdthe.catchright
now:
functioncli(url){
returnnewPromise((resolve,reject)=>{
isOnline(url).then((results)=>{
//printonstdoutforterminalusers
Object.keys(results).forEach((entry)=>{
letmsg='seemstobeofflineornodatabaseserver';
if(results[entry]){
msg='seemstobeonline';
}
console.log(entry,msg);
resolve(results);
});
}).catch(reject);//addthemissingcatch
});
}
exports.cli=cli;
Ournexttryisabitmoresuccessful:
$loungerisonlineragrragr
[Error:InvalidURI"ragrragr"]
Not sure ifyouarehappywith it… I’mnot! Just imagine someone thathasneverusedNode.js or the Terminal. Maybe even someone that is completely new to computers.InvalidURI won’t help themmuch to get their task done. Twenty years ago they
wouldhavehad togetabookfromthe library inorder to findoutwhatanURI is,andtodaytheywouldhavetogoogleforit.UsefultimetheycouldhavefunwithourCLIandgetthingsdone!
Gladlywecanfixitbyaddingvalidationsfortheargumentsinisonline.js.
IftheuserdoesnotprovideaurltotheCLIwearecreatinganewerrorwiththemessageUsage: lounger isonline <url> which describes how they have to use the
command.WearesettingthetypeoftheerrortoEUSAGE,whichwillbeimportantlater.
In lounger all errors that are thrownbecause theusermadewrong input aregetting thetypeEUSAGE.Allothercaseswhereweintroducedbugsdon’tgetthetypeEUSAGE:
functioncli(url){
returnnewPromise((resolve,reject)=>{
if(!url){
consterr=newError('Usage:loungerisonline<url>');
err.type='EUSAGE';
returnreject(err);
}
The less-than sign and greater-than sign around the url indicate thaturl is a required
argumentandnotoptional.The lastcommand in theblock rejects thePromisewithourerrorandreturns,inordertopreventtheexecutionoffollowingcode.
Earlyreturnsareusefultoreducecyclomaticcomplexity.Cyclomaticcomplexityappearswhereifandelseblocks are nested,whichmakes it harder for the human brain to reason about the flow of the programexecution.
Additionallywehavetocheckiftheurlisavalidurl:
if(!/^(http:|https:)/.test(url)){
consterr=newError([
'invalidprotocol,mustbehttpsorhttp',
'Usage:loungerisonline<url>'
].join('\n'));
err.type='EUSAGE';
returnreject(err);
}
In this case we are setting the error type toEUSAGE again and reject the Promise.
Additionallywearetellingtheuserthatweexpectavalidurlwithaprotocolthatisusableforus.
Onournexttrywewillgetaslightlybetterresult:
$./bin/lounger-cliisonlinedsf
{[Error:invalidprotocol,mustbehttpsorhttp
Usage:loungerisonline<url>]type:'EUSAGE'}
Aswe reject the promise, theconsole.error thatwe added in inbin/lounger-
cliprintstheerrorobject.Byaddingafewlinesofcodewecanformatit,sohumanscan
read it better.Wewill install thenpmlog logger for it (hint:npmi is short fornpm
install):
$npmi--savenpmlog
We require it at the top ofbin/lounger-cli, the filewherewe catch the rejected
Promise:
constlog=require('npmlog');
As a next step we add the functionerrorHandler tobin/lounger-cli. If the
errorisausageerror(oftypeEUSAGE),welogthemessageandexitwitherrorcode1.
Allothererrorsareloggedusinglog.error(err)fornow:
functionerrorHandler(err){
if(!err){
process.exit(1);
}
if(err.type==='EUSAGE'){
err.message&&log.error(err.message);
process.exit(1);
}
log.error(err);
process.exit(1);
}
Nowwehave to switch from theoldconsole.error call toournewerrorhandling
function:
lounger.load().then(()=>{
lounger.cli[cmd]
.apply(null,parsed.argv.remain)
.catch(errorHandler);
}).catch(errorHandler);
Cool,let’sseeifitworks:
$loungerisonline
ERR!Usage:loungerisonline<url>
$echo$?
1
Thatlooksalotbetter!
Figure2.AusageerrorresultingfromprovidingwronginputtotheCLI
We still did not touch other errors: errors from dependencieswe use, or evil bugs thatsneak in, like reference errors. To simulate such an error we can add a call to a non-existingfunctionintheclifunction:
functioncli(url){
returnnewPromise((resolve,reject)=>{
doesNotExist();
Ifwenowrunthecommandlineclientweget:
$loungerisonlinehttp://example.com
ERR!ReferenceError:doesNotExistisnotdefined
IfIjustdownloadedthecommandlineclientandtriedtouseit,Iwouldbequitepuzzled.MaybeIgotanewjobandtriedtousethesametoolmycoworkersuse,butdownloadedanewerreleasewithbugs.Iwouldbestuck,withnofurthernoticehowtocontinue.Tobehonest,ifIhadn’tprogrammedinJavaScriptforyearsthisstacktracewouldreallypuzzleme!Formostpeoplethesearerocksintheirwaywheretheyjuststopusingourprogramandswitchtoanalternative.VeryfewgoonthejourneytofindoutwheretheycansubmitanissueorevenwriteaPR.Usuallycomputersaretoofrustratingandpeopledon’treallywanttospendmultiplehourstryingtofindsomeonetohelpthemwithacrypticmessage.Sowhat aboutmaking this process as easy as possible, reducing the frictionwherewecan?
npmitselfsupportsabugspropertyinthepackage.json.Ifweadd:
"bugs":{
"url":"http://example.com/lounger/issues"
},
to thepackage.json of lounger, a call to$ npm bugs will open
http://example.com/lounger/issues in a browser for us. Cool, we got a
centralplacewherewearestoringtheurltoourissuetracker.Wecanalsoaddtheurltoourstacktraces,inordertomakesubmittingbugsforouruserseasier.Weneedtorequirethepackage.jsoninbin/lounger-cli,thefilewhereweprintourerrorsanyway:
constpkg=require('./package.json');
ByalteringourerrorHandlerwemakeitprintfullstacktraces.Additionallyweadd
asktheusertoopenanissueasitisprettyclearrightnowthattheerrorwasnotausageerrorthatwascaughtbyourvalidations:
functionerrorHandler(err){
if(!err){
process.exit(1);
}
if(err.type==='EUSAGE'){
err.message&&log.error(err.message);
process.exit(1);
}
err.message&&log.error(err.message);
if(err.stack){
log.error('',err.stack);
log.error('','');
log.error('','');
log.error('','lounger:',pkg.version,'node:',process.version);
log.error('','pleaseopenanissueincludingthislogon'+
pkg.bugs.url);
}
process.exit(1);
}
Ok,nexttry:
$loungerisonlinehttp://example.com
ERR!doesNotExistisnotdefined
ERR!ReferenceError:doesNotExistisnotdefined
ERR!at/home/rocko/clibook/sourcecode/error-
handling/lib/isonline.js:35:7
ERR!atcli(/home/rocko/clibook/sourcecode/error-
handling/lib/isonline.js:34:10)
ERR!at/home/rocko/clibook/sourcecode/error-handling/bin/lounger-
cli:15:6
ERR!
ERR!
ERR!lounger:1.0.0node:v4.2.3
ERR!pleaseopenanissueincludingthislogon
http://example.com/lounger/issues
Awesome!Thestacktracewith linenumbers isuseful forus.Thecurrentversionof theprogramandtheNode.jsenvironmenthelpus,too.Incasethecommandlineclientreallyhits awall,we receive a lot of informations in order to debug the process. Evenmoreimportant:theuserget’salltheinformationneededtocreateanissue.Weremovealotoffriction from the process by directly pointing to the issue tracker and providing allinformationthatisneededtodescribethebug-nolongbackandforthaboutthecurrentNodeversionorthemissinglogfile!
Thecodeforthissectioncanbefoundatsourcecode/error-handling.
JSONsupportandShorthandsJSONsupportisusefulforallusersthatwanttotaketheoutputfromtheCLIandprocessit programmatically with their own tools (we talked about that in[_it_supports_powerusers_json]inthefirstpartofthebook).Byaddinga--jsonflagto
our commandisonlinewe can add this useful featurewith a few lines of code.We
havetotellourargumentparseraboutit,inthiscase,wearetellingnoptthatwewantto
have--jsonhandledasabooleaninbin/lounger-cli:
constparsed=nopt({
'json':[Boolean]
},{'j':'--json'},process.argv,2);
Based on the typeBoolean nopt will automatically also add--no-json for us,
whichwillcomehandywhenweaddadditionalconfigurationbyfile later.Additionallyweregisterashorthandforourpowerusers,theycanalsouse-jinsteadof—json.
Wearethenpassingtheresultparsedintolounger.load:
constparsed=nopt({
'json':[Boolean]
},{'j':'--json'},process.argv,2);
constcmd=parsed.argv.remain.shift();
lounger.load(parsed).then(()=>{
lounger.cli[cmd]
.apply(null,parsed.argv.remain)
.catch(errorHandler);
}).catch(errorHandler);
longer.loadaddsalounger.config.getcommandandmakesitavailableforus
aspartofthebootstrap:
lounger.load=functionload(opts){
returnnewPromise((resolve,reject)=>{
lounger.config={
get:(key)=>{
returnopts[key];
}
};
fs.readdir(__dirname,(err,files)=>{
Werequirelounger.jsinourfileisonline.js:
constlounger=require('./lounger.js');
As last stepwe the check for the json-flag in our functioncli afterwegot the results
back:
isOnline(url).then((results)=>{
if(lounger.config.get('json')){
console.log(results);
resolve(results);
return;
}
That’sit!Wecantestthecommand:
$loungerisonlinehttp://example.com
http://example.comseemstobeofflineornodatabaseserver
$loungerisonlinehttp://example.com--json
{'http://example.com':false}
$loungerisonlinehttp://example.com-j
{'http://example.com':false}
Ouruserscannowpipetheoutputontheirterminalsintootherconsumersandprocesstheresults.Wealsoaddedourfirstcommandlineflagtoloungertomodifytheexecutionofacommand - great! The code and tests for this section are insourcecode/json-
flags.
DocumentationThelaststeptofinishourcommandisonlineistoaddproperdocumentation.Wewill
write the documentation in Markdown and need documentation for the API and CLIcommands. The API docs will live indoc/api and the CLI commands will live in
doc/cli.Iknowthatmostprogrammershatewritingdocumentation,butitwillhelpus
alot:newuserswillbeabletogetupandrunningeasierandwewon’tlosethembeforethey had the chance to enjoy our product. Additionally, we make our lives easier bydocumenting the functionality once, so people don’t have to open issues or ask in chathow they can use a command. Basically a win-win situation.We start withdoc/api
/lounger-isonline.md, which describes the API that is available at
lounger.commands:
lounger-isonline(3)—checkifadatabaseisonline
====================================================
Theheadingdescribesourcommandaslounger-isonline(3)andthenaddsashort
explanationwhatthecommandisabout.Thenumberinbracketsdescribesthetypeofthesection.Foraman-pageaLibraryFunctionisnotedbya3andaUserCommandwould
bea1(spoiler:ourCLIcommandisaUserCommand).
Thenextsectiondescribeshowouruserscanusethecommand:
##SYNOPSIS
lounger.commands.isonline(url)
Thelastpartisadetaileddescriptionofhowthecommandworks:
##DESCRIPTION
CheckifaCouchDB/PouchDBdatabaseisavailableonthecurrent
network.
url:
Theurlmustbea`String`andmustbeaurlusingthehttporhttps
protocol.
Thecommandreturnsapromise.ThepromisereturnsanObject.Thekey
oftheObjectistheprovidedurlandthevaluesareoftype`Boolean`.
`true`indicatesanonlineCouchDB/PouchDBnode.
That’s it for the API part, we can now add the text fordoc/cli/lounger-
isonline.md:
lounger-isonline(1)—checkifadatabaseisonline
====================================================
loungerisonline<url>[--json]
##DESCRIPTION
<url>:
Checkifadatabasenodeiscurrentlyonlineoravailable.
`isonline`printstheresultashumanreadableoutput.JSONoutputis
alsosupportedbypassingthe`--json`flag.
With the small1 inlounger-isonline(1)we are signalling that this help section
explainsaUserCommand.Theless-thanandgreater-thansymbolsfor<url> showthe
userthaturlisamandatoryargument-withoutitthecommandwon’twork.Thesquare
bracketsof[--json]meanthatthejson-flagisanoptionalcommand.
Aswe got the sources for our documentation,we can start to build our documentationfromoursourceswithmarkedandmarked-man:
$npmi--save-devmarkedmarked-man
AMakefilewould be a great fit for generating the documentation from the source, butsadlyitishardtogetMakefilestoworkonWindows,sowewillwriteourbuildstepsinJavaScript. In therootdirectoryof lounger,wecreate thefilebuild.js.Additionally,
we have to installmkdirp andrimraf,mkdirp provides the functionalitywe know
from the Linux commandmkdir-p in a cross-platform compatibleway: it creates a
directories and subdirectories in a recursive way. The modulerimraf brings us the
equivalent ofrm-rf to theNode.js platform: deleting directories in a recursiveway.
Additionally we will use the moduleglob to match all needed files for our
documentationbuild.
$npmi--save-devmkdirprimrafglob
Ourfirstfunctionwillbeafunctiontocleanafreshfolderstructurewherewecansaveourman-pages:
'usestrict';
constmkdirp=require('mkdirp');
constrimraf=require('rimraf');
constglob=require('glob');
constpath=require('path');
functioncleanUpMan(){
rimraf.sync(__dirname+'/man/');
//recreatethetargetdirectory
mkdirp.sync(__dirname+'/man/');
}
Wehavetofindoutwhichmarkdownfilesareavailableforthecompile.Oursourcesfordocumentationareatdoc/apioratdoc/cli.Additionallywewillhavesomecontent
indoc/websitewhichisspecifictothewebsite,i.e.thecontentfortheindex.html.
The functiongetSourceshelpsus toget thefullpathto themarkdownfilesforeach
typeofour sources (api,doc,website). It returns the relativepathof thematching
globandthenusesthefunctionpath.resolvetogetthefullpathinacross-platform
compatibleway.
functiongetSources(type){
constfiles=glob.sync('doc/'+type+'/*.md');
returnfiles.map(file=>path.resolve(file));
}
Theobjectsourcesstoresanarrayofthefoundfilesforeachtype:
constsources={
api:getSources('api'),
cli:getSources('cli'),
websiteIndex:getSources('website'),
};
Weareabletocleanupourtargetdirectorynowandtogetalistoffilenamesthatwewanttoconvert.Westillneedtofindoutthetargetpathandfilenamefortheconvertedfiles.Man-pages have different file endings depending of the kind of functionality theydescribe.Ourman-pagesfortheCLIwouldgettheending.1(UserCommands)andour
API function would get the ending.3 (Library Functions). Additionally we have to
change the/doc/cli/ and/doc/api/ in thepathof the file toour targetdirectory
/man/.Wehavetotakespecialcareofthepath-separators.OnWindowstheseparators
area\\,insteadof/.Thatmeansthepathdoc/apigetsdoc\\apionWindows.The
goodnewsisthatwecanaccessthecurrentpath-separatorusingpath.sepinNode.js
(theseparatorisprovidedbythecoremodulepath):
functiongetTargetForManpages(currentFile,type){
lettarget;
//settherightsectionforthemanpageonunixsystems
if(type==='cli'){
target=currentFile.replace(/\.md$/,'.1');
}
if(type==='api'){
target=currentFile.replace(/\.md$/,'.3');
}
//replacethesourcedirwiththetargetdir
//doitforthewindowspath(doc\\api)andtheunixpath(doc/api)
target=target
.replace(['doc','cli'].join(path.sep),'man')
.replace(['doc','api'].join(path.sep),'man');
returntarget;
}
Right nowwe justwant to createman-pages from our documents in theapi andcli
folder.
Based on these building blocks we can create the final functionbuildMan that will
finally build our man-pages. It will make use of the functions we just created and
additionally spawnachildprocesswhichcompiles themarkdown filesusingmarked-
man.WewillusethefunctionspawnSyncfromNodecoretospawntheprocesses.As
wewritetheresulttothefilesystem,wehavetorequirethefsmodule,too:
constfs=require('fs');
constspawnSync=require('child_process').spawnSync;
Thefirstjobiscleaninguptogetanewtargetdirectorywithoutanyfilesfrompreviousbuilds.Wetheniterateoveroursourcesandgetthetargetforournewgeneratedfile.Thefile is thenwritten to theharddiskusingfs.writeFileSync.Incaseofthewebsite
indexwestoptheexecutionaswedon’twanttodousethewebsiteindexforaman-pagerightnow.Inthenextiterationwecoulddefinitelyaddamainpagefortheloungerman-pages:
functionbuildMan(){
cleanUpMan();
Object.keys(sources).forEach(type=>{
sources[type].forEach(currentFile=>{
if(type==='websiteIndex'){
return;
}
//convertmarkdowntoman-pages
constout=spawnSync('node',[
'./node_modules/marked-man/bin/marked-man',
currentFile
]);
consttarget=getTargetForManpages(currentFile,type);
//writeoutputtotargetfile
fs.writeFileSync(target,out.stdout,'utf8');
})
});
}
buildMan();
WithbuildMan();inthelastline,wekickoffthebuildprocesseverytimewerunthe
scriptwithNode.Afewmodificationstoourpackage.jsoncouldmakeitanpmscript
andrunthebuildbeforeapublish,soourusersdon’thavetocompileanythingontheir
ownaspartoftheinstallation.Thisensuresthateveryuserreallygetsthesamecontentofthepackageandmakesinstallationsfaster.
I npackage.json we modify thescripts section and add entries forbuild and
prepublish. Theprepublish entry is a special hook for npm – itwill run every
timebeforewepublishthepackagetotheregistry:
"scripts":{
"test":"mocha-Rspec",
"docs":"node./build",
"prepublish":"npmrundocs"
},
npmalsooffersanicefeatureforman-pages:npmcaninstallthemfortheusersotheyareavailable on the terminal withman<command> for Linux/Unix users. In order to do
that, we have to add another entry to ourpackage.json, theman entry in the
directorysection:
"directories":{
"man":"./man"
},
Theentrypointstoourlocalman-pagesdirectory.IfyouareonUnix/Linuxyoucantryiftheman-pagesworknow:
$npmrundocs
>[email protected]/home/rocko/clibook/sourcecode/documentation
>node./build
$npminstall-g.
>[email protected]/home/rocko/clibook/sourcecode/documentation
>npmrundocs
>[email protected]/home/rocko/clibook/sourcecode/documentation
>node./build
/home/rocko/.nvm/versions/node/v4.2.3/bin/lounger->
/home/rocko/.nvm/versions/node/v4.2.3/lib/node_modules/lounger/bin/lounger-
cli
/home/rocko/.nvm/versions/node/v4.2.3/lib
$manlounger-isonline
Figure3.Theman-pageforlounger,ourcommandlineclient
Itisalsopossibletoselectaspecificsection:
$man3lounger-isonline
$man1lounger-isonline
Forourhtml-baseddocumentationwehave toadd thehtmlspecificpartnow.Wehavecreate a folder calledwebsite in the doc folder of our module. It will contain the
templatesforthewebsitethatareusedto“frame“thedocumentoutput.
Intothefoldercalledwebsiteweputafilecalledtemplate.htmlwithsomebasic
markup,andmostimportantly,placeholders!
<!doctypehtml>
<htmllang="en">
<head>
<metacharset="utf-8">
<title>Theloungermanual&documentation</title>
</head>
<body>
<divclass="wrapper">
<ahref="./">lounger</a>
<divclass="content">
__CONTENT__
</div>
</div>
<navclass="toc-container">
<divclass="tocmain-toc">
__TOC__
</div>
</nav>
</body>
</html>
Weneedtocreatesomecontentfortherootofourwebsitetowelcometheuser.Iwilljustprovide a very short example. In general the landingpage shouldgive the user an ideawhatthecommandlineclientisaboutandmaybealsodemoit.Ascreencastisagreatwaytodemosomeofthecorefeatures.Forsomeinspirationwhattoputonthesite,feelfreetovisithttp://apache.github.io/couchdb-nmo,whichisthepageforthecommandlineclientIwrotelastyear.Nexttoourtemplateindoc/websiteweaddthebespokeindexpageas
index.md:
#WelcometoLounger
LoungerisafriendlyadministrationtoolforCouchDBandPouchDB.
```
#youwillneedNode.js>4forlounger
npminstall-glounger
```
Thingsyoucandowithlounger:
```
#checkifaCouchDB/PouchDBinstanceisonline
loungerisonlinehttp://example.com
```
Thepageisminimalisticbutgivesabriefoverviewwhatloungerisabout.ItalsoshowshowourvisitorscaninstallitandgivesahintthattheyneedNode.js–noteveryonehasNode.jsinstalledandsomepeoplemayhaveneverheardofnpm.Remember:ourgoalistomakeeverythingaseasyaspossiblefornewusers–theynevergetstuck.
Inorder tobuild thewebsitewehave toaddsomecode to thebuild.js.Westartby
addinganothercleanup-functionforourwebsite files,sowecanstartwithablankslateeverytime:
functioncleanUpWebsite(){
rimraf.sync(__dirname+'/website/');
mkdirp.sync(__dirname+'/website/');
}
Our website has different targets than the man-pages. We add a new function,getTargetForWebsite.Currentlyourmarkdownfilesareprefixedwithlounger-,
whichcomeshandyfortheman-pages,butisnotveryusefulforourwebsite.Insteadwewanttoprefixthemwiththeirtype,anAPIdocumentwouldgetprefixedbyapi-.This
waywe can put pages for the API next to the ones for the CLI, whichmakes linkingeasier. The first lines of the functiongetTargetForWebsitewill take care of that
taskandreplacethelounger-prefixwithaprefixspecifictothetypeofthedocument.
Afterwereplacedtheprefix,wesetthefileendingfrom.md to.html.Thefinaltarget
folder for our compiled results will be./website, so we have to take care that the
directoriesaresetupright.ThedifferencetogetTargetForManpagesisthatweuse
thewebsite folder instead ofman and that we also take care of the path for the
doc/website/index.mdfile:
functiongetTargetForWebsite(currentFile,type){
lettarget=currentFile;
//modifiythefilenameabitforourhtmlfile:
//prefixallclifunctionswithcli-insteadoflounger-
//prefixallapifunctionswithapi-insteadoflounger-
if(type==='cli'){
target=currentFile.replace(/lounger-/,'cli-');
}
if(type==='api'){
target=currentFile.replace(/lounger-/,'api-');
}
//setthefileendingtohtml
target=target.replace(/\.md$/,'.html');
//replacethesourcedirwiththetargetdir
target=target
.replace(['doc','cli'].join(path.sep),'website')
.replace(['doc','api'].join(path.sep),'website')
.replace(['doc','website'].join(path.sep),'website');
returntarget;
}
The last item which is missing for our website is something like a table of contents.getTocForWebsitewillcreatealistingofourAPIandCLIfunctions.Laterwewill
insertthetableofcontentsintotheTOCplaceholderofthetemplate.
TheTOCitselfgetsanestedlistofthefileswecompile.Inthefutureitmightmakesenseto replace the whole system with a template engine likehandlebars, jade or
Nunjucks. I coulddefinitelywritea secondbookabout staticwebsitegenerationplus
thedifferentpossibletoolchains,butItrytokeepitsimpleandminimalisticfornowandjustuseplainES6templates.
Afterstartinganunorderedlistwith<ul>weiterateoverthetypesofoursourcesagain.
ThefilesofourwebsiteIndextypeareunwantedandwereturnearlyintheircase.For
allotherswetakethetypeofeachsectionanduseitasthefirstlistelement.Itshowsthetypeofeachsection(APIorCLI)andactsasaheading.
Afterwegotaheading,weiterateovereachfilefromthecurrentsection.Asthelinktextdiffersabitfromtheactualhyperlink,wecreateaconstantcalledfileandanadditional
linktext constant. For bothwemust get rid of thelounger- prefix.To create the
referencethatisusedashrefwemustchangethe.mdtoa.htmlending.Thelinktext
shouldnothaveafileendingatall,soweremovethe.mdendingforit:
functiongetTocForWebsite(){
lettoc='<ul>';
Object.keys(sources).forEach(type=>{
//wedon'twanttheindexinourtocfornow
if(type==='websiteIndex'){
return;
}
toc+=`<li><span>${type}</span><ul>`;
sources[type].forEach(currentFile=>{
constprefix=type==='cli'?'cli-':'api-';
constfile=path.basename(currentFile)
.replace('lounger-',prefix)
.replace(/\.md/,'.html');
constlinktext=path.basename(currentFile)
.replace('lounger-','')
.replace(/\.md/,'');
toc+=`<li><ahref="${file}">${linktext}</a></li>`;
});
toc+='</ul></li>';
});
toc+='</ul>';
returntoc;
}
We can now take the small functions we created and create the main build functionbuildWebsitefromthem.TheconstanttemplateFiledescribeswherewecanfind
ourtemplatefiletemplate.htmlwhichwecreatedinthebeginning:
consttemplateFile=__dirname+'/doc/website/template.html';
Thenext step is to replace theplaceholders in the templateswith thegenerated tableofcontentsandthedifferentcontentforeachAPIorCLImethod.
InbuildWebsitewe callcleanUpWebsite toremoveanyoutdatedfilesasafirst
step.Wereadthetemplatewithfs.readFileSyncandgetthetableofcontents.We
then iterateoverour sources.This timewedon’t spawnachildprocesswithmarked-
man,wejustusemarked,whichoutputshtml.Afterwegotthetargetforthecurrentfile
ofthewebsitewereplacetheCONTENTplaceholderwithit.TheTOCgetsreplacedwith
the tableofcontentswhich issaved intocat the topof themainfunction.Movingthe
readoperationofthetemplateandthecreationoftheTOCoutoftheloopshaspositiveeffectsontheperformanceofourbuild,aswedon’thavetocallthemforeverynewfile.Therenderedcontentfinallygetswrittentothediskusingfs.writeFileSyncagain.
The last line in the code snippet finally callsbuildWebsite in order to build our
websitewhenwerunbuild.js.
functionbuildWebsite(){
cleanUpWebsite();
consttemplate=fs.readFileSync(templateFile,'utf8');
consttoc=getTocForWebsite();
Object.keys(sources).forEach(type=>{
sources[type].forEach(currentFile=>{
//convertmarkdowntowebsitecontent
constout=spawnSync('node',[
'./node_modules/marked/bin/marked',
currentFile
]);
consttarget=getTargetForWebsite(currentFile,type);
constrendered=template
.replace('__CONTENT__',out.stdout)
.replace('__TOC__',toc);
//writeoutputtotargetfile
fs.writeFileSync(target,rendered,'utf8');
});
});
}
buildWebsite();
We can now run our build again and take a look at ourwebsite,which appears in thewebsitefolder:
$npmrundocs
Figure4.Theminimalwebsiteforloungergeneratedfromourmarkdown
The site still misses some content and also a nice stylesheet. As I already mentioned,buildingwebsitesisawholetopicforanewbookandIwanttoleaveitlikethisfornow.Feel free to add more content and styles to the website. Nevertheless we just createdsomethingthatwecandeploytoGitHubpagesoranyotherhoster.Thebestthingaboutitis that we can ship it with our command line client as additional documentation – foreveryone who can’t use or doesn’t like man-pages. We will use both types ofdocumentationinournextsectionwhenwecreateahelpsystem.
Thecodeforthissectioncanbefoundatsourcecode/documentation.
MoreHelpWegotsomedocumentation,buttherearestillsomeroughedgesinlounger.Ifwetrytoaccessacommandwhichdoesnotexist,theuserdoesn’tgetanyadvicehowtocontinue:
$loungerfoobar
Ifausercallsloungerwithnoargumentsatalltheyalsodon’tgetsupport:
$lounger
What’s missing is a help page which explains how our users can get their task done.Additionally it would be really awesome if they could open our documentation (web-pagesorman-pages)rightfromtheterminal.Ourusersshouldn’thavetocareaboutman-pagesorhave to findoutwherewehostourwebsite.Thedesiredbehaviourof loungerwouldbe:
1. TheCLI prints a general helpmessage if itwas calledwithout a command or if apassedcommanddoesn’texist.Itgivestheuserahinthowtoproceedfurthertogettheirtaskdone.
2. Itiseasytogetadditionalhelpforacommand
Westartbycreatinganewlibraryfunction,lib/help.js.Thefirstfunctionwillprint
afriendlyhelptexttotheuser.Thehelptextgetsconstructedinahelperfunction(nopunintended).
WecangetallofouravailablecommandsbycallingObject.keys(lounger.cli).
Wechaina.join(',')calltoseparateeachcommandwithacommaandaspace:
functiongetGeneralHelpMessage(){
constcommands=Object.keys(lounger.cli).join(',');
Thenextpartisatemplatestringwhichexplainshowtouselounger.Weaddallavailablecommandswhichareexposedonthecommandlineinterface.Wealsoexplainthatuserswithoutacluecanrunloungerhelptogetherwiththecommandtheyareinterestedin
togetdetailedhelpforacommand.Additionallyweprovideanexamplehowtheywouldcallthehelp.
The final line tells them which version of lounger they run, which comes handy indifferentsituations.Usuallypeopleforgetwhichversionofapackagetheyhaveinstalled.Thereason issimple:nobodycanremember theexactversionofall software theyhaveinstalledrightnow.Imagineauserwhojustreadablogarticleaboutloungerversion2.3andagreatnewfeaturetheywanttotry.Theyinstalledloungersometimeago,butsadlythecommandisjustavailablesinceversion2.3whichgotreleasedthreedaysago.Inthiscasetheygetimmediatefeedbackthattheyrunonanolderversion.
HereisthewholefunctiongetGeneralHelpMessage:
functiongetGeneralHelpMessage(){
constcommands=Object.keys(lounger.cli).join(',');
constmessage=`Usage:lounger<command>
Theavailablecommandsforloungerare:
${commands}
Youcangetmorehelponeachcommandwith:loungerhelp<command>
Example:
loungerhelpisonline
loungerv${lounger.version}onNode.js${process.version}`;
returnmessage;
}
Thenextfunctionwe’llbuildwilltrytoopentheman-pageifpossible.ItwillfallbacktothewebsiteversionforWindowsusers.
Themoduleopenerdoesagreat jobtoopenfilesondifferentoperatingsystems.This
alsoappliestohtmlfiles,aswewanttoopentheminthedefaultbrowserofthecurrentoperatingsystem.Soweinstallopenerasournextdependency:
$npminstall--saveopener
Wethenrequireopeneratthetopofhelp.js:
constopener=require('opener');
We also have to spawn theman command later and find out the absolute path for our
websitefiles:
constspawnSync=require('child_process').spawnSync;
constpath=require('path');
ThecoremoduleoscanhelpustofindoutifwearerunningonWindows:
constisWindows=require('os').platform()==='win32';
We then have to spawn theman command or open the default browser for the desired
functionality.Withstdio:inheritforthespawncommandwecanseeandinteract
withtheoutputofthespawnedprocess:
functionopenDocumentation(command){
if(isWindows){
consthtmlFile=path.resolve(__dirname+'/../website/cli-'+command+
'.html');
returnopener('file:///'+htmlFile);
}
spawnSync('man',['lounger-'+command],{stdio:'inherit'});
}
Thelasttaskisputtingallourhelperfunctionstogether,ifacommandisnotavailable,weprint the general help. If the command exists, we are opening the man-page or thebrowser,dependingontheOperatingSystem:
exports.cli=help;
functionhelp(command){
returnnewPromise((resolve,reject)=>{
if(!lounger.cli[command]){
console.log(getGeneralHelpMessage());
}else{
openDocumentation(command);
}
resolve();
});
}
Ifyouwantyoucanalsoaddsomecodetomakeitconfigurablefortheuserwhichtypeofdocumentationisopenedforthem.MaybeLinuxusersprefertheWebsiteversionofthedocs. We can already try the new help command, as it gets picked up by ourlounger.jsfile:
Figure5.Ourhelpcommandinaction
Ourfinaltaskinthissectionisthatwehavetomakesuretoprintthegeneralhelpincaseswheretheuserdoesnotenteracommandatall.Inbin/lounger-cliwerequirethe
help.jsfileandmodifythelounger.loadcall.Incasewedon’tfindthecommand
inlounger.cli,weprintthegeneralhelp:
consthelp=require('../lib/help.js');
lounger.load(parsed).then(()=>{
if(!lounger.cli[cmd]){
returnhelp.cli();
}
lounger.cli[cmd]
.apply(null,parsed.argv.remain)
.catch(errorHandler);
}).catch(errorHandler);
Congrats!Wearefinishedwiththehelpsystemnowandmadesignificantprogress!Feelfreetoplayaroundwithournewhelpsystembyenteringthesecommands:
$loungerblerg
$lounger
$loungerhelp
$loungerhelpisonline
Thecodeforthissectionislocatedat:sourcecode/help-system.
ConfigurationRequiringourusers toadd their favourite settingsas flagsbyhandevery time theyuselounger can be cumbersome.A configuration file enables our users to save the settingstheyneedeveryday.Itwouldbeniceifwecouldsupportthisfeaturethatwecoveredin[_it_supports_powerusers_configuration]tomakeourpowerusershappier.
Itiscommonpracticethatcommandlinetoolsputtheirconfigurationfilesintothehomedirectoryoftheuser.Thehome-directoryisdifferentforeveryOperatingSystem,butthemoduleosenvcanhelpustofindthecurrenthomedirectory:
$npminstall--saveosenv
Webasicallyhavetoseeifaloungerrcconfigurationexistsinourhomedirectory,and
ifnot,wehavetocreateanemptyone.Wedothatinlounger-cli,inordertokeepthe
APIfunctionsfreeof thesideeffect.This is thechangedlounger-clifilewherewe
added arequire-call forosenv and create a config file right after the argument
parsing.Wealsoaddthepathtotheconfigfiletoourparsedarguments:
#!/usr/bin/envnode
constlounger=require('../lib/lounger.js');
constpkg=require('../package.json');
constlog=require('npmlog');
constnopt=require('nopt');
consthelp=require('../lib/help.js');
constosenv=require('osenv');
constfs=require('fs');
constparsed=nopt({
'json':[Boolean]
},{'j':'--json'},process.argv,2);
consthome=osenv.home();
parsed.loungerconf=home+'/'+'.loungerrc';
if(!fs.existsSync(parsed.loungerconf)){
fs.writeFileSync(parsed.loungerconf,'');
}
The config itself will use ini-formatted config files in order to store and read settings.config-chainisamoduletoloadconfigurationswithdifferentprioritiesbasedonthe
orderweloadthem.Italsosupportsini-formattedfiles.
Let’screateafilelib/config.jsandrequireconfig-chain:
$npmi--saveconfig-chain
'usestrict';
constcc=require('config-chain');
config-chainisabletomanagemultipleconfigurations.Itwilloverrideconfiguration
settingsaccordingtotheorderweloadthem.
Forourusecasewewanttheoptionsprovidedasargumentsonthecommandlinetohavethehighestpriority.Meaning,theyoverridetheonesfromtheconfigfile.Astheloadingofthefileisdoneasync,wehavetolistentotheloadeventemittedbyconfig-chain
onceitisfinished.Fortheusecaseswherenoconfigfilewasset,wedon’ttrytoloadadditasfile.Onallerrorswerejectourpromiseandpasstheerrorobject:
exports.loadConfig=loadConfig;
functionloadConfig(nopts){
returnnewPromise((resolve,reject)=>{
letcfg;
if(!nopts.loungerconf){
cfg=cc(nopts)
.on('load',()=>{
resolve(cfg);
}).on('error',reject);
}else{
cfg=cc(nopts)
.addFile(nopts.loungerconf,'ini','config')
.on('load',()=>{
resolve(cfg);
}).on('error',reject);
}
});
};
Theconfigobjectthatisreturnedfromconfigchainhasnicegetandsetmethods.We
canevensavetheconfigbacktotheconfigurationfileafterchangingtheconfigusingthe
savemethod.FornowwehavetointegrateloadConfigintothebootstrapoflounger.
Do you remember theconfig.get method inlounger.js from the previous
chapter?Wewillreplaceitwiththeconfigobjectthatisreturnedbyourloadfunction.
Asafirststepwehavetoloadourconfiginlounger.js:
constconfig=require('./config.js');
Inlounger.loadwearegoingtoloadtheconfig:
lounger.load=functionload(opts){
returnnewPromise((resolve,reject)=>{
config
.loadConfig(opts)
.then((cfg)=>{
});
Theothercontentoflounger.loadincludingthefs.readdircallismovedintothe
callbackofthechainedthenfunction:
lounger.load=functionload(opts){
returnnewPromise((resolve,reject)=>{
config.loadConfig(opts)
.then((cfg)=>{
lounger.config=cfg;
fs.readdir(__dirname,(err,files)=>{
files.forEach((file)=>{
if(!/\.js$/.test(file)||file==='lounger.js'){
return;
}
constcmd=file.match(/(.*)\.js$/)[1];
constmod=require('./'+file);
if(mod.cli){
cli[cmd]=mod.cli;
}
if(mod.api){
api[cmd]=mod.api;
}
});
lounger.loaded=true;
resolve(lounger);
});
}).catch(reject);
});
};
Ifwenowrunloungeritwillcreateaconfigfileforusinourhomedirectory.OnOSXmyconfigisat~/.loungerrc.
Afterthefilegotcreated(don’tforgettorunloungeratleastonetime)wecansetJSONoutputtotruein~/.loungerrc:
json=true
Just try out$ lounger isonlinehttp://example.com, lounger will print
JSONnow.Wecanstilloverridetheconfigonthecommandline:
$loungerisonline--no-json
As manually editing the~/.loungerrc is not very user friendly, we will build a
loungerconfig command.The config command shouldhave the abilities to show
theconfiganditsvaluesandtosetconfigvalues.
HereistheproposedCLI:
$loungerconfigsetjsontrue
$loungerconfiggetjson
TheAPIcouldlooklikethis:
lounger.commands.set('json',true)
lounger.commands.get('json')
config-chain offers the data provided in the loaded config file as
.sources.config.data.ForJSONformattedoutputwereturn thewholeconfig if
nokeywasprovided:
constdata=lounger.config.sources.config.data;
if(lounger.config.get('json')&&!key){
resolve(data);
return;
}
IfakeywasprovidedwebuildaJSONobjectthatjustcontainsthevalueforourkey:
if(lounger.config.get('json')&&key){
resolve({[key]:data[key]});
return;
}
Givenwedon’twantJSONformattedoutputandprovidedakeywereturnthevalue:
if(key){
resolve(lounger.config.sources.config.data[key]);
return;
}
Inthelastcasewherethejsonsettingissettofalse,andnokeywasprovidedwesimplyreadtheunparsedini-file:
resolve(fs.readFileSync(lounger.config.sources));
Hereisthewholegetfunction:
functionget(key){
returnnewPromise((resolve,reject)=>{
constdata=lounger.config.sources.config.data;
if(lounger.config.get('json')&&!key){
resolve(data);
return;
}
if(lounger.config.get('json')&&key){
resolve({[key]:data[key]});
return;
}
if(key){
resolve(lounger.config.sources.config.data[key]);
return;
}
resolve(fs.readFileSync(lounger.config.sources));
});
}
Modifyingtheconfigisdonebythesetfunction.Ittakesakeyandavalue,callsseton
theconfig-chainobjectandresolvesthepromiseafterthevaluesarewrittentothedisk:
functionset(key,value){
returnnewPromise((resolve,reject)=>{
if(!key&&!value){
reject(newError('keyandvaluerequired'));
return;
}
lounger.config.set(key,value,'config');
lounger.config.on('save',()=>{
resolve();
});
lounger.config.save('config');
});
}
Asalaststepwehavetoexposebothcommands:
exports.api={
get:get,
set:set
};
WebuildtheCLIfunctionalityontopofourAPIfunctions.ThemaindifferencebetweentheAPIandCLIfunctionisthattheCLIfunctionhassideeffectsforthegetcommand:
itprintstheresulttotheconsole.Wealsoaddsomeniceerrormessagestomakeiteasiertouse forourusers.Theyget instructionshow to run thecommand if theydon’tuse itproperly:
exports.cli=cli;
functioncli(cmd,key,value){
returnnewPromise((resolve,reject)=>{
functiongetUsageError(){
consterr=newError([
'Usage:',
'',
'loungerconfigget[<key>]',
'loungerconfigset<key><value>',
].join('\n'));
err.type='EUSAGE';
returnerr;
}
if(!cmd||(cmd!=='get'&&cmd!=='set')){
consterr=getUsageError();
returnreject(err);
}
if(cmd==='get'){
returnget(key).then((result)=>{
console.log(result);
}).catch(reject);
}
if(cmd==='set'){
if(!key&&!value){
consterr=getUsageError();
returnreject(err);
}
returnset(key,value).catch(reject);
}
});
}
Great! We have a config command now! The code for this section is atsourcecode/config. We fulfilled all points from[_it_supports_powerusers] and
[_you_never_get_stuck]andarereadyforaninitialrelease!
Ourfirstrelease&releasetipsYoucanpublishOpenSourcemodulesafter registeringanaccount at thenpm registry.After registering the accountyoubasically just have to typenpmpublish topublish
yourmodule to theregistry.Beforewepublish lounger Iwant tomention that therearesomenicewaystooptimisethepublishedpackages.InthissectionIwillexplainhowtooptimiseyourpackageregardingsizeandinstallationtime.
One way to keep the installation size small is to add a.npmignore file to the root
directory.Itworkssimilartoa.gitignorefileandnpmwon’tincludethelistedfiles
anddirectoriesinthepublishedpackage.
Dependingonthetypeofthefileswhicharen’tneededwhenusingthemoduleweareabletosavealotofspaceontheharddisksofourusers.Wewillsavethemalotofbandwidth,too.
Wecansavesomespaceifweexcludeourunitandintegrationtestsandalsothesourceforthecompileddocumentation:
/test/
/docs/
build.js
.DS_Store
npm-debug.log
Anothergreatwaytospeedupinstallationtimeistoincludeallproductiondependenciesinthepublishedmodule.WecantellnpmtobundlethemwithourmodulebyaddingthemasbundleDependenciestoourpackage.json:
"bundleDependencies":[
"config-chain",
"nopt",
"npmlog",
"opener",
"osenv",
"request"
],
Bundling the dependencies reduces the installation time a lot, aswe omit all the smallHTTPrequestsforeachdependencyandtheirdependenciesduringinstallation.Withthecurrent npm 3 it reduces the installation time from 20 seconds to 5 seconds for abroadbandconnection.Bundlingthedependenciesalsomakessurethatourpackageisstillinstallableevenifamodulewasunpublished.
When we now runnpmpublish wewill publish a highly optimised version of our
package.Thecodeforthissectionisatsourcecode/first-release.
MigrationoflargeamountsofdatausingStreamsSometimeswewant toprocess largeamountsofdata.With traditionalbufferingwerunintomemoryproblemsverysoon.Thewholedatajustdoesn’tfitintothememoryofthecomputer. Streams enable us to process data in small slices.Node.js streamswork likeUnixstreamsontheterminal,whereyoupipedatafromaproducerintoaconsumerusingthepipesymbol|.
Thecatprogramprintsthecontentoffilestostdout.InthisexampleIampipingthe
output of thecat program, intotr to change all letters from ourpackage.json to
uppercaseletters:
$catsourcecode/first-release/package.json|tr'a-z''A-Z'
Itworkswithalloutputonstdout:
$echo"ishout?"|tr'a-z''A-Z'
StreamsinUnixandinNode.jsenableustocomposesmallprogramsormodulesthatdoone thingwell togetour taskdone.Theyhandlebackpressure,whichmeans thata fastproducerwillautomaticallyslowdownifitispipedintoaslowconsumer.
The most used streams in Node are theReadable, Writeable andTransform
streams.Theyarebaseclassesthatcanbeusedtobuildyourowncustomstreams.Thereare also other stream types, like theDuplex stream and thePassthrough stream
whicharen’tcoveredin thebook.TheReadablestreamisusedtoreadinputdata,the
Transform stream is usually used to modify chunks of data and theWriteable
streamsacceptsdatatowriteitsomewhere(e.g.intoafile).
TodaywewillwriteacommandthatwillusestreamsforpipingdatafromCSVfilesintoCouchDB/PouchDB.Wecouldalsowriteanimportertomigratedatafromadatabase,e.g.aPostgresorMongoDB,butwithplainCSVfileswedon’thavetoinstallanewdatabase.Theprincipleappliestobothsourcetypes,filesanddatabases.AttheendofthechapterIwillprovidea link to anexample for aNode.js streampipeline thatmigratesdata fromMongoDBtoCouchDB/PouchDB.
Thefirststream
Building streamscanbeabit tricky sometimes.Westartbycreating the filestream-
example.jsintherootdirofourmodule.
OurfirstiterationwilloutputourCSVcontentstostdout.Wewilluseittolearnhow
streamsworkandbuildourimporterontopofitlater.TodevelopweneedaCSVfile,wecansaveittotest/fixtures/test.csv:
time;location
march;austin,us
april;boston,us
october;bristol,uk
february;hermigua,es
march;hermigua,es
april;havana,cu
Luckilywedon’thavetowriteourownstreamingCSVparser:
$npminstall--savecsv-parse
Werequirethefs,andutilmodule.Theutilmoduleisusedtoinheritfromthebase
objectTransform.ThefsmoduleisneededtoreadtheCSVfilefromdisk.Wealsoneed
torequiretheCSVparser:
constparse=require('csv-parse');
constfs=require('fs');
constTransform=require('stream').Transform;
constutil=require('util');
OurcustomstreamcalledMyTransformStreaminheritsfromstream.Transform.
WesetthestreamintoobjectModetobeabletoprocesstheJSONinputfromtheCSV
parser:
functionMyTransformStream(){
Transform.call(this,{
objectMode:true
});
}
util.inherits(MyTransformStream,Transform);
ATransformstreamhastoimplementonemethod:_transform.Themethodiscalled
for every chunk of data that we are processing. In the_transform method we can
transformthechunkstosomethingnew.Thetransformeddataisthenpushedtothenextconsumerusingthis.push.Oncewearefinishedwecall thedonecallbacktosignal
thatwearefinishedwiththischunk.Rightnowwejustwanttotakealookhowachunklookslike:
MyTransformStream.prototype._transform=transform;
functiontransform(chunk,encoding,done){
console.log('chunk:',chunk);
this.push(chunk);
done();
}
AslaststepwehavetopipetheCSVfileintotheCSVparserandtheparsedoutputintoourcustomstream:
constopts={comment:'#',delimiter:';',columns:true};
constparser=parse(opts);
constinput=fs.createReadStream(__dirname+'/test/fixtures/test.csv');
input
.pipe(parser)
.pipe(newMyTransformStream());
Whenwenowrunnodestreams-example.jswegetthisoutput:
$nodestreams-example.js
chunk:{time:'march',location:'austin,us'}
chunk:{time:'april',location:'boston,us'}
chunk:{time:'october',location:'bristol,uk'}
chunk:{time:'february',location:'hermigua,es'}
chunk:{time:'march',location:'hermigua,es'}
chunk:{time:'april',location:'havana,cu'}
Every chunk is a JSON object and every time we calldone and if there is still input
produced,_transform iscalledwith thenextchunk.Wecould takeeverychunkand
post it against the CouchDB / HTTPAPI now.Wewould keep ourmemory footprintsuperlow,butwewouldalsosendalotofHTTPrequestsandthewholemigrationwould
takeverylong.AhealthycompromiseistobufferafewchunksandpostthemagainstthebulkAPIsofCouchDB/PouchDB.Thiswaywedon’tbufferallexistingdataandrunoutofmemoryandwearefinishedearlierwithourimport,aswedon’thavetosendsomanyHTTPrequests.
The code for this section can be found atsourcecode/streams/streams-
example.js.
TheTransformandWriteablestreamForournext streamwewillcreate the filestreams-bulk-example.js in theroot
directoryoflounger.ItwilltaketheobjectsfromtheCSVparsingstreamandbufferthem.Atagivenlimititwillpassthebufferedobjectstothenextconsumer.Theresultpassedtothenextconsumer is ready togetpostedagainst theCouchDB/PouchDBbulkdocsAPIendpoint. The CouchDB/PouchDB bulk API accepts an array of objects wrapped with{"docs":[]}.
Westartwiththesamesetofmodules:
constparse=require('csv-parse');
constfs=require('fs');
constTransform=require('stream').Transform;
constutil=require('util');
ThenameofourstreamwillbeTransformToBulkDocsanditwilltakeoptionsasan
object.Usingtheoptionswecanspecifytheamountofdocumentstobuffer:
functionTransformToBulkDocs(options){
if(!options){
options={};
}
if(!options.bufferedDocCount){
options.bufferedDocCount=200;
}
Theemptyarrayforthis.bufferwillbeourbuffer:
Transform.call(this,{
objectMode:true
});
this.buffer=[];
this.bufferedDocCount=options.bufferedDocCount;
}
util.inherits(TransformToBulkDocs,Transform);
Hereisthewholeconstructorfunction:
functionTransformToBulkDocs(options){
if(!options){
options={};
}
if(!options.bufferedDocCount){
options.bufferedDocCount=200;
}
Transform.call(this,{
objectMode:true
});
this.buffer=[];
this.bufferedDocCount=options.bufferedDocCount;
}
util.inherits(TransformToBulkDocs,Transform);
Inthe_transformmethodweaddeverychunktoourbuffer:
TransformToBulkDocs.prototype._transform=transform;
functiontransform(chunk,encoding,done){
this.buffer.push(chunk);
Ifthebufferhasgrownbigenoughwecallthemethodthis.push,whichweinherited
from the baseTransform stream.this.push(args) tellsNode thatwewant topass
argstothenextconsumerinourstreampipeline.Wethenemptythebufferfornewdata
thatmightarrive:
if(this.buffer.length>=this.bufferedDocCount){
this.push({docs:this.buffer});
this.buffer=[];
}
done();
}
Hereisthewhole_transformmethod:
TransformToBulkDocs.prototype._transform=transform;
functiontransform(chunk,encoding,done){
this.buffer.push(chunk);
if(this.buffer.length>=this.bufferedDocCount){
this.push({docs:this.buffer});
this.buffer=[];
}
done();
}
Thelastpartofourfileisalmostidenticaltoourfirststreamsexample:
constopts={comment:'#',delimiter:';',columns:true};
constparser=parse(opts);
constinput=fs.createReadStream(__dirname+'/test/fixtures/test.csv');
input
.pipe(parser)
.pipe(newTransformToBulkDocs());
Weaddatemporaryconsole.logtoourcodetoseeifitworks:
if(this.buffer.length>=this.bufferedDocCount){
this.push({docs:this.buffer});
console.log({docs:this.buffer});
this.buffer=[];
}
When we runnodestreams-bulk-example.js we see that… it doesn’t work!
Why?
Theproblemisthatwedon’thaveenoughdocumentstoreachthedefaultdocumentcountof 200. The same applies to the remaining documents of a set. If we have 250 initialdocuments as input, the first 200 hundreds are pushed to the next consumer, but theremaining 50 are lost. Luckily the Node.js developers were aware of the problem andprovided the_flushmethod.Themethod doesn’t have to be implemented tomake a
Transform stream work, like the_transform method. Instead we can chose to
implementit,givenweneedit.
The_flushmethodwillgetcalledat theveryendafteralldatawasconsumedby the
stream, but before the stream emits theend eventwhich signals the endof the stream.
_flushwillgetcalledattheveryendandgivenwestillhaveafewbuffereddocuments,
wepushthemtothenextconsumer:
TransformToBulkDocs.prototype._flush=flush;
functionflush(done){
this.buffer.length&&this.push({docs:this.buffer});
done();
}
That’sourfirstcustomstream!Don’tforgettoremovetheconsole.logcallweadded!
Thenextconsumerinourpipelinewilltakethecollecteddocumentsandpostthemagainstthe CouchDB / PouchDB bulk docs endpoint. As the streams are able to handlebackpressure, the other streamswillwait untilwe successfully added the documents toCouchDB/PouchDB.Theywillcontinuetopassusdatadownthepipelineonceweareabletopullinthenextcollectionofdocuments.
Thenextstreamwewillbuildaccepts thedata fromtheTransformstreamandwrites itintothedatabase.ItisaWriteablestreamandwewilladdittoourstreams-bulk-
example.jsfile:
constWritable=require('stream').Writable;
OurWritablestreamneedstoknowwheretoputthedata,sowewillneedtopassitthedatabaseurl.Asthemethodsofthestreamarecalledforeachchunkwehavetostorethepassedurlasthis.url:
functionCouchBulkImporter(options){
if(!options){
options={};
}
if(!options.url){
constmsg=[
'options.urlmustbeset',
'example:',
"newCouchBulkImporter({url:'http://localhost:5984/baseball'})"
].join('\n')
thrownewError(msg);
}
Writable.call(this,{
objectMode:true
})
//sanitiseurl,removetrailingslash
this.url=options.url.replace(/\/$/,'');
}
util.inherits(CouchBulkImporter,Writable);
ToimplementachildofaWritablestream,wehavetoimplementthe_writemethod.
Like the_transformmethodof theTransformstream, the_writemethod iscalled
foreverychunkthat ispassed to thestreamfromthepreviousproducer. Inourcasewesend the JSON chunks as JSON to the database using request. After we sent the datasuccessful to the/_bulk_docsAPIendpointwecall thedonecallbacktosignalthat
wearereadyforanewchunk:
CouchBulkImporter.prototype._write=write;
functionwrite(chunk,enc,done){
request({
json:true,
uri:this.url+'/_bulk_docs',
method:'POST',
body:chunk
},function(err,res,body){
if(err){
returndone(err);
}
if(!/^2../.test(res.statusCode)){
constmsg='CouchDBserveranswered:\nStatus:'+
res.statusCode+'\nBody:'+JSON.stringify(body);
returndone(newError(msg));
}
done();
});
}
Wealsohavetorequirerequestinourfile:
constrequest=require('request');
To use the stream we have to pipe the data into it. We update the last section ofstreams-bulk-example.js:
input
.pipe(parser)
.pipe(newTransformToBulkDocs())
.pipe(newCouchBulkImporter({url:'http://127.0.0.1:5984/travel'}));
Afterwecreatedthedatabasetravelandrunourscript,wehaveimportedtheCSV:
$curl-XPUThttp://localhost:5984/travel
{"ok":true}
$nodestreams-bulk-example.js
$curlhttp://localhost:5984/travel/_all_docs
{"total_rows":6,"offset":0,"rows":[{"id":"3444bf7c-65c0-438f-f8e8-
7f55124f1736","key":"3444bf7c-65c0-438f-f8e8-7f55124f1736","value":{"rev":"1-
37fcd2e5b40939805b8e043da44f9b1d"}},{"id":"568c3b0f-78fe-43a2-9eac-
06620cfaa595","key":"568c3b0f-78fe-43a2-9eac-06620cfaa595","value":{"rev":"1-
e047788bac9aada0564fc928642d3960"}},{"id":"5d170e63-d845-4e45-f760-
97e30cbc4b21","key":"5d170e63-d845-4e45-f760-97e30cbc4b21","value":{"rev":"1-
6f1794e4eb24b60665fe02c2624d53eb"}},{"id":"9cd34a61-8a48-4b88-afbf-
7fd6e3c9cf42","key":"9cd34a61-8a48-4b88-afbf-7fd6e3c9cf42","value":{"rev":"1-
747b11a103cc0fe31d1c546ef70d69c0"}},{"id":"cc7da504-438a-40c1-842b-
4722b63c9a37","key":"cc7da504-438a-40c1-842b-4722b63c9a37","value":{"rev":"1-
0f9e7add8b7fbb773dc7d3081475d855"}},{"id":"f09811c5-43ec-4a6d-bd3b-
b34b3674676d","key":"f09811c5-43ec-4a6d-bd3b-b34b3674676d","value":{"rev":"1-
d677c5bbbee1bbbe45f6128a9cf1fe8d"}}]}
Wecanaccessasingledocumentusingtheid:
$curlhttp://localhost:5984/travel/3444bf7c-65c0-438f-f8e8-7f55124f1736
{"time":"february","location":"hermigua,es","_id":"3444bf7c-65c0-438f-f8e8-
7f55124f1736","_rev":"1-37fcd2e5b40939805b8e043da44f9b1d"}
Looksgreat!Seemswehaveeverythinginplacetouseourlow-levelstreamingfunctionsin the command line client. The code for this section is located atsourcecode/streams/streams-bulk-example.js.
ThestreamingimportcommandForthelastpartofthischapterwewillreusethecustomstreamimplementationthatwecreated inThe Transform andWriteable stream. In the real world I would create twomodulesforourtwostreamstomakethemreusableacrossmultipleprojects,butfornowwe can copy the code for theCouchBulkImporter and the
TransformToBulkDocs streams intolib/csv.jswhichwill be the homeof our
importcommand:
constparse=require('csv-parse');
constfs=require('fs');
constTransform=require('stream').Transform;
constutil=require('util');
constWritable=require('stream').Writable;
constrequest=require('request');
constlounger=require('./lounger.js');
functionTransformToBulkDocs(options){
if(!options){
options={};
}
if(!options.bufferedDocCount){
options.bufferedDocCount=200;
}
Transform.call(this,{
objectMode:true
});
this.buffer=[];
this.bufferedDocCount=options.bufferedDocCount;
}
util.inherits(TransformToBulkDocs,Transform);
TransformToBulkDocs.prototype._transform=transform;
functiontransform(chunk,encoding,done){
this.buffer.push(chunk);
if(this.buffer.length>=this.bufferedDocCount){
this.push({docs:this.buffer});
this.buffer=[];
}
done();
}
TransformToBulkDocs.prototype._flush=flush;
functionflush(done){
this.buffer.length&&this.push({docs:this.buffer});
done();
}
functionCouchBulkImporter(options){
if(!options){
options={};
}
if(!options.url){
constmsg=[
'options.urlmustbeset',
'example:',
"newCouchBulkImporter({url:'http://localhost:5984/baseball'})"
].join('\n')
thrownewError(msg);
}
Writable.call(this,{
objectMode:true
})
//sanitiseurl,removetrailingslash
this.url=options.url.replace(/\/$/,'');
}
util.inherits(CouchBulkImporter,Writable);
CouchBulkImporter.prototype._write=write;
functionwrite(chunk,enc,done){
request({
json:true,
uri:this.url+'/_bulk_docs',
method:'POST',
body:chunk
},function(err,res,body){
if(err){
returndone(err);
}
if(!/^2../.test(res.statusCode)){
constmsg='CouchDBserveranswered:\nStatus:'+
res.statusCode+'\nBody:'+JSON.stringify(body);
returndone(newError(msg));
}
done();
});
}
Let’s think a bit about the commandwe are going to build.ACSV can have differentdelimiters, some use semicolons as a delimiter, others are using commas or tabs. The
symbolstodenoteacommentcanalsochange.Inourpreviousimplementationsweusedfixedvalues:
constopts={comment:'#',delimiter:';',columns:true};
For a real world use case the symbols for the delimiter and comment must beconfigurable.TheCSVinputisusuallyafile.
HereisapossibleCLI:
$loungercsvtransfer<file><database>[--delimiter=;][--comment=#][--
chunksize=200]
The commandcsv isopen toextensionandcanhostallCSVrelatedcommands in the
future. The command reads quite nicely and is easy to remember:lounger csv
transfer <file> <database> reads almost aslounger [do] csv
transfer[from]<file>[to]<database>.Sanedefaultshelpus toavoid
passingoptionalmodifiersatall,butincaseweneedtomodifythemwecanchangeeveryimportantaspectofourimport.
I’mnotsureifyounoticedit,butwhenweplayedwithourstreamswewouldhavehadtocreatethetargetdatabaseusingcurlinadvance.ItwouldbehandyifourCLIusersdon’thavetocreatethetargetdatabaseontheirown.Ourgoalistohelpthemtosolvetheirtaskas quickly and easily as possible, so we should automatically create databases asnecessary.
The functioncreateTargetDatabase is a helper functionwhichwrapsrequest
intoaPromise.Ifthedatabasewascreated(HTTPcode201or200)orthedatabaseexistsalready(HTTPcode412)weresolve,allotherstatesleadtorejectionofthePromise:
functioncreateTargetDatabase(url){
returnnewPromise((resolve,reject)=>{
request({
json:true,
uri:url,
method:'PUT',
body:{}
},function(er,res,body){
if(er&&(er.code==='ECONNREFUSED'||er.code==='ENOTFOUND')){
consterr=newError(
'Couldnotconnectto'+url+'.Pleasecheckifthedatabaseis
offline'
);
err.type='EUSAGE';
returnreject(err);
}
if(er){
returnreject(er);
}
constcode=res.statusCode;
if(code!==200&&code!==201&&code!==412){
constmsg='CouchDBserveranswered:\nStatus:'+
res.statusCode+'\nBody:'+JSON.stringify(body);
returnreject(newError(msg));
}
resolve();
});
});
}
In case of anECONNREFUSED orENOTFOUND error we can safely assume that the
databaseiscurrentlyofflineandasktheusertotakealookifthedatabaseisavailable.Ican’tstressenoughhowimportantpropererrorhandlingis.Takethisexample,wherewearegettingbackECONNREFUSED:
$./bin/lounger-clicsvtransfertest/fixtures/test.csv
http://127.0.0.1:1337/testimport
ERR!connectECONNREFUSED127.0.0.1:5984
ERR!Error:connectECONNREFUSED127.0.0.1:5984
ERR!atObject.exports._errnoException(util.js:870:11)
ERR!atexports._exceptionWithHostPort(util.js:893:20)
ERR!atTCPConnectWrap.afterConnect[asoncomplete](net.js:1063:14)
ERR!
ERR!
ERR!lounger:1.0.0node:v4.2.4
ERR!pleaseopenanissueincludingthislogon
http://example.com/lounger/issues
Depending on howmuch our users usedNode.js before theywould be very puzzled Iguess.Theonlywaytocontinueforthemwouldbetoaskasearchengineortoopenanissue.After receiving the issueourboring jobwouldbe toclose the issueand tell themthattheyprobablyhadatypointheirurl.
After writing thecreateTargetDatabase function we should have all our
supporting functions in place. As usual we are starting to implement the main CLIfunctions by implementing the API command which we will then wrap with our CLIfunction.
The delimiter and comment options are defined in the config file or are passed on thecommand line. To know what their values are, we have to interact withlounger.config.Toaccesslounger.configwehavetorequireit:
constlounger=require('./lounger.js');
The main API function checks if all necessary arguments were provided and appliesdefaultsifnoconfigurationwaspassedinfromtheconfigfileoronthecommandline.Wecreate the database in case it does not exist yet and delegate to the helper functionimportFromCsvFile:
exports.api={
transfer:bulkdocsImport
};
functionbulkdocsImport(file,targetDb){
returnnewPromise((resolve,reject)=>{
constopts={};
if(!file&&!targetDb){
returnreject(newError('fileand/ortargetDbargumentmissing'));
}
opts.delimiter=lounger.config.get('delimiter')||';';
opts.comment=lounger.config.get('comment')||'#';
opts.chunksize=lounger.config.get('chunksize')||200;
createTargetDatabase(targetDb)
.then(()=>{
returnimportFromCsvFile(file,targetDb,opts);
}).catch(reject);
});
}
importFromCsvFile accepts the source CSV file, url and options and creates the
streampipeline.Themaindifferencetoourpreviouscodeinstreams-example.jsis
thatwehavepropererrorhandlinginplacetocatchallerrors.
functionimportFromCsvFile(file,url,opts){
returnnewPromise((resolve,reject)=>{
constoptions={comment:opts.comment,delimiter:opts.delimiter,
columns:true};
constparser=parse(options);
constinput=fs.createReadStream(file);
input
.pipe(parser)
.on('error',reject)
.pipe(newTransformToBulkDocs({bufferedDocCount:opts.chunksize}))
.on('error',reject)
.pipe(newCouchBulkImporter({url:url}))
.on('error',reject);
});
}
TheCLIfunctionsfinallywrapsourAPImethodandaddsfriendlyerrormessages:
exports.cli=importCli;
functionimportCli(cmd,file,target){
returnnewPromise((resolve,reject)=>{
if(!cmd||cmd!=='transfer'||!file||!target){
consterr=newError(
'Usage:loungercsvtransfer<file><database>[--delimiter=;][--
comment=#][--chunksize=200]'
);
err.type='EUSAGE';
returnreject(err);
}
returnbulkdocsImport(file,target).catch(reject);
});
}
We introduced three new options,delimiter, comment andchunksize.
lounger.config enables our users to set default values using the config file. In
addition we have to take care that the options are parsed on the command line. Inbin/lounger-cliwehavetoregisterouroptionalargumentstonopt:
constparsed=nopt({
'json':[Boolean],
'delimiter':[String],
'comment':[String],
'chunksize':[Number]
},{'j':'--json'},process.argv,2);
That’s it!We created a command that is able to stream large amounts of data into ourdatabase.
If you are interested in a stream pipelinewhichwould stream data fromMongoDB toCouchDB you can take a look athttps://github.com/robertkowalski/couchbulkimporter/blob/master/examples/mongo.js.
Thecodeforthissectionisatsourcecode/streams,enjoy! �
TIPS&TRICKS
ThissectioncollectssometipsandtricksregardingNode.jsdevelopmentingeneral.
TestingWith proper unit and integration tests in place, ensure new features or bugfixes don’tintroduce regressions. There are a lot of great services that provide a hosted CIenvironment. They can test every Pull Request before it is merged, which makesreviewingcodealoteasier.ApopularserviceforhostedCIisTravisCI.TravisCIisfreeforOpenSourceprojects.
SemanticVersioningwithSemVerI would recommend to follow Semantic Versioning with SemVer (http://semver.org/).SemVer divides the version number of a release into three areas:MAJOR.MINOR.PATCH.Theversion3.5.8wouldhave3asMAJORversionlevel,5asMINOR version level and 8 as patchlevel. Given a new release includes a breakingchange,theMAJORversionnumberisbumped.AnewfeaturewouldjustneedaminorversionbumpandbugfixeswouldjustrequireabumpofthePATCHsection.
Example:Mypackage has version 3.5.7 and I add a new featurewhich does not breakbackwardscompatibility.Mynextreleasewouldbe3.6.0.
Thiswayyourusersgetanideaifareleasemightbreaktheirproductioncode(MAJOR),it contains a new feature (MINOR)or a bug fix (PATCH).Agreat tool to help you tomake the right decision for the next version bump is semantic-release(https://www.npmjs.com/package/semantic-release).
GreenkeeperKeeping track which dependencies of your project got a new version and need to getupdated can be tedious. The update itself (bumping the version number in thepackage.json) is not the most interesting task on earth, too. A new and exciting
service ishttp://greenkeeper.io.Once you registered it for your project itwill send youpullrequestswithupdatedversionsofyourdependencies.Ifyouhaveatestsuiteinplaceandeverythingis“green“youjusthavetomergethePullRequestfromtheGreenkeeper
bot.TestingandaCIServicethatautomaticallyrunsthetests,SemVerandGreenkeeperreallyshowtheirstrengthswhencombinedtogether. �
THEEND
IhopeyouenjoyedTheCLIBook!
OurjourneytosuccessfulCLIsdoesn’tendhere,itjustbegins.
I am very happy about feedback. Please send and feedback or corrections [email protected]. You can also contact me on twitter:@robinson_k – pleaserecommendthebooktoothersincaseyoulikeit. �