Beautiful Bash: Let's make reading and writing bash scripts fun again!

  • View
    720

  • Download
    6

Embed Size (px)

Transcript

  • Beautiful Bash: A community driven effortLets make reading & writing Bash scripts fun again!Aaron Zaunerazet@azet.org

    lambda.co.at:Highly-Available, Scalable & Secure Distributed SystemsDevOps/Security Meetup Vienna - 17/12/2014

  • IntroductionWorking towards a community style guideDoing it wrongModern Bash scripting (Welcome to 2014!)Conclusion

  • Caveat EmptorIm not endorsing Bash for large-scale projects, difficult orperformance critical tasks. If your project needs to talk to adatabase, object store, interact with a filesystem or dynamicallyhandle block devices - you SHOULD NOT use Bash in the firstplace. You can. But youll regret it - I speak from years ofexperience doing completely insane stuff in Bash for fun (certainlynot for profit).Bash is useful for one thing and one thing only: as glue!..and its the glue that holds Linux distributions, EmbeddedAppliances and even Commercial networking gear together - so youbetter use the best glue on the market, right?DevOps/Security Meetup Vienna - 17/12/2014 Beautiful Bash: A community driven effortAaron Zauner 1/30

  • Do we really need another style guide?I For starters: Its not only a style guide, but more on that later.I A lot of the internet actually runs on poorly written Bash.I Your company probably depends on a lot of Bash-glue.I Everyone uses it on a daily basis to glue userland utilitiestogether.I Some scripts unintentionally look like they are submissions foran obfuscated code contest.I There are some style guides (e.g. by Google) and tutorials -but nothing definitive.I Most books on the subject are ancient and often reflectpersonal opinions of authors, outdated Bash versions anduserland utilities and most havent been updated in decades.I I dont know a single good book on Bash. The best resource isstill http://wiki.bash-hackers.org.

    DevOps/Security Meetup Vienna - 17/12/2014 Beautiful Bash: A community driven effortAaron Zauner 2/30

  • Working towards a community style guideI Ive started collecting style guides, tutorials, write-ups, toolsand debugging projects during the last couple of years...chose the best ideas and clearest styles and combined theminto one big community driven effort.I People started contributing.I Nothing is written in stone. Come up with a better idea for acertain topic and Ill gladly accept it.I Ive also included a lot of mistakes people do or even rely onwhen writing their (often production) scripts.I Ive also collected a lot of tricks and shortcuts Ive learned overthe years specific to bash scripting and the Linux userland.

    DevOps/Security Meetup Vienna - 17/12/2014 Beautiful Bash: A community driven effortAaron Zauner 3/30

  • Bad Example

    Heres a cool and bad example at the same time. rpm2cpioreimplemented in bash.I As Debian package: Installed-Size: 1044I As Bash script: 4

    DevOps/Security Meetup Vienna - 17/12/2014 Beautiful Bash: A community driven effortAaron Zauner 4/30

  • Bad Example (cont.)

    https://trac.macports.org/attachment/ticket/33444/rpm2cpioDevOps/Security Meetup Vienna - 17/12/2014 Beautiful Bash: A community driven effortAaron Zauner 5/30

  • Common bad style practicesI overusing grep for tasks that Bash can do by itself.I using bourne-shell backticks instead of $() for subshell calls... ever tried to nest backtick subshells? yea. youll have toescape them. instead of e.g.:

    $(util1 $(util2 ${some_variable_as_argument})).I manual argument parsing instead of using the getopts builtin.I using awk for arithmetic operations bash can do very well... same goes for expr(1). please stop using it in bash scripts... same goes for bc(1). please stop using it in bash scripts.

    DevOps/Security Meetup Vienna - 17/12/2014 Beautiful Bash: A community driven effortAaron Zauner 6/30

  • Common bad style practices (cont.)

    I using the echo builtin where printf can (and probablyshould) be used.I using seq 1 15 for range expressions instead of {1..15}I many coreutils you do not need & you save on subshell calls... a lot is set as a variable in your environment already(protip: see what env gives you to work with in the first place)I worst of all: endless and unreadable pipe glue. . . . . . . . . . . .

    DevOps/Security Meetup Vienna - 17/12/2014 Beautiful Bash: A community driven effortAaron Zauner 7/30

  • Common bad style practices (cont.)So what is more readable to you and probably the angry sysadminthat might take over your codebase at some point in time?ls ${long_list_of_parameters} | grep ${foo} | grep -vgrep | pgrep | wc -l | sort | uniq

    orls ${long_list_of_parameters} \

    | grep ${foo} \| grep -v grep \| pgrep \| wc -l \| sort \| uniq

    DevOps/Security Meetup Vienna - 17/12/2014 Beautiful Bash: A community driven effortAaron Zauner 8/30

  • awk(1) for everythingBut why?$ du -sh Downloads | awk { print $1 }366G

    $ folder_size=($(du -sh Downloads))$ echo ${folder_size[1]}Downloads$ echo ${folder_size[0]}366G

    DevOps/Security Meetup Vienna - 17/12/2014 Beautiful Bash: A community driven effortAaron Zauner 9/30

  • DevOps/Security Meetup Vienna - 17/12/2014 Beautiful Bash: A community driven effortAaron Zauner 10/30

  • Debugging is a mess

    One of the reasons nobody should aim for big projects in Bash isthat it is terrible to debug, most of you will know this already.

    This project aims to make it easier for you to debug your scripts.By writing beautiful, solid and testable code.

    DevOps/Security Meetup Vienna - 17/12/2014 Beautiful Bash: A community driven effortAaron Zauner 11/30

  • Modern Bash scriptingMost people dont know that there are a lot of useful paradigms andtools that are used for software engineering in serious languagesavailable also to Bash.

    Lets not kid ourselves: some Bash scripts will run in production,even for years. Theyd better work. And not take your businessoine.

    DevOps/Security Meetup Vienna - 17/12/2014 Beautiful Bash: A community driven effortAaron Zauner 12/30

  • ConventionsIve come up with a few conventions:I use #!/usr/bin/env bashI do not use TABs for (consistently use 2, 3 or 4 spaces)I but conditional and loop clauses on the same line:

    if ..; then instead ofif ...then

    ...fi

    I therere no private functions in Bash, RedHat has a conventionfor that, prepend with two underscores function__my_private_function()

    I as in Ruby, Python; dont use indents in switch (case) blocksI always escape varabiles. Bad: $MyVar, Good: ${MyVar}.DevOps/Security Meetup Vienna - 17/12/2014 Beautiful Bash: A community driven effortAaron Zauner 13/30

  • DocOptDocOpt is a Command-line interface description language withsupport for all popular programming languages.I http://docopt.org/I https://github.com/docopt

    ..also for BashI https://github.com/docopt/docopts

    DevOps/Security Meetup Vienna - 17/12/2014 Beautiful Bash: A community driven effortAaron Zauner 14/30

  • Test Driven Development and Unit tests with Bash#!/usr/bin/env bats

    @test "addition using bc" {result="$(echo 2+2 | bc)"[ "$result" -eq 4 ]

    }

    @test "addition using dc" {result="$(echo 2 2+p | dc)"[ "$result" -eq 4 ]

    }

    . . .DevOps/Security Meetup Vienna - 17/12/2014 Beautiful Bash: A community driven effortAaron Zauner 15/30

  • Test Driven Development and Unit tests with Bash (cont.)1. Sam Stephenson (of rbenv fame) wrote an automated testingsystem for Bash scripts called bats using TAP (Test AnythingProtocol): https://github.com/sstephenson/bats2. Sharness: another TAP library. theres even a Chef cookbookfor it: https://github.com/mlafeldt/sharness3. Cram: a functional testing framework based on Marcurialsunified test format - https://bitheap.org/cram/4. rnt: Automated testing of commandline interfaces -

    https://github.com/roman-neuhauser/rnt5. shUnit2: is a xUnit framework (similar to PyUnit, JUnit etcetera) - https://code.google.com/p/shunit2/6. shpec: Tests/Specs - https://github.com/rylnd/shpec..there are more, but these Ive found to be most useful.DevOps/Security Meetup Vienna - 17/12/2014 Beautiful Bash: A community driven effortAaron Zauner 16/30

  • LintingI A online Bash style linter:

    https://github.com/koalaman/shellcheckI Ubuntu ships with a tool called checkbashisms based onDebians lintian (portability).I shlint tests for portability between zsh, ksh, bash, dash andbourne shell (if need be):

    https://github.com/duggan/shlintI For Node fans: Grunt task that checks if a Bash script is valid(not anything else, btw):

    https://www.npmjs.com/package/grunt-lint-bash

    DevOps/Security Meetup Vienna - 17/12/2014 Beautiful Bash: A community driven effortAaron Zauner 17/30

  • Inter-shell portability

    Personal opinion:Inter-shell portability doesnt matter. Ive spent years writing OSagnostic bourne-shell scripts. Today every modern OS ships with areasonably recent version of Bash. These days Solaris (and FOSSforks like SmartOS) ship even with a GNU userland. Use Bash.I love zsh and it can do a lot more. I still use Bash for (semi-)production scripts. They run basically everywhere when done right.

    DevOps/Security Meetup Vienna - 17/12/2014 Beautiful Bash: A community driven effortAaron Zauner 18/30

  • Defensive Bash programmingI As you would in every other language, write helper functions,test these functions.I Set constants readonly.I Write concise, well defined and tested functions for everyaction.I Use the local keyword for function-local variables.I Prepend every function with the function keyword.I Return proper error codes and check for them.I Write unit tests.I Some people write a function main() as people would withPython. So one can import and test ones main call as well.

    DevOps/Security Meetup Vien