Sphinx autodoc - automated api documentation - PyCon.KR 2015

Sphinx autodocautomated API documentation

Takayuki Shimizukawa

Sphinx co-maintainer

Sphinx-users.jp

1PyCon Korea 2015

안녕하세요 .annyeonghaseyo.나는 시미즈 강입니다 .

naneun shimizukawa ibnida.

2

こんにちはKon-Nichi-Wa

3

Who am I

@shimizukawa

1. Sphinx co-maintainer

2. Sphinx-users.jp

3. PyCon JP committee

BePROUD co, ltd.4

http://pycon.jp/

5

Sphinx autodoc

6

Do you knowthe Sphinx?

7

Are you usingthe Sphinx?

8

Do you know the docstring?

9

1. def dumps(obj, ensure_ascii=True):2. 　　　　 """Serialize ``obj`` to a JSON formatted

``str``.3. """4. 5. ...6. return ...

Line 2,3 is a docstring

You can see the string by "help(dumps)"

Docstring

10

Have you writtenAPI docs

using docstrings?

11

What is the reason you do not write docstrings.

I don't know what/where should I write.

Are there some docstring format spec?

It's not beneficial.

I'll tell you a good way to write the docstrings.

12

Goal of this session

How to generate a doc from Python source code.

re-discovering the meaning of docstrings.

13

Sphinx introduction& How to setup it

14

What is Sphinx?

15

Sphinx is a documentation generator

Sphinx generates doc as several output formats from the reST text

markup

Sphinx

reSTreSTreStructuredText

(reST)reST

Parser

HTML Builder

ePub Builder

LaTeX Builder texlive

HTMLtheme

Favorite Editor

The history of Sphinx (short ver.)

16

The father of Sphinx

Too hard to maintenance

~2007

Easy to writeEasy to

maintenance2007~

Sphinx Before and After

Before

There was no standard ways to write documentsSometime, we need converting markups into other

formats

After

Generate multiple output format from single sourceIntegrated html themes make read docs easierAPI references can be integrated into narrative docsAutomated doc building and hosting by ReadTheDocs

17

Many docs are written by Sphinx

For examples

Python libraries/tools:Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …

Non python libraries/tools:Chef, CakePHP(2.x), MathJax, Selenium, Varnish

18

Many docs are written by Sphinx

For examples

Python libraries/tools:Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …

Non python libraries/tools:Chef, CakePHP(2.x), MathJax, Selenium, Varnish

19

Sphinx extensions (built-in)

Sphinx provides these extensions to support automated API

documentation.

sphinx.ext.autodocsphinx.ext.autosummarysphinx.ext.doctestsphinx.ext.coverage

Sphinx

reST Parser

HTML Builder

ePub Builder

LaTeX Builder

docutils

auto

sum

ma

ry

auto

doc

doctest

coverage

20

library code example

1. "utility functions"2. 　3. def dumps(obj, ensure_ascii=True):4. """Serialize ``obj`` to a JSON formatted

``str``.5. """6. ... 　

deep_thought/utils.py

$ tree /path/to/your-code+- deep_thought| +- __init__.py| +- api.py| +- calc.py| +- utils.py+- setup.py

21

$ pip install sphinx

Your code and sphinx should be in single python environment.

Python version is also important.

How to install Sphinx

22

$ cd /path/to/your-code$ sphinx-quickstart doc -m...Project name: Deep thoughtAuthor name(s): MiceProject version: 0.7.5......Finished

"-m" to generate minimal Makefile/make.bat-m is important to introduce this session easily.

How to start a Sphinx project

Keep pressing ENTER key

23

Create a doc directory

$ cd doc$ make html...Build finished. The HTML pages are in _build/html.

"make html" command generates html files into _build/html.

make html once

24

Current files structure

$ tree /path/to/your-code+- deep_thought| +- __init__.py| +- api.py| +- calc.py| +- utils.py+- doc| +- _build/| | +- html/ | +- _static/| +- _template/| +- conf.py| +- index.rst| +- make.bat| +- Makefile+- setup.py

Scaffold files

Build output

Library files

25

Generate API docs from your python source code

26

$ tree /path/to/your-code+- deep_thought| +- __init__.py| +- api.py| +- calc.py| +- utils.py+- doc| +- _build/| | +- html/ | +- _static/| +- _template/| +- conf.py| +- index.rst| +- make.bat| +- Makefile+- setup.py

1. import os2. import sys3. sys.path.insert(0,

os.path.abspath('..'))4. extensions = [5. 'sphinx.ext.autodoc',6. ]

setup autodoc extensiondoc/conf.py

27

Line-3: add your library path to import them from Sphinx autodoc.

Line-5: add 'sphinx.ext.autodoc' to use the extension.

Add automodule directive to your doc

1. Deep thought API2. ================3. 4. .. automodule:: deep_thought.utils5. :members:6.

1. "utility functions"2. 　3. def dumps(obj, ensure_ascii=True):4. """Serialize ``obj`` to a JSON formatted

``str``.5. """6. ... 　

doc/index.rst

28

deep_thought/utils.pyLine-4: automodule directive import

specified module and inspect the module.

Line-5: :members: option will inspects all members of module not only module docstring.

$ cd doc$ make html...Build finished. The HTML pages are in _build/html.

make html

29

How does it work?

autodoc directive generates intermediate reST internally:1. Deep thought API2. ================3. 　4. .. py:module:: deep_thought.utils5. 　6. utility functions7. 　8. .. py:function:: dumps(obj, ensure_ascii=True)9. :module: deep_thought.utils10. 　11. Serialize ``obj`` to a JSON formatted :class:`str`.

doc/index.rst

IntermediatereST

30

$ make html SPHINXOPTS=-vvv......[autodoc] output:

.. py:module:: deep_thought.utils

utility functions

.. py:function:: dumps(obj, ensure_ascii=True) :module: deep_thought.utils

Serialize ``obj`` to a JSON formatted :class:`str`.

You can see the reST with -vvv option

31

Take care!

Sphinx autodoc import your code

to get docstrings.

It means autodoc will execute code at module global level.

32

Danger code

1. import os2. 3. def delete_world():4. os.system('sudo rm -Rf /')5. 6. delete_world() # will be executed at "make html"

danger.py

33

execution guard on import

1. import os2. 3. def delete_world():4. os.system('sudo rm -Rf /')5. 6. delete_world() # will be executed at "make html"

danger.py

1. import os2. 3. def delete_world():4. os.system('sudo rm -Rf /')5. 6. if __name__ == '__main__':7. delete_world() # doesn't execute at "make

html"

safer.py

Execution guard

34

execution guard on import

1. import os2. 3. def delete_world():4. os.system('sudo rm -Rf /')5. 6. delete_world() # will be executed at "make html"

danger.py

1. import os2. 3. def delete_world():4. os.system('sudo rm -Rf /')5. 6. if __name__ == '__main__':7. delete_world() # doesn't execute at "make

html"

safer.py

Execution guard

35

"Oh, I can't understand the type of arguments and meanings even reading this!"

36

Lacking necessary information

1. def dumps(obj, ensure_ascii=True):2. """Serialize ``obj`` to a JSON formatted3. :class:`str`.4. 5. :param dict obj: dict type object to

serialize.6. :param bool ensure_ascii: Default is True. If7. False, all non-ASCII characters are

not ...8. :return: JSON formatted string9. :rtype: str10. """http://sphinx-doc.org/domains.html#info-field-

lists

"info field lists" for arguments

deep_thought/utils.py

37

def dumps(obj, ensure_ascii=True): """Serialize ``obj`` to a JSON formatted :class:`str`.

:param dict obj: dict type object to serialize. :param bool ensure_ascii: Default is True. If False, all non-ASCII characters are not ... :return: JSON formatted string :rtype: str """ ...

"info field lists" for argumentsdeep_thought/utils.py

38

Cross-reference to functions

1. Examples2. ==========3. 4. This is a usage

of :func:`deep_thought.utils.dumps` blah blah blah. ...

examples.py

reference(hyper link)

39

Detect deviations of the impl and doc

40

Code example in a docstring

1. def dumps(obj, ensure_ascii=True):2. """Serialize ``obj`` to a JSON formatted3. :class:`str`.4. 5. For example:6. 7. >>> from deep_thought.utils import dumps8. >>> data = dict(spam=1, ham='egg')9. >>> dumps(data)10. '{spam: 1, ham: "egg"}'11. 12. :param dict obj: dict type object to

serialize.13. :param bool ensure_ascii: Default is True. If14. False, all non-ASCII characters are

not ...

deep_thought/utils.py

41

doctestblock

You can copy & paste the red lines from python interactive

shell.

Syntax highlighted output

$ make html...

rendereddoctestblock

42

You can expect that developers will update code examples when the interface is changed.

We expect ...

1. def dumps(obj, ensure_ascii=True):2. """Serialize ``obj`` to a JSON formatted3. :class:`str`.4. 5. For example:6. 7. >>> from deep_thought.utils import dumps8. >>> data = dict(spam=1, ham='egg')9. >>> dumps(data)10. '{spam: 1, ham: "egg"}'

The code example is very close from implementation!!

deep_thought/utils.py

43

￣＼ _( ツ )_/ ￣

44

doctest builder

45

1. ...2. 3. extensions = [4. 'sphinx.ext.autodoc',5. 'sphinx.ext.doctest',6. ]7.

Line-5: add 'sphinx.ext.doctest' extension.

setup doctest extension

doc/conf.py

append

46

$ make doctest...Document: api-------------********************************************************File "api.rst", line 11, in defaultFailed example: dumps(data)Expected: '{spam: 1, ham: "egg"}'Got: 'to-be-implemented'...make: *** [doctest] Error 1

Result of "make doctest"

47

Result of doctest

Listing APIs automatically

48

Individual pages for each modulesapi.html

calc.html

utils.html

49

1. deep_thought.utils2. ===================3. .. automodule:: deep_thought.utils4. :members:

doc/deep_thought.utils.rst

1. deep_thought.utils2. ===================3. .. automodule:: deep_thought.utils4. :members:

Add automodule directive to your doc

doc/deep_thought.utils.rst

1. deep_thought.calc2. ==================3. .. automodule:: deep_thought.calc4. :members:

1. deep_thought.api2. ==================3. .. automodule:: deep_thought.api4. :members:

doc/deep_thought.calc.rst

doc/deep_thought.api.rst

And many many reST files ... 50

￣＼ _( ツ )_/ ￣

51

autosummary extension

52

1. ...2. 3. extensions = [4. 'sphinx.ext.autodoc',5. 'sphinx.ext.doctest',6. 'sphinx.ext.autosummary',7. ]8. autodoc_default_flags = ['members']9. autosummary_generate = True10.

Line-9: autosummary_generate = True generates reST files what you will specify with using autosummary directive.

setup autosummary extension

doc/conf.py

append

appendappend

53

1. Deep thought API2. ================3. 4. .. autosummary::5. :toctree: generated6. 7. deep_thought.api8. deep_thought.calc9. deep_thought.utils10.

Replace automodule with autosummary

doc/index.rst

$ make html...

54

Output of autosummary

autosummary directive become TOC for each module pages.

55

Discovering undocumented APIs

56

1. def spam():2. ...3. return res4. 5. def ham():6. ...7. return res8. 9. def egg():10. ...11. return res

How to find undocumented funcs?

doc/nodoc.py

57

coverage extension

58

1. ...2. 3. extensions = [4. 'sphinx.ext.autodoc',5. 'sphinx.ext.doctest',6. 'sphinx.ext.autosummary',7. 'sphinx.ext.coverage',8. ]9. autodoc_default_flags = ['members']10.autosummary_generate = True

setup coverage extension

doc/conf.py

append

Line-7: add 'sphinx.ext.coverage' extension.

59

make coverage and check the result

$ make coverage...Testing of coverage in the sources finished, look at the results in _build\coverage.

$ ls _build/coveragec.txt python.txt undoc.pickle

1. Undocumented Python objects2. ===========================3. deep_thought.utils4. ------------------5. Functions:6. * egg

_build/coverage/python.txt

This function doesn't have a doc!60

CAUTION!

1. Undocumented Python objects2. ===========================3. deep_thought.utils4. ------------------5. Functions:6. * egg

python.txt

$ make coverage...Testing of coverage in the sources finished, look at the results in _build\coverage.

$ ls _build/coveragec.txt python.txt undoc.pickle

The command always return ZERO

coverage.xml is not exist

reST format for whom?

61

Let's make Pull-Request!

We are waiting for your contribution

to solve the problem.(bow)

62

Wrap-up

63

Why don't you write docstrings?

I don't know what/where should I write.Let's write a description, arguments and

doctest blocks at the next line of function signature.

Are there some docstring format spec?Yes, you can use "info field list" for argument

spec and you can use doctest block for code example.

It's not beneficial.You can use autodoc, autosummary,

doctest and coverage to make it beneficial.

64

Options, Tips

65

1. Options

66

Options for autodoc

:members: blahTo document just specified members. Empty is ALL.

:undoc-members: ...To document members which doesn't have docstring.

:private-members: ...To document private members which name starts with

underscore.

:special-members: ...To document starts with underscore underscore.

:inherited-members: ...To document inherited from super class.

67

2. Directives for Web API

sphinxcontrib-httpdomain

68

sphinxcontrib-httpdomain extension

http domain's get directive

render

page

render routing table (index)

http highlighter

It also contains sphinxcontrib.autohttp.flask, .bottle, .tornado extensions

Listing

New Index

69

3. Directives for Diagram

sphinxcontrib-blockdiagsphinxcontrib-sqdiagsphinxcontrib-actdiagsphinxcontrib-nwdiag

70

blockdiag series

71

http://blockdiag.com/

blockdiag seqdiag (sequence)

actdiag (activity) nwdiag (network)

packetdiag (packet) is included in nwdiag

class Request(object): """ .. seqdiag::

{ code [label="Your Code"]; lib [label="Library"]; site [label="Web Site"];

code -> lib [label="Request(url)"]; code <-- lib [label="req"]; code -> lib [label="req.open(auth_cb)"]; lib -> site [label="GET"]; lib <-- site [label="status=401"]; lib => code [label="auth_cb(realm)", ... lib -> site [label="GET/auth header"]; lib <-- site [label="status=200"]; code <-- lib [label="status"]; } """

sphinxcontrib-seqdiag extension

72

deep_thought/api.py

extensions = [ 'sphinxcontrib.seqdiag',]

conf.py

$ pip install sphinxcontrib-seqdiag

4. Document translation

73

Translation into other languages

$ make gettext...Build finished. The message catalogs are in _build/gettext.

$ sphinx-intl update -p _build/gettext -l es

#: ../../../deep_thought/utils.pydocstring of deep_thought.msgid "Serialize ``obj`` to a JSON formatted :class:`str`."msgstr "Serializar ``obj`` a un formato JSON :class:`str`."

msgid "For example:"msgstr "Por ejemplo:"

locale/es/LC_MESSAGES/generated.po

language = 'es'locale_dirs = ['locale']

conf.py

$ make html...Build finished. The HTML pages are in _build/html. 74

Questions?@shimizukawa

Grab me at Office Hour!(After this session)

75

Thanks :)

76