Jump to: navigation, search


Doc Migration from DocBook to RST Overview

Previously, we migrated the <service>-api repos from DocBook to RST. These were API reference documents that were meant for the contributor developer to know what was going into an API. So, the history of that document was forged as a specification. As projects beyond swift and nova were added, projects made similar documents. They're output to HTML at http://docs.openstack.org/api/api-specs.html. However, as we now have a <project>-specs repo, it makes sense to move to those repos. In that move, also it makes sense to migrate to RST rather than DocBook/WADL.

Migration Conventions

Follow the RST markup conventions in the OpenStack Documentation Contributor Guide .


First title in an RST file, use equals signs above and below. Second title, use tilde. Third title, use a series of dashes. Try not to need a Heading 4 if you can help it by rewriting or reorganizing.

Heading 1

Heading 2

Heading 3

File names

As a rule, we want to keep the xml:id identical so that the output HTML files do not need redirection. However, we also at the same time want to get rid of ch_ and section_ nomenclatures while going to a page-based, topical approach. So, first use the XML:ID, and if the XML:ID has ch_ or section_ in it, remove the prefix.

Cross references

Use :ref: when doing cross references so that you don't have to have the exact title. This means you will have to add markup to create anchors, such as:

.. _dashboard-project-tab:

Please use the existing XML:Ids if possible for these cross references.

Also, while intersphinx is an enabled extension for many of the contributor developer docs, we don't have an intersphinx requirement yet for the end user guide or admin user guide. We'll investigate that as we add more guides.

Figures and images

Store the figures in a /figures/ directory with the deliverable itself.

Notes and admonitions

If the conversion outputs Note as a heading, change it to use the .. note:: directive.

Line numbers in code blocks

If your file has lots of code blocks, be sure to set up line numbering for the entire file. Each code-block must have :linenos: after the .. code-block:: directive, the code itself must be indented at the same level as the :lineos: line, and you must have at the start of the file this directive:

.. highlight: python
   :linenothreshold: 5

so that any code-blocks longer than five lines long will have line numbers.

Migration Issues

List of bugs or missing features for the Sphinx template, openstackdocstheme: https://bugs.launchpad.net/openstack-manuals/+bugs?field.tag=openstackdocstheme

Can you have line breaks in something with inline semantic markup like :guilabel: at the end of the line? For example:

#. Select the volume to add to an instance and click :guilabel:`Edit Attachments`.

That line is longer than 79 characters, so Attachments`. must be on the second line, but when you do that, the HTML output is incorrect.

Since RST documents impose a 79 character limit when building files in .rst format. When tables exceed that limit, a list table role can help create content in a new table without exceeding the character limit. The issue is when a table is generated by an OpenStack service, and returned as a result of a command. These tables are constructed with +, -, and | characters. The table created by a list table role has solid lines forming rows, columns, and cells. The content is similar, however users might be surprised by a table with solid lines rather than a table built with +, -, and | characters.

Large scale content inside tables.

When building tables with a list table role, some items that appear inside the table cells might exceed the 79 character limit. Is there a solution to recreating this large scale content in a table such that it does not affect the character limit?

The :orphan: role and references.

Files with an :orphan: role at the top of the file, not included in the toc tree, will not link to another file when the :ref: role is used to link to the file from a different file. Is there a solution to this linking issue?

Pandoc conversion problems.

When converting files with Pandoc, some entries in the .xml file will not convert accurately. Procedure titles and sub section titles will not parse correctly, and will be deleted from the converted document. Pandoc also removes procedure numbering. Currently, the solution is to compare the complete .rst file to the built .xml version, and check the headings, subheadings, and procedure numbers to make sure they are correct, and line up with original. This is an issue that could also be discussed.

  • Contact details for RST issues that we can't solve:
    • Author: David Goodger
    • Contact: docutils-develop@lists.sourceforge.net

Migration How To

Note: If you don't want to use tox, install these prereqs locally: pip install sphinx; pip install openstackdocstheme and then switch to the directory containing a conf.py and run sphinx-build /path/to/source/ path/to/build/ to get html output by default.

Because Sphinx builds have dependent requirements it is best to work with in a virtual environment. Fortunately the openstack-manuals project already has tox set up so that you can create a virtualenv that contains the required dependencies. To use it, do this on a Mac or Ubuntu machine with Python and pip already installed:

  1. Clone the openstack-manuals repository.
  2. Change directories into openstack-manuals.
  3. Run this:
    tox -e py27
  4. When you get a congratulations message, run this:


source .tox/py27/bin/activate


 source .tox/py27/Scripts/activate

Your prompt should now have (py27) as a preface.

Now you have all the pre-requisites installed and can run additional tox commands. To see the list of commands, view or edit tox.ini in the openstack-manuals directory. To build the RST docs, run:

tox -e docs

This will run sphinx-build doc/playground-user-guide/source/ doc/playground-user-guide/build/html.

When the build is finished, you can open doc/playground-user-guide/build/html/index.html to view the resulting output.

Migration Using Oxygen

  1. Open DocBook book file in Oxygen.
  2. Choose Document > Transformation > Configure Transformation Scenario(s).
  3. Select DocBook XHTML - Chunk.
  4. Click Apply associated (1).
  5. Within the /out/xhtml-chunks/ directory that's generated, run the following script:
    for i in *.xhtml
    # Convert from XHTML to RST
    pandoc -s -t rst $i -o $file_name
    sed -i -e '4,16d' $file_name
    sed -i -e '/+--------------------------+$/,$d' $file_name
    sed -i -e '$d' $file_name
    sed -i -e '$d' $file_name
    # Rename file to second line of new RST content, but lowercase and all non-alphanumeric chars renamed to underscores:
    real_file_name=$(sed 's/[^a-zA-Z0-9\-]/_/g;2q;d' $file_name | awk '{print tolower($0)}').rst
    mv $file_name $real_file_name<pre><nowiki>
    # Replace all cross-refs to xhtml files to renamed rst files:
    sed -i '' -e "s/\<$i/fixmefixmefixme/g" *.rst
    # sed -i '' -e "s/\ fixmefixmefixme.*\`__//g" *.rst
  6. Clean up where the fixmefixmefixme is output, it indicates where a cross-reference cannot exist any longer.
  7. Clean up tables where the pandoc conversion just outputs paragraphs.
  8. Remove numbering from Example titles and Table titles.
  9. Rename chapter_ files and ensure they are titled to match the contents of the file, such as "Networking API 2.0 Overview" to networking-api-2.0-overview" for example.
  10. Remove "programlisting" "screen" and "literallayout" from .. code:: lines.
  11. When commiting conversion patches, include the related blueprint in your commit message. For example:
    Convert ch_compute_focus.xml to RST
    Implements: blueprint archguide-mitaka-rst

Migration Using a Text Editor

  1. Install pandoc.
  2. Update your openstack-manuals master branch, and create a branch.
  3. Run the pandoc command to convert the .xml file to .rst. For example:
    pandoc -f docbook -t rst -s ch_compute_focus.xml -o compute-focus.rst
  4. Move converted files to the <guide>/source folder.
  5. Rename converted RST files and ensure they are titled to match the contents of the file, such as "Networking API 2.0 Overview" to networking-api-2.0-overview" for example.
  6. Pandoc has its quirks, so check all the content has been migrated from the source file.
  7. Use a text editor to complete the following tasks:
    1. Clean up tables where the pandoc conversion just outputs paragraphs.
    2. Remove numbering from Example titles and Table titles.
    3. Remove "programlisting" "screen" and "literallayout" from .. code:: lines.
    4. Check the converted RST file follow conventions described in the OpenStack Documentation Contributor Guide
  8. Build the guide locally, to check for errors:
    tox -e docs
  9. Commit your changes, including the related blueprint in your commit message. For example:
    Convert ch_compute_focus.xml to RST
    Implements: blueprint archguide-mitaka-rst

Doc Migration Plan

For the Mitaka release, we migrated the Configuration Reference Guide and the Architecture Design Guide. Refer to the detailed specification for details.

API Reference Plan

See http://docs.openstack.org/contributor-guide/api-guides.html for explanation of the process. Anne has already preprocessed the files, so the other migration steps on this wiki page are not needed at all.

Sign up below for a service, then download the files from this patch: https://review.openstack.org/#/c/311596/. These will require a lot of cleanup manually but gets them as far as the tools can get them. Also, setup a job to publish data like https://review.openstack.org/312184.

Service name Service version Project Name Patch URL Status
Compute v2.1 nova Sean Dague https://review.openstack.org/#/q/status:merged+project:openstack/nova+branch:master+topic:bp/api-ref-in-rst In progress
Identity v2.0 keystone Samuel de Medeiros Queiroz https://review.openstack.org/#/q/project:openstack/keystone+branch:master+topic:migrate-identity-api-ref In progress
Identity v3 (and extensions) keystone Samuel de Medeiros Queiroz https://review.openstack.org/#/q/project:openstack/keystone+branch:master+topic:migrate-identity-api-ref In progress
Bare metal v1 ironic Jim Rollenhagen https://review.openstack.org/#/c/312181/ and https://review.openstack.org/312184 In progress
Block Storage v1 cinder Sheel Rana Insaan https://review.openstack.org/#/c/312113 In progress
Block Storage v2 cinder Sheel Rana Insaan https://review.openstack.org/#/c/312113 In progress
Clustering v1 senlin
Data Processing v1.1 sahara Note: has these REST API docs http://docs.openstack.org/developer/sahara/restapi/rest_api_v1.0.html
Database v1.0 trove Laurel Michaels https://review.openstack.org/#/c/316381 Needs review, can't build locally
Image v1 glance rosmaita https://review.openstack.org/#/c/312259/ and https://review.openstack.org/#/c/315191/ merged
Image v2 glance rosmaita https://review.openstack.org/#/c/332950/ merged
Tasks v2 glance rosmaita https://review.openstack.org/#/c/333029/ merged
Metadefs v2 glance rosmaita https://review.openstack.org/#/c/333031/ merged
Networking v2.0 (and extensions) neutron HenryG, amotoki https://review.openstack.org/#/c/314819/ In progress
Object Storage v1 swift Anne Gentle https://review.openstack.org/312315 In review
Orchestration v1 heat Jay Dobies (jdob) https://review.openstack.org/#/c/312712/ and https://review.openstack.org/312726 Complete :)
Shared File Systems v1 manila Daniel Gonzalez https://review.openstack.org/#/c/313874 and https://review.openstack.org/#/c/313875 Merged
Shared File Systems v2 manila Daniel Gonzalez https://review.openstack.org/#/c/313874 and https://review.openstack.org/#/c/313875 Merged
Telemetry v2 ceilometer Note: has these REST API docs http://docs.openstack.org/developer/ceilometer/webapi/v2.html

Operations Guide

Sign up below for a chapter, then create a patch with RST in doc/ops-guide/source for others to review. Contributions are welcome. Please add "Implements: blueprint ops-guide-rst" in the commit message.

Please make sure this book follows O'Reilly conventions.

Chapter/Section (old filename) Chapter/Section (new filename) Name Patch URL Status
preface_ops.xml preface_ops.rst venkatamahesh https://review.openstack.org/#/c/292071 Merged
part_architecture.xml part_architecture.rst venkatamahesh https://review.openstack.org/#/c/292064 Merged
part_operations.xml operations.rst KATO Tomoyuki https://review.openstack.org/#/c/291987/ Merged
ch_arch_cloud_controller.xml ch_arch_cloud_controller.rst venkatamahesh https://review.openstack.org/#/c/292061 Merged
ch_arch_compute_nodes.xml ch_arch_compute_nodes.rst venkatamahesh https://review.openstack.org/#/c/292059 Merged
ch_arch_examples.xml ch_arch_examples.rst venkatamahesh https://review.openstack.org/#/c/292052 Merged
section_arch_example-neutron.xml section_arch_example-neutron.rst venkatamahesh https://review.openstack.org/#/c/291965 Merged
section_arch_example-nova.xml section_arch_example-nova.rst venkatamahesh https://review.openstack.org/#/c/291783 Merged
ch_arch_network_design.xml ch_arch_network_design.rst venkatamahesh https://review.openstack.org/#/c/291974 Merged
ch_arch_provision.xml ch_arch_provision.rst venkatamahesh https://review.openstack.org/#/c/291976 Merged
ch_arch_scaling.xml ch_arch_scaling.rst venkatamahesh https://review.openstack.org/#/c/291978 Merged
ch_arch_storage.xml ch_arch_storage.rst venkatamahesh https://review.openstack.org/#/c/292007 Merged
ch_ops_advanced_configuration.xml ch_ops_advanced_configuration.rst venkatamahesh https://review.openstack.org/#/c/292008 Merged
ch_ops_backup_recovery.xml ch_ops_backup_recovery.rst venkatamahesh https://review.openstack.org/#/c/292011 Merged
ch_ops_customize.xml ch_ops_customize.rst venkatamahesh https://review.openstack.org/#/c/292013 Merged
ch_ops_lay_of_land.xml lay_of_the_land.rst KATO Tomoyuki https://review.openstack.org/#/c/291996/ Merged
ch_ops_log_monitor.xml ch_ops_log_monitor.rst venkatamahesh https://review.openstack.org/#/c/292015 Merged
ch_ops_maintenance.xml ch_ops_maintenance.rst venkatamahesh https://review.openstack.org/#/c/292022 Merged
ch_ops_network_troubleshooting.xml ch_ops_network_troubleshooting.rst venkatamahesh https://review.openstack.org/#/c/292027 Merged
ch_ops_projects_users.xml projects_users.rst KATO Tomoyuki https://review.openstack.org/#/c/292002/ Merged
ch_ops_resources.xml ch_ops_resources.rst venkatamahesh https://review.openstack.org/#/c/292033 Merged
ch_ops_upgrades.xml ch_ops_upgrades.rst venkatamahesh https://review.openstack.org/#/c/292035 Merged
ch_ops_upstream.xml ch_ops_upstream.rst venkatamahesh https://review.openstack.org/#/c/292036 Merged
ch_ops_user_facing.xml ch_ops_user_facing.rst venkatamahesh https://review.openstack.org/#/c/292050 Merged

Completing conversion to RST

Once we have reviewed the draft guide and think it's ready to publish, we need to do the following steps:

  1. Create a patch with the following steps (for example, see https://review.openstack.org/#/c/248577/ or https://review.openstack.org/211766)
    1. Delete the old guide, here security-guide
    2. Move the RST guide to the location of new guide (security-guide-rst -> security-guide).
    3. Update tools/build-all-rst.sh for the change
    4. Delete the entry from doc/pom.xml
    5. Update RELEASENOTES.rst
    6. If the repository has no further DocBook guides in it, update tox.ini
    7. Update doc-tools-check-languages.conf
    8. Rename localization files to new directory name
    9. If there is a ".tx/config" file, update it (remove RST guide and update paths)
  2. If the repository has no further DocBook guides in it:
  3. Update .gitignore. For example: https://review.openstack.org/212685
  4. If the repository is not the openstack-manuals repo, stop syncing of XML files: https://review.openstack.org/211842 and remove the copied files once that patch is merged: https://review.openstack.org/211904
  5. Fix links on docs.openstack.org and in other guides: https://review.openstack.org/212044
    • Create redirects from content directory to top-level index.html file
    • Update all links so that they go to new guide (security-guide, not security-guide/content)
    • Remove links to PDF of guide
  6. Sync translations from old guide with new guide (needs to be done in transifex by Andreas)
  7. Tell i18n team that conversion is finished and which resource is active.
  8. Remove old guide and draft RST guides from docs.openstack.org (needs docs.openstack.org admin access)
  9. Regenerate sitemap.xml after all changes are in: https://review.openstack.org/212689 and blacklist the change: https://review.openstack.org/212690