Documentation/Migrate

Doc Migration from DocBook to RST Overview
Previously, we migrated the -api repos from DocBook to RST. These were API reference documents that were meant for the contributor developer to know what was going into an API. So, the history of that document was forged as a specification. As projects beyond swift and nova were added, projects made similar documents. They're output to HTML at http://docs.openstack.org/api/api-specs.html. However, as we now have a -specs repo, it makes sense to move to those repos. In that move, also it makes sense to migrate to RST rather than DocBook/WADL.

Migration Conventions
Follow the RST markup conventions in the OpenStack Documentation Contributor Guide.

Headings
First title in an RST file, use equals signs above and below. Second title, use tilde. Third title, use a series of dashes. Try not to need a Heading 4 if you can help it by rewriting or reorganizing.

=
Heading 1

=
Heading 2

Heading 3 -

File names
As a rule, we want to keep the xml:id identical so that the output HTML files do not need redirection. However, we also at the same time want to get rid of ch_ and section_ nomenclatures while going to a page-based, topical approach. So, first use the XML:ID, and if the XML:ID has ch_ or section_ in it, remove the prefix.

Cross references
Use :ref: when doing cross references so that you don't have to have the exact title. This means you will have to add markup to create anchors, such as:

.. _dashboard-project-tab:

Please use the existing XML:Ids if possible for these cross references.

Also, while intersphinx is an enabled extension for many of the contributor developer docs, we don't have an intersphinx requirement yet for the end user guide or admin user guide. We'll investigate that as we add more guides.

Figures and images
Store the figures in a /figures/ directory with the deliverable itself.

Notes and admonitions
If the conversion outputs Note as a heading, change it to use the .. note:: directive.

Line numbers in code blocks
If your file has lots of code blocks, be sure to set up line numbering for the entire file. Each code-block must have :linenos: after the .. code-block:: directive, the code itself must be indented at the same level as the :lineos: line, and you must have at the start of the file this directive:

.. highlight: python :linenothreshold: 5

so that any code-blocks longer than five lines long will have line numbers.

Migration Issues
List of bugs or missing features for the Sphinx template, openstackdocstheme: https://bugs.launchpad.net/openstack-manuals/+bugs?field.tag=openstackdocstheme

Can you have line breaks in something with inline semantic markup like :guilabel: at the end of the line? For example:


 * 1) . Select the volume to add to an instance and click :guilabel:`Edit Attachments`.

That line is longer than 79 characters, so Attachments`. must be on the second line, but when you do that, the HTML output is incorrect.

Since RST documents impose a 79 character limit when building files in .rst format. When tables exceed that limit, a list table role can help create content in a new table without exceeding the character limit. The issue is when a table is generated by an OpenStack service, and returned as a result of a command. These tables are constructed with +, -, and | characters. The table created by a list table role has solid lines forming rows, columns, and cells. The content is similar, however users might be surprised by a table with solid lines rather than a table built with +, -, and | characters.

Large scale content inside tables.

When building tables with a list table role, some items that appear inside the table cells might exceed the 79 character limit. Is there a solution to recreating this large scale content in a table such that it does not affect the character limit?

The :orphan: role and references.

Files with an :orphan: role at the top of the file, not included in the toc tree, will not link to another file when the :ref: role is used to link to the file from a different file. Is there a solution to this linking issue?

Pandoc conversion problems.

When converting files with Pandoc, some entries in the .xml file will not convert accurately. Procedure titles and sub section titles will not parse correctly, and will be deleted from the converted document. Pandoc also removes procedure numbering. Currently, the solution is to compare the complete .rst file to the built .xml version, and check the headings, subheadings, and procedure numbers to make sure they are correct, and line up with original. This is an issue that could also be discussed.


 * Contact details for RST issues that we can't solve:
 * Author:	David Goodger
 * Contact: docutils-develop@lists.sourceforge.net

Migration How To
Note: If you don't want to use tox, install these prereqs locally: pip install sphinx; pip install openstackdocstheme and then switch to the directory containing a conf.py and run sphinx-build /path/to/source/ path/to/build/ to get html output by default.

Because Sphinx builds have dependent requirements it is best to work with in a virtual environment. Fortunately the openstack-manuals project already has tox set up so that you can create a virtualenv that contains the required dependencies. To use it, do this on a Mac or Ubuntu machine with Python and pip already installed:

Mac/Ubuntu: source .tox/py27/bin/activate
 * 1) Clone the openstack-manuals repository.
 * 2) Change directories into openstack-manuals.
 * 3) Run this:
 * tox -e py27
 * 1) When you get a congratulations message, run this:

Windows: source .tox/py27/Scripts/activate

Your prompt should now have (py27) as a preface.

Now you have all the pre-requisites installed and can run additional tox commands. To see the list of commands, view or edit tox.ini in the openstack-manuals directory. To build the RST docs, run:

tox -e docs

This will run sphinx-build doc/playground-user-guide/source/ doc/playground-user-guide/build/html.

When the build is finished, you can open doc/playground-user-guide/build/html/index.html to view the resulting output.

Migration Using Oxygen

 * 1) Open DocBook book file in Oxygen.
 * 2) Choose Document > Transformation > Configure Transformation Scenario(s).
 * 3) Select DocBook XHTML - Chunk.
 * 4) Click Apply associated (1).
 * 5) Within the /out/xhtml-chunks/ directory that's generated, run the following script:
 * for i in *.xhtml
 * do
 * # Convert from XHTML to RST
 * file_name=${i%.*l}.rst
 * pandoc -s -t rst $i -o $file_name
 * sed -i -e '4,16d' $file_name
 * sed -i -e '/+--+$/,$d' $file_name
 * sed -i -e '$d' $file_name
 * sed -i -e '$d' $file_name
 * # Rename file to second line of new RST content, but lowercase and all non-alphanumeric chars renamed to underscores:
 * real_file_name=$(sed 's/[^a-zA-Z0-9\-]/_/g;2q;d' $file_name | awk '{print tolower($0)}').rst
 * mv $file_name $real_file_name
 * # Replace all cross-refs to xhtml files to renamed rst files:
 * sed -i '' -e "s/\<$i/fixmefixmefixme/g" *.rst
 * # sed -i '' -e "s/\ fixmefixmefixme.*\`__//g" *.rst
 * done
 * 1) Clean up where the fixmefixmefixme is output, it indicates where a cross-reference cannot exist any longer.
 * 2) Clean up tables where the pandoc conversion just outputs paragraphs.
 * 3) Remove numbering from Example titles and Table titles.
 * 4) Rename chapter_ files and ensure they are titled to match the contents of the file, such as "Networking API 2.0 Overview" to networking-api-2.0-overview" for example.
 * 5) Remove "programlisting" "screen" and "literallayout" from .. code:: lines.
 * 6) When commiting conversion patches, include the related blueprint in your commit message. For example:
 * Convert ch_compute_focus.xml to RST
 * Implements: blueprint archguide-mitaka-rst
 * Convert ch_compute_focus.xml to RST
 * Implements: blueprint archguide-mitaka-rst

Migration Using a Text Editor

 * 1) Install pandoc.
 * 2) Update your openstack-manuals master branch, and create a branch.
 * 3) Run the pandoc command to convert the .xml file to .rst. For example:
 * pandoc -f docbook -t rst -s ch_compute_focus.xml -o compute-focus.rst
 * 1) Move converted files to the /source folder.
 * 2) Rename converted RST files and ensure they are titled to match the contents of the file, such as "Networking API 2.0 Overview" to networking-api-2.0-overview" for example.
 * 3) Pandoc has its quirks, so check all the content has been migrated from the source file.
 * 4) Use a text editor to complete the following tasks:
 * 5) Clean up tables where the pandoc conversion just outputs paragraphs.
 * 6) Remove numbering from Example titles and Table titles.
 * 7) Remove "programlisting" "screen" and "literallayout" from .. code:: lines.
 * 8) Check the converted RST file follow conventions described in the OpenStack Documentation Contributor Guide
 * 9) Build the guide locally, to check for errors:
 * tox -e docs
 * 1) Commit your changes, including the related blueprint in your commit message. For example:
 * Convert ch_compute_focus.xml to RST
 * Implements: blueprint archguide-mitaka-rst
 * Implements: blueprint archguide-mitaka-rst

Doc Migration Plan
For the Mitaka release, we migrated the Configuration Reference Guide and the Architecture Design Guide. Refer to the detailed specification for details.

API Reference Plan
See http://docs.openstack.org/contributor-guide/api-guides.html for explanation of the process. Anne has already preprocessed the files, so the other migration steps on this wiki page are not needed at all.

Sign up below for a service, then download the files from this patch: https://review.openstack.org/#/c/311596/. These will require a lot of cleanup manually but gets them as far as the tools can get them. Also, setup a job to publish data like https://review.openstack.org/312184.

Operations Guide
Sign up below for a chapter, then create a patch with RST in doc/ops-guide/source for others to review. Contributions are welcome. Please add "Implements: blueprint ops-guide-rst" in the commit message.

Please make sure this book follows O'Reilly conventions.

Completing conversion to RST
Once we have reviewed the draft guide and think it's ready to publish, we need to do the following steps:


 * 1) Create a patch with the following steps (for example, see https://review.openstack.org/#/c/248577/ or https://review.openstack.org/211766)
 * 2) Delete the old guide, here security-guide
 * 3) Move the RST guide to the location of new guide (security-guide-rst -> security-guide).
 * 4) Update tools/build-all-rst.sh for the change
 * 5) Delete the entry from doc/pom.xml
 * 6) Update RELEASENOTES.rst
 * 7) If the repository has no further DocBook guides in it, update tox.ini
 * 8) Update doc-tools-check-languages.conf
 * 9) Rename localization files to new directory name
 * 10) If there is a ".tx/config" file, update it (remove RST guide and update paths)
 * 11) If the repository has no further DocBook guides in it:
 * 12) * Remove jobs from Jenkins. For example: https://review.openstack.org/211843
 * 13) * Cleanup tox.ini. For example: https://review.openstack.org/212127
 * 14) * Remove tools/generatepot. For example: https://review.openstack.org/212129
 * 15) Update .gitignore. For example: https://review.openstack.org/212685
 * 16) If the repository is not the openstack-manuals repo, stop syncing of XML files: https://review.openstack.org/211842 and remove the copied files once that patch is merged: https://review.openstack.org/211904
 * 17) Fix links on docs.openstack.org and in other guides: https://review.openstack.org/212044
 * 18) * Create redirects from content directory to top-level index.html file
 * 19) * Update all links so that they go to new guide (security-guide, not security-guide/content)
 * 20) * Remove links to PDF of guide
 * 21) Sync translations from old guide with new guide (needs to be done in transifex by Andreas)
 * 22) Tell i18n team that conversion is finished and which resource is active.
 * 23) Remove old guide and draft RST guides from docs.openstack.org (needs docs.openstack.org admin access)
 * 24) Regenerate sitemap.xml after all changes are in: https://review.openstack.org/212689 and blacklist the change: https://review.openstack.org/212690