Jump to: navigation, search


< Documentation
Revision as of 00:35, 5 December 2015 by To222 (talk | contribs) (Configuration Reference Guide Migration)

Doc Migration from DocBook to RST Overview

Previously, we migrated the <service>-api repos from DocBook to RST. These were API reference documents that were meant for the contributor developer to know what was going into an API. So, the history of that document was forged as a specification. As projects beyond swift and nova were added, projects made similar documents. They're output to HTML at http://docs.openstack.org/api/api-specs.html. However, as we now have a <project>-specs repo, it makes sense to move to those repos. In that move, also it makes sense to migrate to RST rather than DocBook/WADL.

Migration Conventions

Follow the RST markup conventions in the OpenStack Documentation Contributor Guide .


First title in an RST file, use equals signs above and below. Second title, use tilde. Third title, use a series of dashes. Try not to need a Heading 4 if you can help it by rewriting or reorganizing.

Heading 1

Heading 2

Heading 3

File names

As a rule, we want to keep the xml:id identical so that the output HTML files do not need redirection. However, we also at the same time want to get rid of ch_ and section_ nomenclatures while going to a page-based, topical approach. So, first use the XML:ID, and if the XML:ID has ch_ or section_ in it, remove the prefix.

Cross references

Use :ref: when doing cross references so that you don't have to have the exact title. This means you will have to add markup to create anchors, such as:

.. _dashboard-project-tab:

Please use the existing XML:Ids if possible for these cross references.

Also, while intersphinx is an enabled extension for many of the contributor developer docs, we don't have an intersphinx requirement yet for the end user guide or admin user guide. We'll investigate that as we add more guides.

Figures and images

Store the figures in a /figures/ directory with the deliverable itself.

Notes and admonitions

If the conversion outputs Note as a heading, change it to use the .. note:: directive.

Line numbers in code blocks

If your file has lots of code blocks, be sure to set up line numbering for the entire file. Each code-block must have :linenos: after the .. code-block:: directive, the code itself must be indented at the same level as the :lineos: line, and you must have at the start of the file this directive:

.. highlight: python
   :linenothreshold: 5

so that any code-blocks longer than five lines long will have line numbers.

Migration Issues

List of bugs or missing features for the Sphinx template, openstackdocstheme: https://bugs.launchpad.net/openstack-manuals/+bugs?field.tag=openstackdocstheme

Can you have line breaks in something with inline semantic markup like :guilabel: at the end of the line? For example:

#. Select the volume to add to an instance and click :guilabel:`Edit Attachments`.

That line is longer than 79 characters, so Attachments`. must be on the second line, but when you do that, the HTML output is incorrect.

Since RST documents impose a 79 character limit when building files in .rst format. When tables exceed that limit, a list table role can help create content in a new table without exceeding the character limit. The issue is when a table is generated by an OpenStack service, and returned as a result of a command. These tables are constructed with +, -, and | characters. The table created by a list table role has solid lines forming rows, columns, and cells. The content is similar, however users might be surprised by a table with solid lines rather than a table built with +, -, and | characters.

Large scale content inside tables.

When building tables with a list table role, some items that appear inside the table cells might exceed the 79 character limit. Is there a solution to recreating this large scale content in a table such that it does not affect the character limit?

The :orphan: role and references.

Files with an :orphan: role at the top of the file, not included in the toc tree, will not link to another file when the :ref: role is used to link to the file from a different file. Is there a solution to this linking issue?

Pandoc conversion problems.

When converting files with Pandoc, some entries in the .xml file will not convert accurately. Procedure titles and sub section titles will not parse correctly, and will be deleted from the converted document. Pandoc also removes procedure numbering. Currently, the solution is to compare the complete .rst file to the built .xml version, and check the headings, subheadings, and procedure numbers to make sure they are correct, and line up with original. This is an issue that could also be discussed.

  • Contact details for RST issues that we can't solve:
    • Author: David Goodger
    • Contact: docutils-develop@lists.sourceforge.net

Migration How To

Note: If you don't want to use tox, install these prereqs locally: pip install sphinx; pip install openstackdocstheme and then switch to the directory containing a conf.py and run sphinx-build /path/to/source/ path/to/build/ to get html output by default.

Because Sphinx builds have dependent requirements it is best to work with in a virtual environment. Fortunately the openstack-manuals project already has tox set up so that you can create a virtualenv that contains the required dependencies. To use it, do this on a Mac or Ubuntu machine with Python and pip already installed:

  1. Clone the openstack-manuals repository.
  2. Change directories into openstack-manuals.
  3. Run this:
    tox -e py27
  4. When you get a congratulations message, run this:


source .tox/py27/bin/activate


 source .tox/py27/Scripts/activate

Your prompt should now have (py27) as a preface.

Now you have all the pre-requisites installed and can run additional tox commands. To see the list of commands, view or edit tox.ini in the openstack-manuals directory. To build the RST docs, run:

tox -e docs

This will run sphinx-build doc/playground-user-guide/source/ doc/playground-user-guide/build/html.

When the build is finished, you can open doc/playground-user-guide/build/html/index.html to view the resulting output.

Migration Using Oxygen

  1. Open DocBook book file in Oxygen.
  2. Choose Document > Transformation > Configure Transformation Scenario(s).
  3. Select DocBook XHTML - Chunk.
  4. Click Apply associated (1).
  5. Within the /out/xhtml-chunks/ directory that's generated, run the following script:
    for i in *.xhtml
    # Convert from XHTML to RST
    pandoc -s -t rst $i -o $file_name
    sed -i -e '4,16d' $file_name
    sed -i -e '/+--------------------------+$/,$d' $file_name
    sed -i -e '$d' $file_name
    sed -i -e '$d' $file_name
    # Rename file to second line of new RST content, but lowercase and all non-alphanumeric chars renamed to underscores:
    real_file_name=$(sed 's/[^a-zA-Z0-9\-]/_/g;2q;d' $file_name | awk '{print tolower($0)}').rst
    mv $file_name $real_file_name<pre><nowiki>
    # Replace all cross-refs to xhtml files to renamed rst files:
    sed -i '' -e "s/\<$i/fixmefixmefixme/g" *.rst
    # sed -i '' -e "s/\ fixmefixmefixme.*\`__//g" *.rst
  6. Clean up where the fixmefixmefixme is output, it indicates where a cross-reference cannot exist any longer.
  7. Clean up tables where the pandoc conversion just outputs paragraphs.
  8. Remove numbering from Example titles and Table titles.
  9. Rename chapter_ files and ensure they are titled to match the contents of the file, such as "Networking API 2.0 Overview" to networking-api-2.0-overview" for example.
  10. Remove "programlisting" "screen" and "literallayout" from .. code:: lines.
  11. When commiting conversion patches, include the related blueprint in your commit message. For example:
    Convert ch_compute_focus.xml to RST
    Implements: blueprint archguide-mitaka-rst

Migration Using a Text Editor

  1. Install pandoc.
  2. Update your openstack-manuals master branch, and create a branch.
  3. Run the pandoc command to convert the .xml file to .rst. For example:
    pandoc -f docbook -t rst -s ch_compute_focus.xml -o compute-focus.rst
  4. Move converted files to the <guide>/source folder.
  5. Rename converted RST files and ensure they are titled to match the contents of the file, such as "Networking API 2.0 Overview" to networking-api-2.0-overview" for example.
  6. Pandoc has its quirks, so check all the content has been migrated from the source file.
  7. Use a text editor to complete the following tasks:
    1. Clean up tables where the pandoc conversion just outputs paragraphs.
    2. Remove numbering from Example titles and Table titles.
    3. Remove "programlisting" "screen" and "literallayout" from .. code:: lines.
    4. Check the converted RST file follow conventions described in the OpenStack Documentation Contributor Guide
  8. Build the guide locally, to check for errors:
    tox -e docs
  9. Commit your changes, including the related blueprint in your commit message. For example:
    Convert ch_compute_focus.xml to RST
    Implements: blueprint archguide-mitaka-rst

Doc Migration Plan

For the Mitaka release, we are migrating the Configuration Reference Guide and the Architecture Design Guide. Refer to the detailed specification for details.

The Juno projects where this document needs to migrate to a specification are: nova (compute-api) v2, v3 swift (object-api) v1 glance (image-api) v1, v1.1, v2 keystone (identity-api) v2.0, v3 neutron (netconn-api) v1.0, v2.0 cinder (volume-api) v1.0, v2.0

Juno projects that have this type of document in a separate <service>-api repo are: trove (trove)

Juno projects that do not provide this type of prose-based spec for their API are: ceilometer heat sahara

Incubating projects, this information is just a heads-up so you know how we're thinking about API documentation going forward. ironic zaqar (marconi) barbican designate

I'm going to do the migration work with pandoc and propose the patch to the project's repo. Be on the lookout for those patches.

Configuration Reference Guide Migration

Sign up below for a chapter, then create a patch with RST in doc/config-ref-rst/source for others to review. While the Config Ref Specialty team will be giving the conversion attention, contributions are welcome.

Chapter/Section (old filename) Chapter/Section (new filename) Name Patch URL Status
app_firewalls-ports.xml firewalls-default-ports.rst Gauvain Pocentek https://review.openstack.org/#/c/245419/ Merged
app_policy_json.xml policy-json-file.rst France Berteloot https://review.openstack.org/#/c/245221/ Merged
bk-config-ref.xml index.rst Gauvain Pocentek https://review.openstack.org/#/c/242809/ Merged
block-storage/backup/ceph-backup-driver.xml block-storage/backup/ceph-backup-driver.rst Gauvain Pocentek https://review.openstack.org/249170 Merged
block-storage/backup/nfs-backup-driver.xml block-storage/backup/nfs-backup-driver.rst Gauvain Pocentek https://review.openstack.org/249170 Merged
block-storage/backup/swift-backup-driver.xml block-storage/backup/swift-backup-driver.rst Gauvain Pocentek https://review.openstack.org/249170 Merged
block-storage/backup/tsm-backup-driver.xml block-storage/backup/tsm-backup-driver.rst Gauvain Pocentek https://review.openstack.org/249170 Merged
block-storage/drivers/blockbridge-eps-driver.xml block-storage/drivers/blockbridge-eps-driver.rst France Berteloot https://review.openstack.org/#/c/251462/ In progress
block-storage/drivers/ceph-rbd-volume-driver.xml block-storage/drivers/ceph-rbd-volume-driver.rst France Berteloot https://review.openstack.org/#/c/251475/ In progress
block-storage/drivers/dell-equallogic-driver.xml block-storage/drivers/dell-equallogic-driver.rst France Berteloot In progress
block-storage/drivers/dell-storagecenter-driver.xml block-storage/drivers/dell-storagecenter-driver.rst France Berteloot In progress
block-storage/drivers/dothill-driver.xml block-storage/drivers/dothill-driver.xml France Berteloot In progress
block-storage/drivers/emc-scaleio-driver.xml block-storage/drivers/emc-scaleio-driver.rst France Berteloot In progress
block-storage/drivers/emc-vmax-driver.xml block-storage/drivers/emc-vmax-driver.xml Victor Howard In progress
block-storage/section_backup-drivers.xml block-storage/backup-drivers.rst Gauvain Pocentek https://review.openstack.org/249170 Merged
block-storage/section_block-storage-overview.xml block-storage-overview.rst France Berteloot https://review.openstack.org/#/c/246824/ Merged
block-storage/section_block-storage-sample-configuration-files.xml block-storage-sample-configuration-files.rst France Berteloot In progress
block-storage/section_cinder-log-files.xml cinder-log-files.rst France Berteloot https://review.openstack.org/#/c/249691/ Merged
block-storage/section_fc-zoning.xml fc-zoning.rst France Berteloot https://review.openstack.org/#/c/249899/ Merged
block-storage/section_misc.xml volume-misc.rst France Berteloot https://review.openstack.org/251375 Merged
block-storage/section_volume-drivers.xml volume-drivers.rst France Berteloot https://review.openstack.org/247684 Merged
block-storage/section_volume-encryption.xml block-storage/volume-encryption.rst Andreas Jaeger https://review.openstack.org/251101l
ch_baremetalconfigure.xml baremetal.rst Gauvain Pocentek https://review.openstack.org/246770 Merged
ch_blockstorageconfigure.xml block-storage.rst France Berteloot https://review.openstack.org/#/c/245888/ Merged
ch_computeconfigure.xml compute.rst KATO Tomoyuki https://review.openstack.org/#/c/245523/ Merged
ch_config-overview.xml config-overview.rst KATO Tomoyuki https://review.openstack.org/#/c/243612/ Merged
ch_dashboardconfigure.xml dashboard-configure.rst Alexandra Settle
ch_databaseserviceconfigure.xml databaseservice-configure.rst Alexandra Settle
ch_dataprocessingserviceconfigure.xml dataprocess-configure.rst Alexandra Settle
ch_identityconfigure.xml identity-configure.rst Alexandra Settle
ch_imageservice.xml image-service.rst Alexandra Settle
ch_networkingconfigure.xml networking.rst Andreas Jaeger https://review.openstack.org/251056 Merged
ch_objectstorageconfigure.xml object-storage.rst
ch_orchestrationconfigure.xml orchestration.rst
ch_sharedfilesystemsconfigure.xml shared-file-systems.rst Gauvain Pocentek https://review.openstack.org/#/c/249613/ Merged
ch_telemetryconfigure.xml telemetry.rst Andreas Jaeger https://review.openstack.org/251056 Merged
../common/section_compute_config-api.xml compute/api.rst KATO Tomoyuki https://review.openstack.org/#/c/251774/ Merged
../common/section_compute-configure-ec2.xml compute/ec2-api.rst KATO Tomoyuki https://review.openstack.org/#/c/251774/ Merged
../common/section_fibrechannel.xml compute/fibre-channel.rst KATO Tomoyuki https://review.openstack.org/#/c/252404/ Merged
../common/section_config_format.xml config-format.rst KATO Tomoyuki https://review.openstack.org/#/c/243612/ Merged
compute/samples/cells-resp.json compute/cells.rst KATO Tomoyuki https://review.openstack.org/#/c/250828/ Merged
compute/section_compute-cells.xml compute/cells.rst KATO Tomoyuki https://review.openstack.org/#/c/250828/ Merged
compute/section_compute-conductor.xml compute/conductor.rst KATO Tomoyuki https://review.openstack.org/#/c/250732/ Merged
compute/section_compute-config-samples.xml compute/config-samples.rst KATO Tomoyuki https://review.openstack.org/#/c/250605/ Merged
compute/section_compute-configure-db.xml compute/database-connections.rst KATO Tomoyuki https://review.openstack.org/#/c/251167/ Merged
compute/section_compute-configure-xapi.xml KATO Tomoyuki Merged
compute/section_compute-hypervisors.xml compute/hypervisors.rst KATO Tomoyuki https://review.openstack.org/#/c/252200/ Merged
compute/section_compute-iscsioffload.xml compute/iscsi-offload.rst KATO Tomoyuki https://review.openstack.org/#/c/252404/ Merged
compute/section_compute-options-reference.xml compute/config-options.rst KATO Tomoyuki https://review.openstack.org/#/c/248453/ Merged
compute/section_compute-sample-configuration-files.xml compute/config-samples.rst KATO Tomoyuki https://review.openstack.org/#/c/248453/ Merged
compute/section_compute-scheduler.xml compute/scheduler.rst KATO Tomoyuki https://review.openstack.org/#/c/251000/ Merged
compute/section_hypervisor_hyper-v.xml compute/hyper-v.rst KATO Tomoyuki https://review.openstack.org/#/c/252408/ Merged
compute/section_hypervisor_kvm.xml compute/hypervisor-kvm.rst KATO Tomoyuki https://review.openstack.org/#/c/252427/ Merged
compute/section_hypervisor_lxc.xml compute/hypervisor-lxc.rst KATO Tomoyuki https://review.openstack.org/#/c/252997/ Merged
compute/section_hypervisor_qemu.xml compute/hypervisor-qemu.rst KATO Tomoyuki https://review.openstack.org/#/c/252714/ Merged
compute/section_hypervisor_vmware.xml compute/hypervisor-vmware.rst KATO Tomoyuki https://review.openstack.org/#/c/253341/ In Progress
compute/section_hypervisor_xen_api.xml compute/hypervisor-xen-api.rst KATO Tomoyuki https://review.openstack.org/#/c/253384/ Merged
compute/section_hypervisor_xen_libvirt.xml compute/hypervisor-xen-libvirt.rst KATO Tomoyuki https://review.openstack.org/#/c/253440/ In Progress
compute/section_nova-conf.xml compute/nova-conf.rst KATO Tomoyuki https://review.openstack.org/#/c/245523/ Merged
compute/section_nova-log-files.xml compute/nova-logs.rst KATO Tomoyuki https://review.openstack.org/#/c/250578/ Merged
compute/section_rpc.xml compute/rpc.rst KATO Tomoyuki https://review.openstack.org/#/c/251167/ Merged
compute/section_xapi-ami-setup.xml compute/hypervisor-xen-api.rst KATO Tomoyuki https://review.openstack.org/#/c/253384/ Merged
compute/section_xapi-install-plugins.xml compute/hypervisor-xen-api.rst KATO Tomoyuki https://review.openstack.org/#/c/253384/ Merged
compute/section_xapi-install.xml compute/hypervisor-xen-api.rst KATO Tomoyuki https://review.openstack.org/#/c/253384/ Merged
compute/section_xapi-resize-setup.xml compute/hypervisor-xen-api.rst KATO Tomoyuki https://review.openstack.org/#/c/253384/ Merged
conf-changes/ceilometer.xml Automated
conf-changes/cinder.xml Automated
conf-changes/glance.xml Automated
conf-changes/heat.xml Automated
conf-changes/ironic.xml Automated
conf-changes/keystone.xml Automated
conf-changes/manila.xml Automated
conf-changes/neutron.xml Automated, not done yet.
conf-changes/nova.xml Automated
conf-changes/sahara.xml Automated
conf-changes/swift.xml Automated
conf-changes/trove.xml Automated
database-service/section-databaseservice-db.xml database-service/databaseservice_db.rst Andreas Jaeger https://review.openstack.org/251033 Merged
database-service/section-databaseservice-rpc.xml database-service/databaseservice_api.rst Andreas Jaeger https://review.openstack.org/251033 Merged
image-service/section_image-service-api.xml image-service/image_service_api.rst Andreas Jaeger https://review.openstack.org/251037 Merged
image-service/section_image-service-backends.xml image-service/image_service_backends.rst Andreas Jaeger https://review.openstack.org/251037 Merged
image-service/section_image-service-backend-vmware.xml image-service/image_service_backends.rst Andreas Jaeger https://review.openstack.org/251037 Merged
image-service/section_image-service-ISO-support.xml image-service/image_service_ISO_support.rst Andreas Jaeger https://review.openstack.org/251037 Merged
image-service/section_image-service-rpc.xml image-service/image_service_rpc.rst Andreas Jaeger https://review.openstack.org/251037 Merged
networking/section_networking-log-files.xml networking/networking_log.rst Andreas Jaeger https://review.openstack.org/251103
networking/section_networking-options-reference.xml networking/networking_options_reference.rst Andreas Jaeger https://review.openstack.org/251103
networking/section_networking-plugins-ml2.xml Andreas Jaeger networking/networking_options_reference.rst
networking/section_networking-plugins.xml Andreas Jaeger networking/networking_options_reference.rst
networking/section_rpc-for-networking.xml Andreas Jaeger networking/networking_options_reference.rst
orchestration/section_orchestration-api.xml orchestration/orchestration_api.rst Andreas Jaeger https://review.openstack.org/250655 Merged
orchestration/section_orchestration-clients.xml orchestration/orchestration_clients.rst Andreas Jaeger https://review.openstack.org/250655 Merged
orchestration/section_orchestration-rpc.xml orchestration/orchestration_rpc.rst Andreas Jaeger https://review.openstack.org/250655 Merged
roadmap.rst ../roadmap.rst Andreas Jaeger https://review.openstack.org/251099 Deleted
shared-file-systems/drivers/emc-isilon-driver.xml shared-file-systems/drivers/emc-isilon-driver.rst Gauvain Pocentek https://review.openstack.org/251108 Merged
shared-file-systems/drivers/emc-vnx-driver.xml shared-file-systems/drivers/emc-vnx-driver.rst Gauvain Pocentek https://review.openstack.org/251108 Merged
shared-file-systems/drivers/generic-driver.xml shared-file-systems/drivers/generic-driver.rst Gauvain Pocentek https://review.openstack.org/251108 Merged
shared-file-systems/drivers/glusterfs-driver.xml shared-file-systems/drivers/glusterfs-driver.rst Gauvain Pocentek https://review.openstack.org/251108 Merged
shared-file-systems/drivers/glusterfs-native-driver.xml shared-file-systems/drivers/glusterfs-native-driver.rst Gauvain Pocentek https://review.openstack.org/251108 Merged
shared-file-systems/drivers/hdfs-native-driver.xml shared-file-systems/drivers/hdfs-native-driver.rst Gauvain Pocentek https://review.openstack.org/251108 Merged
shared-file-systems/drivers/hp-3par-share-driver.xml shared-file-systems/drivers/hp-3par-share-driver.rst Gauvain Pocentek https://review.openstack.org/251108 Merged
shared-file-systems/drivers/huawei-nas-driver.xml shared-file-systems/drivers/huawei-nas-driver.rst Gauvain Pocentek https://review.openstack.org/251108 Merged
shared-file-systems/drivers/ibm-gpfs-driver.xml shared-file-systems/drivers/ibm-gpfs-driver.rst Gauvain Pocentek https://review.openstack.org/251108 Merged
shared-file-systems/drivers/netapp-cluster-mode-driver.xml shared-file-systems/drivers/netapp-cluster-mode-driver.rst Gauvain Pocentek https://review.openstack.org/251108 Merged
shared-file-systems/section_manila-log-files.xml shared-file-systems/log-files.rst Gauvain Pocentek https://review.openstack.org/#/c/249613/ Merged
shared-file-systems/section_manila-misc.xml shared-file-systems/misc.rst Gauvain Pocentek https://review.openstack.org/#/c/249613/ Merged
shared-file-systems/section_manila-sample-configuration-files.xml shared-file-systems/sample-configuration-files.rst Gauvain Pocentek https://review.openstack.org/#/c/249613/ Merged
shared-file-systems/section_shared-file-systems-overview.xml shared-file-systems/overview.rst Gauvain Pocentek https://review.openstack.org/#/c/249613/ Merged
shared-file-systems/section_share-drivers.xml shared-file-systems/drivers.rst Gauvain Pocentek https://review.openstack.org/#/c/249613/ Merged
table_default-ports-peripheral-services.xml firewall-default-ports.rst Merged
table_default-ports-primary-services.xml firewall-default-ports.rst Merged
telemetry/section_telemetry-alarming-service-config-opts.xml telemetry/alarming_service_config_opts.rst Andreas Jaeger https://review.openstack.org/251034 Merged
telemetry/section_telemetry-sample-configuration-files.xml telemetry/sampl-configuration-files.rst Gauvain Pocentek https://review.openstack.org/251509
telemetry/section_telemetry-service-config-opts.xml telemetry/telemetry_service_config_opts.rst Andreas Jaeger https://review.openstack.org/251034 Merged

Completing conversion to RST

Once we have reviewed the draft guide and think it's ready to publish, we need to do the following steps:

  1. Create a patch with the following steps (for example, see https://review.openstack.org/#/c/248577/ or https://review.openstack.org/211766)
    1. Delete the old guide, here security-guide
    2. Move the RST guide to the location of new guide (security-guide-rst -> security-guide).
    3. Update tools/build-all-rst.sh for the change
    4. Delete the entry from doc/pom.xml
    5. Update RELEASENOTES.rst
    6. If the repository has no further DocBook guides in it, update tox.ini
    7. Update doc-tools-test-languages.conf
    8. Rename localization files to new directory name
    9. If there is a ".tx/config" file, update it (remove RST guide and update paths)
  2. If the repository has no further DocBook guides in it:
  3. Update .gitignore. For example: https://review.openstack.org/212685
  4. If the repository is not the openstack-manuals repo, stop syncing of XML files: https://review.openstack.org/211842 and remove the copied files once that patch is merged: https://review.openstack.org/211904
  5. Fix links on docs.openstack.org and in other guides: https://review.openstack.org/212044
    • Create redirects from content directory to top-level index.html file
    • Update all links so that they go to new guide (security-guide, not security-guide/content)
    • Remove links to PDF of guide
  6. Sync translations from old guide with new guide (needs to be done in transifex by Andreas)
  7. Tell i18n team that conversion is finished and which resource is active.
  8. Remove old guide and draft RST guides from docs.openstack.org (needs docs.openstack.org admin access)
  9. Regenerate sitemap.xml after all changes are in: https://review.openstack.org/212689 and blacklist the change: https://review.openstack.org/212690