Jump to: navigation, search

Difference between revisions of "Documentation/Migrate"

(Publishing User Guides)
(Publishing User Guides)
Line 453: Line 453:
#** Update entries to index.html pages pointing to new content (annegentle, merged): https://review.openstack.org/#/c/174380/
#** Update entries to index.html pages pointing to new content (annegentle, merged): https://review.openstack.org/#/c/174380/
#* Removal of old content, 12:00 UTC or later, annegentle will approve and do fixes as needed:  
#* Removal of old content, 12:00 UTC or later, annegentle will approve and do fixes as needed:  
#** Update sitemap.xml (annegentle)
#** Update sitemap.xml (annegentle) https://review.openstack.org/#/c/174837/
#** Add redirects for moved pages removing extra /content/ directory (annegentle)
#** Add redirects for moved pages removing extra /content/ directory (annegentle) https://review.openstack.org/#/c/174835/
#** Remove old pages from /content/ directory for both user-guide and user-guide-admin on FTP server (annegentle)
#** Remove old pages from /content/ directory for both user-guide and user-guide-admin on FTP server (annegentle) Will wait for redirects to be in place.
# Search for old links. Gothrough our repos for  (just search for "user-guide/content" and "user-guide-admin/content") and replace links with new location.
# Search for old links. Go through our repos for  (just search for "user-guide/content" and "user-guide-admin/content") and replace links with new location.
Please review and enhance so that we don't miss anything!
Please review and enhance so that we don't miss anything!

Revision as of 14:37, 17 April 2015

Doc Migration from DocBook to RST Overview

We are currently migrating the End User Guide and Admin User Guide to RST. Feel free to sign up for a chapter in the section below. We plan to enable more migrations with the End User Guide and Admin User Guide as our first phase.

Previously, we migrated the <service>-api repos from DocBook to RST. These were API reference documents that were meant for the contributor developer to know what was going into an API. So, the history of that document was forged as a specification. As projects beyond swift and nova were added, projects made similar documents. They're output to HTML at http://docs.openstack.org/api/api-specs.html. However, as we now have a <project>-specs repo, it makes sense to move to those repos. In that move, also it makes sense to migrate to RST rather than DocBook/WADL.

Migration Conventions

Follow the existing existing markup conventions for RST.

When in doubt, use simpler markup. This guidance means revise the text to not use tables, and avoid cross-references that are difficult to maintain.


First title in an RST file, use equals signs above and below. Second title, use tilde. Third title, use a series of dashes. Try not to need a Heading 4 if you can help it by rewriting or reorganizing.

Heading 1

Heading 2

Heading 3

File names

As a rule, we want to keep the xml:id identical so that the output HTML files do not need redirection. However, we also at the same time want to get rid of ch_ and section_ nomenclatures while going to a page-based, topical approach. So, first use the XML:ID, and if the XML:ID has ch_ or section_ in it, remove the prefix.

Cross references

Use :ref: when doing cross references so that you don't have to have the exact title. This means you will have to add markup to create anchors, such as:

.. _dashboard-project-tab:

Please use the existing XML:Ids if possible for these cross references.

Also, while intersphinx is an enabled extension for many of the contributor developer docs, we don't have an intersphinx requirement yet for the end user guide or admin user guide. We'll investigate that as we add more guides.

Figures and images

Store the figures in a /figures/ directory with the deliverable itself.

Notes and admonitions

If the conversion outputs Note as a heading, change it to use the .. note:: directive.

There is a styling bug at https://bugs.launchpad.net/openstack-manuals/+bug/1416572 where the admonitions are not output like the prototype.

Line numbers in code blocks

If your file has lots of code blocks, be sure to set up line numbering for the entire file. Each code-block must have :linenos: after the .. code-block:: directive, the code itself must be indented at the same level as the :lineos: line, and you must have at the start of the file this directive:

.. highlight: python
   :linenothreshold: 5

so that any code-blocks longer than five lines long will have line numbers.

Migration Issues

List of bugs or missing features for the Sphinx template, openstackdocstheme: https://bugs.launchpad.net/openstack-manuals/+bugs?field.tag=openstackdocstheme

How to get numbered list continuation to work when you have a table in a step?

How to get numbered list continuation to work when you have a bulleted list between list items?

How do we address converting files with conditional or audience profiling ?

How to get get embedded .. note: directives to work between numbered list items?

Can you have line breaks in something with inline semantic markup like :guilabel: at the end of the line? For example:

#. Select the volume to add to an instance and click :guilabel:`Edit Attachments`.

That line is longer than 79 characters, so Attachments`. must be on the second line, but when you do that, the HTML output is incorrect.

Migration How To

Note: If you don't want to use tox, install these prereqs locally: pip install sphinx; pip install openstackdocstheme and then switch to the directory containing a conf.py and run sphinx-build /path/to/source/ path/to/build/ to get html output by default.

Because Sphinx builds have dependent requirements it is best to work with in a virtual environment. Fortunately the openstack-manuals project already has tox set up so that you can create a virtualenv that contains the required dependencies. To use it, do this on a Mac or Ubuntu machine with Python and pip already installed:

Clone the openstack-manuals repo.

Change directories into openstack-manuals.

Run this:

tox -e py27

When you get a congratulations message, run this:


source .tox/py27/bin/activate


source .tox/py27/Scripts/activate

Your prompt should now have (py27) as a preface.

Now you have all the pre-requisites installed and can run additional tox commands. To see the list of commands, view or edit tox.ini in the openstack-manuals directory. To build the RST docs, run:

tox -e docs

This will run sphinx-build doc/playground-user-guide/source/ doc/playground-user-guide/build/html.

When the build is finished, you can open doc/playground-user-guide/build/html/index.html to view the resulting output.

End User Guide Migration

First, get a local copy of the End User Guide from openstack-manuals. Create a branch to work within.

These files should use this markup in the first two lines to ensure they are only output in the user guide version:

.. meta::
    :scope: user_only

Next, either use the process documented on this wiki page or your own methods with pandoc (http://johnmacfarlane.net/pandoc/try/) to convert from DocBook to RST. Pick any chapter or section and look for errors in the automation, then manually clean up. Once you have cleaned up the RST, build it within playground-user-guide by running tox -echeckbuild or if you already have Sphinx installed, run sphinx-build doc/playground-user-guide/source/ doc/playground-user-guide/build/html from within openstack-manuals. This will apply the oslosphinx theme currently.

Sign up below for a chapter, then create a patch with RST in doc/playground-user-guide/source for others to review. Use the xml:id from the DocBook file for the RST file name. Once the patch merges, change the Status from In Review to Merged.

Chapter/Section (old filename) Chapter/Section (new filename) Name Patch URL Status
xxx section_dashboard_access_and_security.rst Andreas Jaeger https://review.openstack.org/#/c/142437/ Merged
xxx section_dashboard_access Anne Gentle https://review.openstack.org/151289 Merged
xxx launching_instances_using_dashboard Anne Gentle https://review.openstack.org/#/c/170617/ Merged
xxx upload_manage_images_using_dashboard Anne Gentle https://review.openstack.org/#/c/170593/ Merged
section_dashboard_databases.xml dashboard_databases Anne Gentle https://review.openstack.org/#/c/151846/ Merged
xxx dashboard_create_networks Anne Gentle https://review.openstack.org/#/c/152582/ Merged
xxx dashboard_manage_containers Anne Gentle https://review.openstack.org/#/c/153349/ Merged
xxx dashboard_manage_volumes Anne Gentle https://review.openstack.org/#/c/153577 Merged
section_dashboard_stacks.xml dashboard_stacks.rst Olena Logvinova https://review.openstack.org/168863 Merged
section_cli_trove.xml trove-manage-db.rst Laurel Michaels https://review.openstack.org/#/c/156087/ Merged
section_cli_trove.xml create_db.rst Laurel Michaels https://review.openstack.org/156266/ Merged
section_cli_trove.xml backup_db.rst Laurel Michaels https://review.openstack.org/#/c/164791/ Merged
section_cli_trove.xml backup_db_incremental.rst Laurel Michaels https://review.openstack.org/#/c/167649/ Merged
section_cli_trove.xml manage_db_config.rst Laurel Michaels https://review.openstack.org/#/c/169002/ Merged
section_cli_trove.xml set_up_replication.rst Laurel Michaels https://review.openstack.org/#/c/170560/ Merged
section_cli_trove.xml set_up_clustering.rst Laurel Michaels https://review.openstack.org/#/c/170734/ Merged
ch_cli.xml cli Karin Levenstein https://review.openstack.org/#/c/154235/ Merged
section_cli_nova_boot cli_launch_instances Karin Levenstein https://review.openstack.org/#/c/154235/ Merged
ch_sdk.xml sdk Darren Chan https://review.openstack.org/#/c/154412/ Merged
section_sdk_install sdk_install Darren Chan https://review.openstack.org/#/c/154412/ Merged
section_sdk_authenticate sdk_authenticate Darren Chan https://review.openstack.org/#/c/154412/ Merged
section_sdk_manage_images.xml sdk_manage _images Darren Chan https://review.openstack.org/#/c/154412/ Merged
section_sdk_configure_instances.xml sdk_configure _access_security_instances Darren Chan https://review.openstack.org/#/c/154412/ Merged
section_sdk_neutron.xml sdk_authenticate_networking_endpoint Darren Chan https://review.openstack.org/#/c/154412/ Merged
section_sdk_nova.xml sdk_authenticate_compute_endpoint Darren Chan https://review.openstack.org/#/c/154412/ Merged
section_cli_swift_howto.xml managing-openstack-object-storage-with-swift-cli Erik Wilson https://review.openstack.org/#/c/165132/ Merged
xxx cli_create_containers.rst Erik Wilson https://review.openstack.org/#/c/165132/ Merged
xxx cli_swift_manage_access_swift.rst Erik Wilson https://review.openstack.org/#/c/165132/ Merged
xxx cli_swift_manage_objects.rst Erik Wilson https://review.openstack.org/#/c/165132/ Merged
section_object-api-env-vars.xml cli_swift_env-vars.rst Erik Wilson https://review.openstack.org/#/c/165132/ Merged
section_object-api-versioning.xml cli_swift_set-object-versions.rst Erik Wilson https://review.openstack.org/#/c/165132/ Merged
section_object-api-response-formats.xml cli_swift_serialized-response-formats.rst Erik Wilson https://review.openstack.org/#/c/165132/ Merged
section_object-api-large-lists.xml cli_swift_large-lists.rst Erik Wilson https://review.openstack.org/#/c/165132/ Merged
section_object-api-pseudo-hier-folders.xml cli_swift_pseudo-hierarachical-folders-directories.rst Erik Wilson https://review.openstack.org/#/c/165132/ Merged
section_object-api-discoverability.xml cli_swift_discoverability.rst Erik Wilson https://review.openstack.org/#/c/165132/ Merged
section_object-api-create-large-objects.xml cli_swift_large-object-creation.rst Erik Wilson https://review.openstack.org/#/c/165132/ Merged
section_object-api-create-large-objects.xml cli_swift_archive-auto-extract.rst Erik Wilson https://review.openstack.org/#/c/165132/ Merged
section_object-api-bulk-delete.xml cli_swift_bulk-delete.rst Erik Wilson https://review.openstack.org/#/c/165132/ Merged
section_object-api-create-website.xml cli_swift_static-website.rst Erik Wilson https://review.openstack.org/#/c/165132/ Merged
section_object-api-cors-headers.xml sdk_assign_cors_headers Darren Chan https://review.openstack.org/#/c/156451/ Merged
section_cli_nova_manage_instances.xml cli_manage_instances_hosts Darren Chan https://review.openstack.org/#/c/157270/ Merged
section_cli_nova_floating_ips.xml cli_manage_ip_addresses Darren Chan https://review.openstack.org/#/c/157270/ Merged
section_cli_nova_resizerebuild.xml cli_change_the_size_of_your_server Darren Chan https://review.openstack.org/#/c/157270/ Merged
section_cli_nova_startstop.xml cli_stop_and_start_an_instance Darren Chan https://review.openstack.org/#/c/157270/ Merged
section_cli_nova_userdata.xml cli_provide_user_data_to_instances Darren Chan https://review.openstack.org/#/c/157270/ Merged
appendix_a__openstack_command-line_interface_cheat_sheet.xml cli_quick_start Anne Gentle https://review.openstack.org/158142/ Merged
section_object-api-expire-objects.xml sdk_schedule_objects_for_deletion Darren Chan https://review.openstack.org/#/c/158961/ Merged
section_sdk_auth_keystone.xml sdk_authenticate_against_identity_endpoint Darren Chan https://review.openstack.org/#/c/158961/ Merged
section_sdk_auth_glance.xml sdk_authenticate_against_image_service_endpoint Darren Chan https://review.openstack.org/#/c/158961/ Merged
section_sdk_auth_nova.xml sdk_authenticate_against_compute_endpoint Darren Chan https://review.openstack.org/#/c/158961/ Merged
section_sdk_auth_neutron.xml sdk_authenticate_against_networking_endpoint Darren Chan https://review.openstack.org/#/c/158961/ Merged
section_cli_nova_reboot.xml cli_reboot_an_instance Darren Chan https://review.openstack.org/#/c/160578/ Merged
section_cli_nova_terminate.xml cli_delete_an_instance Darren Chan https://review.openstack.org/#/c/160578/ Merged
section_cli_nova_get_console.xml cli_access_instance_through_a_console Darren Chan https://review.openstack.org/#/c/160578/ Merged
section_cli_nova_baremetal.xml cli_manage_bare_metal_nodes Darren Chan https://review.openstack.org/#/c/160578/ Merged
section_cli_nova_search_ip.xml cli_search_instance_with_ip_address Darren Chan https://review.openstack.org/#/c/160578/ Merged
section_cli_nova_migrate_instances.xml cli_use_snapshots_to_migrate _instances Darren Chan https://review.openstack.org/#/c/160578/ Merged
common/section_cli_overview.xml common/cli_overview Darren Chan https://review.openstack.org/#/c/164060/ Merged
common/section_cli_install.xml common/cli_install_openstack_command_line _clients Darren Chan https://review.openstack.org/#/c/167111/ Merged
common/section_cli_version.xml common/cli_discover_version_number_for _a_client Darren Chan https://review.openstack.org/#/c/166075/ Merged
common/section_cli_openrc.xml common/cli_set_environment_variables_using_openstack_rc Darren Chan https://review.openstack.org/#/c/166075/ Merged
common/section_cli_neutron_manage_networks.xml common/cli_create_and_manage_networks Darren Chan https://review.openstack.org/#/c/166075/ Merged
common/section_cli_glance_manage_images.xml common/cli_manage_images Darren Chan https://review.openstack.org/#/c/166075/ Merged
common/section_cli_cinder_manage_volumes.xml common/cli_manage_volumes Darren Chan https://review.openstack.org/#/c/166075/ Merged
common/section_cli_nova_usage_statistics.xml common/cli_nova_show_usage_statistic_for_hosts_instances Darren Chan https://review.openstack.org/#/c/166085/ Merged
section_cli_nova_configure_instances.xml cli_nova_configure_access_security_for_instances Darren Chan https://review.openstack.org/#/c/168772/ Merged
common/section_cli_nova_boot_from_volume.xml cli_nova_launch_instance_from_volume Darren Chan https://review.openstack.org/#/c/168772/ Merged
hot-guide/*.rst Gauvain Pocentek https://review.openstack.org/#/c/156351/ Merged
section_cli_nova_config-drive.xml cli_config_drive Anne Gentle https://review.openstack.org/#/c/171025/ Merged
section_cli_heat.xml cli_create_and_manage_stacks Darren Chan https://review.openstack.org/#/c/171446/ Merged

Admin User Guide Migration

Sign up below for a chapter, then create a patch with RST in doc/playground-user-guide/source for others to review.

These files should use this markup in the first two lines to ensure they are only output in the admin version:

.. meta::
    :scope: admin_only

Chapter/Section (old filename) Chapter/Section (new filename) Name Patch URL Status
section_cli_admin_manage_environment.xml cli_admin_manage_environment.rst Olga Gusarenko https://review.openstack.org/#/c/165121 Merged
section_cli_admin_manage_stacks.xml cli_admin_manage_stacks.rst Olga Gusarenko https://review.openstack.org/#/c/165121 Merged
section_cli_cinder_quotas.xml cli_cinder_quotas.rst Olga Gusarenko https://review.openstack.org/#/c/165803 Merged
section_cli_keystone_services.xml cli_keystone_manage_services.rst Olga Gusarenko https://review.openstack.org/#/c/166786 Merged
section_cli_keystone_set_quotas.xml cli_set_quotas.rst Olga Gusarenko https://review.openstack.org/#/c/165803 Merged
section_cli_manage_services.xml cli_manage_services.rst Olga Gusarenko https://review.openstack.org/#/c/166786 Merged
section_cli_nova_floating_ips.xml cli_admin_manage_ip_addresses.rst Olga Gusarenko https://review.openstack.org/#/c/168296 Merged
section_cli_nova_host_servers_migrate.xml xmlid.rst Olga Gusarenko Not used currently or find a proper place for it Deferred
section_cli_nova_manage_flavors cli_manage_flavors.rst Olga Gusarenko https://review.openstack.org/#/c/168797 In Progress
section_cli_nova_manage_projects_security nova_cli_manage_projects_security.rst Anne Gentle https://review.openstack.org/#/c/156877 Merged
section_cli_nova_migrate.xml cli_nova_migrate.rst Maria Zlatkova https://review.openstack.org/#/c/167178 Merged
section_cli_nova_services.xml cli_nova_manage_services.rst Olga Gusarenko https://review.openstack.org/#/c/166786 Merged
section_cli_nova_specify_host.xml cli_nova_specify_host.rst Olga Gusarenko https://review.openstack.org/#/c/168869 Merged
section_dashboard_admin_manage_flavors.xml dashboard_manage_flavors.rst Olena Logvinova https://review.openstack.org/#/c/167154 Merged
section_dashboard_admin_manage_host_aggregates.xml dashboard_manage_host_aggregates.rst Anne Gentle https://review.openstack.org/163198 Merged
section_dashboard_admin_manage_images.xml dashboard_manage_images.rst Olena Logvinova https://review.openstack.org/#/c/167950/2 Merged
section_dashboard_admin_manage_resources.xml dashboard_manage_resources.rst Karen Bradshaw https://review.openstack.org/#/c/165583/ Merged
section_dashboard_admin_manage_roles.xml dashboard_admin_manage_roles.rst Olena Logvinova https://review.openstack.org/166241 Merged
section_dashboard_admin_manage_services.xml dashboard_manage_services.rst Karen Bradshaw https://review.openstack.org/#/c/165486/ Merged
section_dashboard_admin_manage_stacks.xml dashboard_manage_stacks.rst Olena Logvinova https://review.openstack.org/#/c/168303 Merged
section_dashboard_admin_set_quotas.xml dashboard_set_quotas.rst Karen Bradshaw https://review.openstack.org/#/c/166306 Merged
section_dashboard_admin_view_cloud_resources.xml dashboard_view_cloud_resources.rst Karen Bradshaw https://review.openstack.org/#/c/165499/ Merged
section_cli_keystone_manage_projects_users_roles.xml admin_cli_manage_projects_users.rst Anne Gentle https://review.openstack.org/#/c/156877/ Merged
section_cli_keystone_manage_projects_users_roles.xml nova_cli_manage_projects_security.rst Anne Gentle https://review.openstack.org/#/c/156877/ Merged
section_dashboard_admin_manage_instances.xml dashboard_manage_instances.rst Anne Gentle https://review.openstack.org/163082 Merged
section_dashboard_admin_manage_volumes.xml dashboard_manage_volumes.rst Anne Gentle https://review.openstack.org/163087 Merged
section_cli_swift_analyze_log_files.xml analyzing-log-files-with-swift-cli.rst Anne Gentle https://review.openstack.org/161092 Merged

Doc Migration Plan

In Kilo we are migrating the End User Guide and the Admin User Guide. Refer to the detailed specification for the plan.

The Juno projects where this document needs to migrate to a specification are: nova (compute-api) v2, v3 swift (object-api) v1 glance (image-api) v1, v1.1, v2 keystone (identity-api) v2.0, v3 neutron (netconn-api) v1.0, v2.0 cinder (volume-api) v1.0, v2.0

Juno projects that have this type of document in a separate <service>-api repo are: trove (trove)

Juno projects that do not provide this type of prose-based spec for their API are: ceilometer heat sahara

Incubating projects, this information is just a heads-up so you know how we're thinking about API documentation going forward. ironic zaqar (marconi) barbican designate

I'm going to do the migration work with pandoc and propose the patch to the project's repo. Be on the lookout for those patches.

Doc Migration Steps

Open docbook book file in Oxygen.

Choose Document > Transformation > Configure Transformation Scenario(s).

Select DocBook XHTML - Chunk.

Click Apply associated (1).

Within the /out/xhtml-chunks/ directory that's generated, run the following script:

for i in *.xhtml
  # Convert from XHTML to RST
  pandoc -s -t rst $i -o $file_name
  sed -i -e '4,16d' $file_name 
  sed -i -e '/+--------------------------+$/,$d' $file_name
  sed -i -e '$d' $file_name
  sed -i -e '$d' $file_name

  # Rename file to second line of new RST content, but lowercase and
  # all non-alphanumeric chars renamed to underscores
  real_file_name=$(sed 's/[^a-zA-Z0-9\-]/_/g;2q;d' $file_name | awk '{print tolower($0)}').rst
  mv $file_name $real_file_name
  # Replace all cross-refs to xhtml files to renamed rst files
  sed -i '' -e "s/\<$i/fixmefixmefixme/g" *.rst
#  sed -i '' -e "s/\ fixmefixmefixme.*\`__//g" *.rst

Clean up where the fixmefixmefixme is output, it indicates where a cross-reference cannot exist any longer.

Clean up tables where the pandoc conversion just outputs paragraphs.

Remove numbering from Example titles and Table titles.

Rename chapter_ files and ensure they are titled to match the contents of the file, such as "Networking API 2.0 Overview" to networking_api_2.0_overview" for example.

Remove "programlisting" "screen" and "literallayout" from .. code:: lines.

Publishing User Guides

Proposed next steps for publishing the user guides on the 9th of April:

  1. Fix openstackdocstheme "log a bug" and release new openstackdocstheme (annegentle, done) (independent of other steps but should be done before release)
  2. Review that all content exists (annegentle, karenb, done)
  3. Rename playground-user-guide directory to user-guides (ajaeger, done)
  4. Rename on transifex as well (ajaeger, done)
  5. Split files in doc/user-guides/source into (ajaeger, done):
    • doc/user-guides/source/common: files shared by both guides
    • doc/user-guides/source/adminuser: files only for Admin User Guide
    • doc/user-guides/source/enduser: files only for End User Guide
  6. Publish content on docs.openstack.org for review (ajaeger, done):
    • End User Guide: docs.openstack.org/draft/user-guide/
    • Admin User Guide: docs.openstack.org/draft/user-guide-admin/
  7. Review contents and fix (karenb, starting with admin guide, can continue after publication)
  8. On the 17th of April, publish converted RST User Guides to:
    • End User Guide: docs.openstack.org/user-guide/
    • Admin User Guide: docs.openstack.org/user-guide-admin/
  9. Steps for this:
    • Publishing new content, 6:00 UTC or later, ajaeger will approve and do fixes as needed:
      • Remove entries from .tx/config for guides (ajaeger, done)
      • Remove doc/user-guide, doc/user-guide-admin, doc/hot-guide/ directories (ajaeger, merged): https://review.openstack.org/171683
      • publish new User Guides to proper place (ajaeger): part of patch above
      • Update entries to index.html pages pointing to new content (annegentle, merged): https://review.openstack.org/#/c/174380/
    • Removal of old content, 12:00 UTC or later, annegentle will approve and do fixes as needed:
  10. Search for old links. Go through our repos for (just search for "user-guide/content" and "user-guide-admin/content") and replace links with new location.

Please review and enhance so that we don't miss anything!