Difference between revisions of "I18n/Tools"

Revision as of 19:04, 14 September 2015

Note: References below to Transifex are out-dated. We switched to using Zanata for the Liberty cycle. Updated documentation is coming soon. Please direct questions to the openstack-i18n mailing list until then.

I18N uses many different tools, across several OpenStack Projects to meet its goals.

Tools related with document translation

OpenStack documents are in DocBook format. In order to make the document translation management consistent with message translation, we need to convert DocBook into PO format.

The document translation management include following steps:

• Slicing: slice DocBook into string segment with a Python script, and generate a PO template
• Uploading: Upload the translation resources to Zanta by automation scripts
• Translating: manage the translation in Zanata including the translation memory and glossary management
• Merging: merge the translated string into Docbook with a python script
• Uploading: Download the translated results by automation scripts.

Slicing script

The slicing script has dependency to package "xml2po". So you have to install gnome-doc-utils before running it. On Ubuntu, do:

 sudo apt-get install gnome-doc-utils

or on openSUSE:

 sudo zypper install xml2po

The script can scan a specific folder under openstack-manuals/doc/src/docbkx, collect all the text segment in *.xml and combine them into a PO template, with command:

./tools/generatepot [foldername]

The PO template will be put in a "locale" folder under the specific docbook folder, i.e. openstack-manuals/doc/src/docbkx/[bookname]/locale.

Merging script

The merging script has dependency to package "xml2po". So you have to install gnome-doc-utils before running it. On Ubuntu, do:

 sudo apt-get install gnome-doc-utils

or on openSUSE:

 sudo zypper install xml2po

The script can merge the translation strings into DocBook and generate a DocBook in the specific language.

tox -e buildlang -- $LANG

Uploading script

The uploading script is an automation job running in Jenkins, which will be triggered by "commit" event of openstack-manuals repository. See here.

Downloading script

The downloading script is an automation job running daily in Jenkins. See here.

Tools related with RST document translation

(We don't use Transfix any more. So the descriptions related with Transifex part is out of date. It needs to be updated. )

Some of OpenStack documents are using RST format. The steps to translate RST documents are similar with DocBook documents, including:

• Slicing: generate PO templates from RST documents
• Uploading: Upload the translation resources to Transifex
• Translating: manage the translation in Transifex, including the translation memory and glossary management
• Downloading: Download the translated results by automation scripts.
• Building: Build HTML from RST documents and the translated results.

Sphinx is a tool to translate RST source files to various output formats, including POT and HTML. You need to install Sphinx before you go to below steps.

pip install Sphinx

Slicing

We use sphinx-build to translate RST files to POT files. Because we want to have a single POT file per document, we use msgcat to merge those POTs after sphinx-build.

sphinx-build -b gettext doc/[docname]/source/ doc/[docname]/source/locale/
msgcat doc/[docname]/source/locale/*.pot > doc/[docname]/source/locale/[docname].pot

Uploading

We use Transifex client to upload the POT file to www.transifex.com

tx set --auto-local -r openstack-i18n.[docname] "doc/[docname]/source/locale/<lang>/LC_MESSAGES/[docname].po"  --source-lang en --source-file doc/[docname]/source/locale/[docname].pot -t PO --execute
tx push -s

It could be automatic executed by Jenkins jobs.

Downloading

We use Transifex client to download the translated PO files from www.transifex.com

tx set --auto-local -r openstack-i18n.[docname] "doc/[docname]/source/locale/<lang>/LC_MESSAGES/[docname].po"  --source-lang en --source-file doc/[docname]/source/locale/[docname].pot -t PO --execute
tx pull -l [lang]

Building

Before use sphinx-build to build HTML file, we need to feed the translations from the single PO file into those small PO files. For example:

msgmerge -o doc/[docname]/source/locale/zh_CN/LC_MESSAGES/A.po doc/[docname]/source/locale/zh_CN/LC_MESSAGES/[docname].po doc/[docname]/source/locale/A.pot

Then, for evey PO file, we should execute the following command to build into MO file:

msgfmt "doc/[docname]/source/locale/zh_CN/LC_MESSAGES/A.po" -o "doc/[docname]/source/locale/zh_CN/LC_MESSAGES/A.mo"

Finally, we could generate HTML files by

sphinx-build -D "language='zh_CN' doc/[docname]/source/ doc/[docname]/build/html

Tools related with message translation

For most of the Python projects, the preferred tools for I18N are gettext and babel. The gettext module provides internationalization (I18N) and localization (L10N) services for your Python modules and applications. Babel are a collection of tools for internationalizing Python applications.

You can extract the messages in code to PO template with pybabel:

pybabel extract -o nova/locale/nova.pot nova/

Uploading script

For each Python project in OpenStack, there is an automation job to extract the messages , generate PO template and upload to Transifex, which is triggered by the "commit" event. See here.

Downloading script

For each Python project in OpenStack, there is an automation job daily to download the translations in po file to the "locale" folder under the source folder of each project. See here. It will generate a review request in Gerrit. After review, the translation in po file will be merged.