Jump to: navigation, search

Difference between revisions of "Translations"

(Integrating translations into your project)
 
(40 intermediate revisions by 15 users not shown)
Line 1: Line 1:
__NOTOC__
+
''Note: We switched in September 2015 to using [https://translate.openstack.org/ Zanata] for the Liberty cycle. Please direct questions to the  [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-i18n openstack-i18n mailing list] and update the documentation below to fully explain Zanata.''
= Translation, Internationalization and Localization in [[OpenStack]] =
 
  
[[OpenStack]] is committed to broad international support, and as such there must be
+
= Translation, Internationalization and Localization in OpenStack =
an ongoing concern with making [[OpenStack]] usable for all audiences. This includes
+
 
 +
OpenStack is committed to broad international support, and as such there must be
 +
an ongoing concern with making OpenStack usable for all audiences. This includes
 
proper use of internationalization and localization tools by developers, and
 
proper use of internationalization and localization tools by developers, and
 
high-quality translations for both user-facing messages and documentation.
 
high-quality translations for both user-facing messages and documentation.
Line 11: Line 12:
 
Let's start with a working definition: translation is the act of taking the
 
Let's start with a working definition: translation is the act of taking the
 
written materials in one language and converting them into another language in
 
written materials in one language and converting them into another language in
the most meaningful way possible. In terms of [[OpenStack]], translation happens on
+
the most meaningful way possible. In terms of OpenStack, translation happens on
 
both the written documentation and on strings marked for translation in the
 
both the written documentation and on strings marked for translation in the
 
projects' codebases.
 
projects' codebases.
Line 17: Line 18:
 
''NOTE: information on how to prepare your code or documentation for translation, see the section on internationalization below.''
 
''NOTE: information on how to prepare your code or documentation for translation, see the section on internationalization below.''
  
=== Transifex ===
+
=== Zanata ===
  
[[OpenStack]] has adopted [https://www.transifex.net/projects/p/openstack/ Transifex] as its translation management platform of
+
OpenStack is using a Zanata instance running at [https://translate.openstack.org/ https://translate.openstack.org/] as translation management platform.
choice. Transifex's overall selection of management tools, features, community
 
support, solid [[GitHub]] integration and proven track record made it the winning
 
candidate in the end.
 
  
The following sections should introduce you to the core concepts for
+
=== Downloading translation files ===
contributing translations to [[OpenStack]].
 
  
==== Project Hub ====
+
If you wish to download the translation files (.po files) you can do so by
 
+
selecting the language you're interested in, then clicking on the name of
The [https://www.transifex.net/projects/p/openstack/ OpenStack Project Hub] is the starting point for [[OpenStack]] on Transifex.
+
the project resource you wish to download. In the modal dialog which appears,
It tracks all the registered [[OpenStack]] projects and allows for coordinated
+
you can use any of the "download" options depending on your use case.
management and shared translation capabilities (such as shared language teams
 
and shared translation memory).
 
 
 
At present the [[OpenStack]] project is a "free for all" where anyone can
 
contribute any translation without officially joining a translation team, so
 
feel free to jump right in and start contributing.
 
 
 
==== Individual projects ====
 
 
 
Within the [[OpenStack]] project hub, each individual project can manage its
 
own translation resources. Setting up new projects is fairly simple,
 
and the goal is to have ''all'' core [[OpenStack]] projects fully integrated into
 
the internationalization process.
 
  
 
=== Translating on the site ===
 
=== Translating on the site ===
  
Translation is most efficiently done right on Transifex's site. You don't
+
Translation is most efficiently done right on Zanata's site. You don't
 
need to download any files or applications to get started.
 
need to download any files or applications to get started.
  
If your language already exists, simply click on it in the list of available
+
If your language already exists, select a project, select the "master" version, select your language and then select a document to start translating.
langauges for [[OpenStack]], and then click on the name of a project resource
 
which you'd like to start working on. In the modal dialog that appears, click
 
the "Translate Now" button to begin working.
 
  
If the language you wish to contribute to doesn't already exist, you'll need to
+
=== Downloading translation files ===
navigate to a specific project's resource files to click the "add language"
 
button. Clicking the "Explore" link in the navigation bar, switching to the
 
"Projects" tab, and filtering on the tag "openstack" will get you a list of all
 
the available [[OpenStack]] projects. From the project you wish to work on, click
 
the "Resources" tab on the project page, select a resource (if there's more
 
than one), and then click the "add language" button on that page.
 
  
Once you're satisfied with your translations, click the "Save all" or "Save and
+
You can also download translation files and translate locally.
exit" button to finalize your contributions.
 
  
=== Downloading translation files ===
+
'''TODO: Explain exactly how this is done with Zanata.'''
 
 
If you wish to download the translation files (.po files) you can do so by
 
selecting the language you're interested in, then clicking on the name of
 
the project resource you wish to download. In the modal dialog which appears,
 
you can use any of the "download" options depending on your use case.
 
  
 
== Release cycle ==
 
== Release cycle ==
Line 81: Line 50:
 
=== String Freeze ===
 
=== String Freeze ===
  
''NOTE: [[OpenStack]]'s string freeze happens at the close of the final milestone in the development cycle, giving translators the entire RC period to update translations.''
+
''NOTE: OpenStack's string freeze happens at the close of the final milestone in the development cycle, giving translators the entire RC period to update translations.''
  
 
At a predefined time during the release cycle there will be a "string freeze",
 
At a predefined time during the release cycle there will be a "string freeze",
Line 87: Line 56:
 
codebase can no longer be changed except in the case of critical-priority bugs.
 
codebase can no longer be changed except in the case of critical-priority bugs.
  
Once the string freeze is in effect, the translation files in Transifex can
+
Once the string freeze is in effect, the translation files in Zanata can
 
be assumed to be static, and translation efforts should happen in full force.
 
be assumed to be static, and translation efforts should happen in full force.
 
This is not to say that translation can't happen all the time. But during the
 
This is not to say that translation can't happen all the time. But during the
Line 96: Line 65:
 
do not alter or add translation strings, or else coordinated with translators
 
do not alter or add translation strings, or else coordinated with translators
 
to ensure that changes are handled appropriately.
 
to ensure that changes are handled appropriately.
 +
 +
Check out http://docs.openstack.org/project-team-guide/release-management.html for more details.
  
 
=== Re-incorporating Translations ===
 
=== Re-incorporating Translations ===
  
The [[OpenStack]] Infrastructure team has set up automatic generation of reviews
+
The OpenStack Infrastructure team has set up automatic generation of reviews
 
for translations so that they can be re-incorporated with minimal effort
 
for translations so that they can be re-incorporated with minimal effort
at any time.
+
at any time. For each project where this is setup in our CI infrastructure, every day a job is run. This job regenerates the original pot file and imports all well enough translated files and then proposes them to the project as patches. Only files that have 75 per cent or more translated strings are downloaded.
 +
 
 +
The list of current open proposed imports is available at [https://review.openstack.org/#/q/status:open++branch:master+topic:transifex/translations,n,z review.openstack.org].
  
 
Most importantly though, immediately prior to the release of each Release
 
Most importantly though, immediately prior to the release of each Release
Line 109: Line 82:
  
 
At present it is the responsibility of each project's PTL or appointed
 
At present it is the responsibility of each project's PTL or appointed
translation manager to make sure this happens, though [[OpenStack]]'s release
+
translation manager to make sure this happens, though OpenStack's release
 
managers, translation team coordinators, etc. are also encouraged to help
 
managers, translation team coordinators, etc. are also encouraged to help
 
ensure that this happens smoothly.
 
ensure that this happens smoothly.
Line 130: Line 103:
 
==== Python Projects (General) ====
 
==== Python Projects (General) ====
  
For most of the [[OpenStack]] core projects (and any that use Python), the preferred
+
For most of the OpenStack core projects (and any that use Python), the preferred
tools for internationalization are [http://docs.python.org/library/gettext.html gettext] and [http://babel.edgewall.org/ babel]. Getting started is
+
tools for internationalization are [http://docs.python.org/library/gettext.html gettext] and [http://babel.edgewall.org/ babel] (Debian/Ubuntu package name ''python-pybabel''). Getting started is
 
pretty easy:
 
pretty easy:
  
1. Enable the gettext module everywhere in your application by installing it in the root `<u>init</u>.py` file like so:
+
===== Adopt oslo.i18n =====
 
+
First step is to adopt oslo.i18n in your project - [http://docs.openstack.org/developer/oslo.i18n/usage.html How to Use oslo.i18n in Your Application or Library]
 
 
<pre><nowiki>#!highlight python
 
import gettext
 
gettext.install(<project name>, unicode=1)
 
</nowiki></pre>
 
 
 
 
 
This makes the `_()` function available everywhere in your codebase as a shortcut to mark strings for translation.
 
 
 
2. Configure your project to use Babel to easily create your translation files. First, add `Babel` to your pip-requires file (or wherever you track dependencies). Second, create a `babel.cfg` file in the root of your project; at it's simplest it can just contain this line:
 
  
 +
===== Extract messages =====
 +
Once you have some [http://docs.openstack.org/developer/oslo.i18n/guidelines.html#choosing-a-marker-function messages to translate], we need to extract those messages using Babel. The easiest way is to run "python setup.py extract_messages" in say the py27 venv.
  
<pre><nowiki>
+
Configure your project to use Babel to easily create your translation files. First, add `Babel` to your requirements.txt file (or wherever you track dependencies). Second, create a `babel.cfg` file in the root of your project; at it's simplest it can just contain this line:
 +
  <pre><nowiki>
 
[python: **.py]
 
[python: **.py]
 
</nowiki></pre>
 
</nowiki></pre>
 
  
 
Finally, add the following to your `setup.cfg` file:
 
Finally, add the following to your `setup.cfg` file:
 
 
 
<pre><nowiki>
 
<pre><nowiki>
 
[extract_messages]
 
[extract_messages]
Line 162: Line 125:
 
output_file = <project name>/locale/<project name>.pot
 
output_file = <project name>/locale/<project name>.pot
 
</nowiki></pre>
 
</nowiki></pre>
 
  
 
That will allow you to run `python setup.py extract_messages` and have it automatically generate the base translation resource file for your project.
 
That will allow you to run `python setup.py extract_messages` and have it automatically generate the base translation resource file for your project.
  
3. In your codebase, mark user-facing strings (API messages, etc.) for for translation by wrapping them with the underscore function like so:
+
Now you are ready to merge the generated files into your project (see example [https://review.openstack.org/#/c/182848/ review]). Note that an initial file needs to be imported into your project for the scripts that interact with the translation site.
 
 
 
 
<pre><nowiki>
 
my_internationalized_string = _("I'm internationalized!")
 
</nowiki></pre>
 
 
 
  
You can use internationalized strings anywhere that unicode is safe.
+
===== Setup Zanata server, import and export of translations =====
  
4. Once you've marked all your strings for translation, you can use the aforementioned `extract_messages` command to generate the base translation resource file. That file can then be uploaded to Transifex to serve as the basis for your project's translation efforts.
+
Now you are ready to setup Zanata and the CI infrastructure. Read the [https://docs.openstack.org/infra/manual/creators.html#enabling-translation-infrastructure Infra manual] on how to do it.
 
 
Once the project is set up with Transifex, the [[OpenStack]] Infrastructure team can make sure that any changes to translation files are automatically pushed to Transifex in real-time.
 
  
 
==== Horizon (Django) ====
 
==== Horizon (Django) ====
Line 201: Line 155:
 
While developer documentation for projects can generally be maintained solely in
 
While developer documentation for projects can generally be maintained solely in
 
English, user-oriented documentation such as that produced and maintained by
 
English, user-oriented documentation such as that produced and maintained by
[[OpenStack]]'s Docs team is also a high-priority for translation. This includes
+
OpenStack's Docs team is also a high-priority for translation. This includes
 
installation and administration manuals.
 
installation and administration manuals.
  
 
''NOTE: For the first release this does not include API documentation. Typically these are sourced in the `openstack-manuals` project.''
 
''NOTE: For the first release this does not include API documentation. Typically these are sourced in the `openstack-manuals` project.''
  
For specifics on translation of [[OpenStack]] Documentation, please refer to the
+
For specifics on translation of OpenStack Documentation, please refer to the
[https://gist.github.com/3037139 OpenStack Document Translation Guide].
+
[[Documentation/Translation]].
  
 
==== What To Translate ====
 
==== What To Translate ====
Line 214: Line 168:
 
API messages, CLI responses, documentation, help text, etc.
 
API messages, CLI responses, documentation, help text, etc.
  
There has been a lack of consensus about the translation of log messages;
+
See [[LoggingStandards#Log_Translation]] for information about translating log messages.
the current ruling is that while it is not against policy to mark log messages
 
for translation if your project feels strongly about it, translating log
 
messages is not actively encouraged.
 
  
Exception text should ''not'' be marked for translation, becuase if an exception
+
Exception text should ''not'' be marked for translation, because if an exception
 
occurs there is no guarantee that the translation machinery will be functional.
 
occurs there is no guarantee that the translation machinery will be functional.
  
Line 243: Line 194:
 
Numbers:
 
Numbers:
 
     1,000.42 == One thousand and 42 hundredths (US)
 
     1,000.42 == One thousand and 42 hundredths (US)
     1.000,42 == One thousand and 42 hundredths (UK)
+
     1.000,42 == One thousand and 42 hundredths (EU)
 
</nowiki></pre>
 
</nowiki></pre>
  
Line 253: Line 204:
 
convert them back to the user's expected format for display.
 
convert them back to the user's expected format for display.
  
Another less common (for [[OpenStack]]) issue related to localization revolves
+
Another less common (for OpenStack) issue related to localization revolves
 
around name formats, which vary culturally. The western style of "first name"
 
around name formats, which vary culturally. The western style of "first name"
 
and "last name" doesn't fit for many cultural naming conventions. This isn't
 
and "last name" doesn't fit for many cultural naming conventions. This isn't
Line 274: Line 225:
 
to localize a Python project with some effort. More information on this will
 
to localize a Python project with some effort. More information on this will
 
be added in the future.
 
be added in the future.
 +
 +
 +
== Translation infrastructure ==
 +
The translation infrastructure and workflow is documented on the [[Translations/Infrastructure]] page.
 +
 +
----
 +
[[Category:I18n]]

Latest revision as of 07:27, 23 June 2017

Note: We switched in September 2015 to using Zanata for the Liberty cycle. Please direct questions to the openstack-i18n mailing list and update the documentation below to fully explain Zanata.

Translation, Internationalization and Localization in OpenStack

OpenStack is committed to broad international support, and as such there must be an ongoing concern with making OpenStack usable for all audiences. This includes proper use of internationalization and localization tools by developers, and high-quality translations for both user-facing messages and documentation.

Translation & Management

Let's start with a working definition: translation is the act of taking the written materials in one language and converting them into another language in the most meaningful way possible. In terms of OpenStack, translation happens on both the written documentation and on strings marked for translation in the projects' codebases.

NOTE: information on how to prepare your code or documentation for translation, see the section on internationalization below.

Zanata

OpenStack is using a Zanata instance running at https://translate.openstack.org/ as translation management platform.

Downloading translation files

If you wish to download the translation files (.po files) you can do so by selecting the language you're interested in, then clicking on the name of the project resource you wish to download. In the modal dialog which appears, you can use any of the "download" options depending on your use case.

Translating on the site

Translation is most efficiently done right on Zanata's site. You don't need to download any files or applications to get started.

If your language already exists, select a project, select the "master" version, select your language and then select a document to start translating.

Downloading translation files

You can also download translation files and translate locally.

TODO: Explain exactly how this is done with Zanata.

Release cycle

One of the most challenging aspects of managing translations in an Open Source project is handling the interplay between translators and developers during the release cycle. The key piece of this equation is the "string freeze".

String Freeze

NOTE: OpenStack's string freeze happens at the close of the final milestone in the development cycle, giving translators the entire RC period to update translations.

At a predefined time during the release cycle there will be a "string freeze", which means that after this point strings marked for translation in the codebase can no longer be changed except in the case of critical-priority bugs.

Once the string freeze is in effect, the translation files in Zanata can be assumed to be static, and translation efforts should happen in full force. This is not to say that translation can't happen all the time. But during the development process strings may change and translation efforts may end up being wasted.

Any changes during the RC period should be carefully vetted to ensure they do not alter or add translation strings, or else coordinated with translators to ensure that changes are handled appropriately.

Check out http://docs.openstack.org/project-team-guide/release-management.html for more details.

Re-incorporating Translations

The OpenStack Infrastructure team has set up automatic generation of reviews for translations so that they can be re-incorporated with minimal effort at any time. For each project where this is setup in our CI infrastructure, every day a job is run. This job regenerates the original pot file and imports all well enough translated files and then proposes them to the project as patches. Only files that have 75 per cent or more translated strings are downloaded.

The list of current open proposed imports is available at review.openstack.org.

Most importantly though, immediately prior to the release of each Release Candidate, and before cutting the Final Release for each version, the translation files should be merged back into their respective projects to make sure they are properly distributed with the release.

At present it is the responsibility of each project's PTL or appointed translation manager to make sure this happens, though OpenStack's release managers, translation team coordinators, etc. are also encouraged to help ensure that this happens smoothly.

Stable Releases and Backports

At present, changes to translations will not be backported to stable release branches. Doing so would require maintaining wholly separate copies of each set of translations and massively increases the burden on translators.

Internationalization (i18n)

The term internationalization is used to broadly describe coding practices that allow software to be adapted to the linguistic and technical differences of various regions. This includes practices such as marking strings for translation, supporting non-ASCII character sets, etc.

Integrating translations into your project

Python Projects (General)

For most of the OpenStack core projects (and any that use Python), the preferred tools for internationalization are gettext and babel (Debian/Ubuntu package name python-pybabel). Getting started is pretty easy:

Adopt oslo.i18n

First step is to adopt oslo.i18n in your project - How to Use oslo.i18n in Your Application or Library

Extract messages

Once you have some messages to translate, we need to extract those messages using Babel. The easiest way is to run "python setup.py extract_messages" in say the py27 venv.

Configure your project to use Babel to easily create your translation files. First, add `Babel` to your requirements.txt file (or wherever you track dependencies). Second, create a `babel.cfg` file in the root of your project; at it's simplest it can just contain this line:

[python: **.py]

Finally, add the following to your `setup.cfg` file:

[extract_messages]
keywords = _ gettext ngettext l_ lazy_gettext
mapping_file = babel.cfg
output_file = <project name>/locale/<project name>.pot

That will allow you to run `python setup.py extract_messages` and have it automatically generate the base translation resource file for your project.

Now you are ready to merge the generated files into your project (see example review). Note that an initial file needs to be imported into your project for the scripts that interact with the translation site.

Setup Zanata server, import and export of translations

Now you are ready to setup Zanata and the CI infrastructure. Read the Infra manual on how to do it.

Horizon (Django)

Django has built-in internationalization tools that go well-beyond the basics of `gettext` to ensure proper unicode support throughout the entire codebase and to make advanced features more accessible. As such, Horizon uses Django's family of `ugettext` functions from `django.utils.translation`. It is preferrable to explicitly import the translation function you wish to use:


#!highlight python
from django.utils.translation import ugettext, ugettext_lazy  # ..., etc.


For more information on the internationalization tools Django makes available, see the Django i18n Docs.

Documentation (DocBook)

While developer documentation for projects can generally be maintained solely in English, user-oriented documentation such as that produced and maintained by OpenStack's Docs team is also a high-priority for translation. This includes installation and administration manuals.

NOTE: For the first release this does not include API documentation. Typically these are sourced in the `openstack-manuals` project.

For specifics on translation of OpenStack Documentation, please refer to the Documentation/Translation.

What To Translate

At present the convention is to translate all user-facing strings. This means API messages, CLI responses, documentation, help text, etc.

See LoggingStandards#Log_Translation for information about translating log messages.

Exception text should not be marked for translation, because if an exception occurs there is no guarantee that the translation machinery will be functional.

Localization (L10n)

The term localization is used more specifically than internationalization to cover coding practices that allow a software's input and output characteristics to adjust to variances in style from region to region. This includes things like number and date formatting, especially.

Dates, Numbers, and Other Concerns

Going beyond What is accomplished by Internationalization, the most important aspect to consider is regional differences in formatting for dates and numbers. For example::


Dates:
    04/01/2012 == April 1st, 2012 (US)
    04/01/2012 == January 4th, 2012 (UK)

Numbers:
    1,000.42 == One thousand and 42 hundredths (US)
    1.000,42 == One thousand and 42 hundredths (EU)


Accepting any format and naively passing it into our code would horribly break things. Accepting only one format leaves out large chunks of the world. Therefore, we use localization tools to accept these formats and normalize them into data structures Python can handle universally on input, and to convert them back to the user's expected format for display.

Another less common (for OpenStack) issue related to localization revolves around name formats, which vary culturally. The western style of "first name" and "last name" doesn't fit for many cultural naming conventions. This isn't something a software tool can account for, so for problems such as these the best solution is to simply accept the broadest range of inputs (e.g. a single "name" field).

How To Localize Your Project

Horizon (Django)

Horizon has excellent localization tools available since it is built on top of the Django web framework. Most conversions happen automatically when the localization framework is active. Full support for a localized user dashboard experience is a high-priority feature.

Other OpenStack Projects

Python's `locale` and `gettext` modules offer most of the tools necessary to localize a Python project with some effort. More information on this will be added in the future.


Translation infrastructure

The translation infrastructure and workflow is documented on the Translations/Infrastructure page.