Copyright 2016, SUSE LLC

This work is licensed under a Creative Commons Attribution 3.0
Unported License.
http://creativecommons.org/licenses/by/3.0/legalcode

Improve Translation Setup

https://storyboard.openstack.org/#!/story/2000452

Projects that want to setup translations for their repositories run into a few common problems and we should address these to make the setup for them easier. The problems they run into come from limitations of the current infra scripts, so we need to rework the infra scripts.

We should also have a simple and consistent setup for translations - with a sensible consistency. Right now it’s consistent for python projects but not sensible in many cases.

Note that manuals are also translated but are not part of this spec.

Problem Description

Initially only the core projects, like horizon, keystone, neutron, nova and swift, have been translated. With the growth of OpenStack more and more projects want to have translations. So, now we have translations for libraries like oslo.i18n, for python clients like python-novaclient or python-openstackclient, for dashboard plug-ins like designate-dashboard, or for networking plug-ins like networking-l2gw or networking-ovn.

Setting up translations for new projects currently has the following challenges:

  1. For “python” repositories, we expect that the locale file is located at $repo/locale/$repo.pot - without any change possible. This leads to python-novaclient/locale/python-novaclient.pot and oslo.log/locale/oslo.log.pot - in both cases the python module has a different name, novaclient and oslo_log.

    Currently everybody makes it wrong and we have to help them use the proper file names.

    There’s also an inconsistency on how the gettext domainname is set, file setup.cfg contains an entry and also oslo.i18n’s initialization. For some projects these are not in sync.

  2. For dashboard repos using django, there’s no common entry point yet and some are even setup like “python” repositories (like openstack/designate-dashboard). Each one is done differently, so needs special casing in the infra scripts.

  3. There’s also discussion going on to add to the various networking plug-in repositories also dashboard content so that they contain both python and Django directories. The vitrage-dashboard repository is an existing one that is set up this way but not yet translated. We should define how to translate these.

Proposed Change

The major changes are a common set up for the Django repositories as well as a new way to determine the location of the POT files:

  1. We will identify the modulename of each repository in the following way: The file setup.cfg is parsed and the packages entry in the [files] section is used as name for the module. If there are multiple entries, the following steps are applied for each entry. Note that the entries have to be valid python object paths and the scripts will fail with invalid ones.

    For some repositories, we might need to special case this and add an exact modulename - but this should be a rare exception.

  2. Instead of using $repo/locale/$repo.pot, the CI scripts will use $modulename/locale/$modulename.pot as location for POT files of python projects.

  3. Repositories using the django framework use two domains, one called django.pot for strings from Python files and one called djangojs.pot for strings from Javascript files.

    Since these jobs run on a slave that has access to passwords, we will not call scripts inside the repositories but let the CI scripts do the work.

    We will update the CI scripts to call pybabel directly like the following:

    KEYWORDS="-k gettext_noop -k gettext_lazy -k ngettext_lazy:1,2"
KEYWORDS+=" -k ugettext_noop -k ugettext_lazy -k ungettext_lazy:1,2"
KEYWORDS+=" -k npgettext:1c,2,3 -k pgettext_lazy:1c,2 -k npgettext_lazy:1c,2,3"
pybabel extract -F babel-${DOMAIN}.cfg \
  -o ${modulename}/locale/${DOMAIN}.pot $KEYWORDS ${modulename}

    For this, the files babel-django.cfg and babel-djangojs.cfg need to be setup in the repositories as follows:

    File babel-django.cfg:

    [extractors]
django = django_babel.extract:extract_django

[python: **.py]
[django: templates/**.html]
[django: **/templates/**.csv]

    File babel-djangojs.cfg:

    [extractors]
# We use a custom extractor to find translatable strings in AngularJS
# templates. The extractor is included in horizon.utils for now.
# See http://babel.pocoo.org/docs/messages/#referencing-extraction-methods for
# details on how this works.
angular = horizon.utils.babel_extract_angular:extract_angular

[javascript: **.js]

# We need to look into all static folders for HTML files.
# The **/static ensures that we also search within
# /openstack_dashboard/dashboards/XYZ/static which will ensure
# that plugins are also translated.
[angular: **/static/**.html]

    This uses the Babel, django-babel, and horizon packages, these will be installed by the CI scripts in a virtual environment for calling pybabel.

    Teams do not have to set up any tox environment for their repositories but if they add a tox environment it should be called extractmessages and it should use pybabel like the CI scripts do.

  4. All repositories ending in -dashboard, -horizon, or -ui will get the treatment from the previous step. Projects that do not have special requirements will continue to use python setup.py extract_messages.

    horizon repository has two modules unlike other django repositories. The scripts will support multiple modules and run the translation job for each module, so horizon repository will no longer be treated specially.

    This Django setup will be used for the following repositories that are currently setup (or planned to be setup) in project-config repository for translations: horizon, django_openstack_auth, designate-dashboard, horizon-cisco-ui, magnum-ui, murano-dashboard, trove-dashboard, zaqar-ui.

  5. Repositories like the networking plug-in ones that will have both Python and Django code will all be treated the same way. The CI scripts will run both Python and Django translations extraction. The exact way to configure the names of the Python module and the Django directory will be evaluated and documented.

    One idea to evaluate is supporting an optional section in file setup.cfg like the following:

    [openstack_translations]
django_modules = module1
python_modules = module2 module3

  6. As a review policy, changes to enable translations in the project-config repository should only be done after a project is set up for translations, otherwise jobs are triggered that always fail.

Alternatives

We can keep the status quo with leads to individual solutions for each repository and education of teams on how to properly set up repositories.

Implementation

Assignee(s)

Primary assignee:

  • jaegerandi

Also:

  • amotoki

Gerrit Topic

Use Gerrit topic “translation_setup” for all patches related to this spec.

git-review -t translation_setup

Work Items

  • Freeze adding of new translation setup for repositories until the spec is finished.

  • Update the Infra scripts. Implement the new setup for one or two repositories and test everything, then add further repositories. As part of this we need to rename modules in Zanata and in repositories.

    There might be a freeze for translations sync while the renaming happens.

  • Update documentation:

  • Review cookiecutter projects and see whether those need to be updated for these changes.

Repositories

This will be done in existing repositories. Work will touch the project-config repository for the infra scripts and might touch repositories that are translated.

Servers

No new servers are needed.

DNS Entries

No update for DNS entries is needed.

Documentation

The page http://docs.openstack.org/infra/system-config/translate.html as well as the Infra Manual should get updated.

Security

No security impact.

Testing

Manual tests need to be done to check that the scripts work.

Dependencies

No dependencies exist.