Difference between revisions of "Horizon/Packaging"

Latest revision as of 11:29, 22 September 2015

Packaging generalities

Packaging can easily be understood as: you're an engineer building a car. Unfortunately, you don't have anything else, other than a manual and very few tools. Any specific tool you'll require to actually build your car have to be created, too.

As example, if you you're going to add a library named "foo", it has to be

• Free Software
• all tools required to build it, have to be provided (as package) as well
• it has to be maintained, i.e. there needs to be an upstream
• it should NOT require a specific file system layout, FHS applies

Embedded copies

Imagine if every packages had a local copy of Jquery. If a security hole was discover in Jquery, then we would have to write more than 90 patches in Debian (as more than 90 packages are depending on Jquery). This is simply not practical. Therefore, it is not acceptable to purely copy code from other repositories into any software code (and no, Horizon cannot be a special case). This tends to become a fork, diverging from its real upstream. The main reason for this is, it's not being maintained, and if a bug is discovered in original upstream, if can't easily be fixed by updating just a single package.

Why did we decide to use the XStatic things?

There's a bunch of problems that are addressed by XStatic, which would have to be addressed if we are to find a new solution. Non-hexaustively, XStatic provides these things which are inherently not available by default with systems like NPM, Grunt and such:

• Dependency checks: The general rule makes it possible to check that dependencies (including fonts, javascript libs, etc.) are available in downstream distributions. The current XStatic system enables this type of checks.
• Reusable components cross projects: The XStatic system makes sure components are reusable by other packages (ie: non-Horizon things, like Fuel for example)
• System-wide registry of static content: The XStatic system brings a system-wide registry of components, so it is easy to check if one is missing (ie: no egg-info, broken package dependency, etc.). Currently, NPM doesn't offer this.
• Not embedded contents: The XStatic system makes sure we aren't embedding files which are already available in the distribution (ie: libjs-* or fonts-* packages).

If you want to redesign a new system to replace XStatic, please keep the above 4 points in mind. The new system *must* keep these features: please don't sacrifice them just because it's more convenient for Horizon developers. Yes, XStatic are making your life harder, we understand that. But we really need the above.

Free software

Red Hat, Debian and Ubuntu are made only of free software (free as in Libre, or free speech). This means that not only the software we include in our repository is free, but also all the tools that are used are free as well, and also available in the distribution. As package maintainers care about the quality of the packages they upload, they do run the unit tests which are available from upstream repositories. So if you do care about downstream distribution, please make sure that Horizon is only using the tools which can be uploaded, including these which are used for unit testing.

One famous example is Selenium. For a long time, it was only available from the non-free repositories of Debian. The reason was that upstream included some .xpi binaries. These .xpi included some Windows .dll and Linux .so files. As they couldn't be rebuilt from source, the whole of python-selenium was declared non-free. If we made Horizon build-depends on python-selenium, this would mean Horizon wouldn't be in Debian main anymore (remember: contrib and non-free are *not* considered part of Debian). Recently, the package maintainer of python-selenium decided to remove the .xpi files from python-selenium, and upload it to Debian Experimental (this time, in main, not in non-free). So if it was possible for Horizon to use python-selenium (without the non-free .xpi files), then we could run Selenium tests at package build time (I'm not sure if that is the case).

Running unit tests at build time

Why do package maintainers insist on running unit tests at package build time? Well, there's a very easy answer: because the build environment inside a distribution isn't exactly the same as the one in the OpenStack gate. For example, versions of a given library can be slightly different from the one in the gate. And we do want to detect when this is a problem, so it can be fixed. So whenever possible, try to make the lives of package maintainer easy, and allow them (or help them) to run unit tests when it is possible.

Minified Javascript policy

In all free software distribution which actively maintain OpenStack packages (ie: at least Red Hat, Debian and Ubuntu), minified javascripts are considered non-free. This means that they should *not* be present in upstream source code, or at least, a non-minified version should be present next to the minified version. Also, you should be aware of potential security issues with minifiers. This blog post explains it very well.

Component version

One very important thing to take care about, is the version of all the components you will use in your app. Since it is not acceptable to embed a given component within Horizon, then we must use what's in the distribution (including all fonts, javascript, etc.). This is where it becomes a bit tricky.

In most distribution, it is not acceptable to have multiple version of the same piece of software. Yes, in Red Hat system, it is technically possible to install 2 versions of one library at the same time. However, package maintainers try to avoid this as much as possible, because it is a security (or bug fix) nightmare. In Debian, it is simply completely forbidden, unless we are working on a transition from one version to the next. Discussing the mater with FTP masters about the Jquery version, their view is that it is acceptable to have 2 versions of Jquery at a given time in Unstable, but only to manage such a transition. IE, all packages should, at some point, support the higher version, and then another transition can start. For the Stable Debian, only a single version of a library should be available.

This has consequences for an upstream author willing to integrate his software in a downstream distribution. The best situation is when it is possible to support whatever version is currently available in the target distributions, up to the latest version upstream. And no, declaring lower and upper bounds within your requirements.txt doesn't solve the issue: it just hides the consequences in the OpenStack gate.

When it's not possible to support all version of a library (because it would be too much work, or when it would then be very hard to test in the gate), then the best recommendation is to use whatever is available inside the target distributions. For example Horizon currently supports Jquery >= 1.7.2, as this is what is currently available in Debian Jessie (the current stable) and Ubuntu Trusty (the last LTS). However, as Jquery 1.11.3 is now in Debian Unstable, and it will almost surely be synced to the next Ubuntu LTS 16.04. Horizon should then be tight to these versions. If we cannot support version 1.7.2 anymore, then supporting version 1.11.3 is what should be done, then package maintainers will simply do a backport of version 1.11.3 to Ubuntu cloud archive and Jessie backports.

Filesystem Hierarchy Standards

Every distribution has to comply with the FHS (Filesystem Hierarchy Standards). This defines a set of rules which we *must* follow as package maintainers. Some of the most important ones are:

• /usr should be considered as read only, and no software should write in it at runtime (however, it is fine for a package post installation script to write there). As a consequence, distributions had to write many tricks to convince horizon to write in /var/lib only (for example: writing symlinks to /var/lib/openstack-dashboard, or patch the default local_settings.py to write the SECRET_KEY in /var).
• Configuration should always be in /etc, no mater what. As a consequence, package maintainers (at least in Debian and Ubuntu) had to configure symlinks to /etc/openstack-dashboard/local_settings.py instead of using directly /usr/share/openstack-dashboard/openstack_dashboard/local/local_settings.py which Horizon expects.