Upgrading YPlan to Python 3 with Zero Downtime

Making our dependencies compatible

The first step to make our code base compatible was to ensure that all our dependencies were! We have nearly 200 of them at this point, so auditing them all took some time. Luckily there’s the useful caniusepython3 tool which checks for the appropriate PyPI classifiers on your installed packages. We found some cases where packages were just missing the correct classifiers, or where the Python 3 compatibility had been recently released recently and we needed to upgrade, but there were about twenty packages were just not compatible and needed fixing.

We got to work making pull requests. Unfortunately in a few cases the original libraries were totally abandoned; we found for them that we were only using a fraction of the library so porting it all ourselves didn’t look attractive, and thus we made our own to fill in the gaps (e.g. treepoem and poor-smime-sign).

The other thing to look out for is where forks already exist for Python 3, but the original package is abandoned and hasn’t been updated to link to the fork. The main one for us was the abandoned python-openid which has been forked to python3-openid - it was only by an almost accidental web search that we found out about this! Because of this, we ended up with two requirements files, one for each python version, and a base file that they both included with the line -r requirements-base.txt.

After all the dependencies were confirmed compatible, we started adding backports packages on Python 2 to use some Python 3 features and compatibility done early. For example, backports.csv is very useful for getting the Python 3 Unicode-friendliness of the csv module on Python 2 - we’d actually already written a wrapper around the Python 2 csv and could delete that just by using this library.

Making our code compatible

Once all that was done, we had to make sure our own code was Python 3 ready. For a long time we’d been adding all the __future__ headers possible on Python 2 (absolute_import, division, print_function, unicode_literals) as they all helped with preventing bugs. We do this with excellent isort plus this config in our setup.cfg:

[isort]
add_imports =
    from __future__ import absolute_import
    from __future__ import division
    from __future__ import print_function
    from __future__ import unicode_literals

We had also been forcing all files to be UTF-8 for a long time, using the encoding header # -*- encoding:utf-8 -*-, enforced by flake8-coding with this config in setup.cfg:

[flake8]
; flake8-coding
accept-encodings = utf-8

To get to fully cross-compatible code, we added another linter to our toolchain: python-modernize. This uses the power of 2to3, but rather than just convert once, it moves your code to be compatible with both Pythons, via the amazing six.

We activated modernize one rule at a time, every few days, starting with the easier ones and working our way up. This allowed everyone to be synchronized on the compatible style by learning a few rules at a time, e.g. “use six.text_type instead of unicode”, rather than getting them all at once.

Another thing that we found important was ensuring that the six.moves aliases and the backport packages we added were used. To help with this I added a new option to my flake8 plugin flake8-tidy-imports and then added some rules on the YPlan code base, which were configured in our setup.cfg like so:

[flake8]
; flake8-tidy-imports
banned-modules =
    cStringIO = Use io.BytesIO or io.StringIO!
    csv = Use yplan.compat.csv instead!
    mock = Use unittest.mock!
    StringIO = Use io.BytesIO or io.StringIO!
    urllib.urlencode = Use six.moves.urllib.parse.urlencode!
    urlparse = Use six.moves.urllib.parse!

The cStringIO/StringIO -> io.BytesIO / io.StringIO one is especially important; the io module is a lot closer to how Python 3 thinks about bytes versus str, and allows you to detect errors a lot sooner. We found several cases where both bytes and str data was mixed with Python 2’s StringIO because it allows you to be sloppier.

Making our tests pass

Having added all these linting rules there were still other problems with compatibility, for example some small things that python-modernize doesn’t check for. Luckily we have very high test coverage with about 5000 tests, so we could be reasonably sure that things were working on Python 3 as long as all the tests passed.

Once we’d fixed some base failures, such as code that just wouldn’t import, we took the test suite and worked on it piece by piece to make it pass, mostly on one Django app at a time. In a final four day push, we got the test failures down from 1400 to 0, as tracked by white board:

One thing we found at this stage was that there were a few small bits that couldn’t be solved with six. Instead, we had to write code that ran differently on each python version, for example in one class we had:

if six.PY2:

    def __nonzero__(self):
        return True

else:

    def __bool__(self):
        return True

This was still easy to clean up after the deployment of Python 3, as it continued to mention six, so we’d easily come across it.

After this stage, we modified our Ansible to always install a Python 3 virtualenv alongside the Python 2 one, on all development and testing machines. We then didn’t regress on our test failures - no commit was allowed to land that broke Python 3 compatibility, as enforced by Jenkins.