'pip install' From a Git Repository

2019-03-11 Cat with ball

It’s quite common to want to pip install a version of a package that hasn’t been released to PyPI, but is available on its Git repository host, such as GitHub. If the package is pure Python or has a relatively simple build process integrated with setup.py, it can be installed from source.

Installing via Git

Pip can speak git to do this through its VCS Support. For example, to install Django’s most recent commit at time of writing, we can run:

$ python -m pip install git+https://github.com/django/django.git@45dfb3641aa4d9828a7c5448d11aa67c7cbd7966

Update (2020-04-11): Why python -m pip instead of just pip? See my post on why it has fewer problems.

Installing via tarballs

An alternative that avoids Git is to install from a tarball URL, that the major hosted Git solutions provide, for example:

$ # GitHub
$ python -m pip install https://github.com/django/django/archive/45dfb3641aa4d9828a7c5448d11aa67c7cbd7966.tar.gz
$ # GitLab
$ python -m pip install https://gitlab.com/pycqa/flake8/-/archive/3.7.7/flake8-3.7.7.tar.gz
$ # Bitbucket
$ python -m pip install https://bitbucket.org/hpk42/tox/get/2.3.1.tar.gz

Either format is slower than using a package on PyPI, because the download will be executed every time the install is run, which you might do with pip install -r requirements.txt on every build. However the tarball formats save a lot of downloading because only the tip of the repository will be downloaded, and not the full history, which can be very big for something like Django!

Also if you need a package with extra requirements, for example django[argon2] which ensures Django gets all the requirements it needs for password hashing, add the #egg argument on the end of the URL. For example:

$ # Git repo
$ python -m pip install git+https://github.com/django/django.git@45dfb3641aa4d9828a7c5448d11aa67c7cbd7966#egg=django[argon2]
$ # Tarball
$ python -m pip install https://github.com/django/django/archive/45dfb3641aa4d9828a7c5448d11aa67c7cbd7966.tar.gz#egg=django[argon2]

Timing

For comparison, on a 100MB/s internet connection, I find the git clone for Django takes 36s while the tarball install takes 15s - less than half the time. Installing from PyPI, where a wheel is available, takes just 5s.

The ideal situation is thus to install from PyPI. If an upstream package hasn’t released in a while, it’s worth asking them to, or offering to help with maintenance. Otherwise it is also possible to fork it and publish it on PyPI with a namespace prefix, for example publish your django fork as acme-corp-django.

More info

There’s a lot more options for sources in the pip install docs, but installing from tarball URL’s is the most common piece of advice I find myself giving.

Hope that helps,

Adam

Working on a Django project? Check out my book Speed Up Your Django Tests which covers loads of best practices so you can write faster, more accurate tests.

One summary email a week, no spam, I pinky promise.

Related posts:

Tags: python