‘pip install’ From a Git Repository

2019-03-11 Cat with ball

Update (2022-09-28): Rewrote to cover Git support rather than tarball URL’s, reflecting speed improvements in newer pip versions.

It’s quite common to want to pip install a version of a package that hasn’t been released to PyPI, but is available on its Git repository host, such as GitHub. If the package is pure Python or has a relatively simple build process, you can normally install it directly via Git.

Installing via Git

Pip can speak git to do this through its Version Control System (VCS) Support. (Git is one version control system.) For example, to install Django’s most recent commit at time of writing, run:

$ python -m pip install 'django @ git+https://github.com/django/django@80d38de52bb2721a7b44fce4057bcff571afc23a'

You can put any kind of Git ref after the second @, such as a branch or tag. In most cases, you’ll want to stick to immutable refs (tags or SHA’s), since they cannot change between installs.

Using quotes lets you put spaces around the first at sign, making things a bit clearer. Quotes also avoid misinterpreting some of the syntax we’ll cover in following sections.

(Why python -m pip instead of just pip? See my post on why it has fewer problems.)

Inside requirements files

You can use this same syntax inside requirements files, alongside your other requirement specifiers:

django @ git+https://github.com/django/django@80d38de52bb2721a7b44fce4057bcff571afc23a
psycopg2-binary==2.9.3

Nice.

Installing with extras

If you need to specify package extras, add #egg=<pkg>[<extras>] on the end. For example, to install Django with its argon2 extra:

$ python -m pip install 'django @ git+https://github.com/django/django@80d38de52bb2721a7b44fce4057bcff571afc23a#django[argon2]'

Easy peasy.

Installing from subdirectories

Sometimes the project you want to install is not at the root of the repository, but in a subdirectory. In this case, add #subdirectory=<path> at the end. For exmaple, to install this project from the repo subdirectory django_stubs_ext:

$ python -m pip install 'django-stubs-ext @ git+https://github.com/typeddjango/django-stubs@9a41aa63ba22e599aa8f355eed6cb60c90f896c6#subdirectory=django_stubs_ext'

In this example, there’s a main project, django-stubs, and a sub-project, django-stubs-ext. This command installs django-stubs-ext from within its subdirectory.

If you need to combine extras and a subdirectory, list both options with a &:

$ python -m pip install 'django-stubs-ext @ git+https://github.com/typeddjango/django-stubs@9a41aa63ba22e599aa8f355eed6cb60c90f896c6#egg=django-stubs-ext[example]&subdirectory=django_stubs_ext'

(The 'example' extra doesn’t actually exist in django-stubs-ext. I just couldn’t find a real world example!)

How pip does it

On initial install, pip will:

Clone the repository
Fetch and check out the desired commit
Build the package into a wheel (the pip wheel command)
Install that wheel (plus any dependencies)

pip’s output logs the steps in this process:

$ python -m pip install 'django @ git+https://github.com/django/django@80d38de52bb2721a7b44fce4057bcff571afc23a'
Collecting django@ git+https://github.com/django/django@80d38de52bb2721a7b44fce4057bcff571afc23a
  Cloning https://github.com/django/django (to revision 80d38de52bb2721a7b44fce4057bcff571afc23a) to /private/var/folders/20/lzgtdyzs7wj5fc_90w1h3bhc0000gn/T/pip-install-sirxc970/django_b245ad848cac4a16a8639862854e9b94
  Running command git clone --filter=blob:none --quiet https://github.com/django/django /private/var/folders/20/lzgtdyzs7wj5fc_90w1h3bhc0000gn/T/pip-install-sirxc970/django_b245ad848cac4a16a8639862854e9b94
  Running command git rev-parse -q --verify 'sha^80d38de52bb2721a7b44fce4057bcff571afc23a'
  Running command git fetch -q https://github.com/django/django 80d38de52bb2721a7b44fce4057bcff571afc23a
  Resolved https://github.com/django/django to commit 80d38de52bb2721a7b44fce4057bcff571afc23a
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting asgiref>=3.5.2
  Using cached asgiref-3.5.2-py3-none-any.whl (22 kB)
Collecting sqlparse>=0.2.2
  Using cached sqlparse-0.4.3-py3-none-any.whl (42 kB)
Building wheels for collected packages: django
  Building wheel for django (pyproject.toml) ... done
  Created wheel for django: filename=Django-4.2.dev20220928074807-py3-none-any.whl size=7843439 sha256=c7801e84dfa16f0f944fd875b2287936b09dff9914f41786d8733441442f63a2
  Stored in directory: /Users/chainz/Library/Caches/pip/wheels/2c/fe/82/922c04ff998d31a0c66cbb08f691112b5e6a5a218eb4103011
Successfully built django
Installing collected packages: sqlparse, asgiref, django
Successfully installed asgiref-3.5.2 django-4.2.dev20220928074807 sqlparse-0.4.3

If you re-install the same repository at the same commit, pip will reuse the wheel it built previously:

$ python -m pip install 'django @ git+https://github.com/django/django@80d38de52bb2721a7b44fce4057bcff571afc23a'
Collecting django@ git+https://github.com/django/django@80d38de52bb2721a7b44fce4057bcff571afc23a
  Using cached Django-4.2.dev20220928074807-py3-none-any.whl
Requirement already satisfied: asgiref>=3.5.2 in ./venv/lib/python3.10/site-packages (from django@ git+https://github.com/django/django@80d38de52bb2721a7b44fce4057bcff571afc23a) (3.5.2)
Requirement already satisfied: sqlparse>=0.2.2 in ./venv/lib/python3.10/site-packages (from django@ git+https://github.com/django/django@80d38de52bb2721a7b44fce4057bcff571afc23a) (0.4.3)

This makes it nice and fast when you’re reinstalling a requirements file.

Installing via tarballs

An alternative that avoids Git is to install from a tarball URL, that the major hosted Git solutions provide, for example:

$ python -m pip install https://github.com/django/django/archive/80d38de52bb2721a7b44fce4057bcff571afc23a.tar.gz

The original 2019 version of this article recommended using tarball URL’s over Git installs, because they used to be faster. Pip has since optimized its Git install mechanism, so that is normally faster now. Tarball URL’s may still be useful in niche situations though, for example if you need to install on a system without Git installed.

Read the docs

See pip’s VCS Support docs for full information, such as other VCS’s and editable mode. The pip install page also has many examples of different specifiers.

Fin

Hope that helps,

Adam

Improve your Django develompent experience with my new book.

One summary email a week, no spam, I pinky promise.

Related posts:

Tags: pip, python