Adam Johnson

Home | Blog | Training | Projects | Colophon | Contact

Maintaining Multiple Python Projects With myrepos

2020-04-02

I maintain several open source Python projects, each in its own GitHub repository. I like to keep them all up to date according to a kind of template - similarity increases maintainability.

I have thought about putting them in a monorepo, but this would be unidiomatic and make it harder for contributors to join in. A tool I discovered recently to make working with multiple repositories more like using a monorepo is myrepos.

It allows you to run commands in multiple repositories, group the output by repository, and see a summary of successes/failures. It’s a relatively simple tool - not much more than a complex invocation of GNU parallel - but its integration with VCS’s does sometimes come in useful. It let me replace a flaky shell loop I was copy-pasting between uses.

It installs as the mr command, which has a bunch of subcommands. The most useful one is run, to run arbitrary commands in each repository.

For example, to see git status in all repositories, you can run:

$ mr run git status
mr run: /Users/chainz/Documents/Projects/apig-wsgi
On branch master
Your branch is up to date with 'origin/master'.


nothing to commit, working tree clean


mr run: /Users/chainz/Documents/Projects/django-cors-headers
On branch master
Your branch is up to date with 'origin/master'.


nothing to commit, working tree clean


...

mr run: finished (19 ok)

(Output cropped because I have 19 repositories!)

Configuration

You can install myrepos from most package managers (docs). On macOS I installed it with brew install myrepos.

You then need to register your repositories by cding to their folder and running mr register. This adds them to the default configuration file ~/.mrconfig.

I added one change to my configuration file, to make 8 jobs to run in parallel by default:

[DEFAULT]
jobs = 8

Most of the changes I make are parallel-safe and this gives quite a speed boost. You can switch back to serial for a single command with --jobs 1 before the subcommand.

There is some documentation on the home page and more on the man page.

Example Change: check-manifest

myrepos has saved me a heaps of time and reduced errors for many different changes on my projects. For example, recently I wanted to update the MANIFEST.in files in all my repositories. This controls which files are added to the built Python packages. I also wanted to run the check-manifest tool in my projects’ testing environments, to ensure their MANIFEST.in files were correct.

I started by cd’ing to one of my smaller projects, and figuring out the steps that would do this.

First, I looked through the files and manually made the necessary edits. I then converted them to equivalent shell commands, like rm and sed. I then copied those into a shell script I called add-check-manifest.sh, placed in my ~/scripts folder, which is on my $PATH so myrepos can use it.

I iterated on the script by resetting the repository and rerunning the script, until it worked all the way through. The final version was this:

#!/bin/sh
set -e


rm MANIFEST.in
echo 'global-exclude *.py[cod]
graft tests
prune __pycache__
prune requirements
include HISTORY.rst
include LICENSE
include README.rst
exclude .editorconfig
exclude pyproject.toml
exclude pytest.ini
exclude tox.ini' > MANIFEST.in
cd requirements
echo "check-manifest ; python_version == '3.8.*'" >> requirements.in
sort requirements.in -o requirements.in
./compile.sh
cd ..
sed -E -i '' -e 's/    multilint/    multilint\
    check-manifest/g' tox.ini
tox -r -e py38-codestyle
git switch -c check-manifest
git add --all
git commit -m "Test MANIFEST.in with check-manifest"

Notes:

I knew it wouldn’t work perfectly on all the repositories, but it was much faster than making all these changes manually.

I then ran it across all repositories with mr run add-check-manifest.sh. tox failed on a few, as I expected, so I went and patched them individually.

After each repository was ready, I ran another script I have called pushupr. This pushes a branch and creates a GitHub pull request for it using GitHub’s hub. I then needed CI to pass on all the PR’s, to be sure, so I took a break to go get on with life.

When I came back to my computer later, all the PR’s had passed. I tabbed through all the pull requests and hit the merge buttons.

I could then run a few myrepos commands to pull those locally and clean up the branches:

$ mr run git checkout master
$ mr run git pull --prune
$ mr run git branch -D check-manifest

Writing the script took a little initial investment, but saved me an hour or two of work. In fact, without it, I would probably have never have bothered to do such a mundane task across all my projects. A great example of automation as an enabler!

Fin

I hope this helps you set up maintain your projects,

—Adam


Are your Django project's tests slow? Read Speed Up Your Django Tests now!


Subscribe via RSS, Twitter, or email:

One summary email a week, no spam, I pinky promise.

Related posts:

Tags: commandline