Shell: benchmark the difference between two Git branches with hyperfine

hyperfine is a neat tool for benchmarking commands. If you provide it multiple commands, its output includes a comparison, saying which is the fastest, and by how much.

I often find I want to benchmark before and after some change, by running the same test command on my main Git branch and a feature branch. Here is my hyperfine recipe to do that, using the --prepare option to switch branches before each benchmark:

hyperfine \
    --warmup 1 \
    --command-name before \
    --prepare 'git switch main' \
    '<test-command>' \
    --command-name after \
    --prepare 'git switch <feature-branch>' \
    '<test-command>'

A quick breakdown:

• --warmup 1: run one warmup iteration of <test-command> before recording results. This is useful for ensuring caches are filled, such as file system caches and Python’s __pycache__ directories.
• --command-name: names each benchmark. Without this, they use the command as the name, which provides no information given we are running the same command twice.
• --prepare: runs a command before each benchmark, in this case to switch to the target branches.
• git switch is the new Git command for changing branches, added in Git 2.23 (2019-08-16). I recommend it over the lgeacy git checkout command, which confusingly does several things.
• <test-command>: replace this with the command you want to benchmark.
• <feature-branch>: replace this with the name of the branch you want to compare against your main.

For an example: I recently tried improving test runtime on a client project by altering garbage collection settings. To check for a performance improvement, I benchmarked some tests with pytest like:

$  hyperfine \
    --warmup 1 \
    --command-name before \
    --prepare 'git switch main' \
    'pytest example/tests/test_something.py' \
    --command-name after \
    --prepare 'git switch infrequent_gc' \
    'pytest example/tests/test_something.py'
Benchmark 1: before
  Time (mean ± σ):      4.414 s ±  0.175 s    [User: 3.183 s, System: 0.529 s]
  Range (min … max):    4.273 s …  4.783 s    10 runs

Benchmark 2: after
  Time (mean ± σ):      4.264 s ±  0.064 s    [User: 3.010 s, System: 0.526 s]
  Range (min … max):    4.160 s …  4.347 s    10 runs

Summary
  after ran
    1.04 ± 0.04 times faster than before

The final output shows that the change makes the test command ~4% faster, although with a fairly wide spread (a standard deviation of 0.04). This data gave me encouragement that the change probably helps, so I went on to benchmark the full test suite, which is still running as I write this post…

A couple of other options that you may wish to use:

• -N (--shell=none): run the command directly, without an intermediate shell. This removes some features, like variables or running multiple commands, but makes results more accurate by removing the overhead of launching your shell. (It seems this will be the default from hyperfine 2.0.)
• -r <n>: specify the number of runs. By default, hyperfine picks a number of runs based on the time taken, in order to get a good estimate. For quick tests, I like to pick just -r 1 or -r 3 to see if there’s an obvious win, before I run longer tests.

😸😸😸 Check out my new book on using GitHub effectively, Boost Your GitHub DX! 😸😸😸

One summary email a week, no spam, I pinky promise.

Related posts:

• Shell: tricks for repeatedly running flaky tests
• Git: generate statistics with shortlog
• Git: the basics of git bisect

Tags: commandline, git