Python: fix BrokenPipeError when piping output to other commands

Now there’s one pipe you don’t want to break!

If you’ve written a Python script that outputs a lot of data, then piped that output into another command that only reads part of it, you might have encountered a BrokenPipeError. For example, take this script:

for i in range(10_000):
    print(f"This is line {i} ")

When piped into head -n3, which only reads the first 10 lines, it raises a BrokenPipeError:

$ python example.py | head -n3
This is line 0
This is line 1
This is line 2
Traceback (most recent call last):
  File "/.../example.py", line 2, in <module>
    print(f"This is line {i} ")
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^
BrokenPipeError: [Errno 32] Broken pipe
Exception ignored on flushing sys.stdout:
BrokenPipeError: [Errno 32] Broken pipe

This happens because head stops reading its input after the first 10 lines and closes its input stream, which is the pipe from Python. The operating system sends a SIGPIPE signal to the Python process, indicating that it can no longer write to the pipe because it’s closed. Python translates this signal into a BrokenPipeError when it tries to write the next line to the closed pipe.

The exception occurs a second time as well, when Python shuts down, as it tries to flush the output stream. That’s reported in the final two lines, where the exception recurred but is ignored because it occurs during deletion of the sys.stdout object at shutdown.

To fix these BrokenPipeErrors, the Python documentation has a recommendation. Applying this to our example, we get:

import os
import sys

try:
    for i in range(10_000):
        print(f"This is line {i} ")

    sys.stdout.flush()
except BrokenPipeError:
    # Python flushes standard streams on exit; redirect remaining output
    # to devnull to avoid another BrokenPipeError at shutdown
    devnull = os.open(os.devnull, os.O_WRONLY)
    os.dup2(devnull, sys.stdout.fileno())

Here’s what’s new:

  1. The output loop is wrapped in a try / except PipeError block.
  2. The try branch now flushes sys.stdout, to trigger any BrokenPipeErrors that might occur during the loop.
  3. The except branch handles the BrokenPipeError by redirecting the remaining output to /dev/null. It does this by opening the os.devnull special file (/dev/null on Unix-like systems, or NUL on Windows) for writing, and then using the arcane system call os.dup2() to redirect the standard output stream to this file descriptor.

Python’s documentation also suggests sys.exit(1) in the except block, but I didn’t include that because I think it’s unnecessary. Exiting with a non-zero exit code indicates an error, but there’s not really an error when stopping early. Many programs use a zero exit code when finishing early (cat, grep, rg) while others use exit code 141 (like yes or git log) It doesn’t seem common to use exit code 1, though.

After these changes, the example exits cleanly:

$ python example.py | head -n3
This is line 0
This is line 1
This is line 2

…and we can check the exit code using the pipestatus variable in Bash/Zsh:

$ echo $pipestatus 0 0

This variable lists the exit codes of all commands in the last one, and here they’re both 0.

As a context manager

It’s a bit fiddly to wrap the output section of your script in such a try / except block. Here‘s a version that bundles the logic into a context manager:

import os
import sys
from contextlib import contextmanager
from typing import Generator


@contextmanager
def handle_broken_pipe() -> Generator[None]:
    """
    Prevent BrokenPipeError when the output stream is closed early, such as
    when piping to head.

    https://adamj.eu/tech/2025/07/20/python-fix-brokenpipeerror/
    """
    try:
        yield
        sys.stdout.flush()
    except BrokenPipeError:
        # Python flushes standard streams on exit; redirect remaining output
        # to devnull to avoid another BrokenPipeError at shutdown
        devnull = os.open(os.devnull, os.O_WRONLY)
        os.dup2(devnull, sys.stdout.fileno())

Use it like so:

with handle_broken_pipe():
    for i in range(10_000):
        print(f"This is line {i} ")

I like this approach because it separates the error handling into a neat package. Also, if you put the context manager on your outer main() function, you can scatter output statements throughout your code without worrying about BrokenPipeErrors.

What about stderr?

The above code only handles stdout being closed. It’s also possible, though less common, to pipe stderr to a process that closes it early. For example, we can modify our example to write to stderr:

import sys

for i in range(10_000):
    print(f"This is line {i} ", file=sys.stderr)

With some Zsh redirection, we can pipe stderr to head like so:

$ python example.py 2> >(head -n3)
This is line 0
This is line 1
This is line 2

This time, there’s no exception displayed because Python would print it to stderr, but that stream has been closed early. Still, we can see a failure exit code:

$ echo $? 120

It’s possible to adapt the previous example to handle stderr as well, but I am not sure if it’s a great idea. stderr being closed early is less of a “normal” case, as it prevents normal error reporting from working, so perhaps it’s better to exit with a non-zero status code (at least).

(I’m not sure where the value 120 comes from.)

Fin

May your only broken pipes be virtual!

—Adam

😸😸😸 Check out my new book on using GitHub effectively, Boost Your GitHub DX! 😸😸😸

Subscribe via RSS, Twitter, Mastodon, or email:

One summary email a week, no spam, I pinky promise.

Related posts:

Tags: