subprocess communicate deadlock: cause and resolution

Deadlock in subprocess.communicate often appears in production data pipelines or log‑aggregation scripts that pipe large payloads to external tools, where the child process writes enough to stdout or stderr to fill the OS pipe buffer. This blocks the parent, causing the call to hang and silently breaking downstream processing.

# Example showing the issue
import subprocess, os

data = b'A' * 10_000_000  # 10 MiB payload
proc = subprocess.Popen(
    ['gzip'],
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE
)
# This call deadlocks because gzip writes to stdout while we are still feeding stdin
out, err = proc.communicate(input=data)
print(f'Compressed size: {len(out)} bytes')
# The script hangs indefinitely here

The child process writes more data to stdout or stderr than the OS pipe buffer can hold (typically 64 KiB). While the buffer is full, the child blocks, and the parent blocks in communicate waiting for the child to finish reading its stdin. This circular wait creates a deadlock. This behavior follows POSIX pipe semantics and is identical to how shells block on large pipelines. Related factors:

  • Large stdin payloads supplied to communicate()
  • Child process producing output without flushing
  • No asynchronous reading of stdout/stderr

To diagnose this in your code:

# Simple detection using a timeout
import subprocess, sys
try:
    proc = subprocess.Popen(['some_long_running_cmd'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    out, err = proc.communicate(timeout=5)
except subprocess.TimeoutExpired:
    print('Deadlock detected: communicate timed out')
    proc.kill()
    sys.exit(1)

Fixing the Issue

The quickest fix is to avoid feeding huge data via communicate; instead, write the input to a temporary file and let the child read it:

import subprocess, tempfile

data = b'A' * 10_000_000
with tempfile.NamedTemporaryFile(delete=False) as tmp:
    tmp.write(data)
    tmp_path = tmp.name
proc = subprocess.run(['gzip', '-c', tmp_path], capture_output=True)
print(f'Compressed size: {len(proc.stdout)} bytes')

For production‑ready code, read streams asynchronously so the pipe never fills up:

import subprocess, threading, logging

def _reader(pipe, buffer):
    for line in iter(pipe.readline, b''):
        buffer.append(line)
    pipe.close()

data = b'A' * 10_000_000
proc = subprocess.Popen(
    ['gzip'],
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE
)
stdout_buf, stderr_buf = [], []
threading.Thread(target=_reader, args=(proc.stdout, stdout_buf), daemon=True).start()
threading.Thread(target=_reader, args=(proc.stderr, stderr_buf), daemon=True).start()
proc.stdin.write(data)
proc.stdin.close()
proc.wait()
out = b''.join(stdout_buf)
logging.info('Compressed size: %d bytes', len(out))

When to use: The file‑based shortcut is fine for quick scripts; the threaded reader is the safe choice for long‑running services that handle large streams.

Why better: It prevents the pipe buffer from filling, eliminates the deadlock, and logs any unexpected stderr output.

What Doesn’t Work

❌ Using proc.wait() then proc.communicate(): proc.wait() blocks forever if the pipe is full.

❌ Calling proc.stdout.read() without a separate thread: blocks the main thread and deadlocks.

❌ Setting stdout=None to discard output: you lose error information and may mask real failures.

  • Calling proc.wait() before reading stdout/stderr.
  • Using proc.stdout.read() after communicate(), which never returns.
  • Feeding large data via communicate() without considering pipe limits.

When NOT to optimize

  • Tiny inputs: Under a few kilobytes the buffer never fills.
  • One‑off scripts: Quick utilities where occasional hang is acceptable.
  • Controlled environments: When the child process is guaranteed not to produce output.
  • Testing prototypes: During early prototyping you may accept the risk.

Frequently Asked Questions

Q: Can I set a larger pipe buffer to avoid the deadlock?

No; the buffer size is fixed by the OS and cannot be increased from Python.

Q: Is subprocess.run immune to this problem?

run() uses communicate() under the hood, so the same pipe‑buffer limitation applies.

Pipe‑buffer deadlocks are a classic gotcha in systems that glue together command‑line tools. By streaming data or using temporary files you keep the buffers clear and your service responsive. Adding a timeout check gives you early warning before a silent hang propagates downstream.

→ Fix gRPC Python streaming deadlock in production → Fix Python async concurrency issues → Fix multiprocessing shared memory duplicate rows in pandas → Why itertools tee can cause memory leak