Itertools tee memory leak: detection and resolution

Memory spikes when using itertools.tee often surface in production pipelines that stream millions of log lines or API events, where the original iterator feeds multiple downstream processors. The buffering behavior of tee can retain all unseen items, quickly exhausting RAM.

# Example showing the issue
import itertools
import sys

def large_generator():
    for i in range(10_000_000):
        yield i

# Create two dependent iterators
it1, it2 = itertools.tee(large_generator())

# Consume only the first iterator fully
for _ in it1:
    pass

# Second iterator is never consumed
print(f'Unconsumed items still buffered: {sys.getsizeof(it2)} bytes')
# Output shows a huge buffer size, indicating memory growth

itertools.tee works by storing items from the original iterator in an internal deque until every tee’d iterator has retrieved them. If one iterator lags, the deque grows without bound, effectively leaking memory. This behavior is documented in the itertools module and mirrors the classic producer‑consumer buffering pattern. Related factors:

  • One iterator consumes far slower than the other
  • Original iterator is large or infinite
  • No explicit limit on the internal buffer

To diagnose this in your code:

# Simple memory check while using tee
import psutil, os, itertools, time

proc = psutil.Process(os.getpid())
orig = itertools.count()
it_a, it_b = itertools.tee(orig)

# Consume a few items from it_a only
for _ in range(1000):
    next(it_a)
    time.sleep(0.001)  # artificial slowdown on the other side

print(f'Current RSS memory: {proc.memory_info().rss / 1024 ** 2:.2f} MB')
# Sudden increase indicates the buffer is growing

Fixing the Issue

Quick Fix (1‑Liner): consume the tee’d iterators together so the buffer never builds up:

for a, b in zip(it1, it2):
    process(a, b)

When to use: debugging or scripts where both streams can be processed in lockstep.

Best Practice Solution (Production‑Ready): avoid tee for large or unbalanced streams; instead, explicitly fan‑out using a generator that yields to multiple consumers or use a bounded queue.

import itertools, queue, threading

source = (i for i in range(10_000_000))
q = queue.Queue(maxsize=10_000)  # bounded buffer prevents unlimited growth

def producer():
    for item in source:
        q.put(item)
    q.put(None)  # sentinel

def consumer(name):
    while True:
        item = q.get()
        if item is None:
            q.put(None)  # pass sentinel to other consumers
            break
        # replace with real work
        if name == 'A':
            handle_a(item)
        else:
            handle_b(item)

threading.Thread(target=producer, daemon=True).start()
threading.Thread(target=consumer, args=('A',), daemon=True).start()
threading.Thread(target=consumer, args=('B',), daemon=True).start()

When to use: production pipelines, multi‑consumer processing, or any scenario where one consumer may fall behind. The bounded queue caps memory usage, and the sentinel cleanly shuts down all workers.

This approach removes the hidden deque, gives you explicit back‑pressure, and lets you monitor queue size to catch bottlenecks early.

What Doesn’t Work

❌ Converting the original iterator to a list before tee: lst = list(gen); it1, it2 = itertools.tee(lst). This loads everything into memory upfront, defeating the purpose of streaming.

❌ Calling itertools.tee repeatedly inside a loop: each call adds a new buffer, rapidly exhausting RAM.

❌ Using itertools.chain to duplicate the iterator: it1 = chain(gen); it2 = chain(gen). This creates two independent generators that both re‑iterate the source, causing duplicate work and hidden memory usage.

  • Using tee on an infinite generator without a consumer catching up.
  • Assuming tee creates independent copies that don’t share a buffer.
  • Neglecting to monitor memory when one consumer is slower than the other.

When NOT to optimize

  • Small datasets: Under a few thousand items, the buffer overhead is negligible.
  • One‑off scripts: Quick analyses or ad‑hoc data pulls where performance isn’t critical.
  • Synchronous processing: When you can guarantee both consumers advance at the same pace.
  • Known one‑to‑many relationships: If you intentionally need every item duplicated, the memory cost is expected.

Frequently Asked Questions

Q: Can I limit tee’s internal buffer size?

No; tee uses an unbounded deque, so you must avoid unbalanced consumption.

Q: Is there a built‑in way to reset the buffer?

Only by exhausting all tee’d iterators; otherwise the buffer persists.

Understanding how itertools.tee buffers data is essential when building scalable streaming applications. By either consuming the duplicated iterators in lockstep or replacing tee with a bounded producer‑consumer pattern, you can prevent hidden memory bloat and keep your pipelines robust. Choose the approach that matches your workload’s size and concurrency requirements.

→ Why Python objects consume excess memory → Fix subprocess communicate deadlock in Python → Why objgraph misses circular reference leaks → Fix Python async concurrency issues