objgraph circular reference leaks: detection and resolution

Unexpected memory growth in long‑running Python services often stems from circular references that survive garbage collection. This shows up in production logs and container metrics when objects that reference each other are never freed. The silent retention can break downstream processing and inflate heap size.

# Example showing the issue
import gc, objgraph

class Node:
    def __init__(self, name):
        self.name = name
        self.partner = None
    def __repr__(self):
        return f"Node({self.name})"

# create a hidden cycle
n1 = Node('a')
n2 = Node('b')
n1.partner = n2
n2.partner = n1

# drop external refs
n1 = None
n2 = None

print('Before gc:', len(objgraph.by_type('Node')))
# Force a collection
collected = gc.collect()
print('gc.collect() ->', collected)
print('After gc:', len(objgraph.by_type('Node')))
# Output:
# Before gc: 2
# gc.collect() -> 0
# After gc: 2   # the cycle remains alive

Circular references that include objects defining a del method are exempt from automatic collection. The CPython garbage collector only reclaims cycles composed of objects without finalizers, so the leak persists silently. This behavior is documented in the Python “gc” module reference and mirrors standard reference‑counting semantics. Related factors:

  • del methods block cycle collection
  • gc thresholds may delay collection
  • objgraph only reports reachable objects, not those trapped in uncollectable cycles

To diagnose this in your code:

import gc, objgraph, sys

# Enable debugging flags to expose uncollectable objects
gc.set_debug(gc.DEBUG_UNCOLLECTABLE)

# Run your workload, then force a collection
gc.collect()

# Show objects that the collector could not free
if gc.garbage:
    print('Uncollectable objects:', len(gc.garbage))
    objgraph.show_backrefs(gc.garbage, max_depth=3, filename='leak.png')
else:
    print('No uncollectable objects detected')

Fixing the Issue

Quick Fix (force collection)

import gc
gc.collect()  # runs a full collection pass

When to use: Interactive debugging or short scripts where you just need to clear a known leak. Trade‑off: Doesn’t address the root cause; the cycle will re‑appear if the code path runs again.

Best Practice Solution (design‑time prevention)

import weakref, gc

class Cache:
    def __init__(self):
        self._store = weakref.WeakValueDictionary()
    def get(self, key, factory):
        obj = self._store.get(key)
        if obj is None:
            obj = factory()
            self._store[key] = obj
        return obj

# Avoid __del__ in objects that may participate in cycles
class SafeNode:
    def __init__(self, name):
        self.name = name
        self.partner = None
    # No __del__ – rely on weak refs or explicit breakage
    def break_cycle(self):
        self.partner = None

When to use: Production services, long‑running daemons, or any code that repeatedly creates graph‑like structures. Why better: Using weak references prevents cycles from becoming roots, and removing del lets the GC reclaim the objects automatically. Adding explicit break_cycle calls gives deterministic cleanup for legacy classes.

The gotcha we hit in production was a cache object that stored heavy pandas DataFrames; the DataFrames held back‑references to the cache via a callback, forming a hidden cycle that objgraph never flagged because the cache had a __del__ method. Replacing the cache with a WeakValueDictionary eliminated the leak.

What Doesn’t Work

❌ Wrapping the suspect code in a try/except and ignoring MemoryError: it masks the leak and may corrupt later state.

❌ Switching the join to an outer join in pandas to “see more rows”: unrelated to Python circular references and just inflates the dataset.

❌ Calling .copy() on every object to break references: creates new objects but the original cycle still exists, leading to higher memory use.

  • Adding gc.disable() to “speed up” code, which prevents any cycle collection.
  • Assuming objgraph.show_most_common_types() will list uncollectable objects – it only shows reachable ones.
  • Calling del obj without breaking internal references, leaving the cycle intact.

When NOT to optimize

  • One‑off scripts: If the process exits after a few seconds, the OS will reclaim memory anyway.
  • Tiny data structures: Cycles under a few kilobytes rarely impact overall footprint.
  • Prototyping: During early experimentation, the cost of refactoring may outweigh the temporary memory bloat.
  • Read‑only imports: Modules that only import data without creating long‑lived objects generally don’t need cycle‑break logic.

Frequently Asked Questions

Q: Does enabling gc.DEBUG_UNCOLLECTABLE affect performance?

It adds a small overhead to each collection cycle but is negligible in most services.

Q: Can objgraph detect leaks caused by C extensions?

Only if the extension exposes Python objects; otherwise use specialized profilers.

The key insight is that objgraph only visualizes what the GC considers reachable – it won’t magically surface objects stuck behind a del barrier. Once you eliminate finalizers or employ weak references, the collector does its job and your memory stays flat. We learned this the hard way after a nightly job grew by gigabytes until we added an explicit cycle break.

→ Why CPython ref counting vs GC impacts memory → Why Python objects consume excess memory → Fix How to Detect Open File Descriptor Leaks in Python → Why itertools tee can cause memory leak