Why pandas fillna does not work on specific columns (and how to fix it)

Missing values in pandas fillna usually persist in real-world datasets from CSV exports or APIs, where the fill method does not handle the specific column type. This leads to pandas preserving NaN values, often silently breaking downstream logic.

Quick Answer

Pandas fillna does not work for specific columns due to incorrect fill method or data type. Fix by using the correct fill strategy or converting data types.

TL;DR

  • Pandas fillna fails when fill method is incorrect
  • Column data type affects fillna behavior
  • Always validate fillna results

Problem Example

import pandas as pd

df = pd.DataFrame({'A': [1, 2, None, 4], 'B': [None, 'b', 'c', None]})
print(f"Before fillna:\n{df}")
df_filled = df.fillna(0)
print(f"After fillna:\n{df_filled}")
# Output shows NaN values still present in column 'B'

Root Cause Analysis

The fill method used does not handle the specific column data type. Pandas performs fill operations based on the column type, and incorrect fill methods result in NaN values persisting. This behavior is consistent with pandas’ type-safe fill operations and often surprises developers transitioning from simple fill approaches to type-aware fills. Related factors:

  • Incorrect fill method for column type
  • Column data type not suitable for fill method
  • No validation on fill results

How to Detect This Issue

# Check for NaN values after fillna
nan_count = df_filled.isnull().sum().sum()
print(f'NaN values after fillna: {nan_count}')

# Show NaN values
if nan_count > 0:
    print(df_filled[df_filled.isnull().any(axis=1)])

Solutions

Solution 1: Use correct fill method

df_filled = df.fillna({'A': 0, 'B': 'unknown'})

Solution 2: Convert data type before fill

df['B'] = df['B'].astype(str)
df_filled = df.fillna(0)

Solution 3: Validate fill results

df_filled = df.fillna(0)
if df_filled.isnull().values.any():
    print('Fill operation did not remove all NaN values')

Why validate Parameter Fails

Using fillna() will not remove NaN values when the fill method is incorrect for the column type. This is not a bug — it is pandas preserving type safety. If the fill operation does not remove all NaN values, check the fill method and column types.

Production-Safe Pattern

df_filled = df.fillna({'A': 0, 'B': 'unknown'})
assert not df_filled.isnull().values.any(), 'Fill operation did not remove all NaN values'

Wrong Fixes That Make Things Worse

❌ Forcing fillna with incorrect methods: This corrupts your data

❌ Ignoring NaN values after fillna: Always validate fill results

❌ Not checking column types before fillna: This leads to type mismatches

Common Mistakes to Avoid

  • Not checking for NaN values after fillna
  • Using incorrect fill method for column type
  • Not validating fill results

Frequently Asked Questions

Q: Why does pandas fillna not remove all NaN values?

When the fill method does not handle the specific column data type, pandas preserves NaN values.

Q: Is this a pandas bug?

No. This behavior follows pandas’ type-safe fill operations. Pandas is correctly preserving type safety.

Q: How do I ensure fillna works for all columns?

Use the correct fill method for each column type or convert data types before fill operations.

→ Fix pandas fillna not working with inplace=True → Fix pandas groupby count includes NaN → Fix pandas outer join creates NaN rows → Why pandas map vs replace give different results

Next Steps

After fixing this issue:

  1. Validate your merge with the validate parameter
  2. Add unit tests to catch similar issues
  3. Review related merge problems above