Python SSL verification pitfalls: detection and resolution

Unexpected SSL verification failures in Python HTTP clients often appear in production services that call external APIs or micro‑services, where the server presents a self‑signed or expired certificate. This results in request exceptions that can silently stop data ingestion pipelines.

# Example showing the issue
import requests
url = 'https://self-signed.badssl.com/'
try:
    r = requests.get(url, verify=True, timeout=5)
    print(f'Status: {r.status_code}, length: {len(r.content)}')
except requests.exceptions.SSLError as e:
    print('SSL verification failed:', e)
# Quick test with verification disabled
r_no_verify = requests.get(url, verify=False, timeout=5)
print(f'Status (no verify): {r_no_verify.status_code}, length: {len(r_no_verify.content)}')

The server presents a certificate that is not in Python’s trusted root store. By default, the requests library uses the certifi bundle, so any missing or outdated CA certificates cause verification to fail. This follows the TLS RFC 5280 validation model and often catches developers off guard when deploying to environments without the standard bundle. Related factors:

  • Custom OpenSSL builds lacking system CAs
  • Container images that omit the certifi package
  • Corporate MITM proxies injecting their own certificates

To diagnose this in your code:

# Detect SSL verification problems at runtime
import requests
from requests.exceptions import SSLError
url = 'https://example.com/'
try:
    resp = requests.get(url, timeout=5)
    print('Response received, length:', len(resp.content))
except SSLError as exc:
    print('Detected SSL verification issue:', exc)
    # Show which CA bundle is being used
    import certifi
    print('Using certifi bundle at:', certifi.where())

Fixing the Issue

Quick Fix (disable verification for debugging):

import requests
resp = requests.get('https://example.com/', verify=False)
print('Length:', len(resp.content))

When to use: Local testing, temporary debugging. Trade‑off: Disables security checks, vulnerable to MITM.

Best Practice Solution (production‑ready):

import requests, os, logging, certifi
# Prefer the certifi bundle, but allow an override via env var
ca_path = os.getenv('REQUESTS_CA_BUNDLE') or certifi.where()
if not os.path.exists(ca_path):
    logging.error(f'CA bundle not found at {ca_path}')
    raise FileNotFoundError('Missing CA bundle')

try:
    resp = requests.get('https://example.com/', verify=ca_path, timeout=5)
    resp.raise_for_status()
    print('Length:', len(resp.content))
except requests.exceptions.SSLError as e:
    logging.exception('SSL verification failed')
    # Optionally fallback to a known good bundle or abort
    raise

When to use: Production services, CI pipelines, any code that contacts external HTTPS endpoints. Why better: Explicit CA bundle, logging on failure, respects corporate proxy certificates, and never disables verification silently.

The gotcha here is that many container bases ship without the certifi package; adding it back or mounting the host’s CA store fixes the silent failures we saw during a recent micro‑service rollout.

What Doesn’t Work

❌ Setting verify=‘false’ (string) instead of a boolean: requests treats it as True and verification still runs

❌ Catching BaseException and continuing silently: hides the real SSL problem and makes debugging harder

❌ Adding urllib3.disable_warnings() only: suppresses warnings but does not fix the underlying verification failure

  • Disabling verification globally with verify=False
  • Hard‑coding a path to a CA file that may not exist in all environments
  • Catching all exceptions and suppressing SSLError without logging

When NOT to optimize

  • One‑off scripts: Short‑lived utilities where security risk is negligible.
  • Trusted internal network: Calls to services behind a firewall with known certificates.
  • Exploratory notebooks: Quick data pulls during analysis, not production code.
  • Legacy systems with fixed certificates: When the certificate never changes and risk is accepted.

Frequently Asked Questions

Q: Can I safely set verify=False in production?

No. It removes TLS authentication and opens the connection to man‑in‑the‑middle attacks.

Q: Why does requests still fail after installing certifi?

Because the runtime may be using a custom OpenSSL build that ignores certifi’s bundle; set REQUESTS_CA_BUNDLE explicitly.

SSL verification issues are easy to miss until a deployment hits a self‑signed endpoint. By pinning a reliable CA bundle and logging failures, you keep your data pipelines resilient without sacrificing security. Remember, the safest default is to verify; only disable it in controlled, temporary scenarios.

→ Why Python native extension compilation fails in Linux CI → Fix Python dict get vs [] KeyError