Python Excel AutomationPractical guides for developers
Start learning
Guide
Automating Reporting WorkflowsDeep dive

Retry a Failed Excel Report Job in Python

Retry the failures worth retrying — exponential backoff with jitter, a decorator you can reuse, per-step retry boundaries, idempotent writes and a cap that ends in a real alert.

Retries turn a class of failures into a non-event: the share blinked, the file was open for thirty seconds, the mail server was busy. Applied indiscriminately they do the opposite, hiding real problems behind four identical failures and a delayed alert. This guide, part of Error Handling and Logging in Excel Automation, covers what to retry, how to wait, and where the retry boundary belongs.

Deciding whether an error is worth retrying Ask whether the same call could succeed unchanged in ten seconds. If yes — a lock, a timeout, a temporary network failure — retry with backoff. If no — a missing column, a validation failure, bad credentials — stop and report, because every attempt will fail identically. Could the identical call succeed in ten seconds, unchanged? yes no retry with backoff PermissionError · TimeoutError ConnectionError · SMTP 4xx bounded attempts, then alert stop and report KeyError · ValueError auth rejected · SMTP 5xx retrying only delays the alert

Prerequisites

Bash
pip install pandas openpyxl

Everything else is standard library: time, random, functools and logging.

A reusable retry decorator

Put the policy in one place so every call site does not reinvent it:

Python
import functools
import logging
import random
import time

log = logging.getLogger("report")

TRANSIENT = (PermissionError, TimeoutError, ConnectionError, OSError)

def retry(attempts=4, base_delay=2.0, max_delay=30.0, exceptions=TRANSIENT):
    """Retry a callable on transient errors with exponential backoff and jitter."""
    def decorator(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            for attempt in range(1, attempts + 1):
                try:
                    return fn(*args, **kwargs)
                except exceptions as exc:
                    if attempt == attempts:
                        log.error("%s failed after %d attempt(s): %s",
                                  fn.__name__, attempts, exc)
                        raise
                    delay = min(base_delay * (2 ** (attempt - 1)), max_delay)
                    delay *= 0.5 + random.random()          # jitter: 50%–150%
                    log.warning("%s attempt %d/%d failed (%s) — retrying in %.1fs",
                                fn.__name__, attempt, attempts, exc, delay)
                    time.sleep(delay)
        return wrapper
    return decorator

functools.wraps keeps the wrapped function's name and docstring, which matters because the log messages use fn.__name__. The jitter multiplier spreads retries that would otherwise collide — two regional jobs that both failed on the same locked file at 06:00 would, without it, retry at exactly the same instants and fail together every time.

OSError is in the transient list because PermissionError and ConnectionError are both subclasses of it, and because network-share hiccups surface as bare OSError on some platforms. That breadth is deliberate here, but it is also why the decorator takes an exceptions argument: narrow it wherever you can be specific.

Apply it at the right boundary

The unit of retry should be the smallest step that can fail on its own — not the whole pipeline:

Python
from pathlib import Path

import pandas as pd
from openpyxl import load_workbook

@retry(attempts=4)
def read_source(path):
    return pd.read_excel(path, sheet_name="Orders")

@retry(attempts=5, base_delay=2.0)
def save_report(wb, target):
    wb.save(target)
    return Path(target)

@retry(attempts=3, base_delay=5.0)
def send_report(path, to):
    deliver(path, to)                    # your SMTP call

Wrapping the whole job instead would re-read the source, recompute everything and possibly re-send an email that already went out. Small boundaries also let each step carry its own policy: five patient attempts for a save that is probably blocked by a colleague, three for an SMTP send where a delay of a minute is already unhelpful.

Retrying one step versus retrying the whole job Retrying the whole pipeline repeats the read and the transform and risks sending a duplicate email. Retrying only the failing save repeats a few seconds of work and has no side effects. retry the whole job read transform save ✗ email all four repeat — duplicate email risk retry the failing step read transform save ↻ email only the save repeats — no side effects Rule: the retry boundary is the smallest independently repeatable step anything with a side effect gets its own boundary and its own guard

Do not retry a data failure

The mistake that costs the most is catching too much. A KeyError for a missing column, a ValueError from a validation gate — these are facts about the file, and four attempts produce four identical failures and a four-minute delay before anyone is told:

Python
class DataProblem(Exception):
    """Raised for anything that will fail identically on a second attempt."""

@retry(attempts=4)                       # only transient errors are caught
def load_and_check(path):
    df = pd.read_excel(path, sheet_name="Orders")     # may raise PermissionError → retry
    required = {"Order_ID", "Region", "Quantity"}
    missing = required - set(df.columns)
    if missing:
        raise DataProblem(f"missing column(s): {sorted(missing)}")   # not retried
    return df

Because DataProblem is not in the transient tuple, it propagates immediately on the first attempt. Defining your own exception type for this is worth the three lines: it documents the intent, and it stops a future edit to the transient tuple accidentally making validation failures retryable.

Make retried steps safe to repeat

A retry runs the same code twice, so any step with a side effect needs a guard. For file writes, that means writing to a temporary name and renaming; for deliveries, a marker:

Python
import os
from pathlib import Path

@retry(attempts=5)
def atomic_save(wb, target):
    target = Path(target)
    temp = target.with_name(f".{target.stem}.tmp{target.suffix}")
    wb.save(temp)                 # a failed attempt leaves only a temp file
    os.replace(temp, target)
    return target

def send_once(path, to, marker_dir="state"):
    marker = Path(marker_dir) / f"{Path(path).stem}.sent"
    if marker.exists():
        log.info("already delivered %s at %s", path, marker.read_text().strip())
        return False
    send_report(path, to)         # itself retried
    Path(marker_dir).mkdir(exist_ok=True)
    marker.write_text(str(Path(path).stat().st_mtime))
    return True

The marker turns "send the report" into an idempotent operation, which matters as soon as anything above it retries. Storing the file's modification time in the marker also gives you a cheap way to detect a regenerated report that genuinely should be sent again.

Give up loudly

A retry policy that ends in silence is worse than no retries at all. The final attempt should produce an alert that names the step, the target and the last error:

Python
def run():
    try:
        df = load_and_check(SOURCE)
        report = build(df)
        atomic_save(report, TARGET)
        send_once(TARGET, "finance@example.com")
        return 0
    except DataProblem as exc:
        alert("Report blocked — data problem", f"{SOURCE}: {exc}")
        return 1
    except TRANSIENT as exc:
        alert("Report failed after retries", f"{type(exc).__name__}: {exc}")
        return 2

Two exit codes, two different messages, and both say what a human should do next. The data problem needs the file fixed; the transient failure needs someone to check whether the share is still down.

Choosing the delay curve

The shape of the wait matters as much as the number of attempts. Three curves cover almost every case:

Fixed, linear and exponential backoff compared A fixed delay retries at the same short interval and gives up quickly. Linear backoff grows steadily. Exponential backoff waits far longer by the last attempt, covering a slow-clearing problem within a bounded total time. fixed · 2s each 8s total good for a fast flap gives up too soon on a lock linear · 2, 4, 6, 8s 20s total predictable, easy to reason about a middle ground exponential · 2, 4, 8, 16s 30s total covers a slow-clearing lock the sensible default

Exponential is the default for a reason: the early attempts are cheap enough to catch a momentary blip, and the later ones wait long enough to outlast a colleague reading a spreadsheet. Cap it — without a max_delay a sixth attempt waits over a minute, by which point an alert would have been more useful than another try.

Common pitfalls and gotchas

SymptomCauseFix
Same failure four times, alert delayedRetrying a data problemKeep validation errors out of the transient tuple
Duplicate emails after a retryRetry boundary wrapped the deliveryGuard delivery with a marker; retry the send alone
Retries all collide againNo jitter in the delayRandomise the wait by 50–150%
Truncated output fileRetried a save that writes in placeWrite to a temp file and os.replace
Job hangs for minutesToo many attempts or no max_delayThree to five attempts, cap the delay
Retry loop swallows the errorexcept without a re-raise on the last attemptRe-raise when attempts are exhausted
Wrapped function name lost in logsMissing functools.wrapsAdd it to the decorator

Performance and scale notes

Retries cost only waiting, and the waiting is bounded by design: four attempts with a two-second base and a thirty-second cap is under a minute in the worst case. The cost that surprises people is repeated work — a retry boundary around an expensive step re-does that step, so keep the boundary tight and put it after the expensive part where you can. In a batch job, retry per unit rather than per batch, so one locked regional file does not re-run eleven successful reports.

Record every attempt

A retry that succeeds on the third try should still leave a trace. Logging each attempt with its error and delay makes a pattern visible — the same file locked every Monday, an SMTP server that always needs two tries — and that pattern is usually the real problem worth fixing. Retries that succeed silently hide exactly the information that would let you stop needing them.

Conclusion

Retry the failures that time can fix and nothing else. Put the policy in one decorator, wrap the smallest independently repeatable step, add jitter so parallel jobs do not collide, and make every retried step idempotent — atomic writes for files, markers for deliveries. Cap the attempts at three to five, and make the final failure an alert that names the step and the reason, so the retries buy resilience rather than delay.

Frequently asked questions

Which errors should be retried? Ones caused by timing rather than by content — a locked file, an unreachable share, an SMTP timeout. A missing column or a failed validation will fail identically every time.

Why add jitter to the delay? So several jobs that failed at the same moment do not retry in lockstep. Randomising the wait spreads the load and avoids a repeated collision on the same resource.

Should I retry the whole job or one step? One step. Re-running an entire pipeline to recover from a locked save wastes the work already done and can duplicate side effects such as emails.

How many attempts are sensible? Three to five, with exponential backoff, so the total wait is tens of seconds. Beyond that you are delaying an alert someone needs to see.

Up to the parent guide:

Related guides: