Automating & delivering reports
Scheduling & orchestration
cron, Task Scheduler, APScheduler, Airflow, GitHub Actions and Docker.
Start with the full guides
The long-form treatment of scheduling & orchestration, with the reasoning the shorter pages assume.
- OverviewAutomating Reporting Workflows with Python and ExcelBuild an end-to-end Python reporting pipeline: ingest data, transform it with pandas, generate a styled Excel workbook, then schedule and email the result.
- GuideScheduling Python Excel Scripts with CronRun Python Excel report scripts unattended with cron: write a robust generator, decode crontab syntax, invoke the venv Python directly, log, and prevent overlap.
Step-by-step how-tos
One task each, with code you can copy, run and adapt.
- How-toBackfill Excel Reports for Past Dates with PythonRegenerate a daily or monthly report for a range of past dates — after a bug fix, a missed week or a new report that needs history. Make the report take an as-of date instead of reading the clock, give it a date-range command line, skip what already exists, keep going past failures, rerun only what failed, and never email anyone during a backfill.
- How-toChoose a Scheduler for Python Excel ReportsCron, systemd timers, Task Scheduler, launchd, APScheduler, Celery beat, GitHub Actions, Airflow, Prefect, Lambda, Azure Functions, Cloud Run jobs and Kubernetes CronJobs compared for running Excel reports: where each runs, what it does about missed runs, retries, history and alerts, and a scheduler-agnostic script design that keeps the choice reversible.
- How-toOrchestrate Excel Reports with Apache AirflowTurn a scheduled Excel script into a DAG: importable report functions, storage keys through XCom, a deferrable sensor for upstream data, and logical dates that make reruns safe.
- How-toOrchestrate Excel Reports with DagsterRun an Excel reporting pipeline as Dagster software-defined assets: read the source workbook, clean it and write the report as separate assets, pass the output folder in as a resource, partition by month for backfills, validate with asset checks, schedule the job, and test it all in-process with materialize.
- How-toOrchestrate Excel Reports with PrefectTurn a report script into a Prefect flow: tasks for extract, build and publish, automatic retries for the flaky steps, a report date as a parameter so reruns and backfills are one command, failure hooks that alert someone, and a cron schedule served by a long-running process or a work pool — all testable locally.
- How-toRun a Python Excel Report in DockerContainerise a reporting job: a small Dockerfile, pinned dependencies, timezone and locale, mounted volumes for input and output, secrets, and scheduling the container.
- How-toRun a Python Excel Report in GitHub ActionsSchedule a reporting job without a server — a cron workflow, pinned dependencies, secrets, LibreOffice for PDF, artifacts, and the timezone and skew details that bite.
- How-toRun a Report on the Last Business Day of the MonthCron cannot say "the last working day of the month", and a plain month-end offset forgets public holidays. Schedule the job every weekday and let Python decide: find the last business day with numpy and a holiday calendar, handle the nth business day for close reporting, use the business's time zone, and catch up if the server was down on the day.
- How-toRun an Excel Report as a Kubernetes CronJobSchedule a containerised Python Excel report on a Kubernetes cluster: a CronJob manifest with a time zone, no overlapping runs, a deadline for late starts, retries and a hard time limit, credentials from a Secret, output to object storage, clean-up of finished jobs, a strict schema check before applying, and manual test runs.
- How-toRun a Python Excel Report in AWS LambdaBuild an Excel report in an AWS Lambda function and save it to S3 on a schedule — a handler that works in memory, a package that fits Lambda's size limits, memory and timeout settings that suit pandas, an EventBridge Scheduler rule in local time, retries that do not produce duplicate reports, and tests that run without an AWS account.
- How-toRun an Excel Report in Azure FunctionsSchedule a Python Excel report as an Azure Function: a timer trigger with a six-field NCRONTAB schedule, the workbook built in memory and written to Blob Storage through an output binding, a retry policy, time zones and execution time limits, local testing of the handler and its bindings, and deployment with Core Tools.
- How-toRun an Excel Report on Google Cloud Run JobsPackage a Python Excel report as a container and run it on a schedule with Cloud Run jobs and Cloud Scheduler: a slim image, a script that exits with a meaningful status, parallel tasks that split work by CLOUD_RUN_TASK_INDEX, retries and timeouts, a service account instead of keys, and a local test with Docker before anything is deployed.
- How-toRun a Python Excel Script on Windows Task SchedulerRegister a Python Excel report with Windows Task Scheduler: a self-contained generator, schtasks /Create triggers, the venv interpreter, a .bat wrapper, and logging.
- How-toSchedule a Python Excel Script on macOS with launchdRun a Python report on a Mac on a schedule the way macOS intends: a launch agent property list generated and checked with plistlib, StartCalendarInterval for weekly and weekday schedules, the virtualenv's Python by absolute path, logs to files, what happens when the Mac was asleep, privacy permissions for Documents and Desktop, and loading, testing and removing the job with launchctl.
- How-toSchedule an Excel Report with systemd TimersRun a Python Excel report on a Linux server with a systemd service and timer instead of cron — a oneshot service with its own working directory and environment file, calendar expressions checked before deployment, catch-up runs after downtime, logs in the journal, failure status you can query and an OnFailure alert hook.
- How-toSchedule Excel Reports with Celery BeatRun recurring Excel reports inside an application that already uses Celery: a report task with automatic retries and time limits, a beat schedule with crontab entries in the right time zone, the rule that exactly one beat process may run, testing the task with a real in-process worker, and database-backed schedules people can edit.
- How-toSchedule Recurring Excel Reports with APSchedulerSchedule recurring Excel reports inside Python with APScheduler: a BlockingScheduler, cron and interval triggers, misfire handling, overlap prevention, and timezones.
- How-toSchedule Reports Correctly Across Time Zones and DSTKeep an 08:00 Excel report at 08:00 local time all year: why a UTC cron entry drifts by an hour when daylight saving starts and ends, time-zone-aware schedules in systemd, cronie, Kubernetes and Python, the skipped and repeated hours to avoid, a guard that works on UTC-only schedulers, and report periods that are 23 or 25 hours long.
- How-toTrigger an Excel Report from a WebhookBuild an Excel report the moment something happens instead of on a timer: a small FastAPI endpoint that receives a webhook, verifies its HMAC signature and timestamp, ignores duplicate deliveries, answers within milliseconds with 202 Accepted, and builds the workbook in the background — tested end to end with FastAPI's test client.
- How-toWait for Upstream Data Before Running a ReportStop a scheduled Excel report from running on yesterday's data: before building, check that every input is ready — an export file dated today and finished, a marker file, a database load-status row — poll with growing intervals until a deadline, and then either run, or send a clear "data is late" notice instead of a wrong report. Tested with a simulated clock.
- How-toWatch a Folder and Process New Excel Files with PythonProcess each workbook as soon as it lands in a drop folder. Watch the folder with watchdog, ignore Excel's lock and temporary files, wait until a file has finished copying, hand files to a single worker through a queue, move them to processed or failed folders, pick up anything that arrived while the watcher was down, and poll instead on network shares.
- How-toDownload New Excel Files from an SFTP ServerCollect the spreadsheets a partner drops on an SFTP server with paramiko: connect with a pinned host key, list the folder with sizes and modification times, ignore partial uploads and files still being written, download only files not fetched before — or fetched before but changed since — into a temporary name and rename, check each workbook opens, keep a small state file, and test it all against a local SFTP server.
- How-toRead Excel Attachments into a Database NightlyLoad the spreadsheets that arrive by email into a database every night without duplicates or half-loaded files: take the saved attachments from an inbox folder, skip files already loaded by their content hash, validate each workbook's columns and values, write each file in its own transaction with an upsert on the business key, record every load in a log table, move rejected files to a quarantine folder, and report the night's results.
- How-toRefresh an Excel Report from a Database on a ScheduleKeep a workbook current without anyone asking: an incremental query with a watermark, an atomic write nobody can catch half-finished, a freshness stamp readers can see, and a check that the refresh actually ran.
- How-toRun a Report When an Excel File Changes in OneDriveTrigger a report as soon as someone saves a workbook in OneDrive or SharePoint, instead of on a fixed schedule: poll the file's cTag cheaply, use the Graph delta query to find every changed workbook in a folder, or receive webhook notifications and answer Microsoft's validation handshake — with debouncing so a burst of autosaves runs the report once, and the code tested against mocks.
- How-toSend Report Emails at Each Recipient's Local TimeDeliver a report email at 08:00 in each recipient's own time zone from one scheduled Python job: keep an IANA time zone per recipient in the distribution list, run the job hourly, decide who is due with zoneinfo, skip local weekends and holidays, record sends so nobody gets the report twice, and check the logic across daylight saving changes.
- How-toDetect a Scheduled Report That Did Not RunCatch the failure that error handling cannot: a report job that never started. Check each report's output against its schedule — the last time it should have run plus a grace period — and send heartbeat pings to an external monitor that alerts when a ping is late, with start, success and failure signals and a freshness sheet for humans.
- How-toPrevent Overlapping Runs of an Excel Report JobStop a scheduled report from starting while the previous run is still writing the workbook. Take an operating-system lock at the start of the job, skip or wait when another run holds it, record who holds it, use one lock per report so unrelated jobs still run in parallel, and choose a different lock when jobs run on more than one machine.
- How-toSkip a Report Run When the Input File Has Not ChangedStop a scheduled Excel report from rebuilding and resending the same numbers — fingerprint the input workbook so re-saves without real changes are ignored, check file metadata first to avoid reading large files, record the fingerprint only after a successful run, skip files that are still being written, and alert when the input goes stale.
- How-toReturn an Excel File from AWS Lambda Through API GatewayServe a generated Excel workbook as a download from an AWS Lambda function behind API Gateway: build the file in memory, return it base64-encoded with isBase64Encoded, the right headers and filename, the extra binary media type setting a REST API needs, validation of query parameters, the 6 MB response limit and a pre-signed S3 redirect for bigger files, tested locally with event dictionaries and moto.
- How-toRun Excel COM Automation from Task Scheduler Without a Logged-In UserWhy a pywin32 script that drives Excel works when you run it by hand and fails or hangs from Task Scheduler — session 0, the missing systemprofile Desktop folder, DCOM identity, dialogs nobody can click — and the setups that work, plus how to avoid needing Excel at all.
Compare the options
Which tool or approach to pick, and why.
Follow it through a track
Prefer an ordered route? These tracks cover scheduling & orchestration along the way, in the order that builds on itself.
Related topics
Subjects that keep turning up on the same pages as this one, most overlap first.
Scheduling & orchestration is part of Automating & delivering reports.