[{"data":1,"prerenderedAt":2607},["ShallowReactive",2],{"doc:\u002Fautomating-reporting-workflows\u002Fscheduling-python-excel-scripts-with-cron\u002Forchestrate-excel-reports-with-apache-airflow":3,"surround:\u002Fautomating-reporting-workflows\u002Fscheduling-python-excel-scripts-with-cron\u002Forchestrate-excel-reports-with-apache-airflow":2600},{"id":4,"title":5,"body":6,"dateModified":2573,"datePublished":2573,"description":2574,"extension":2575,"faq":2576,"meta":2585,"navigation":260,"path":2593,"seo":2594,"slug":2596,"stem":2597,"type":2598,"__hash__":2599},"docs\u002Fautomating-reporting-workflows\u002Fscheduling-python-excel-scripts-with-cron\u002Forchestrate-excel-reports-with-apache-airflow\u002Findex.md","Orchestrate Excel Reports with Apache Airflow",{"type":7,"value":8,"toc":2558},"minimark",[9,19,145,150,184,191,195,198,613,621,625,1340,1350,1354,1451,1461,1531,1534,1538,1541,1675,1685,1689,1700,1781,1788,1792,1795,1992,1999,2002,2053,2057,2060,2177,2180,2184,2282,2286,2365,2368,2371,2469,2476,2480,2483,2487,2494,2500,2506,2512,2516,2554],[10,11,12,13,18],"p",{},"A report that is one script on a timer belongs in cron. A report that has to wait for an upstream\nload, build several outputs, validate them and then distribute them is a pipeline, and pipelines\nbenefit from something that tracks dependencies, retries individual steps and keeps a history you can\nlook at. This guide, part of\n",[14,15,17],"a",{"href":16},"\u002Fautomating-reporting-workflows\u002Fscheduling-python-excel-scripts-with-cron\u002F","Scheduling Python Excel Scripts with Cron",",\nputs an Excel reporting job into an Airflow DAG without letting the orchestration swallow the logic.",[20,21,29,30,29,34,29,38,29,45,29,55,29,62,29,70,29,76,29,81,29,86,29,91,29,97,29,101,29,106,29,109,29,111,29,114,29,118,29,123,29,128,29,131,29,133,29,136,29,140],"svg",{"viewBox":22,"role":23,"ariaLabelledBy":24,"xmlns":27,"style":28},"0 0 760 264","img",[25,26],"af-when-t","af-when-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;max-width:760px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif","\n  ",[31,32,33],"title",{"id":25},"Cron or an orchestrator",[35,36,37],"desc",{"id":26},"A single script on a timer is a cron job, a chain of dependent steps with retries and backfills is a DAG, and an in-process schedule inside a long-running service is APScheduler.",[39,40],"rect",{"x":41,"y":41,"width":42,"height":43,"fill":44},"0","760","264","#ffffff",[39,46],{"x":47,"y":48,"width":49,"height":50,"rx":51,"fill":52,"stroke":53,"style":54},"230.0","26","300","56","12","#ebebfd","var(--brand,#5b5cf0)","stroke-width:2px",[56,57,61],"text",{"x":58,"y":59,"style":60},"380.0","60","font-size:13.5px;font-weight:700;fill:var(--brand-strong,#4338ca);text-anchor:middle","What shape is the job?",[39,63],{"x":64,"y":65,"width":66,"height":67,"rx":51,"fill":68,"stroke":69,"style":54},"18.0","148","226.7","84","#d9f4f1","var(--teal,#0f9488)",[56,71,75],{"x":72,"y":73,"style":74},"131.35","182","font-size:13px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","cron",[56,77,80],{"x":72,"y":78,"style":79},"204","font-size:11.5px;font-weight:400;fill:var(--muted,#5b6780);text-anchor:middle","keep it simple",[82,83],"line",{"x1":72,"y1":84,"x2":72,"y2":85,"stroke":53,"style":54},"87","141",[87,88],"polygon",{"points":89,"fill":90},"131.35,141 126.35,132 136.35,132","#5b5cf0",[56,92,96],{"x":93,"y":94,"style":95},"137.35","116","font-size:11px;font-weight:600;fill:var(--muted,#5b6780);text-anchor:start","one script",[39,98],{"x":99,"y":65,"width":66,"height":67,"rx":51,"fill":100,"stroke":53,"style":54},"266.7","#f0f4ff",[56,102,105],{"x":103,"y":73,"style":104},"380.04999999999995","font-size:13px;font-weight:700;fill:var(--brand-strong,#4338ca);text-anchor:middle","Airflow",[56,107,108],{"x":103,"y":78,"style":79},"retries and history",[82,110],{"x1":103,"y1":84,"x2":103,"y2":85,"stroke":53,"style":54},[87,112],{"points":113,"fill":90},"380.04999999999995,141 375.04999999999995,132 385.04999999999995,132",[56,115,117],{"x":116,"y":94,"style":95},"386.04999999999995","dependent steps",[39,119],{"x":120,"y":65,"width":66,"height":67,"rx":51,"fill":121,"stroke":122,"style":54},"515.4","#fdefd8","var(--gold,#b4740a)",[56,124,127],{"x":125,"y":73,"style":126},"628.75","font-size:13px;font-weight:700;fill:var(--gold-ink,#7a4e06);text-anchor:middle","APScheduler",[56,129,130],{"x":125,"y":78,"style":79},"no extra infrastructure",[82,132],{"x1":125,"y1":84,"x2":125,"y2":85,"stroke":53,"style":54},[87,134],{"points":135,"fill":90},"628.75,141 623.75,132 633.75,132",[56,137,139],{"x":138,"y":94,"style":95},"634.75","inside a service",[56,141,144],{"x":58,"y":142,"style":143},"252","font-size:12.5px;font-weight:400;fill:var(--muted,#5b6780);text-anchor:middle","most reports never outgrow the first column",[146,147,149],"h2",{"id":148},"prerequisites","Prerequisites",[151,152,157],"pre",{"className":153,"code":154,"language":155,"meta":156,"style":156},"language-bash shiki shiki-themes github-light github-dark-high-contrast","pip install \"apache-airflow==2.10.*\" pandas openpyxl xlsxwriter\n","bash","",[158,159,160],"code",{"__ignoreMap":156},[161,162,164,168,172,175,178,181],"span",{"class":82,"line":163},1,[161,165,167],{"class":166},"sMTad","pip",[161,169,171],{"class":170},"srMev"," install",[161,173,174],{"class":170}," \"apache-airflow==2.10.*\"",[161,176,177],{"class":170}," pandas",[161,179,180],{"class":170}," openpyxl",[161,182,183],{"class":170}," xlsxwriter\n",[10,185,186,187,190],{},"An Airflow instance — Astronomer, MWAA, a Helm deployment or a local ",[158,188,189],{},"airflow standalone"," — and\nsomewhere shared to put files, which the examples below treat as S3.",[146,192,194],{"id":193},"keep-the-report-logic-out-of-the-dag","Keep the report logic out of the DAG",[10,196,197],{},"The most important design decision is made before any Airflow code: the report should be ordinary\nfunctions in an ordinary module, importable and testable without an Airflow installation. The DAG\nthen becomes a thin description of order and scheduling.",[151,199,203],{"className":200,"code":201,"language":202,"meta":156,"style":156},"language-python shiki shiki-themes github-light github-dark-high-contrast","# reports\u002Fregional.py — no Airflow imports anywhere in this file\nfrom datetime import date\nfrom pathlib import Path\nimport pandas as pd\n\ndef build(run_date: date, source: Path, out_dir: Path) -> Path:\n    frame = pd.read_excel(source, engine=\"calamine\")\n    summary = frame.groupby(\"Region\", as_index=False)[\"Revenue\"].sum()\n    target = out_dir \u002F f\"regional-{run_date:%Y-%m-%d}.xlsx\"\n    with pd.ExcelWriter(target, engine=\"xlsxwriter\") as writer:\n        summary.to_excel(writer, sheet_name=\"Summary\", index=False)\n        frame.to_excel(writer, sheet_name=\"Detail\", index=False)\n    return target\n\ndef validate(path: Path, minimum_rows: int = 1) -> dict:\n    summary = pd.read_excel(path, sheet_name=\"Summary\")\n    if len(summary) \u003C minimum_rows:\n        raise ValueError(f\"{path.name}: summary has {len(summary)} rows\")\n    return {\"rows\": len(summary), \"total\": float(summary[\"Revenue\"].sum())}\n","python",[158,204,205,211,228,241,255,262,275,299,334,384,408,433,456,465,470,499,517,535,578],{"__ignoreMap":156},[161,206,207],{"class":82,"line":163},[161,208,210],{"class":209},"s-wDw","# reports\u002Fregional.py — no Airflow imports anywhere in this file\n",[161,212,214,218,222,225],{"class":82,"line":213},2,[161,215,217],{"class":216},"s-kum","from",[161,219,221],{"class":220},"skGVy"," datetime ",[161,223,224],{"class":216},"import",[161,226,227],{"class":220}," date\n",[161,229,231,233,236,238],{"class":82,"line":230},3,[161,232,217],{"class":216},[161,234,235],{"class":220}," pathlib ",[161,237,224],{"class":216},[161,239,240],{"class":220}," Path\n",[161,242,244,246,249,252],{"class":82,"line":243},4,[161,245,224],{"class":216},[161,247,248],{"class":220}," pandas ",[161,250,251],{"class":216},"as",[161,253,254],{"class":220}," pd\n",[161,256,258],{"class":82,"line":257},5,[161,259,261],{"emptyLinePlaceholder":260},true,"\n",[161,263,265,268,272],{"class":82,"line":264},6,[161,266,267],{"class":216},"def",[161,269,271],{"class":270},"s_Opv"," build",[161,273,274],{"class":220},"(run_date: date, source: Path, out_dir: Path) -> Path:\n",[161,276,278,281,284,287,291,293,296],{"class":82,"line":277},7,[161,279,280],{"class":220},"    frame ",[161,282,283],{"class":216},"=",[161,285,286],{"class":220}," pd.read_excel(source, ",[161,288,290],{"class":289},"sa561","engine",[161,292,283],{"class":216},[161,294,295],{"class":170},"\"calamine\"",[161,297,298],{"class":220},")\n",[161,300,302,305,307,310,313,316,319,321,325,328,331],{"class":82,"line":301},8,[161,303,304],{"class":220},"    summary ",[161,306,283],{"class":216},[161,308,309],{"class":220}," frame.groupby(",[161,311,312],{"class":170},"\"Region\"",[161,314,315],{"class":220},", ",[161,317,318],{"class":289},"as_index",[161,320,283],{"class":216},[161,322,324],{"class":323},"sP0c6","False",[161,326,327],{"class":220},")[",[161,329,330],{"class":170},"\"Revenue\"",[161,332,333],{"class":220},"].sum()\n",[161,335,337,340,342,345,348,351,354,358,361,364,367,370,373,375,378,381],{"class":82,"line":336},9,[161,338,339],{"class":220},"    target ",[161,341,283],{"class":216},[161,343,344],{"class":220}," out_dir ",[161,346,347],{"class":216},"\u002F",[161,349,350],{"class":216}," f",[161,352,353],{"class":170},"\"regional-",[161,355,357],{"class":356},"sSjpA","{",[161,359,360],{"class":220},"run_date:",[161,362,363],{"class":216},"%",[161,365,366],{"class":220},"Y",[161,368,369],{"class":216},"-%",[161,371,372],{"class":220},"m",[161,374,369],{"class":216},[161,376,377],{"class":220},"d",[161,379,380],{"class":356},"}",[161,382,383],{"class":170},".xlsx\"\n",[161,385,387,390,393,395,397,400,403,405],{"class":82,"line":386},10,[161,388,389],{"class":216},"    with",[161,391,392],{"class":220}," pd.ExcelWriter(target, ",[161,394,290],{"class":289},[161,396,283],{"class":216},[161,398,399],{"class":170},"\"xlsxwriter\"",[161,401,402],{"class":220},") ",[161,404,251],{"class":216},[161,406,407],{"class":220}," writer:\n",[161,409,411,414,417,419,422,424,427,429,431],{"class":82,"line":410},11,[161,412,413],{"class":220},"        summary.to_excel(writer, ",[161,415,416],{"class":289},"sheet_name",[161,418,283],{"class":216},[161,420,421],{"class":170},"\"Summary\"",[161,423,315],{"class":220},[161,425,426],{"class":289},"index",[161,428,283],{"class":216},[161,430,324],{"class":323},[161,432,298],{"class":220},[161,434,436,439,441,443,446,448,450,452,454],{"class":82,"line":435},12,[161,437,438],{"class":220},"        frame.to_excel(writer, ",[161,440,416],{"class":289},[161,442,283],{"class":216},[161,444,445],{"class":170},"\"Detail\"",[161,447,315],{"class":220},[161,449,426],{"class":289},[161,451,283],{"class":216},[161,453,324],{"class":323},[161,455,298],{"class":220},[161,457,459,462],{"class":82,"line":458},13,[161,460,461],{"class":216},"    return",[161,463,464],{"class":220}," target\n",[161,466,468],{"class":82,"line":467},14,[161,469,261],{"emptyLinePlaceholder":260},[161,471,473,475,478,481,484,487,490,493,496],{"class":82,"line":472},15,[161,474,267],{"class":216},[161,476,477],{"class":270}," validate",[161,479,480],{"class":220},"(path: Path, minimum_rows: ",[161,482,483],{"class":323},"int",[161,485,486],{"class":216}," =",[161,488,489],{"class":323}," 1",[161,491,492],{"class":220},") -> ",[161,494,495],{"class":323},"dict",[161,497,498],{"class":220},":\n",[161,500,502,504,506,509,511,513,515],{"class":82,"line":501},16,[161,503,304],{"class":220},[161,505,283],{"class":216},[161,507,508],{"class":220}," pd.read_excel(path, ",[161,510,416],{"class":289},[161,512,283],{"class":216},[161,514,421],{"class":170},[161,516,298],{"class":220},[161,518,520,523,526,529,532],{"class":82,"line":519},17,[161,521,522],{"class":216},"    if",[161,524,525],{"class":323}," len",[161,527,528],{"class":220},"(summary) ",[161,530,531],{"class":216},"\u003C",[161,533,534],{"class":220}," minimum_rows:\n",[161,536,538,541,544,547,550,553,555,558,560,563,565,568,571,573,576],{"class":82,"line":537},18,[161,539,540],{"class":216},"        raise",[161,542,543],{"class":323}," ValueError",[161,545,546],{"class":220},"(",[161,548,549],{"class":216},"f",[161,551,552],{"class":170},"\"",[161,554,357],{"class":356},[161,556,557],{"class":220},"path.name",[161,559,380],{"class":356},[161,561,562],{"class":170},": summary has ",[161,564,357],{"class":356},[161,566,567],{"class":323},"len",[161,569,570],{"class":220},"(summary)",[161,572,380],{"class":356},[161,574,575],{"class":170}," rows\"",[161,577,298],{"class":220},[161,579,581,583,586,589,592,594,597,600,602,605,608,610],{"class":82,"line":580},19,[161,582,461],{"class":216},[161,584,585],{"class":220}," {",[161,587,588],{"class":170},"\"rows\"",[161,590,591],{"class":220},": ",[161,593,567],{"class":323},[161,595,596],{"class":220},"(summary), ",[161,598,599],{"class":170},"\"total\"",[161,601,591],{"class":220},[161,603,604],{"class":323},"float",[161,606,607],{"class":220},"(summary[",[161,609,330],{"class":170},[161,611,612],{"class":220},"].sum())}\n",[10,614,615,616,620],{},"Everything Airflow-specific stays in the DAG file. That separation is what lets the report be run\nfrom a laptop during development and covered by the tests in\n",[14,617,619],{"href":618},"\u002Fautomating-reporting-workflows\u002Ftesting-and-packaging-excel-automation-scripts\u002Ftest-excel-output-with-pytest\u002F","Test Excel Output with pytest",".",[146,622,624],{"id":623},"the-dag","The DAG",[151,626,628],{"className":200,"code":627,"language":202,"meta":156,"style":156},"from datetime import datetime, timedelta\nfrom pathlib import Path\nimport pendulum\nfrom airflow.decorators import dag, task\nfrom airflow.providers.amazon.aws.hooks.s3 import S3Hook\n\nfrom reports import regional\n\nBUCKET = \"reporting-artifacts\"\n\n@dag(\n    dag_id=\"regional_revenue_report\",\n    schedule=\"0 6 * * 1-5\",\n    start_date=pendulum.datetime(2026, 8, 1, tz=\"Europe\u002FLondon\"),\n    catchup=False,\n    max_active_runs=1,\n    default_args={\"retries\": 2, \"retry_delay\": timedelta(minutes=5)},\n    tags=[\"excel\", \"reporting\"],\n)\ndef regional_revenue_report():\n\n    @task\n    def build(logical_date=None) -> str:\n        hook = S3Hook()\n        local_dir = Path(\"\u002Ftmp\u002Freports\")\n        local_dir.mkdir(exist_ok=True)\n        source = Path(hook.download_file(\n            key=f\"exports\u002Forders-{logical_date:%Y-%m-%d}.xlsx\",\n            bucket_name=BUCKET, local_path=str(local_dir), preserve_file_name=True,\n        ))\n        built = regional.build(logical_date.date(), source, local_dir)\n        key = f\"reports\u002F{built.name}\"\n        hook.load_file(filename=str(built), key=key, bucket_name=BUCKET, replace=True)\n        return key\n\n    @task\n    def check(key: str) -> dict:\n        hook = S3Hook()\n        local = Path(hook.download_file(key=key, bucket_name=BUCKET,\n                                        local_path=\"\u002Ftmp\", preserve_file_name=True))\n        return regional.validate(local)\n\n    @task\n    def distribute(key: str, stats: dict) -> None:\n        print(f\"sending {key}: {stats['rows']} regions, total {stats['total']:,.0f}\")\n\n    built_key = build()\n    distribute(built_key, check(built_key))\n\nregional_revenue_report()\n",[158,629,630,641,651,658,670,682,686,698,702,712,716,724,737,749,785,796,807,843,864,868,879,884,890,913,924,940,956,967,1004,1035,1041,1052,1075,1117,1126,1131,1136,1155,1164,1189,1211,1219,1224,1229,1252,1307,1312,1323,1329,1334],{"__ignoreMap":156},[161,631,632,634,636,638],{"class":82,"line":163},[161,633,217],{"class":216},[161,635,221],{"class":220},[161,637,224],{"class":216},[161,639,640],{"class":220}," datetime, timedelta\n",[161,642,643,645,647,649],{"class":82,"line":213},[161,644,217],{"class":216},[161,646,235],{"class":220},[161,648,224],{"class":216},[161,650,240],{"class":220},[161,652,653,655],{"class":82,"line":230},[161,654,224],{"class":216},[161,656,657],{"class":220}," pendulum\n",[161,659,660,662,665,667],{"class":82,"line":243},[161,661,217],{"class":216},[161,663,664],{"class":220}," airflow.decorators ",[161,666,224],{"class":216},[161,668,669],{"class":220}," dag, task\n",[161,671,672,674,677,679],{"class":82,"line":257},[161,673,217],{"class":216},[161,675,676],{"class":220}," airflow.providers.amazon.aws.hooks.s3 ",[161,678,224],{"class":216},[161,680,681],{"class":220}," S3Hook\n",[161,683,684],{"class":82,"line":264},[161,685,261],{"emptyLinePlaceholder":260},[161,687,688,690,693,695],{"class":82,"line":277},[161,689,217],{"class":216},[161,691,692],{"class":220}," reports ",[161,694,224],{"class":216},[161,696,697],{"class":220}," regional\n",[161,699,700],{"class":82,"line":301},[161,701,261],{"emptyLinePlaceholder":260},[161,703,704,707,709],{"class":82,"line":336},[161,705,706],{"class":323},"BUCKET",[161,708,486],{"class":216},[161,710,711],{"class":170}," \"reporting-artifacts\"\n",[161,713,714],{"class":82,"line":386},[161,715,261],{"emptyLinePlaceholder":260},[161,717,718,721],{"class":82,"line":410},[161,719,720],{"class":270},"@dag",[161,722,723],{"class":220},"(\n",[161,725,726,729,731,734],{"class":82,"line":435},[161,727,728],{"class":289},"    dag_id",[161,730,283],{"class":216},[161,732,733],{"class":170},"\"regional_revenue_report\"",[161,735,736],{"class":220},",\n",[161,738,739,742,744,747],{"class":82,"line":458},[161,740,741],{"class":289},"    schedule",[161,743,283],{"class":216},[161,745,746],{"class":170},"\"0 6 * * 1-5\"",[161,748,736],{"class":220},[161,750,751,754,756,759,762,764,767,769,772,774,777,779,782],{"class":82,"line":467},[161,752,753],{"class":289},"    start_date",[161,755,283],{"class":216},[161,757,758],{"class":220},"pendulum.datetime(",[161,760,761],{"class":323},"2026",[161,763,315],{"class":220},[161,765,766],{"class":323},"8",[161,768,315],{"class":220},[161,770,771],{"class":323},"1",[161,773,315],{"class":220},[161,775,776],{"class":289},"tz",[161,778,283],{"class":216},[161,780,781],{"class":170},"\"Europe\u002FLondon\"",[161,783,784],{"class":220},"),\n",[161,786,787,790,792,794],{"class":82,"line":472},[161,788,789],{"class":289},"    catchup",[161,791,283],{"class":216},[161,793,324],{"class":323},[161,795,736],{"class":220},[161,797,798,801,803,805],{"class":82,"line":501},[161,799,800],{"class":289},"    max_active_runs",[161,802,283],{"class":216},[161,804,771],{"class":323},[161,806,736],{"class":220},[161,808,809,812,814,816,819,821,824,826,829,832,835,837,840],{"class":82,"line":519},[161,810,811],{"class":289},"    default_args",[161,813,283],{"class":216},[161,815,357],{"class":220},[161,817,818],{"class":170},"\"retries\"",[161,820,591],{"class":220},[161,822,823],{"class":323},"2",[161,825,315],{"class":220},[161,827,828],{"class":170},"\"retry_delay\"",[161,830,831],{"class":220},": timedelta(",[161,833,834],{"class":289},"minutes",[161,836,283],{"class":216},[161,838,839],{"class":323},"5",[161,841,842],{"class":220},")},\n",[161,844,845,848,850,853,856,858,861],{"class":82,"line":537},[161,846,847],{"class":289},"    tags",[161,849,283],{"class":216},[161,851,852],{"class":220},"[",[161,854,855],{"class":170},"\"excel\"",[161,857,315],{"class":220},[161,859,860],{"class":170},"\"reporting\"",[161,862,863],{"class":220},"],\n",[161,865,866],{"class":82,"line":580},[161,867,298],{"class":220},[161,869,871,873,876],{"class":82,"line":870},20,[161,872,267],{"class":216},[161,874,875],{"class":270}," regional_revenue_report",[161,877,878],{"class":220},"():\n",[161,880,882],{"class":82,"line":881},21,[161,883,261],{"emptyLinePlaceholder":260},[161,885,887],{"class":82,"line":886},22,[161,888,889],{"class":270},"    @task\n",[161,891,893,896,898,901,903,906,908,911],{"class":82,"line":892},23,[161,894,895],{"class":216},"    def",[161,897,271],{"class":270},[161,899,900],{"class":220},"(logical_date",[161,902,283],{"class":216},[161,904,905],{"class":323},"None",[161,907,492],{"class":220},[161,909,910],{"class":323},"str",[161,912,498],{"class":220},[161,914,916,919,921],{"class":82,"line":915},24,[161,917,918],{"class":220},"        hook ",[161,920,283],{"class":216},[161,922,923],{"class":220}," S3Hook()\n",[161,925,927,930,932,935,938],{"class":82,"line":926},25,[161,928,929],{"class":220},"        local_dir ",[161,931,283],{"class":216},[161,933,934],{"class":220}," Path(",[161,936,937],{"class":170},"\"\u002Ftmp\u002Freports\"",[161,939,298],{"class":220},[161,941,943,946,949,951,954],{"class":82,"line":942},26,[161,944,945],{"class":220},"        local_dir.mkdir(",[161,947,948],{"class":289},"exist_ok",[161,950,283],{"class":216},[161,952,953],{"class":323},"True",[161,955,298],{"class":220},[161,957,959,962,964],{"class":82,"line":958},27,[161,960,961],{"class":220},"        source ",[161,963,283],{"class":216},[161,965,966],{"class":220}," Path(hook.download_file(\n",[161,968,970,973,975,977,980,982,985,987,989,991,993,995,997,999,1002],{"class":82,"line":969},28,[161,971,972],{"class":289},"            key",[161,974,283],{"class":216},[161,976,549],{"class":216},[161,978,979],{"class":170},"\"exports\u002Forders-",[161,981,357],{"class":356},[161,983,984],{"class":220},"logical_date:",[161,986,363],{"class":216},[161,988,366],{"class":220},[161,990,369],{"class":216},[161,992,372],{"class":220},[161,994,369],{"class":216},[161,996,377],{"class":220},[161,998,380],{"class":356},[161,1000,1001],{"class":170},".xlsx\"",[161,1003,736],{"class":220},[161,1005,1007,1010,1012,1014,1016,1019,1021,1023,1026,1029,1031,1033],{"class":82,"line":1006},29,[161,1008,1009],{"class":289},"            bucket_name",[161,1011,283],{"class":216},[161,1013,706],{"class":323},[161,1015,315],{"class":220},[161,1017,1018],{"class":289},"local_path",[161,1020,283],{"class":216},[161,1022,910],{"class":323},[161,1024,1025],{"class":220},"(local_dir), ",[161,1027,1028],{"class":289},"preserve_file_name",[161,1030,283],{"class":216},[161,1032,953],{"class":323},[161,1034,736],{"class":220},[161,1036,1038],{"class":82,"line":1037},30,[161,1039,1040],{"class":220},"        ))\n",[161,1042,1044,1047,1049],{"class":82,"line":1043},31,[161,1045,1046],{"class":220},"        built ",[161,1048,283],{"class":216},[161,1050,1051],{"class":220}," regional.build(logical_date.date(), source, local_dir)\n",[161,1053,1055,1058,1060,1062,1065,1067,1070,1072],{"class":82,"line":1054},32,[161,1056,1057],{"class":220},"        key ",[161,1059,283],{"class":216},[161,1061,350],{"class":216},[161,1063,1064],{"class":170},"\"reports\u002F",[161,1066,357],{"class":356},[161,1068,1069],{"class":220},"built.name",[161,1071,380],{"class":356},[161,1073,1074],{"class":170},"\"\n",[161,1076,1078,1081,1084,1086,1088,1091,1094,1096,1099,1102,1104,1106,1108,1111,1113,1115],{"class":82,"line":1077},33,[161,1079,1080],{"class":220},"        hook.load_file(",[161,1082,1083],{"class":289},"filename",[161,1085,283],{"class":216},[161,1087,910],{"class":323},[161,1089,1090],{"class":220},"(built), ",[161,1092,1093],{"class":289},"key",[161,1095,283],{"class":216},[161,1097,1098],{"class":220},"key, ",[161,1100,1101],{"class":289},"bucket_name",[161,1103,283],{"class":216},[161,1105,706],{"class":323},[161,1107,315],{"class":220},[161,1109,1110],{"class":289},"replace",[161,1112,283],{"class":216},[161,1114,953],{"class":323},[161,1116,298],{"class":220},[161,1118,1120,1123],{"class":82,"line":1119},34,[161,1121,1122],{"class":216},"        return",[161,1124,1125],{"class":220}," key\n",[161,1127,1129],{"class":82,"line":1128},35,[161,1130,261],{"emptyLinePlaceholder":260},[161,1132,1134],{"class":82,"line":1133},36,[161,1135,889],{"class":270},[161,1137,1139,1141,1144,1147,1149,1151,1153],{"class":82,"line":1138},37,[161,1140,895],{"class":216},[161,1142,1143],{"class":270}," check",[161,1145,1146],{"class":220},"(key: ",[161,1148,910],{"class":323},[161,1150,492],{"class":220},[161,1152,495],{"class":323},[161,1154,498],{"class":220},[161,1156,1158,1160,1162],{"class":82,"line":1157},38,[161,1159,918],{"class":220},[161,1161,283],{"class":216},[161,1163,923],{"class":220},[161,1165,1167,1170,1172,1175,1177,1179,1181,1183,1185,1187],{"class":82,"line":1166},39,[161,1168,1169],{"class":220},"        local ",[161,1171,283],{"class":216},[161,1173,1174],{"class":220}," Path(hook.download_file(",[161,1176,1093],{"class":289},[161,1178,283],{"class":216},[161,1180,1098],{"class":220},[161,1182,1101],{"class":289},[161,1184,283],{"class":216},[161,1186,706],{"class":323},[161,1188,736],{"class":220},[161,1190,1192,1195,1197,1200,1202,1204,1206,1208],{"class":82,"line":1191},40,[161,1193,1194],{"class":289},"                                        local_path",[161,1196,283],{"class":216},[161,1198,1199],{"class":170},"\"\u002Ftmp\"",[161,1201,315],{"class":220},[161,1203,1028],{"class":289},[161,1205,283],{"class":216},[161,1207,953],{"class":323},[161,1209,1210],{"class":220},"))\n",[161,1212,1214,1216],{"class":82,"line":1213},41,[161,1215,1122],{"class":216},[161,1217,1218],{"class":220}," regional.validate(local)\n",[161,1220,1222],{"class":82,"line":1221},42,[161,1223,261],{"emptyLinePlaceholder":260},[161,1225,1227],{"class":82,"line":1226},43,[161,1228,889],{"class":270},[161,1230,1232,1234,1237,1239,1241,1244,1246,1248,1250],{"class":82,"line":1231},44,[161,1233,895],{"class":216},[161,1235,1236],{"class":270}," distribute",[161,1238,1146],{"class":220},[161,1240,910],{"class":323},[161,1242,1243],{"class":220},", stats: ",[161,1245,495],{"class":323},[161,1247,492],{"class":220},[161,1249,905],{"class":323},[161,1251,498],{"class":220},[161,1253,1255,1258,1260,1262,1265,1267,1269,1271,1273,1275,1278,1281,1284,1286,1289,1291,1293,1296,1298,1301,1303,1305],{"class":82,"line":1254},45,[161,1256,1257],{"class":323},"        print",[161,1259,546],{"class":220},[161,1261,549],{"class":216},[161,1263,1264],{"class":170},"\"sending ",[161,1266,357],{"class":356},[161,1268,1093],{"class":220},[161,1270,380],{"class":356},[161,1272,591],{"class":170},[161,1274,357],{"class":356},[161,1276,1277],{"class":220},"stats[",[161,1279,1280],{"class":170},"'rows'",[161,1282,1283],{"class":220},"]",[161,1285,380],{"class":356},[161,1287,1288],{"class":170}," regions, total ",[161,1290,357],{"class":356},[161,1292,1277],{"class":220},[161,1294,1295],{"class":170},"'total'",[161,1297,1283],{"class":220},[161,1299,1300],{"class":216},":,.0f",[161,1302,380],{"class":356},[161,1304,552],{"class":170},[161,1306,298],{"class":220},[161,1308,1310],{"class":82,"line":1309},46,[161,1311,261],{"emptyLinePlaceholder":260},[161,1313,1315,1318,1320],{"class":82,"line":1314},47,[161,1316,1317],{"class":220},"    built_key ",[161,1319,283],{"class":216},[161,1321,1322],{"class":220}," build()\n",[161,1324,1326],{"class":82,"line":1325},48,[161,1327,1328],{"class":220},"    distribute(built_key, check(built_key))\n",[161,1330,1332],{"class":82,"line":1331},49,[161,1333,261],{"emptyLinePlaceholder":260},[161,1335,1337],{"class":82,"line":1336},50,[161,1338,1339],{"class":220},"regional_revenue_report()\n",[10,1341,1342,1345,1346,1349],{},[158,1343,1344],{},"max_active_runs=1"," prevents two runs of a report writing the same output at once, and ",[158,1347,1348],{},"catchup=False","\nstops a newly deployed DAG from immediately running every day since the start date — both defaults\nworth setting deliberately rather than discovering.",[146,1351,1353],{"id":1352},"passing-files-between-tasks","Passing files between tasks",[20,1355,29,1360,29,1363,29,1366,29,1369,29,1376,29,1381,29,1387,29,1392,29,1396,29,1400,29,1403,29,1407,29,1411,29,1414,29,1417,29,1420,29,1427,29,1432,29,1437,29,1440,29,1444,29,1447],{"viewBox":1356,"role":23,"ariaLabelledBy":1357,"xmlns":27,"style":28},"0 0 760 201",[1358,1359],"af-xcom-t","af-xcom-d",[31,1361,1362],{"id":1358},"What travels between tasks",[35,1364,1365],{"id":1359},"Tasks may run on different workers, so a file written to local disk in one is not visible to the next; the key goes through XCom and the file goes through shared storage.",[39,1367],{"x":41,"y":41,"width":42,"height":1368,"fill":44},"201",[39,1370],{"x":1371,"y":1372,"width":1373,"height":1374,"rx":1375,"fill":68,"stroke":69,"style":54},"20","28","270.0","139","14",[56,1377,1380],{"x":1378,"y":1379,"style":74},"155.0","54","through XCom",[82,1382],{"x1":1383,"y1":1384,"x2":1385,"y2":1384,"stroke":69,"style":1386},"36","64","274.0","stroke-width:1px",[56,1388,1391],{"x":1378,"y":1389,"style":1390},"86","font-size:11.5px;font-weight:400;fill:var(--text,#172033);text-anchor:middle","a storage key",[56,1393,1395],{"x":1378,"y":1394,"style":1390},"109","a small stats dict",[56,1397,1399],{"x":1378,"y":1398,"style":1390},"132","strings and numbers",[39,1401],{"x":1402,"y":1372,"width":1373,"height":1374,"rx":1375,"fill":121,"stroke":122,"style":54},"470.0",[56,1404,1406],{"x":1405,"y":1379,"style":126},"605.0","through storage",[82,1408],{"x1":1409,"y1":1384,"x2":1410,"y2":1384,"stroke":122,"style":1386},"486.0","724.0",[56,1412,1413],{"x":1405,"y":1389,"style":1390},"the workbook itself",[56,1415,1416],{"x":1405,"y":1394,"style":1390},"the source export",[56,1418,1419],{"x":1405,"y":1398,"style":1390},"anything large",[39,1421],{"x":1422,"y":1423,"width":1424,"height":1425,"rx":1426,"fill":52,"stroke":53},"316.0","78.5","128","38","19",[56,1428,1431],{"x":58,"y":1429,"style":1430},"102.5","font-size:12.5px;font-weight:700;fill:var(--brand-strong,#4338ca);text-anchor:middle","key only",[82,1433],{"x1":1434,"y1":1435,"x2":1436,"y2":1435,"stroke":53,"style":54},"295.0","97.5","309.0",[87,1438],{"points":1439,"fill":90},"309.0,97.5 300.0,92.5 300.0,102.5",[82,1441],{"x1":1442,"y1":1435,"x2":1443,"y2":1435,"stroke":53,"style":54},"449.0","463.0",[87,1445],{"points":1446,"fill":90},"463.0,97.5 454.0,92.5 454.0,102.5",[56,1448,1450],{"x":58,"y":1449,"style":143},"187","a DataFrame in XCom is a slow scheduler waiting to happen",[10,1452,1453,1454,1457,1458,620],{},"Tasks may run on different workers, so anything written to local disk in one task may not exist in\nthe next. The rule is that XCom carries a ",[1455,1456,1093],"em",{}," and shared storage carries the ",[1455,1459,1460],{},"file",[151,1462,1464],{"className":200,"code":1463,"language":202,"meta":156,"style":156},"@task\ndef build() -> str:\n    ...\n    return key                    # a short string — fine for XCom\n\n@task\ndef check(key: str) -> dict:\n    ...                           # download from the key\n",[158,1465,1466,1471,1484,1489,1499,1503,1507,1523],{"__ignoreMap":156},[161,1467,1468],{"class":82,"line":163},[161,1469,1470],{"class":270},"@task\n",[161,1472,1473,1475,1477,1480,1482],{"class":82,"line":213},[161,1474,267],{"class":216},[161,1476,271],{"class":270},[161,1478,1479],{"class":220},"() -> ",[161,1481,910],{"class":323},[161,1483,498],{"class":220},[161,1485,1486],{"class":82,"line":230},[161,1487,1488],{"class":323},"    ...\n",[161,1490,1491,1493,1496],{"class":82,"line":243},[161,1492,461],{"class":216},[161,1494,1495],{"class":220}," key                    ",[161,1497,1498],{"class":209},"# a short string — fine for XCom\n",[161,1500,1501],{"class":82,"line":257},[161,1502,261],{"emptyLinePlaceholder":260},[161,1504,1505],{"class":82,"line":264},[161,1506,1470],{"class":270},[161,1508,1509,1511,1513,1515,1517,1519,1521],{"class":82,"line":277},[161,1510,267],{"class":216},[161,1512,1143],{"class":270},[161,1514,1146],{"class":220},[161,1516,910],{"class":323},[161,1518,492],{"class":220},[161,1520,495],{"class":323},[161,1522,498],{"class":220},[161,1524,1525,1528],{"class":82,"line":301},[161,1526,1527],{"class":323},"    ...",[161,1529,1530],{"class":209},"                           # download from the key\n",[10,1532,1533],{},"Returning a DataFrame or file bytes from a task pushes them into Airflow's metadata database, which\nis the reliable way to make a scheduler slow and an operations team unhappy. The same argument\napplies to reading the source: fetch it in the task that needs it rather than at module level, since\nmodule-level code runs on every DAG parse.",[146,1535,1537],{"id":1536},"waiting-for-the-upstream-file","Waiting for the upstream file",[10,1539,1540],{},"The dependency that motivates Airflow in the first place is usually \"do not build until the export\nhas landed\". A sensor expresses that, and deferrable mode means it costs nothing while it waits.",[151,1542,1544],{"className":200,"code":1543,"language":202,"meta":156,"style":156},"from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor\n\nwait = S3KeySensor(\n    task_id=\"wait_for_export\",\n    bucket_name=BUCKET,\n    bucket_key=\"exports\u002Forders-{{ ds }}.xlsx\",\n    deferrable=True,\n    timeout=60 * 60 * 3,\n    poke_interval=300,\n)\nwait >> built_key\n",[158,1545,1546,1558,1562,1572,1584,1595,1617,1628,1650,1661,1665],{"__ignoreMap":156},[161,1547,1548,1550,1553,1555],{"class":82,"line":163},[161,1549,217],{"class":216},[161,1551,1552],{"class":220}," airflow.providers.amazon.aws.sensors.s3 ",[161,1554,224],{"class":216},[161,1556,1557],{"class":220}," S3KeySensor\n",[161,1559,1560],{"class":82,"line":213},[161,1561,261],{"emptyLinePlaceholder":260},[161,1563,1564,1567,1569],{"class":82,"line":230},[161,1565,1566],{"class":220},"wait ",[161,1568,283],{"class":216},[161,1570,1571],{"class":220}," S3KeySensor(\n",[161,1573,1574,1577,1579,1582],{"class":82,"line":243},[161,1575,1576],{"class":289},"    task_id",[161,1578,283],{"class":216},[161,1580,1581],{"class":170},"\"wait_for_export\"",[161,1583,736],{"class":220},[161,1585,1586,1589,1591,1593],{"class":82,"line":257},[161,1587,1588],{"class":289},"    bucket_name",[161,1590,283],{"class":216},[161,1592,706],{"class":323},[161,1594,736],{"class":220},[161,1596,1597,1600,1602,1604,1607,1610,1613,1615],{"class":82,"line":264},[161,1598,1599],{"class":289},"    bucket_key",[161,1601,283],{"class":216},[161,1603,979],{"class":170},[161,1605,1606],{"class":356},"{{",[161,1608,1609],{"class":170}," ds ",[161,1611,1612],{"class":356},"}}",[161,1614,1001],{"class":170},[161,1616,736],{"class":220},[161,1618,1619,1622,1624,1626],{"class":82,"line":277},[161,1620,1621],{"class":289},"    deferrable",[161,1623,283],{"class":216},[161,1625,953],{"class":323},[161,1627,736],{"class":220},[161,1629,1630,1633,1635,1637,1640,1643,1645,1648],{"class":82,"line":301},[161,1631,1632],{"class":289},"    timeout",[161,1634,283],{"class":216},[161,1636,59],{"class":323},[161,1638,1639],{"class":216}," *",[161,1641,1642],{"class":323}," 60",[161,1644,1639],{"class":216},[161,1646,1647],{"class":323}," 3",[161,1649,736],{"class":220},[161,1651,1652,1655,1657,1659],{"class":82,"line":336},[161,1653,1654],{"class":289},"    poke_interval",[161,1656,283],{"class":216},[161,1658,49],{"class":323},[161,1660,736],{"class":220},[161,1662,1663],{"class":82,"line":386},[161,1664,298],{"class":220},[161,1666,1667,1669,1672],{"class":82,"line":410},[161,1668,1566],{"class":220},[161,1670,1671],{"class":216},">>",[161,1673,1674],{"class":220}," built_key\n",[10,1676,1677,1680,1681,1684],{},[158,1678,1679],{},"deferrable=True"," hands the wait to the triggerer rather than occupying a worker slot for three\nhours, which matters as soon as several reports wait in parallel. ",[158,1682,1683],{},"{{ ds }}"," is the logical date,\nwhich is what makes a backfill look for the right day's file rather than today's.",[146,1686,1688],{"id":1687},"making-the-dag-idempotent","Making the DAG idempotent",[10,1690,1691,1692,1695,1696,1699],{},"Naming every artefact by ",[158,1693,1694],{},"logical_date"," rather than ",[158,1697,1698],{},"datetime.now()"," is what makes a rerun safe. A\ntask that fails at 06:10 and retries at 06:15 must produce the same file name, and a backfill for\nlast Tuesday must produce Tuesday's.",[151,1701,1703],{"className":200,"code":1702,"language":202,"meta":156,"style":156},"target = out_dir \u002F f\"regional-{logical_date:%Y-%m-%d}.xlsx\"     # correct\ntarget = out_dir \u002F f\"regional-{datetime.now():%Y-%m-%d}.xlsx\"   # wrong on every retry\n",[158,1704,1705,1743],{"__ignoreMap":156},[161,1706,1707,1710,1712,1714,1716,1718,1720,1722,1724,1726,1728,1730,1732,1734,1736,1738,1740],{"class":82,"line":163},[161,1708,1709],{"class":220},"target ",[161,1711,283],{"class":216},[161,1713,344],{"class":220},[161,1715,347],{"class":216},[161,1717,350],{"class":216},[161,1719,353],{"class":170},[161,1721,357],{"class":356},[161,1723,984],{"class":220},[161,1725,363],{"class":216},[161,1727,366],{"class":220},[161,1729,369],{"class":216},[161,1731,372],{"class":220},[161,1733,369],{"class":216},[161,1735,377],{"class":220},[161,1737,380],{"class":356},[161,1739,1001],{"class":170},[161,1741,1742],{"class":209},"     # correct\n",[161,1744,1745,1747,1749,1751,1753,1755,1757,1759,1762,1764,1766,1768,1770,1772,1774,1776,1778],{"class":82,"line":213},[161,1746,1709],{"class":220},[161,1748,283],{"class":216},[161,1750,344],{"class":220},[161,1752,347],{"class":216},[161,1754,350],{"class":216},[161,1756,353],{"class":170},[161,1758,357],{"class":356},[161,1760,1761],{"class":220},"datetime.now():",[161,1763,363],{"class":216},[161,1765,366],{"class":220},[161,1767,369],{"class":216},[161,1769,372],{"class":220},[161,1771,369],{"class":216},[161,1773,377],{"class":220},[161,1775,380],{"class":356},[161,1777,1001],{"class":170},[161,1779,1780],{"class":209},"   # wrong on every retry\n",[10,1782,1783,1784,1787],{},"Uploading with ",[158,1785,1786],{},"replace=True"," completes the property: rerunning overwrites rather than failing or\naccumulating duplicates. Between the two, a rerun becomes something an operator can do without\nthinking about it, which is most of Airflow's practical value.",[146,1789,1791],{"id":1790},"alerting-on-the-runs-that-matter","Alerting on the runs that matter",[10,1793,1794],{},"Airflow's default failure email is a poor alert: it fires per task, says little, and is easy to\nfilter into a folder nobody opens. A callback that posts a short message naming the DAG, the task and\nthe logical date is far more useful, and it lives in one place.",[151,1796,1798],{"className":200,"code":1797,"language":202,"meta":156,"style":156},"def on_failure(context) -> None:\n    task = context[\"task_instance\"]\n    message = (\n        f\"{task.dag_id}.{task.task_id} failed for {context['logical_date']:%Y-%m-%d} \"\n        f\"(try {task.try_number} of {task.max_tries + 1})\"\n    )\n    notify_channel(message)          # Teams, Slack, PagerDuty — whatever the team reads\n\ndefault_args = {\n    \"retries\": 2,\n    \"retry_delay\": timedelta(minutes=5),\n    \"on_failure_callback\": on_failure,\n}\n",[158,1799,1800,1814,1830,1840,1894,1926,1931,1939,1943,1953,1964,1979,1987],{"__ignoreMap":156},[161,1801,1802,1804,1807,1810,1812],{"class":82,"line":163},[161,1803,267],{"class":216},[161,1805,1806],{"class":270}," on_failure",[161,1808,1809],{"class":220},"(context) -> ",[161,1811,905],{"class":323},[161,1813,498],{"class":220},[161,1815,1816,1819,1821,1824,1827],{"class":82,"line":213},[161,1817,1818],{"class":220},"    task ",[161,1820,283],{"class":216},[161,1822,1823],{"class":220}," context[",[161,1825,1826],{"class":170},"\"task_instance\"",[161,1828,1829],{"class":220},"]\n",[161,1831,1832,1835,1837],{"class":82,"line":230},[161,1833,1834],{"class":220},"    message ",[161,1836,283],{"class":216},[161,1838,1839],{"class":220}," (\n",[161,1841,1842,1845,1847,1849,1852,1854,1856,1858,1861,1863,1866,1868,1871,1874,1877,1879,1881,1883,1885,1887,1889,1891],{"class":82,"line":243},[161,1843,1844],{"class":216},"        f",[161,1846,552],{"class":170},[161,1848,357],{"class":356},[161,1850,1851],{"class":220},"task.dag_id",[161,1853,380],{"class":356},[161,1855,620],{"class":170},[161,1857,357],{"class":356},[161,1859,1860],{"class":220},"task.task_id",[161,1862,380],{"class":356},[161,1864,1865],{"class":170}," failed for ",[161,1867,357],{"class":356},[161,1869,1870],{"class":220},"context[",[161,1872,1873],{"class":170},"'logical_date'",[161,1875,1876],{"class":220},"]:",[161,1878,363],{"class":216},[161,1880,366],{"class":220},[161,1882,369],{"class":216},[161,1884,372],{"class":220},[161,1886,369],{"class":216},[161,1888,377],{"class":220},[161,1890,380],{"class":356},[161,1892,1893],{"class":170}," \"\n",[161,1895,1896,1898,1901,1903,1906,1908,1911,1913,1916,1919,1921,1923],{"class":82,"line":257},[161,1897,1844],{"class":216},[161,1899,1900],{"class":170},"\"(try ",[161,1902,357],{"class":356},[161,1904,1905],{"class":220},"task.try_number",[161,1907,380],{"class":356},[161,1909,1910],{"class":170}," of ",[161,1912,357],{"class":356},[161,1914,1915],{"class":220},"task.max_tries ",[161,1917,1918],{"class":216},"+",[161,1920,489],{"class":323},[161,1922,380],{"class":356},[161,1924,1925],{"class":170},")\"\n",[161,1927,1928],{"class":82,"line":264},[161,1929,1930],{"class":220},"    )\n",[161,1932,1933,1936],{"class":82,"line":277},[161,1934,1935],{"class":220},"    notify_channel(message)          ",[161,1937,1938],{"class":209},"# Teams, Slack, PagerDuty — whatever the team reads\n",[161,1940,1941],{"class":82,"line":301},[161,1942,261],{"emptyLinePlaceholder":260},[161,1944,1945,1948,1950],{"class":82,"line":336},[161,1946,1947],{"class":220},"default_args ",[161,1949,283],{"class":216},[161,1951,1952],{"class":220}," {\n",[161,1954,1955,1958,1960,1962],{"class":82,"line":386},[161,1956,1957],{"class":170},"    \"retries\"",[161,1959,591],{"class":220},[161,1961,823],{"class":323},[161,1963,736],{"class":220},[161,1965,1966,1969,1971,1973,1975,1977],{"class":82,"line":410},[161,1967,1968],{"class":170},"    \"retry_delay\"",[161,1970,831],{"class":220},[161,1972,834],{"class":289},[161,1974,283],{"class":216},[161,1976,839],{"class":323},[161,1978,784],{"class":220},[161,1980,1981,1984],{"class":82,"line":435},[161,1982,1983],{"class":170},"    \"on_failure_callback\"",[161,1985,1986],{"class":220},": on_failure,\n",[161,1988,1989],{"class":82,"line":458},[161,1990,1991],{"class":220},"}\n",[10,1993,1994,1995,1998],{},"Including the try number is what keeps the alert honest: a first failure that will be retried in five\nminutes is information, and a final one after every retry is an incident. Sending only on the last\nattempt — by checking ",[158,1996,1997],{},"task.try_number > task.max_tries"," — is the variant worth adopting once the\nalert volume becomes noticeable.",[10,2000,2001],{},"The complementary signal is an SLA: a report that must be on somebody's desk by 07:00 should say so,\nso that a run which is merely slow is caught as well as one which failed.",[151,2003,2005],{"className":200,"code":2004,"language":202,"meta":156,"style":156},"@task(sla=timedelta(minutes=45))\ndef build(logical_date=None) -> str:\n    ...\n",[158,2006,2007,2031,2049],{"__ignoreMap":156},[161,2008,2009,2012,2014,2017,2019,2022,2024,2026,2029],{"class":82,"line":163},[161,2010,2011],{"class":270},"@task",[161,2013,546],{"class":220},[161,2015,2016],{"class":289},"sla",[161,2018,283],{"class":216},[161,2020,2021],{"class":220},"timedelta(",[161,2023,834],{"class":289},[161,2025,283],{"class":216},[161,2027,2028],{"class":323},"45",[161,2030,1210],{"class":220},[161,2032,2033,2035,2037,2039,2041,2043,2045,2047],{"class":82,"line":213},[161,2034,267],{"class":216},[161,2036,271],{"class":270},[161,2038,900],{"class":220},[161,2040,283],{"class":216},[161,2042,905],{"class":323},[161,2044,492],{"class":220},[161,2046,910],{"class":323},[161,2048,498],{"class":220},[161,2050,2051],{"class":82,"line":230},[161,2052,1488],{"class":323},[146,2054,2056],{"id":2055},"testing-a-dag-without-running-airflow","Testing a DAG without running Airflow",[10,2058,2059],{},"Because the report logic lives outside the DAG, most of the testing is ordinary pytest against those\nfunctions. What remains worth testing about the DAG itself is that it imports cleanly and has no\ncycles — a check that catches the majority of deployment failures and runs in under a second.",[151,2061,2063],{"className":200,"code":2062,"language":202,"meta":156,"style":156},"from airflow.models import DagBag\n\ndef test_dags_import_cleanly():\n    bag = DagBag(dag_folder=\"dags\", include_examples=False)\n    assert not bag.import_errors, bag.import_errors\n    dag = bag.get_dag(\"regional_revenue_report\")\n    assert dag is not None\n    assert dag.default_args[\"retries\"] >= 1\n",[158,2064,2065,2077,2081,2090,2119,2130,2144,2159],{"__ignoreMap":156},[161,2066,2067,2069,2072,2074],{"class":82,"line":163},[161,2068,217],{"class":216},[161,2070,2071],{"class":220}," airflow.models ",[161,2073,224],{"class":216},[161,2075,2076],{"class":220}," DagBag\n",[161,2078,2079],{"class":82,"line":213},[161,2080,261],{"emptyLinePlaceholder":260},[161,2082,2083,2085,2088],{"class":82,"line":230},[161,2084,267],{"class":216},[161,2086,2087],{"class":270}," test_dags_import_cleanly",[161,2089,878],{"class":220},[161,2091,2092,2095,2097,2100,2103,2105,2108,2110,2113,2115,2117],{"class":82,"line":243},[161,2093,2094],{"class":220},"    bag ",[161,2096,283],{"class":216},[161,2098,2099],{"class":220}," DagBag(",[161,2101,2102],{"class":289},"dag_folder",[161,2104,283],{"class":216},[161,2106,2107],{"class":170},"\"dags\"",[161,2109,315],{"class":220},[161,2111,2112],{"class":289},"include_examples",[161,2114,283],{"class":216},[161,2116,324],{"class":323},[161,2118,298],{"class":220},[161,2120,2121,2124,2127],{"class":82,"line":257},[161,2122,2123],{"class":216},"    assert",[161,2125,2126],{"class":216}," not",[161,2128,2129],{"class":220}," bag.import_errors, bag.import_errors\n",[161,2131,2132,2135,2137,2140,2142],{"class":82,"line":264},[161,2133,2134],{"class":220},"    dag ",[161,2136,283],{"class":216},[161,2138,2139],{"class":220}," bag.get_dag(",[161,2141,733],{"class":170},[161,2143,298],{"class":220},[161,2145,2146,2148,2151,2154,2156],{"class":82,"line":277},[161,2147,2123],{"class":216},[161,2149,2150],{"class":220}," dag ",[161,2152,2153],{"class":216},"is",[161,2155,2126],{"class":216},[161,2157,2158],{"class":323}," None\n",[161,2160,2161,2163,2166,2168,2171,2174],{"class":82,"line":301},[161,2162,2123],{"class":216},[161,2164,2165],{"class":220}," dag.default_args[",[161,2167,818],{"class":170},[161,2169,2170],{"class":220},"] ",[161,2172,2173],{"class":216},">=",[161,2175,2176],{"class":323}," 1\n",[10,2178,2179],{},"Running that in continuous integration means a typo in a DAG file fails the pull request rather than\nthe 06:00 schedule. It is the cheapest test in the repository and catches the failure mode that\ncosts the most, which is a DAG that stops appearing in the interface without anybody noticing.",[146,2181,2183],{"id":2182},"common-pitfalls","Common pitfalls",[2185,2186,2187,2203],"table",{},[2188,2189,2190],"thead",{},[2191,2192,2193,2197,2200],"tr",{},[2194,2195,2196],"th",{},"Symptom",[2194,2198,2199],{},"Cause",[2194,2201,2202],{},"Fix",[2204,2205,2206,2218,2229,2244,2260,2271],"tbody",{},[2191,2207,2208,2212,2215],{},[2209,2210,2211],"td",{},"Task fails with a missing file",[2209,2213,2214],{},"The previous task wrote to a different worker's disk",[2209,2216,2217],{},"Pass a storage key, not a path",[2191,2219,2220,2223,2226],{},[2209,2221,2222],{},"Scheduler slows down over weeks",[2209,2224,2225],{},"Large XCom payloads in the metadata database",[2209,2227,2228],{},"Return keys and small dicts only",[2191,2230,2231,2234,2241],{},[2209,2232,2233],{},"A backfill produces today's date in every file",[2209,2235,2236,2238,2239],{},[158,2237,1698],{}," used instead of ",[158,2240,1694],{},[2209,2242,2243],{},"Use the logical date everywhere",[2191,2245,2246,2249,2255],{},[2209,2247,2248],{},"DAG runs hundreds of times on deploy",[2209,2250,2251,2254],{},[158,2252,2253],{},"catchup"," left at its default",[2209,2256,2257,2259],{},[158,2258,1348],{}," unless backfilling is intended",[2191,2261,2262,2265,2268],{},[2209,2263,2264],{},"Import errors at parse time",[2209,2266,2267],{},"Heavy imports or I\u002FO at module level",[2209,2269,2270],{},"Import inside the task function",[2191,2272,2273,2276,2279],{},[2209,2274,2275],{},"PDF conversion fails on some workers",[2209,2277,2278],{},"LibreOffice not in every worker image",[2209,2280,2281],{},"Isolate that task on its own queue or image",[146,2283,2285],{"id":2284},"performance-and-scale","Performance and scale",[20,2287,29,2293,29,2296,29,2299,29,2303,29,2307,29,2314,29,2321,29,2327,29,2331,29,2333,29,2337,29,2342,29,2346,29,2348,29,2351,29,2356,29,2360],{"viewBox":2288,"role":23,"ariaLabelledBy":2289,"xmlns":27,"style":2292},"0 0 720 196",[2290,2291],"af-tasks-t","af-tasks-d","width:100%;max-width:720px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif",[31,2294,2295],{"id":2290},"Overhead is per task, not per row",[35,2297,2298],{"id":2291},"Each task costs a scheduling decision, a worker slot and database writes, so splitting a report into many trivial tasks costs far more than keeping it to the genuinely separate steps.",[39,2300],{"x":41,"y":41,"width":2301,"height":2302,"fill":44},"720","196",[56,2304,2306],{"x":1371,"y":50,"style":2305},"font-size:12px;font-weight:600;fill:var(--text,#172033);text-anchor:start","50 trivial tasks",[39,2308],{"x":2309,"y":2310,"width":58,"height":48,"rx":2311,"fill":2312,"stroke":2313},"200","40","6","#e7ebef","var(--line,#cdd5e6)",[39,2315],{"x":1368,"y":2316,"width":2317,"height":2318,"rx":839,"fill":2319,"stroke":2320},"41","378.0","24","#fee8f2","var(--accent,#d81b73)",[56,2322,2326],{"x":2323,"y":2324,"style":2325},"592.0","58","font-size:12px;font-weight:700;fill:var(--accent,#d81b73);text-anchor:start","50 slots",[56,2328,2330],{"x":1371,"y":2329,"style":2305},"100","5 meaningful tasks",[39,2332],{"x":2309,"y":67,"width":58,"height":48,"rx":2311,"fill":2312,"stroke":2313},[39,2334],{"x":1368,"y":2335,"width":2336,"height":2318,"rx":839,"fill":68,"stroke":69},"85","38.0",[56,2338,2341],{"x":2323,"y":2339,"style":2340},"102","font-size:12px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:start","5 slots",[56,2343,2345],{"x":1371,"y":2344,"style":2305},"144","1 script in cron",[39,2347],{"x":2309,"y":1424,"width":58,"height":48,"rx":2311,"fill":2312,"stroke":2313},[39,2349],{"x":1368,"y":2350,"width":1372,"height":2318,"rx":839,"fill":121,"stroke":122},"129",[56,2352,2355],{"x":2323,"y":2353,"style":2354},"146","font-size:12px;font-weight:700;fill:var(--gold-ink,#7a4e06);text-anchor:start","no scheduler",[56,2357,2359],{"x":1371,"y":1371,"style":2358},"font-size:11.5px;font-weight:600;fill:var(--muted,#5b6780);text-anchor:start","relative cost",[56,2361,2364],{"x":2362,"y":2363,"style":143},"360.0","186","split by dependency, not by sheet",[10,2366,2367],{},"Airflow's overhead is per task, not per row: each one is a scheduling decision, a worker slot and a\ndatabase write. That makes a DAG of five meaningful tasks much healthier than one of fifty trivial\nones, and it argues against the instinct to split a report into a task per sheet.",[10,2369,2370],{},"Where a report genuinely fans out — one workbook per region, thirty regions — dynamic task mapping\nexpresses it without writing thirty tasks:",[151,2372,2374],{"className":200,"code":2373,"language":202,"meta":156,"style":156},"@task\ndef regions() -> list[str]:\n    return [\"North\", \"South\", \"East\", \"West\"]\n\n@task\ndef build_one(region: str) -> str:\n    ...\n\nbuild_one.expand(region=regions())\n",[158,2375,2376,2380,2395,2422,2426,2430,2448,2452,2456],{"__ignoreMap":156},[161,2377,2378],{"class":82,"line":163},[161,2379,1470],{"class":270},[161,2381,2382,2384,2387,2390,2392],{"class":82,"line":213},[161,2383,267],{"class":216},[161,2385,2386],{"class":270}," regions",[161,2388,2389],{"class":220},"() -> list[",[161,2391,910],{"class":323},[161,2393,2394],{"class":220},"]:\n",[161,2396,2397,2399,2402,2405,2407,2410,2412,2415,2417,2420],{"class":82,"line":230},[161,2398,461],{"class":216},[161,2400,2401],{"class":220}," [",[161,2403,2404],{"class":170},"\"North\"",[161,2406,315],{"class":220},[161,2408,2409],{"class":170},"\"South\"",[161,2411,315],{"class":220},[161,2413,2414],{"class":170},"\"East\"",[161,2416,315],{"class":220},[161,2418,2419],{"class":170},"\"West\"",[161,2421,1829],{"class":220},[161,2423,2424],{"class":82,"line":243},[161,2425,261],{"emptyLinePlaceholder":260},[161,2427,2428],{"class":82,"line":257},[161,2429,1470],{"class":270},[161,2431,2432,2434,2437,2440,2442,2444,2446],{"class":82,"line":264},[161,2433,267],{"class":216},[161,2435,2436],{"class":270}," build_one",[161,2438,2439],{"class":220},"(region: ",[161,2441,910],{"class":323},[161,2443,492],{"class":220},[161,2445,910],{"class":323},[161,2447,498],{"class":220},[161,2449,2450],{"class":82,"line":277},[161,2451,1488],{"class":323},[161,2453,2454],{"class":82,"line":301},[161,2455,261],{"emptyLinePlaceholder":260},[161,2457,2458,2461,2464,2466],{"class":82,"line":336},[161,2459,2460],{"class":220},"build_one.expand(",[161,2462,2463],{"class":289},"region",[161,2465,283],{"class":216},[161,2467,2468],{"class":220},"regions())\n",[10,2470,2471,2472,620],{},"Mapped tasks run in parallel up to the pool's limit, which is the right way to shorten a fan-out. The\nper-region generation itself is covered in\n",[14,2473,2475],{"href":2474},"\u002Fautomating-reporting-workflows\u002Fgenerating-excel-reports-from-templates\u002Fgenerate-one-excel-report-per-region-in-a-loop\u002F","Generate One Excel Report per Region in a Loop",[146,2477,2479],{"id":2478},"conclusion","Conclusion",[10,2481,2482],{},"Keep the report as importable functions and let the DAG describe only order, scheduling and retries.\nPass storage keys between tasks rather than files, name every artefact by the logical date so retries\nand backfills are safe, wait for upstream data with a deferrable sensor, and keep the task count\nproportional to the number of genuinely separate steps. Cron remains the right answer for a single\nscript; this is what to reach for when the report has become a pipeline.",[146,2484,2486],{"id":2485},"frequently-asked-questions","Frequently asked questions",[10,2488,2489,2493],{},[2490,2491,2492],"strong",{},"Why use Airflow rather than cron for an Excel report?","\nFor a single script on a schedule, cron is simpler and you should keep it. Airflow earns its keep when the report has dependencies — wait for a warehouse load, then build, then check, then distribute — and when you need retries, backfills and a visible history of which runs succeeded.",[10,2495,2496,2499],{},[2490,2497,2498],{},"Should the workbook be passed between tasks?","\nNever through XCom, which is for small values. Write the file to shared storage — S3, a mounted volume, a blob container — and pass its key between tasks. XCom then carries a string, which is what it is designed for.",[10,2501,2502,2505],{},[2490,2503,2504],{},"How do I stop two runs writing the same file?","\nName outputs by the logical date rather than the wall clock: the run for 1 September writes report-2026-09-01.xlsx whether it runs on time or three days late during a backfill. That single convention makes the whole DAG idempotent.",[10,2507,2508,2511],{},[2490,2509,2510],{},"Do I need a separate worker image with LibreOffice?","\nIf any task converts to PDF, yes — and it is worth isolating it. Use a KubernetesPodOperator or a dedicated queue so only the conversion task needs the heavier image, rather than growing every worker.",[146,2513,2515],{"id":2514},"related","Related",[2517,2518,2519,2526,2533,2540,2547],"ul",{},[2520,2521,2522,2523,2525],"li",{},"Up one level: ",[14,2524,17],{"href":16}," — the simpler scheduling options and when they suffice.",[2520,2527,2528,2532],{},[14,2529,2531],{"href":2530},"\u002Fautomating-reporting-workflows\u002Fscheduling-python-excel-scripts-with-cron\u002Fschedule-recurring-excel-reports-with-apscheduler\u002F","Schedule Recurring Excel Reports with APScheduler"," — in-process scheduling without an orchestrator.",[2520,2534,2535,2539],{},[14,2536,2538],{"href":2537},"\u002Fautomating-reporting-workflows\u002Fscheduling-python-excel-scripts-with-cron\u002Frun-a-python-excel-report-in-docker\u002F","Run a Python Excel Report in Docker"," — the image the workers run.",[2520,2541,2542,2546],{},[14,2543,2545],{"href":2544},"\u002Fautomating-reporting-workflows\u002Fpublishing-excel-reports-to-cloud-storage\u002Fupload-an-excel-report-to-amazon-s3-with-boto3\u002F","Upload an Excel Report to Amazon S3 with boto3"," — the storage layer tasks hand keys through.",[2520,2548,2549,2553],{},[14,2550,2552],{"href":2551},"\u002Fautomating-reporting-workflows\u002Ferror-handling-and-logging-in-excel-automation\u002Fretry-a-failed-excel-report-job-in-python\u002F","Retry a Failed Excel Report Job in Python"," — in-script retries, and where the orchestrator's take over.",[2555,2556,2557],"style",{},"html pre.shiki code .sMTad, html code.shiki .sMTad{--shiki-default:#6F42C1;--shiki-dark:#FFB757}html pre.shiki code .srMev, html code.shiki .srMev{--shiki-default:#032F62;--shiki-dark:#ADDCFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .s-wDw, html code.shiki .s-wDw{--shiki-default:#6A737D;--shiki-dark:#BDC4CC}html pre.shiki code .s-kum, html code.shiki .s-kum{--shiki-default:#D73A49;--shiki-dark:#FF9492}html pre.shiki code .skGVy, html code.shiki .skGVy{--shiki-default:#24292E;--shiki-dark:#F0F3F6}html pre.shiki code .s_Opv, html code.shiki .s_Opv{--shiki-default:#6F42C1;--shiki-dark:#DBB7FF}html pre.shiki code .sa561, html code.shiki .sa561{--shiki-default:#E36209;--shiki-dark:#FFB757}html pre.shiki code .sP0c6, html code.shiki .sP0c6{--shiki-default:#005CC5;--shiki-dark:#91CBFF}html pre.shiki code .sSjpA, html code.shiki .sSjpA{--shiki-default:#005CC5;--shiki-dark:#FF9492}",{"title":156,"searchDepth":213,"depth":213,"links":2559},[2560,2561,2562,2563,2564,2565,2566,2567,2568,2569,2570,2571,2572],{"id":148,"depth":213,"text":149},{"id":193,"depth":213,"text":194},{"id":623,"depth":213,"text":624},{"id":1352,"depth":213,"text":1353},{"id":1536,"depth":213,"text":1537},{"id":1687,"depth":213,"text":1688},{"id":1790,"depth":213,"text":1791},{"id":2055,"depth":213,"text":2056},{"id":2182,"depth":213,"text":2183},{"id":2284,"depth":213,"text":2285},{"id":2478,"depth":213,"text":2479},{"id":2485,"depth":213,"text":2486},{"id":2514,"depth":213,"text":2515},"2026-09-04","Turn a scheduled Excel script into a DAG: importable report functions, storage keys through XCom, a deferrable sensor for upstream data, and logical dates that make reruns safe.","md",[2577,2579,2581,2583],{"q":2492,"a":2578},"For a single script on a schedule, cron is simpler and you should keep it. Airflow earns its keep when the report has dependencies — wait for a warehouse load, then build, then check, then distribute — and when you need retries, backfills and a visible history of which runs succeeded.",{"q":2498,"a":2580},"Never through XCom, which is for small values. Write the file to shared storage — S3, a mounted volume, a blob container — and pass its key between tasks. XCom then carries a string, which is what it is designed for.",{"q":2504,"a":2582},"Name outputs by the logical date rather than the wall clock: the run for 1 September writes report-2026-09-01.xlsx whether it runs on time or three days late during a backfill. That single convention makes the whole DAG idempotent.",{"q":2510,"a":2584},"If any task converts to PDF, yes — and it is worth isolating it. Use a KubernetesPodOperator or a dedicated queue so only the conversion task needs the heavier image, rather than growing every worker.",{"breadcrumb":2586},[2587,2589,2592],{"name":2588,"item":347},"Home",{"name":2590,"item":2591},"Automating Reporting Workflows","\u002Fautomating-reporting-workflows\u002F",{"name":17,"item":16},"\u002Fautomating-reporting-workflows\u002Fscheduling-python-excel-scripts-with-cron\u002Forchestrate-excel-reports-with-apache-airflow",{"title":5,"description":2595},"Move an Excel reporting script into Airflow — keep the logic importable, pass S3 keys not files, wait with a deferrable sensor, and name outputs by logical date for idempotent reruns.","orchestrate-excel-reports-with-apache-airflow","automating-reporting-workflows\u002Fscheduling-python-excel-scripts-with-cron\u002Forchestrate-excel-reports-with-apache-airflow\u002Findex","how-to","WAjQCRnlORKOrHWX-IZqYLQhuPs-VioQaEUds92Rpr8",[2601,2604],{"title":17,"path":2602,"stem":2603,"children":-1},"\u002Fautomating-reporting-workflows\u002Fscheduling-python-excel-scripts-with-cron","automating-reporting-workflows\u002Fscheduling-python-excel-scripts-with-cron\u002Findex",{"title":2538,"path":2605,"stem":2606,"children":-1},"\u002Fautomating-reporting-workflows\u002Fscheduling-python-excel-scripts-with-cron\u002Frun-a-python-excel-report-in-docker","automating-reporting-workflows\u002Fscheduling-python-excel-scripts-with-cron\u002Frun-a-python-excel-report-in-docker\u002Findex",1788710162938]