[{"data":1,"prerenderedAt":4172},["ShallowReactive",2],{"doc:\u002Fautomating-reporting-workflows\u002Ferror-handling-and-logging-in-excel-automation":3,"surround:\u002Fautomating-reporting-workflows\u002Ferror-handling-and-logging-in-excel-automation":4164},{"id":4,"title":5,"body":6,"dateModified":4138,"datePublished":4138,"description":4139,"extension":4140,"faq":4141,"meta":4150,"navigation":212,"path":4157,"seo":4158,"slug":4160,"stem":4161,"type":4162,"__hash__":4163},"docs\u002Fautomating-reporting-workflows\u002Ferror-handling-and-logging-in-excel-automation\u002Findex.md","Error Handling and Logging in Excel Automation",{"type":7,"value":8,"toc":4119},"minimark",[9,19,122,127,156,177,181,184,476,479,483,486,773,780,783,923,926,988,992,995,1309,1316,1320,1325,1496,1503,1507,1510,1603,1618,1691,1695,1698,2069,2072,2076,2079,2480,2491,2495,2498,2809,2816,2822,2826,2829,3013,3024,3028,3031,3253,3256,3266,3270,3273,3529,3532,3536,3641,3645,3648,3969,3976,3980,4014,4018,4025,4031,4037,4043,4047,4050,4057,4060,4090,4093,4115],[10,11,12,13,18],"p",{},"A report that runs on your laptop needs to work. A report that runs at 6am on a server needs to fail well: to say what broke, to leave last week's file intact, to retry the things worth retrying, and to tell someone in a form they will read. This guide, part of ",[14,15,17],"a",{"href":16},"\u002Fautomating-reporting-workflows\u002F","Automating Reporting Workflows",", covers the patterns that turn a working script into an unattended job.",[20,21,29,30,29,34,29,38,29,45,29,55,29,62,29,67,29,72,29,77,29,82,29,85,29,89,29,94,29,99,29,103,29,108,29,112,29,116,29,119],"svg",{"viewBox":22,"role":23,"ariaLabelledBy":24,"xmlns":27,"style":28},"0 0 760 268","img",[25,26],"eh-taxo-t","eh-taxo-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;max-width:760px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif","\n  ",[31,32,33],"title",{"id":25},"Four kinds of failure and the right response to each",[35,36,37],"desc",{"id":26},"Transient failures such as a locked file or network blip should be retried. Data failures such as a missing column should stop the run and alert. Environment failures such as a missing directory should stop and alert. Output failures such as an empty report should block delivery and keep the previous file.",[39,40],"rect",{"x":41,"y":41,"width":42,"height":43,"fill":44},"0","760","268","#ffffff",[39,46],{"x":47,"y":48,"width":49,"height":50,"rx":51,"fill":52,"stroke":53,"style":54},"20","30","352","102","14","#fdefd8","var(--gold,#b4740a)","stroke-width:2px",[56,57,61],"text",{"x":58,"y":59,"style":60},"44","58","font-size:13px;font-weight:700;fill:var(--gold,#b4740a)","transient",[56,63,66],{"x":58,"y":64,"style":65},"82","font-size:11.5px;fill:var(--text,#172033)","file locked · share unavailable · SMTP timeout",[56,68,71],{"x":58,"y":69,"style":70},"106","font-size:11.5px;font-weight:700;fill:var(--teal,#0f9488)","→ retry with backoff, then alert",[39,73],{"x":74,"y":48,"width":49,"height":50,"rx":51,"fill":75,"stroke":76,"style":54},"388","#fce9e9","var(--danger,#dc2626)",[56,78,81],{"x":79,"y":59,"style":80},"412","font-size:13px;font-weight:700;fill:var(--danger,#dc2626)","data",[56,83,84],{"x":79,"y":64,"style":65},"missing column · unparsable dates · empty sheet",[56,86,88],{"x":79,"y":69,"style":87},"font-size:11.5px;font-weight:700;fill:var(--danger,#dc2626)","→ never retry; stop and report",[39,90],{"x":47,"y":91,"width":49,"height":50,"rx":51,"fill":92,"stroke":93},"146","#ebebfd","var(--line,#cdd5e6)",[56,95,98],{"x":58,"y":96,"style":97},"174","font-size:13px;font-weight:700;fill:var(--brand-strong,#4338ca)","environment",[56,100,102],{"x":58,"y":101,"style":65},"198","path missing · no credentials · disk full",[56,104,107],{"x":58,"y":105,"style":106},"222","font-size:11.5px;font-weight:700;fill:var(--brand-strong,#4338ca)","→ fail fast, before any work",[39,109],{"x":74,"y":91,"width":49,"height":50,"rx":51,"fill":110,"stroke":111,"style":54},"#d9f4f1","var(--teal,#0f9488)",[56,113,115],{"x":79,"y":96,"style":114},"font-size:13px;font-weight:700;fill:var(--teal,#0f9488)","output",[56,117,118],{"x":79,"y":101,"style":65},"zero rows · totals absurd · file tiny",[56,120,121],{"x":79,"y":105,"style":70},"→ block delivery, keep last good",[123,124,126],"h2",{"id":125},"install-the-dependencies","Install the dependencies",[128,129,134],"pre",{"className":130,"code":131,"language":132,"meta":133,"style":133},"language-bash shiki shiki-themes github-light github-dark-high-contrast","pip install pandas openpyxl\n","bash","",[135,136,137],"code",{"__ignoreMap":133},[138,139,142,146,150,153],"span",{"class":140,"line":141},"line",1,[138,143,145],{"class":144},"sMTad","pip",[138,147,149],{"class":148},"srMev"," install",[138,151,152],{"class":148}," pandas",[138,154,155],{"class":148}," openpyxl\n",[10,157,158,159,162,163,162,166,162,169,172,173,176],{},"Everything else in this guide is standard library: ",[135,160,161],{},"logging",", ",[135,164,165],{},"pathlib",[135,167,168],{},"tempfile",[135,170,171],{},"smtplib"," and ",[135,174,175],{},"time",".",[123,178,180],{"id":179},"fail-fast-on-the-environment","Fail fast on the environment",[10,182,183],{},"The cheapest failure is the one that happens before any work. Check the things a job depends on in the first few lines, and say precisely what is missing:",[128,185,189],{"className":186,"code":187,"language":188,"meta":133,"style":133},"language-python shiki shiki-themes github-light github-dark-high-contrast","from pathlib import Path\n\nSOURCE = Path(\"\u002Fdata\u002Fincoming\u002Forders.xlsx\")\nOUTPUT_DIR = Path(\"\u002Fdata\u002Freports\")\n\ndef preflight():\n    problems = []\n    if not SOURCE.exists():\n        problems.append(f\"source file not found: {SOURCE}\")\n    elif SOURCE.stat().st_size == 0:\n        problems.append(f\"source file is empty: {SOURCE}\")\n    if not OUTPUT_DIR.exists():\n        problems.append(f\"output directory missing: {OUTPUT_DIR}\")\n    elif not OUTPUT_DIR.is_dir():\n        problems.append(f\"output path is not a directory: {OUTPUT_DIR}\")\n    if problems:\n        raise SystemExit(\"preflight failed:\\n  \" + \"\\n  \".join(problems))\n\npreflight()\n","python",[135,190,191,207,214,233,248,253,266,278,293,319,339,359,371,391,403,423,431,465,470],{"__ignoreMap":133},[138,192,193,197,201,204],{"class":140,"line":141},[138,194,196],{"class":195},"s-kum","from",[138,198,200],{"class":199},"skGVy"," pathlib ",[138,202,203],{"class":195},"import",[138,205,206],{"class":199}," Path\n",[138,208,210],{"class":140,"line":209},2,[138,211,213],{"emptyLinePlaceholder":212},true,"\n",[138,215,217,221,224,227,230],{"class":140,"line":216},3,[138,218,220],{"class":219},"sP0c6","SOURCE",[138,222,223],{"class":195}," =",[138,225,226],{"class":199}," Path(",[138,228,229],{"class":148},"\"\u002Fdata\u002Fincoming\u002Forders.xlsx\"",[138,231,232],{"class":199},")\n",[138,234,236,239,241,243,246],{"class":140,"line":235},4,[138,237,238],{"class":219},"OUTPUT_DIR",[138,240,223],{"class":195},[138,242,226],{"class":199},[138,244,245],{"class":148},"\"\u002Fdata\u002Freports\"",[138,247,232],{"class":199},[138,249,251],{"class":140,"line":250},5,[138,252,213],{"emptyLinePlaceholder":212},[138,254,256,259,263],{"class":140,"line":255},6,[138,257,258],{"class":195},"def",[138,260,262],{"class":261},"s_Opv"," preflight",[138,264,265],{"class":199},"():\n",[138,267,269,272,275],{"class":140,"line":268},7,[138,270,271],{"class":199},"    problems ",[138,273,274],{"class":195},"=",[138,276,277],{"class":199}," []\n",[138,279,281,284,287,290],{"class":140,"line":280},8,[138,282,283],{"class":195},"    if",[138,285,286],{"class":195}," not",[138,288,289],{"class":219}," SOURCE",[138,291,292],{"class":199},".exists():\n",[138,294,296,299,302,305,309,311,314,317],{"class":140,"line":295},9,[138,297,298],{"class":199},"        problems.append(",[138,300,301],{"class":195},"f",[138,303,304],{"class":148},"\"source file not found: ",[138,306,308],{"class":307},"sSjpA","{",[138,310,220],{"class":219},[138,312,313],{"class":307},"}",[138,315,316],{"class":148},"\"",[138,318,232],{"class":199},[138,320,322,325,327,330,333,336],{"class":140,"line":321},10,[138,323,324],{"class":195},"    elif",[138,326,289],{"class":219},[138,328,329],{"class":199},".stat().st_size ",[138,331,332],{"class":195},"==",[138,334,335],{"class":219}," 0",[138,337,338],{"class":199},":\n",[138,340,342,344,346,349,351,353,355,357],{"class":140,"line":341},11,[138,343,298],{"class":199},[138,345,301],{"class":195},[138,347,348],{"class":148},"\"source file is empty: ",[138,350,308],{"class":307},[138,352,220],{"class":219},[138,354,313],{"class":307},[138,356,316],{"class":148},[138,358,232],{"class":199},[138,360,362,364,366,369],{"class":140,"line":361},12,[138,363,283],{"class":195},[138,365,286],{"class":195},[138,367,368],{"class":219}," OUTPUT_DIR",[138,370,292],{"class":199},[138,372,374,376,378,381,383,385,387,389],{"class":140,"line":373},13,[138,375,298],{"class":199},[138,377,301],{"class":195},[138,379,380],{"class":148},"\"output directory missing: ",[138,382,308],{"class":307},[138,384,238],{"class":219},[138,386,313],{"class":307},[138,388,316],{"class":148},[138,390,232],{"class":199},[138,392,394,396,398,400],{"class":140,"line":393},14,[138,395,324],{"class":195},[138,397,286],{"class":195},[138,399,368],{"class":219},[138,401,402],{"class":199},".is_dir():\n",[138,404,406,408,410,413,415,417,419,421],{"class":140,"line":405},15,[138,407,298],{"class":199},[138,409,301],{"class":195},[138,411,412],{"class":148},"\"output path is not a directory: ",[138,414,308],{"class":307},[138,416,238],{"class":219},[138,418,313],{"class":307},[138,420,316],{"class":148},[138,422,232],{"class":199},[138,424,426,428],{"class":140,"line":425},16,[138,427,283],{"class":195},[138,429,430],{"class":199}," problems:\n",[138,432,434,437,440,443,446,449,452,455,458,460,462],{"class":140,"line":433},17,[138,435,436],{"class":195},"        raise",[138,438,439],{"class":219}," SystemExit",[138,441,442],{"class":199},"(",[138,444,445],{"class":148},"\"preflight failed:",[138,447,448],{"class":307},"\\n",[138,450,451],{"class":148},"  \"",[138,453,454],{"class":195}," +",[138,456,457],{"class":148}," \"",[138,459,448],{"class":307},[138,461,451],{"class":148},[138,463,464],{"class":199},".join(problems))\n",[138,466,468],{"class":140,"line":467},18,[138,469,213],{"emptyLinePlaceholder":212},[138,471,473],{"class":140,"line":472},19,[138,474,475],{"class":199},"preflight()\n",[10,477,478],{},"Half an hour into a job is a bad moment to discover the output directory does not exist, because everything computed so far is thrown away. Preflight also gives you somewhere obvious to check credentials, disk space and the presence of an SMTP host — all the things that are constant during a run and fatal if absent.",[123,480,482],{"id":481},"logging-that-answers-questions","Logging that answers questions",[10,484,485],{},"The purpose of a log is to answer \"what happened at 06:04?\" without re-running anything. That means one line per meaningful step, with the identifiers a person would need:",[128,487,489],{"className":186,"code":488,"language":188,"meta":133,"style":133},"import logging\nfrom pathlib import Path\n\ndef setup_logging(name=\"report\", logdir=\"logs\", level=logging.INFO):\n    Path(logdir).mkdir(parents=True, exist_ok=True)\n    logger = logging.getLogger(name)\n    logger.setLevel(level)\n    logger.handlers.clear()\n\n    fmt = logging.Formatter(\n        \"%(asctime)s %(levelname)-8s %(name)s %(message)s\",\n        datefmt=\"%Y-%m-%d %H:%M:%S\",\n    )\n    file_handler = logging.FileHandler(Path(logdir) \u002F f\"{name}.log\", encoding=\"utf-8\")\n    file_handler.setFormatter(fmt)\n    stream_handler = logging.StreamHandler()\n    stream_handler.setFormatter(fmt)\n\n    logger.addHandler(file_handler)\n    logger.addHandler(stream_handler)\n    return logger\n\nlog = setup_logging()\nlog.info(\"run started source=%s\", SOURCE)\n",[135,490,491,498,508,512,549,574,584,589,594,598,608,630,648,653,693,698,708,713,717,722,728,737,742,753],{"__ignoreMap":133},[138,492,493,495],{"class":140,"line":141},[138,494,203],{"class":195},[138,496,497],{"class":199}," logging\n",[138,499,500,502,504,506],{"class":140,"line":209},[138,501,196],{"class":195},[138,503,200],{"class":199},[138,505,203],{"class":195},[138,507,206],{"class":199},[138,509,510],{"class":140,"line":216},[138,511,213],{"emptyLinePlaceholder":212},[138,513,514,516,519,522,524,527,530,532,535,538,540,543,546],{"class":140,"line":235},[138,515,258],{"class":195},[138,517,518],{"class":261}," setup_logging",[138,520,521],{"class":199},"(name",[138,523,274],{"class":195},[138,525,526],{"class":148},"\"report\"",[138,528,529],{"class":199},", logdir",[138,531,274],{"class":195},[138,533,534],{"class":148},"\"logs\"",[138,536,537],{"class":199},", level",[138,539,274],{"class":195},[138,541,542],{"class":199},"logging.",[138,544,545],{"class":219},"INFO",[138,547,548],{"class":199},"):\n",[138,550,551,554,558,560,563,565,568,570,572],{"class":140,"line":250},[138,552,553],{"class":199},"    Path(logdir).mkdir(",[138,555,557],{"class":556},"sa561","parents",[138,559,274],{"class":195},[138,561,562],{"class":219},"True",[138,564,162],{"class":199},[138,566,567],{"class":556},"exist_ok",[138,569,274],{"class":195},[138,571,562],{"class":219},[138,573,232],{"class":199},[138,575,576,579,581],{"class":140,"line":255},[138,577,578],{"class":199},"    logger ",[138,580,274],{"class":195},[138,582,583],{"class":199}," logging.getLogger(name)\n",[138,585,586],{"class":140,"line":268},[138,587,588],{"class":199},"    logger.setLevel(level)\n",[138,590,591],{"class":140,"line":280},[138,592,593],{"class":199},"    logger.handlers.clear()\n",[138,595,596],{"class":140,"line":295},[138,597,213],{"emptyLinePlaceholder":212},[138,599,600,603,605],{"class":140,"line":321},[138,601,602],{"class":199},"    fmt ",[138,604,274],{"class":195},[138,606,607],{"class":199}," logging.Formatter(\n",[138,609,610,613,616,619,622,625,627],{"class":140,"line":341},[138,611,612],{"class":148},"        \"",[138,614,615],{"class":307},"%(asctime)s",[138,617,618],{"class":307}," %(levelname)-8s",[138,620,621],{"class":307}," %(name)s",[138,623,624],{"class":307}," %(message)s",[138,626,316],{"class":148},[138,628,629],{"class":199},",\n",[138,631,632,635,637,640,643,646],{"class":140,"line":361},[138,633,634],{"class":556},"        datefmt",[138,636,274],{"class":195},[138,638,639],{"class":148},"\"%Y-%m-",[138,641,642],{"class":307},"%d",[138,644,645],{"class":148}," %H:%M:%S\"",[138,647,629],{"class":199},[138,649,650],{"class":140,"line":373},[138,651,652],{"class":199},"    )\n",[138,654,655,658,660,663,666,669,671,673,676,678,681,683,686,688,691],{"class":140,"line":393},[138,656,657],{"class":199},"    file_handler ",[138,659,274],{"class":195},[138,661,662],{"class":199}," logging.FileHandler(Path(logdir) ",[138,664,665],{"class":195},"\u002F",[138,667,668],{"class":195}," f",[138,670,316],{"class":148},[138,672,308],{"class":307},[138,674,675],{"class":199},"name",[138,677,313],{"class":307},[138,679,680],{"class":148},".log\"",[138,682,162],{"class":199},[138,684,685],{"class":556},"encoding",[138,687,274],{"class":195},[138,689,690],{"class":148},"\"utf-8\"",[138,692,232],{"class":199},[138,694,695],{"class":140,"line":405},[138,696,697],{"class":199},"    file_handler.setFormatter(fmt)\n",[138,699,700,703,705],{"class":140,"line":425},[138,701,702],{"class":199},"    stream_handler ",[138,704,274],{"class":195},[138,706,707],{"class":199}," logging.StreamHandler()\n",[138,709,710],{"class":140,"line":433},[138,711,712],{"class":199},"    stream_handler.setFormatter(fmt)\n",[138,714,715],{"class":140,"line":467},[138,716,213],{"emptyLinePlaceholder":212},[138,718,719],{"class":140,"line":472},[138,720,721],{"class":199},"    logger.addHandler(file_handler)\n",[138,723,725],{"class":140,"line":724},20,[138,726,727],{"class":199},"    logger.addHandler(stream_handler)\n",[138,729,731,734],{"class":140,"line":730},21,[138,732,733],{"class":195},"    return",[138,735,736],{"class":199}," logger\n",[138,738,740],{"class":140,"line":739},22,[138,741,213],{"emptyLinePlaceholder":212},[138,743,745,748,750],{"class":140,"line":744},23,[138,746,747],{"class":199},"log ",[138,749,274],{"class":195},[138,751,752],{"class":199}," setup_logging()\n",[138,754,756,759,762,765,767,769,771],{"class":140,"line":755},24,[138,757,758],{"class":199},"log.info(",[138,760,761],{"class":148},"\"run started source=",[138,763,764],{"class":307},"%s",[138,766,316],{"class":148},[138,768,162],{"class":199},[138,770,220],{"class":219},[138,772,232],{"class":199},[10,774,775,776,779],{},"Two handlers, deliberately. The file survives the run for later investigation; the stream is what a scheduler captures and what you watch while testing. Clearing ",[135,777,778],{},"logger.handlers"," first makes the function safe to call twice, which matters in a job that is also imported by tests.",[10,781,782],{},"Log the shape of the data at each boundary — it turns most investigations into a single glance:",[128,784,786],{"className":186,"code":785,"language":188,"meta":133,"style":133},"import pandas as pd\n\ndf = pd.read_excel(SOURCE, sheet_name=\"Orders\")\nlog.info(\"read source rows=%d columns=%d\", len(df), len(df.columns))\n\nclean = df.dropna(subset=[\"Order_ID\"])\nlog.info(\"after cleaning rows=%d dropped=%d\", len(clean), len(df) - len(clean))\n",[135,787,788,801,805,829,858,862,886],{"__ignoreMap":133},[138,789,790,792,795,798],{"class":140,"line":141},[138,791,203],{"class":195},[138,793,794],{"class":199}," pandas ",[138,796,797],{"class":195},"as",[138,799,800],{"class":199}," pd\n",[138,802,803],{"class":140,"line":209},[138,804,213],{"emptyLinePlaceholder":212},[138,806,807,810,812,815,817,819,822,824,827],{"class":140,"line":216},[138,808,809],{"class":199},"df ",[138,811,274],{"class":195},[138,813,814],{"class":199}," pd.read_excel(",[138,816,220],{"class":219},[138,818,162],{"class":199},[138,820,821],{"class":556},"sheet_name",[138,823,274],{"class":195},[138,825,826],{"class":148},"\"Orders\"",[138,828,232],{"class":199},[138,830,831,833,836,838,841,843,845,847,850,853,855],{"class":140,"line":235},[138,832,758],{"class":199},[138,834,835],{"class":148},"\"read source rows=",[138,837,642],{"class":307},[138,839,840],{"class":148}," columns=",[138,842,642],{"class":307},[138,844,316],{"class":148},[138,846,162],{"class":199},[138,848,849],{"class":219},"len",[138,851,852],{"class":199},"(df), ",[138,854,849],{"class":219},[138,856,857],{"class":199},"(df.columns))\n",[138,859,860],{"class":140,"line":250},[138,861,213],{"emptyLinePlaceholder":212},[138,863,864,867,869,872,875,877,880,883],{"class":140,"line":255},[138,865,866],{"class":199},"clean ",[138,868,274],{"class":195},[138,870,871],{"class":199}," df.dropna(",[138,873,874],{"class":556},"subset",[138,876,274],{"class":195},[138,878,879],{"class":199},"[",[138,881,882],{"class":148},"\"Order_ID\"",[138,884,885],{"class":199},"])\n",[138,887,888,890,893,895,898,900,902,904,906,909,911,914,917,920],{"class":140,"line":268},[138,889,758],{"class":199},[138,891,892],{"class":148},"\"after cleaning rows=",[138,894,642],{"class":307},[138,896,897],{"class":148}," dropped=",[138,899,642],{"class":307},[138,901,316],{"class":148},[138,903,162],{"class":199},[138,905,849],{"class":219},[138,907,908],{"class":199},"(clean), ",[138,910,849],{"class":219},[138,912,913],{"class":199},"(df) ",[138,915,916],{"class":195},"-",[138,918,919],{"class":219}," len",[138,921,922],{"class":199},"(clean))\n",[10,924,925],{},"When a report is wrong three days later, \"rows=48211 dropped=0\" on Monday and \"rows=12 dropped=48199\" on Tuesday tells you exactly where to look, without a debugger and without the source file.",[20,927,29,932,29,935,29,938,29,941,29,948,29,954,29,960,29,966,29,970,29,975,29,978,29,983],{"viewBox":928,"role":23,"ariaLabelledBy":929,"xmlns":27,"style":28},"0 0 760 236",[930,931],"eh-log-t","eh-log-d",[31,933,934],{"id":930},"A log line that helps and one that does not",[35,936,937],{"id":931},"The unhelpful line reads error occurred with no context. The helpful line names the step, the file, the row count and the specific problem, so the reader knows what broke and where without re-running the job.",[39,939],{"x":41,"y":41,"width":42,"height":940,"fill":44},"236",[39,942],{"x":943,"y":944,"width":945,"height":946,"rx":947,"fill":75,"stroke":76,"style":54},"24","34","712","76","12",[56,949,953],{"x":950,"y":951,"style":952},"48","60","font-size:12px;font-weight:700;fill:var(--danger,#dc2626)","unhelpful",[39,955],{"x":950,"y":956,"width":957,"height":958,"rx":959,"fill":44,"stroke":93},"70","664","28","6",[56,961,965],{"x":962,"y":963,"style":964},"62","89","font-size:11.5px;fill:var(--text,#172033);font-family:ui-monospace,Menlo,monospace","2026-03-02 06:04:11 ERROR report Error occurred",[39,967],{"x":943,"y":968,"width":945,"height":969,"rx":947,"fill":110,"stroke":111,"style":54},"126","88",[56,971,974],{"x":950,"y":972,"style":973},"152","font-size:12px;font-weight:700;fill:var(--teal,#0f9488)","useful",[39,976],{"x":950,"y":977,"width":957,"height":958,"rx":959,"fill":44,"stroke":93},"162",[56,979,982],{"x":962,"y":980,"style":981},"181","font-size:11px;fill:var(--text,#172033);font-family:ui-monospace,Menlo,monospace","06:04:11 ERROR report step=read file=orders.xlsx sheet=Orders rows=0",[56,984,987],{"x":962,"y":985,"style":986},"206","font-size:11px;fill:var(--muted,#5b6780);font-family:ui-monospace,Menlo,monospace","   reason=\"sheet has headers but no data rows\"",[123,989,991],{"id":990},"retry-only-what-is-worth-retrying","Retry only what is worth retrying",[10,993,994],{},"A locked file usually unlocks; a missing column never appears. Retrying the wrong class of failure wastes time and buries the real message:",[128,996,998],{"className":186,"code":997,"language":188,"meta":133,"style":133},"import time\n\nTRANSIENT = (PermissionError, OSError, TimeoutError)\n\ndef with_retries(fn, attempts=4, base_delay=2.0, logger=None):\n    \"\"\"Retry transient failures with exponential backoff. Re-raise everything else.\"\"\"\n    for attempt in range(1, attempts + 1):\n        try:\n            return fn()\n        except TRANSIENT as exc:\n            if attempt == attempts:\n                if logger:\n                    logger.error(\"giving up after %d attempt(s): %s\", attempts, exc)\n                raise\n            delay = base_delay * (2 ** (attempt - 1))\n            if logger:\n                logger.warning(\"attempt %d\u002F%d failed (%s) — retrying in %.0fs\",\n                               attempt, attempts, exc, delay)\n            time.sleep(delay)\n\nimport pandas as pd\ndf = with_retries(lambda: pd.read_excel(SOURCE, sheet_name=\"Orders\"), logger=log)\n",[135,999,1000,1007,1011,1036,1040,1073,1078,1108,1115,1123,1137,1149,1157,1177,1182,1213,1219,1249,1254,1259,1263,1273],{"__ignoreMap":133},[138,1001,1002,1004],{"class":140,"line":141},[138,1003,203],{"class":195},[138,1005,1006],{"class":199}," time\n",[138,1008,1009],{"class":140,"line":209},[138,1010,213],{"emptyLinePlaceholder":212},[138,1012,1013,1016,1018,1021,1024,1026,1029,1031,1034],{"class":140,"line":216},[138,1014,1015],{"class":219},"TRANSIENT",[138,1017,223],{"class":195},[138,1019,1020],{"class":199}," (",[138,1022,1023],{"class":219},"PermissionError",[138,1025,162],{"class":199},[138,1027,1028],{"class":219},"OSError",[138,1030,162],{"class":199},[138,1032,1033],{"class":219},"TimeoutError",[138,1035,232],{"class":199},[138,1037,1038],{"class":140,"line":235},[138,1039,213],{"emptyLinePlaceholder":212},[138,1041,1042,1044,1047,1050,1052,1055,1058,1060,1063,1066,1068,1071],{"class":140,"line":250},[138,1043,258],{"class":195},[138,1045,1046],{"class":261}," with_retries",[138,1048,1049],{"class":199},"(fn, attempts",[138,1051,274],{"class":195},[138,1053,1054],{"class":219},"4",[138,1056,1057],{"class":199},", base_delay",[138,1059,274],{"class":195},[138,1061,1062],{"class":219},"2.0",[138,1064,1065],{"class":199},", logger",[138,1067,274],{"class":195},[138,1069,1070],{"class":219},"None",[138,1072,548],{"class":199},[138,1074,1075],{"class":140,"line":255},[138,1076,1077],{"class":148},"    \"\"\"Retry transient failures with exponential backoff. Re-raise everything else.\"\"\"\n",[138,1079,1080,1083,1086,1089,1092,1094,1097,1100,1103,1106],{"class":140,"line":268},[138,1081,1082],{"class":195},"    for",[138,1084,1085],{"class":199}," attempt ",[138,1087,1088],{"class":195},"in",[138,1090,1091],{"class":219}," range",[138,1093,442],{"class":199},[138,1095,1096],{"class":219},"1",[138,1098,1099],{"class":199},", attempts ",[138,1101,1102],{"class":195},"+",[138,1104,1105],{"class":219}," 1",[138,1107,548],{"class":199},[138,1109,1110,1113],{"class":140,"line":280},[138,1111,1112],{"class":195},"        try",[138,1114,338],{"class":199},[138,1116,1117,1120],{"class":140,"line":295},[138,1118,1119],{"class":195},"            return",[138,1121,1122],{"class":199}," fn()\n",[138,1124,1125,1128,1131,1134],{"class":140,"line":321},[138,1126,1127],{"class":195},"        except",[138,1129,1130],{"class":219}," TRANSIENT",[138,1132,1133],{"class":195}," as",[138,1135,1136],{"class":199}," exc:\n",[138,1138,1139,1142,1144,1146],{"class":140,"line":341},[138,1140,1141],{"class":195},"            if",[138,1143,1085],{"class":199},[138,1145,332],{"class":195},[138,1147,1148],{"class":199}," attempts:\n",[138,1150,1151,1154],{"class":140,"line":361},[138,1152,1153],{"class":195},"                if",[138,1155,1156],{"class":199}," logger:\n",[138,1158,1159,1162,1165,1167,1170,1172,1174],{"class":140,"line":373},[138,1160,1161],{"class":199},"                    logger.error(",[138,1163,1164],{"class":148},"\"giving up after ",[138,1166,642],{"class":307},[138,1168,1169],{"class":148}," attempt(s): ",[138,1171,764],{"class":307},[138,1173,316],{"class":148},[138,1175,1176],{"class":199},", attempts, exc)\n",[138,1178,1179],{"class":140,"line":393},[138,1180,1181],{"class":195},"                raise\n",[138,1183,1184,1187,1189,1192,1195,1197,1200,1203,1206,1208,1210],{"class":140,"line":405},[138,1185,1186],{"class":199},"            delay ",[138,1188,274],{"class":195},[138,1190,1191],{"class":199}," base_delay ",[138,1193,1194],{"class":195},"*",[138,1196,1020],{"class":199},[138,1198,1199],{"class":219},"2",[138,1201,1202],{"class":195}," **",[138,1204,1205],{"class":199}," (attempt ",[138,1207,916],{"class":195},[138,1209,1105],{"class":219},[138,1211,1212],{"class":199},"))\n",[138,1214,1215,1217],{"class":140,"line":425},[138,1216,1141],{"class":195},[138,1218,1156],{"class":199},[138,1220,1221,1224,1227,1229,1231,1233,1236,1238,1241,1244,1247],{"class":140,"line":433},[138,1222,1223],{"class":199},"                logger.warning(",[138,1225,1226],{"class":148},"\"attempt ",[138,1228,642],{"class":307},[138,1230,665],{"class":148},[138,1232,642],{"class":307},[138,1234,1235],{"class":148}," failed (",[138,1237,764],{"class":307},[138,1239,1240],{"class":148},") — retrying in ",[138,1242,1243],{"class":307},"%.0f",[138,1245,1246],{"class":148},"s\"",[138,1248,629],{"class":199},[138,1250,1251],{"class":140,"line":467},[138,1252,1253],{"class":199},"                               attempt, attempts, exc, delay)\n",[138,1255,1256],{"class":140,"line":472},[138,1257,1258],{"class":199},"            time.sleep(delay)\n",[138,1260,1261],{"class":140,"line":724},[138,1262,213],{"emptyLinePlaceholder":212},[138,1264,1265,1267,1269,1271],{"class":140,"line":730},[138,1266,203],{"class":195},[138,1268,794],{"class":199},[138,1270,797],{"class":195},[138,1272,800],{"class":199},[138,1274,1275,1277,1279,1282,1285,1288,1290,1292,1294,1296,1298,1301,1304,1306],{"class":140,"line":739},[138,1276,809],{"class":199},[138,1278,274],{"class":195},[138,1280,1281],{"class":199}," with_retries(",[138,1283,1284],{"class":195},"lambda",[138,1286,1287],{"class":199},": pd.read_excel(",[138,1289,220],{"class":219},[138,1291,162],{"class":199},[138,1293,821],{"class":556},[138,1295,274],{"class":195},[138,1297,826],{"class":148},[138,1299,1300],{"class":199},"), ",[138,1302,1303],{"class":556},"logger",[138,1305,274],{"class":195},[138,1307,1308],{"class":199},"log)\n",[10,1310,1311,1312,1315],{},"The exponential delay matters when the cause is a colleague with the file open in Excel: retrying four times in four seconds fails four times, while waiting 2, 4 and 8 seconds often succeeds. A ",[135,1313,1314],{},"ValueError"," from a validation check passes straight through, which is exactly right — no amount of waiting will fix a missing column.",[123,1317,1319],{"id":1318},"permission-and-lock-errors-deserve-their-own-handling","Permission and lock errors deserve their own handling",[10,1321,1322,1324],{},[135,1323,1023],{}," on Windows almost always means the file is open in Excel, and the message a reader needs is that sentence rather than the traceback:",[128,1326,1328],{"className":186,"code":1327,"language":188,"meta":133,"style":133},"from pathlib import Path\n\ndef save_workbook(wb, target, logger=log):\n    target = Path(target)\n    try:\n        wb.save(target)\n    except PermissionError:\n        lock = target.with_name(f\"~${target.name}\")\n        hint = \" (it looks open in Excel)\" if lock.exists() else \"\"\n        logger.error(\"cannot write %s%s — close the file and re-run\", target, hint)\n        raise SystemExit(2)\n    logger.info(\"wrote %s (%.1f KB)\", target, target.stat().st_size \u002F 1024)\n",[135,1329,1330,1340,1344,1359,1369,1376,1381,1391,1417,1439,1456,1468],{"__ignoreMap":133},[138,1331,1332,1334,1336,1338],{"class":140,"line":141},[138,1333,196],{"class":195},[138,1335,200],{"class":199},[138,1337,203],{"class":195},[138,1339,206],{"class":199},[138,1341,1342],{"class":140,"line":209},[138,1343,213],{"emptyLinePlaceholder":212},[138,1345,1346,1348,1351,1354,1356],{"class":140,"line":216},[138,1347,258],{"class":195},[138,1349,1350],{"class":261}," save_workbook",[138,1352,1353],{"class":199},"(wb, target, logger",[138,1355,274],{"class":195},[138,1357,1358],{"class":199},"log):\n",[138,1360,1361,1364,1366],{"class":140,"line":235},[138,1362,1363],{"class":199},"    target ",[138,1365,274],{"class":195},[138,1367,1368],{"class":199}," Path(target)\n",[138,1370,1371,1374],{"class":140,"line":250},[138,1372,1373],{"class":195},"    try",[138,1375,338],{"class":199},[138,1377,1378],{"class":140,"line":255},[138,1379,1380],{"class":199},"        wb.save(target)\n",[138,1382,1383,1386,1389],{"class":140,"line":268},[138,1384,1385],{"class":195},"    except",[138,1387,1388],{"class":219}," PermissionError",[138,1390,338],{"class":199},[138,1392,1393,1396,1398,1401,1403,1406,1408,1411,1413,1415],{"class":140,"line":280},[138,1394,1395],{"class":199},"        lock ",[138,1397,274],{"class":195},[138,1399,1400],{"class":199}," target.with_name(",[138,1402,301],{"class":195},[138,1404,1405],{"class":148},"\"~$",[138,1407,308],{"class":307},[138,1409,1410],{"class":199},"target.name",[138,1412,313],{"class":307},[138,1414,316],{"class":148},[138,1416,232],{"class":199},[138,1418,1419,1422,1424,1427,1430,1433,1436],{"class":140,"line":295},[138,1420,1421],{"class":199},"        hint ",[138,1423,274],{"class":195},[138,1425,1426],{"class":148}," \" (it looks open in Excel)\"",[138,1428,1429],{"class":195}," if",[138,1431,1432],{"class":199}," lock.exists() ",[138,1434,1435],{"class":195},"else",[138,1437,1438],{"class":148}," \"\"\n",[138,1440,1441,1444,1447,1450,1453],{"class":140,"line":321},[138,1442,1443],{"class":199},"        logger.error(",[138,1445,1446],{"class":148},"\"cannot write ",[138,1448,1449],{"class":307},"%s%s",[138,1451,1452],{"class":148}," — close the file and re-run\"",[138,1454,1455],{"class":199},", target, hint)\n",[138,1457,1458,1460,1462,1464,1466],{"class":140,"line":341},[138,1459,436],{"class":195},[138,1461,439],{"class":219},[138,1463,442],{"class":199},[138,1465,1199],{"class":219},[138,1467,232],{"class":199},[138,1469,1470,1473,1476,1478,1480,1483,1486,1489,1491,1494],{"class":140,"line":361},[138,1471,1472],{"class":199},"    logger.info(",[138,1474,1475],{"class":148},"\"wrote ",[138,1477,764],{"class":307},[138,1479,1020],{"class":148},[138,1481,1482],{"class":307},"%.1f",[138,1484,1485],{"class":148}," KB)\"",[138,1487,1488],{"class":199},", target, target.stat().st_size ",[138,1490,665],{"class":195},[138,1492,1493],{"class":219}," 1024",[138,1495,232],{"class":199},[10,1497,1498,1499,1502],{},"Checking for the ",[135,1500,1501],{},"~$"," lock file turns a guess into a statement. It is also worth writing reports to a path nobody browses to, and copying the finished file to the shared folder afterwards — a job that writes directly into a directory people work in will hit this every few weeks.",[123,1504,1506],{"id":1505},"write-atomically-so-a-crash-cannot-destroy-the-last-good-file","Write atomically so a crash cannot destroy the last good file",[10,1508,1509],{},"The default failure mode of \"open the output and write into it\" is a truncated workbook where a working one used to be. Write beside it and rename:",[128,1511,1513],{"className":186,"code":1512,"language":188,"meta":133,"style":133},"import os\nfrom pathlib import Path\n\ndef atomic_save(wb, target):\n    target = Path(target)\n    temp = target.with_name(f\".{target.name}.tmp\")\n    wb.save(temp)                       # a crash here leaves the original untouched\n    os.replace(temp, target)            # atomic on the same filesystem\n    return target\n",[135,1514,1515,1522,1532,1536,1546,1554,1579,1588,1596],{"__ignoreMap":133},[138,1516,1517,1519],{"class":140,"line":141},[138,1518,203],{"class":195},[138,1520,1521],{"class":199}," os\n",[138,1523,1524,1526,1528,1530],{"class":140,"line":209},[138,1525,196],{"class":195},[138,1527,200],{"class":199},[138,1529,203],{"class":195},[138,1531,206],{"class":199},[138,1533,1534],{"class":140,"line":216},[138,1535,213],{"emptyLinePlaceholder":212},[138,1537,1538,1540,1543],{"class":140,"line":235},[138,1539,258],{"class":195},[138,1541,1542],{"class":261}," atomic_save",[138,1544,1545],{"class":199},"(wb, target):\n",[138,1547,1548,1550,1552],{"class":140,"line":250},[138,1549,1363],{"class":199},[138,1551,274],{"class":195},[138,1553,1368],{"class":199},[138,1555,1556,1559,1561,1563,1565,1568,1570,1572,1574,1577],{"class":140,"line":255},[138,1557,1558],{"class":199},"    temp ",[138,1560,274],{"class":195},[138,1562,1400],{"class":199},[138,1564,301],{"class":195},[138,1566,1567],{"class":148},"\".",[138,1569,308],{"class":307},[138,1571,1410],{"class":199},[138,1573,313],{"class":307},[138,1575,1576],{"class":148},".tmp\"",[138,1578,232],{"class":199},[138,1580,1581,1584],{"class":140,"line":268},[138,1582,1583],{"class":199},"    wb.save(temp)                       ",[138,1585,1587],{"class":1586},"s-wDw","# a crash here leaves the original untouched\n",[138,1589,1590,1593],{"class":140,"line":280},[138,1591,1592],{"class":199},"    os.replace(temp, target)            ",[138,1594,1595],{"class":1586},"# atomic on the same filesystem\n",[138,1597,1598,1600],{"class":140,"line":295},[138,1599,733],{"class":195},[138,1601,1602],{"class":199}," target\n",[10,1604,1605,1608,1609,1613,1614,1617],{},[135,1606,1607],{},"os.replace"," is atomic within a filesystem, so a reader either sees the old file or the new one and never a half-written one. Keeping the temporary file in the ",[1610,1611,1612],"em",{},"same directory"," is what guarantees that — a temp file in ",[135,1615,1616],{},"\u002Ftmp"," and a target on a network share means a copy, not a rename, and the atomicity is lost.",[20,1619,29,1624,29,1627,29,1630,29,1633,29,1637,29,1642,29,1648,29,1653,29,1658,29,1663,29,1667,29,1670,29,1675,29,1678,29,1681,29,1684,29,1688],{"viewBox":1620,"role":23,"ariaLabelledBy":1621,"xmlns":27,"style":28},"0 0 760 218",[1622,1623],"eh-atomic-t","eh-atomic-d",[31,1625,1626],{"id":1622},"Writing in place versus writing then renaming",[35,1628,1629],{"id":1623},"Writing directly into the target means a crash halfway leaves a truncated file and the previous good report is gone. Writing to a temporary file in the same directory and renaming means a crash leaves the previous report intact and the rename is instantaneous.",[39,1631],{"x":41,"y":41,"width":42,"height":1632,"fill":44},"218",[39,1634],{"x":943,"y":48,"width":1635,"height":1636,"rx":51,"fill":75,"stroke":76,"style":54},"340","160",[56,1638,1641],{"x":1639,"y":59,"style":1640},"194","font-size:13px;font-weight:700;fill:var(--danger,#dc2626);text-anchor:middle","write in place",[39,1643],{"x":1644,"y":1645,"width":1646,"height":48,"rx":1647,"fill":44,"stroke":93},"52","74","284","7",[56,1649,1652],{"x":1639,"y":1650,"style":1651},"94","font-size:11.5px;fill:var(--text,#172033);text-anchor:middle","report.xlsx opened for writing",[56,1654,1657],{"x":1639,"y":1655,"style":1656},"128","font-size:11.5px;fill:var(--danger,#dc2626);text-anchor:middle","crash at 60% →",[56,1659,1662],{"x":1639,"y":1660,"style":1661},"150","font-size:11.5px;font-weight:700;fill:var(--danger,#dc2626);text-anchor:middle","truncated file, no fallback",[56,1664,1666],{"x":1639,"y":96,"style":1665},"font-size:11px;fill:var(--muted,#5b6780);text-anchor:middle","last good report is already gone",[39,1668],{"x":1669,"y":48,"width":1635,"height":1636,"rx":51,"fill":110,"stroke":111,"style":54},"396",[56,1671,1674],{"x":1672,"y":59,"style":1673},"566","font-size:13px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","write then rename",[39,1676],{"x":1677,"y":1645,"width":1646,"height":48,"rx":1647,"fill":44,"stroke":93},"424",[56,1679,1680],{"x":1672,"y":1650,"style":1651},".report.xlsx.tmp in the same folder",[56,1682,1657],{"x":1672,"y":1655,"style":1683},"font-size:11.5px;fill:var(--teal,#0f9488);text-anchor:middle",[56,1685,1687],{"x":1672,"y":1660,"style":1686},"font-size:11.5px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","previous report still there",[56,1689,1690],{"x":1672,"y":96,"style":1665},"os.replace swaps it in instantly",[123,1692,1694],{"id":1693},"check-the-output-before-anyone-receives-it","Check the output before anyone receives it",[10,1696,1697],{},"A job can succeed technically and produce a useless file. Assert the properties a good report has, and refuse to deliver one that fails:",[128,1699,1701],{"className":186,"code":1700,"language":188,"meta":133,"style":133},"from pathlib import Path\n\nimport pandas as pd\n\ndef validate_output(path, min_rows=1, min_bytes=5_000, expected_sheets=(\"Summary\",)):\n    path = Path(path)\n    problems = []\n    if not path.exists():\n        return [f\"{path} was not created\"]\n    if path.stat().st_size \u003C min_bytes:\n        problems.append(f\"file is only {path.stat().st_size} bytes\")\n\n    book = pd.ExcelFile(path)\n    for sheet in expected_sheets:\n        if sheet not in book.sheet_names:\n            problems.append(f\"missing sheet {sheet!r}\")\n            continue\n        frame = book.parse(sheet)\n        if len(frame) \u003C min_rows:\n            problems.append(f\"sheet {sheet!r} has {len(frame)} row(s), expected >= {min_rows}\")\n    return problems\n\nissues = validate_output(\"reports\u002Fmarch.xlsx\")\nif issues:\n    log.error(\"output validation failed: %s\", \"; \".join(issues))\n    raise SystemExit(1)\nlog.info(\"output validated\")\n",[135,1702,1703,1713,1717,1727,1731,1766,1776,1784,1793,1818,1831,1852,1856,1866,1878,1894,1918,1923,1933,1947,1990,1997,2001,2016,2024,2045,2059],{"__ignoreMap":133},[138,1704,1705,1707,1709,1711],{"class":140,"line":141},[138,1706,196],{"class":195},[138,1708,200],{"class":199},[138,1710,203],{"class":195},[138,1712,206],{"class":199},[138,1714,1715],{"class":140,"line":209},[138,1716,213],{"emptyLinePlaceholder":212},[138,1718,1719,1721,1723,1725],{"class":140,"line":216},[138,1720,203],{"class":195},[138,1722,794],{"class":199},[138,1724,797],{"class":195},[138,1726,800],{"class":199},[138,1728,1729],{"class":140,"line":235},[138,1730,213],{"emptyLinePlaceholder":212},[138,1732,1733,1735,1738,1741,1743,1745,1748,1750,1753,1756,1758,1760,1763],{"class":140,"line":250},[138,1734,258],{"class":195},[138,1736,1737],{"class":261}," validate_output",[138,1739,1740],{"class":199},"(path, min_rows",[138,1742,274],{"class":195},[138,1744,1096],{"class":219},[138,1746,1747],{"class":199},", min_bytes",[138,1749,274],{"class":195},[138,1751,1752],{"class":219},"5_000",[138,1754,1755],{"class":199},", expected_sheets",[138,1757,274],{"class":195},[138,1759,442],{"class":199},[138,1761,1762],{"class":148},"\"Summary\"",[138,1764,1765],{"class":199},",)):\n",[138,1767,1768,1771,1773],{"class":140,"line":255},[138,1769,1770],{"class":199},"    path ",[138,1772,274],{"class":195},[138,1774,1775],{"class":199}," Path(path)\n",[138,1777,1778,1780,1782],{"class":140,"line":268},[138,1779,271],{"class":199},[138,1781,274],{"class":195},[138,1783,277],{"class":199},[138,1785,1786,1788,1790],{"class":140,"line":280},[138,1787,283],{"class":195},[138,1789,286],{"class":195},[138,1791,1792],{"class":199}," path.exists():\n",[138,1794,1795,1798,1801,1803,1805,1807,1810,1812,1815],{"class":140,"line":295},[138,1796,1797],{"class":195},"        return",[138,1799,1800],{"class":199}," [",[138,1802,301],{"class":195},[138,1804,316],{"class":148},[138,1806,308],{"class":307},[138,1808,1809],{"class":199},"path",[138,1811,313],{"class":307},[138,1813,1814],{"class":148}," was not created\"",[138,1816,1817],{"class":199},"]\n",[138,1819,1820,1822,1825,1828],{"class":140,"line":321},[138,1821,283],{"class":195},[138,1823,1824],{"class":199}," path.stat().st_size ",[138,1826,1827],{"class":195},"\u003C",[138,1829,1830],{"class":199}," min_bytes:\n",[138,1832,1833,1835,1837,1840,1842,1845,1847,1850],{"class":140,"line":341},[138,1834,298],{"class":199},[138,1836,301],{"class":195},[138,1838,1839],{"class":148},"\"file is only ",[138,1841,308],{"class":307},[138,1843,1844],{"class":199},"path.stat().st_size",[138,1846,313],{"class":307},[138,1848,1849],{"class":148}," bytes\"",[138,1851,232],{"class":199},[138,1853,1854],{"class":140,"line":361},[138,1855,213],{"emptyLinePlaceholder":212},[138,1857,1858,1861,1863],{"class":140,"line":373},[138,1859,1860],{"class":199},"    book ",[138,1862,274],{"class":195},[138,1864,1865],{"class":199}," pd.ExcelFile(path)\n",[138,1867,1868,1870,1873,1875],{"class":140,"line":393},[138,1869,1082],{"class":195},[138,1871,1872],{"class":199}," sheet ",[138,1874,1088],{"class":195},[138,1876,1877],{"class":199}," expected_sheets:\n",[138,1879,1880,1883,1885,1888,1891],{"class":140,"line":405},[138,1881,1882],{"class":195},"        if",[138,1884,1872],{"class":199},[138,1886,1887],{"class":195},"not",[138,1889,1890],{"class":195}," in",[138,1892,1893],{"class":199}," book.sheet_names:\n",[138,1895,1896,1899,1901,1904,1906,1909,1912,1914,1916],{"class":140,"line":425},[138,1897,1898],{"class":199},"            problems.append(",[138,1900,301],{"class":195},[138,1902,1903],{"class":148},"\"missing sheet ",[138,1905,308],{"class":307},[138,1907,1908],{"class":199},"sheet",[138,1910,1911],{"class":195},"!r",[138,1913,313],{"class":307},[138,1915,316],{"class":148},[138,1917,232],{"class":199},[138,1919,1920],{"class":140,"line":433},[138,1921,1922],{"class":195},"            continue\n",[138,1924,1925,1928,1930],{"class":140,"line":467},[138,1926,1927],{"class":199},"        frame ",[138,1929,274],{"class":195},[138,1931,1932],{"class":199}," book.parse(sheet)\n",[138,1934,1935,1937,1939,1942,1944],{"class":140,"line":472},[138,1936,1882],{"class":195},[138,1938,919],{"class":219},[138,1940,1941],{"class":199},"(frame) ",[138,1943,1827],{"class":195},[138,1945,1946],{"class":199}," min_rows:\n",[138,1948,1949,1951,1953,1956,1958,1960,1962,1964,1967,1969,1971,1974,1976,1979,1981,1984,1986,1988],{"class":140,"line":724},[138,1950,1898],{"class":199},[138,1952,301],{"class":195},[138,1954,1955],{"class":148},"\"sheet ",[138,1957,308],{"class":307},[138,1959,1908],{"class":199},[138,1961,1911],{"class":195},[138,1963,313],{"class":307},[138,1965,1966],{"class":148}," has ",[138,1968,308],{"class":307},[138,1970,849],{"class":219},[138,1972,1973],{"class":199},"(frame)",[138,1975,313],{"class":307},[138,1977,1978],{"class":148}," row(s), expected >= ",[138,1980,308],{"class":307},[138,1982,1983],{"class":199},"min_rows",[138,1985,313],{"class":307},[138,1987,316],{"class":148},[138,1989,232],{"class":199},[138,1991,1992,1994],{"class":140,"line":730},[138,1993,733],{"class":195},[138,1995,1996],{"class":199}," problems\n",[138,1998,1999],{"class":140,"line":739},[138,2000,213],{"emptyLinePlaceholder":212},[138,2002,2003,2006,2008,2011,2014],{"class":140,"line":744},[138,2004,2005],{"class":199},"issues ",[138,2007,274],{"class":195},[138,2009,2010],{"class":199}," validate_output(",[138,2012,2013],{"class":148},"\"reports\u002Fmarch.xlsx\"",[138,2015,232],{"class":199},[138,2017,2018,2021],{"class":140,"line":755},[138,2019,2020],{"class":195},"if",[138,2022,2023],{"class":199}," issues:\n",[138,2025,2027,2030,2033,2035,2037,2039,2042],{"class":140,"line":2026},25,[138,2028,2029],{"class":199},"    log.error(",[138,2031,2032],{"class":148},"\"output validation failed: ",[138,2034,764],{"class":307},[138,2036,316],{"class":148},[138,2038,162],{"class":199},[138,2040,2041],{"class":148},"\"; \"",[138,2043,2044],{"class":199},".join(issues))\n",[138,2046,2048,2051,2053,2055,2057],{"class":140,"line":2047},26,[138,2049,2050],{"class":195},"    raise",[138,2052,439],{"class":219},[138,2054,442],{"class":199},[138,2056,1096],{"class":219},[138,2058,232],{"class":199},[138,2060,2062,2064,2067],{"class":140,"line":2061},27,[138,2063,758],{"class":199},[138,2065,2066],{"class":148},"\"output validated\"",[138,2068,232],{"class":199},[10,2070,2071],{},"This is the check that stops the classic embarrassment: a source system returned nothing, every stage handled the empty frame gracefully, and a beautifully formatted workbook of zeros went out to forty people. Comparing against the previous run — row counts within a sensible band — catches the subtler version, where the file is not empty but is missing a third of the data.",[123,2073,2075],{"id":2074},"exit-codes-and-alerts-a-human-will-read","Exit codes and alerts a human will read",[10,2077,2078],{},"Distinguish outcomes so a scheduler can act, and alert on the outcome rather than on the exception:",[128,2080,2082],{"className":186,"code":2081,"language":188,"meta":133,"style":133},"import smtplib\nimport sys\nfrom email.message import EmailMessage\n\nOK, DATA_PROBLEM, ENVIRONMENT_PROBLEM = 0, 1, 2\n\ndef alert(subject, body, to=\"ops@example.com\", host=\"smtp.example.com\"):\n    msg = EmailMessage()\n    msg[\"Subject\"] = subject\n    msg[\"From\"] = \"reports@example.com\"\n    msg[\"To\"] = to\n    msg.set_content(body)\n    with smtplib.SMTP(host, 587, timeout=30) as smtp:\n        smtp.starttls()\n        smtp.send_message(msg)\n\ndef main():\n    try:\n        preflight()\n        report = build_report()                 # your pipeline\n        issues = validate_output(report)\n        if issues:\n            alert(\"Report NOT sent — output failed validation\",\n                  \"The March report was not delivered.\\n\\n\" + \"\\n\".join(issues))\n            return DATA_PROBLEM\n        deliver(report)                          # email, upload, copy to share\n        log.info(\"run complete report=%s\", report)\n        return OK\n    except SystemExit:\n        raise\n    except Exception:\n        log.exception(\"unhandled failure\")       # traceback goes to the log\n        alert(\"Report job failed\", \"See logs\u002Freport.log for the traceback.\")\n        return ENVIRONMENT_PROBLEM\n\nif __name__ == \"__main__\":\n    sys.exit(main())\n",[135,2083,2084,2091,2098,2110,2114,2142,2146,2171,2181,2197,2211,2225,2230,2258,2263,2268,2272,2281,2287,2292,2305,2315,2321,2331,2351,2358,2366,2381,2389,2398,2404,2414,2429,2445,2453,2458,2474],{"__ignoreMap":133},[138,2085,2086,2088],{"class":140,"line":141},[138,2087,203],{"class":195},[138,2089,2090],{"class":199}," smtplib\n",[138,2092,2093,2095],{"class":140,"line":209},[138,2094,203],{"class":195},[138,2096,2097],{"class":199}," sys\n",[138,2099,2100,2102,2105,2107],{"class":140,"line":216},[138,2101,196],{"class":195},[138,2103,2104],{"class":199}," email.message ",[138,2106,203],{"class":195},[138,2108,2109],{"class":199}," EmailMessage\n",[138,2111,2112],{"class":140,"line":235},[138,2113,213],{"emptyLinePlaceholder":212},[138,2115,2116,2119,2121,2124,2126,2129,2131,2133,2135,2137,2139],{"class":140,"line":250},[138,2117,2118],{"class":219},"OK",[138,2120,162],{"class":199},[138,2122,2123],{"class":219},"DATA_PROBLEM",[138,2125,162],{"class":199},[138,2127,2128],{"class":219},"ENVIRONMENT_PROBLEM",[138,2130,223],{"class":195},[138,2132,335],{"class":219},[138,2134,162],{"class":199},[138,2136,1096],{"class":219},[138,2138,162],{"class":199},[138,2140,2141],{"class":219},"2\n",[138,2143,2144],{"class":140,"line":255},[138,2145,213],{"emptyLinePlaceholder":212},[138,2147,2148,2150,2153,2156,2158,2161,2164,2166,2169],{"class":140,"line":268},[138,2149,258],{"class":195},[138,2151,2152],{"class":261}," alert",[138,2154,2155],{"class":199},"(subject, body, to",[138,2157,274],{"class":195},[138,2159,2160],{"class":148},"\"ops@example.com\"",[138,2162,2163],{"class":199},", host",[138,2165,274],{"class":195},[138,2167,2168],{"class":148},"\"smtp.example.com\"",[138,2170,548],{"class":199},[138,2172,2173,2176,2178],{"class":140,"line":280},[138,2174,2175],{"class":199},"    msg ",[138,2177,274],{"class":195},[138,2179,2180],{"class":199}," EmailMessage()\n",[138,2182,2183,2186,2189,2192,2194],{"class":140,"line":295},[138,2184,2185],{"class":199},"    msg[",[138,2187,2188],{"class":148},"\"Subject\"",[138,2190,2191],{"class":199},"] ",[138,2193,274],{"class":195},[138,2195,2196],{"class":199}," subject\n",[138,2198,2199,2201,2204,2206,2208],{"class":140,"line":321},[138,2200,2185],{"class":199},[138,2202,2203],{"class":148},"\"From\"",[138,2205,2191],{"class":199},[138,2207,274],{"class":195},[138,2209,2210],{"class":148}," \"reports@example.com\"\n",[138,2212,2213,2215,2218,2220,2222],{"class":140,"line":341},[138,2214,2185],{"class":199},[138,2216,2217],{"class":148},"\"To\"",[138,2219,2191],{"class":199},[138,2221,274],{"class":195},[138,2223,2224],{"class":199}," to\n",[138,2226,2227],{"class":140,"line":361},[138,2228,2229],{"class":199},"    msg.set_content(body)\n",[138,2231,2232,2235,2238,2241,2243,2246,2248,2250,2253,2255],{"class":140,"line":373},[138,2233,2234],{"class":195},"    with",[138,2236,2237],{"class":199}," smtplib.SMTP(host, ",[138,2239,2240],{"class":219},"587",[138,2242,162],{"class":199},[138,2244,2245],{"class":556},"timeout",[138,2247,274],{"class":195},[138,2249,48],{"class":219},[138,2251,2252],{"class":199},") ",[138,2254,797],{"class":195},[138,2256,2257],{"class":199}," smtp:\n",[138,2259,2260],{"class":140,"line":393},[138,2261,2262],{"class":199},"        smtp.starttls()\n",[138,2264,2265],{"class":140,"line":405},[138,2266,2267],{"class":199},"        smtp.send_message(msg)\n",[138,2269,2270],{"class":140,"line":425},[138,2271,213],{"emptyLinePlaceholder":212},[138,2273,2274,2276,2279],{"class":140,"line":433},[138,2275,258],{"class":195},[138,2277,2278],{"class":261}," main",[138,2280,265],{"class":199},[138,2282,2283,2285],{"class":140,"line":467},[138,2284,1373],{"class":195},[138,2286,338],{"class":199},[138,2288,2289],{"class":140,"line":472},[138,2290,2291],{"class":199},"        preflight()\n",[138,2293,2294,2297,2299,2302],{"class":140,"line":724},[138,2295,2296],{"class":199},"        report ",[138,2298,274],{"class":195},[138,2300,2301],{"class":199}," build_report()                 ",[138,2303,2304],{"class":1586},"# your pipeline\n",[138,2306,2307,2310,2312],{"class":140,"line":730},[138,2308,2309],{"class":199},"        issues ",[138,2311,274],{"class":195},[138,2313,2314],{"class":199}," validate_output(report)\n",[138,2316,2317,2319],{"class":140,"line":739},[138,2318,1882],{"class":195},[138,2320,2023],{"class":199},[138,2322,2323,2326,2329],{"class":140,"line":744},[138,2324,2325],{"class":199},"            alert(",[138,2327,2328],{"class":148},"\"Report NOT sent — output failed validation\"",[138,2330,629],{"class":199},[138,2332,2333,2336,2339,2341,2343,2345,2347,2349],{"class":140,"line":755},[138,2334,2335],{"class":148},"                  \"The March report was not delivered.",[138,2337,2338],{"class":307},"\\n\\n",[138,2340,316],{"class":148},[138,2342,454],{"class":195},[138,2344,457],{"class":148},[138,2346,448],{"class":307},[138,2348,316],{"class":148},[138,2350,2044],{"class":199},[138,2352,2353,2355],{"class":140,"line":2026},[138,2354,1119],{"class":195},[138,2356,2357],{"class":219}," DATA_PROBLEM\n",[138,2359,2360,2363],{"class":140,"line":2047},[138,2361,2362],{"class":199},"        deliver(report)                          ",[138,2364,2365],{"class":1586},"# email, upload, copy to share\n",[138,2367,2368,2371,2374,2376,2378],{"class":140,"line":2061},[138,2369,2370],{"class":199},"        log.info(",[138,2372,2373],{"class":148},"\"run complete report=",[138,2375,764],{"class":307},[138,2377,316],{"class":148},[138,2379,2380],{"class":199},", report)\n",[138,2382,2384,2386],{"class":140,"line":2383},28,[138,2385,1797],{"class":195},[138,2387,2388],{"class":219}," OK\n",[138,2390,2392,2394,2396],{"class":140,"line":2391},29,[138,2393,1385],{"class":195},[138,2395,439],{"class":219},[138,2397,338],{"class":199},[138,2399,2401],{"class":140,"line":2400},30,[138,2402,2403],{"class":195},"        raise\n",[138,2405,2407,2409,2412],{"class":140,"line":2406},31,[138,2408,1385],{"class":195},[138,2410,2411],{"class":219}," Exception",[138,2413,338],{"class":199},[138,2415,2417,2420,2423,2426],{"class":140,"line":2416},32,[138,2418,2419],{"class":199},"        log.exception(",[138,2421,2422],{"class":148},"\"unhandled failure\"",[138,2424,2425],{"class":199},")       ",[138,2427,2428],{"class":1586},"# traceback goes to the log\n",[138,2430,2432,2435,2438,2440,2443],{"class":140,"line":2431},33,[138,2433,2434],{"class":199},"        alert(",[138,2436,2437],{"class":148},"\"Report job failed\"",[138,2439,162],{"class":199},[138,2441,2442],{"class":148},"\"See logs\u002Freport.log for the traceback.\"",[138,2444,232],{"class":199},[138,2446,2448,2450],{"class":140,"line":2447},34,[138,2449,1797],{"class":195},[138,2451,2452],{"class":219}," ENVIRONMENT_PROBLEM\n",[138,2454,2456],{"class":140,"line":2455},35,[138,2457,213],{"emptyLinePlaceholder":212},[138,2459,2461,2463,2466,2469,2472],{"class":140,"line":2460},36,[138,2462,2020],{"class":195},[138,2464,2465],{"class":219}," __name__",[138,2467,2468],{"class":195}," ==",[138,2470,2471],{"class":148}," \"__main__\"",[138,2473,338],{"class":199},[138,2475,2477],{"class":140,"line":2476},37,[138,2478,2479],{"class":199},"    sys.exit(main())\n",[10,2481,2482,2485,2486,2490],{},[135,2483,2484],{},"log.exception"," records the traceback where an engineer will look; the email says what a reader needs. Sending the traceback itself to a distribution list is how alerting becomes noise — after a few weeks the messages are filtered away and the next real failure goes unnoticed. The ",[14,2487,2489],{"href":2488},"\u002Fautomating-reporting-workflows\u002Femailing-excel-reports-with-smtplib\u002F","emailing Excel reports"," guide covers the delivery side properly.",[123,2492,2494],{"id":2493},"isolate-failures-in-a-batch","Isolate failures in a batch",[10,2496,2497],{},"When one job produces many reports — one per region, one per client — a single bad input should not take the others down. Process each unit inside its own error boundary and summarise at the end:",[128,2499,2501],{"className":186,"code":2500,"language":188,"meta":133,"style":133},"def run_batch(units):\n    succeeded, failed = [], []\n    for unit in units:\n        try:\n            output = build_report_for(unit)          # your per-unit pipeline\n            issues = validate_output(output)\n            if issues:\n                raise ValueError(\"; \".join(issues))\n            succeeded.append((unit, output))\n            log.info(\"unit=%s ok output=%s\", unit, output)\n        except Exception as exc:                      # noqa: BLE001 - boundary is deliberate\n            failed.append((unit, f\"{type(exc).__name__}: {exc}\"))\n            log.exception(\"unit=%s failed\", unit)\n\n    log.info(\"batch complete ok=%d failed=%d\", len(succeeded), len(failed))\n    if failed:\n        alert(\n            f\"{len(failed)} of {len(units)} reports failed\",\n            \"\\n\".join(f\"{unit}: {reason}\" for unit, reason in failed),\n        )\n    return succeeded, failed\n",[135,2502,2503,2513,2523,2535,2541,2554,2564,2570,2584,2589,2609,2623,2659,2674,2678,2707,2714,2719,2752,2797,2802],{"__ignoreMap":133},[138,2504,2505,2507,2510],{"class":140,"line":141},[138,2506,258],{"class":195},[138,2508,2509],{"class":261}," run_batch",[138,2511,2512],{"class":199},"(units):\n",[138,2514,2515,2518,2520],{"class":140,"line":209},[138,2516,2517],{"class":199},"    succeeded, failed ",[138,2519,274],{"class":195},[138,2521,2522],{"class":199}," [], []\n",[138,2524,2525,2527,2530,2532],{"class":140,"line":216},[138,2526,1082],{"class":195},[138,2528,2529],{"class":199}," unit ",[138,2531,1088],{"class":195},[138,2533,2534],{"class":199}," units:\n",[138,2536,2537,2539],{"class":140,"line":235},[138,2538,1112],{"class":195},[138,2540,338],{"class":199},[138,2542,2543,2546,2548,2551],{"class":140,"line":250},[138,2544,2545],{"class":199},"            output ",[138,2547,274],{"class":195},[138,2549,2550],{"class":199}," build_report_for(unit)          ",[138,2552,2553],{"class":1586},"# your per-unit pipeline\n",[138,2555,2556,2559,2561],{"class":140,"line":255},[138,2557,2558],{"class":199},"            issues ",[138,2560,274],{"class":195},[138,2562,2563],{"class":199}," validate_output(output)\n",[138,2565,2566,2568],{"class":140,"line":268},[138,2567,1141],{"class":195},[138,2569,2023],{"class":199},[138,2571,2572,2575,2578,2580,2582],{"class":140,"line":280},[138,2573,2574],{"class":195},"                raise",[138,2576,2577],{"class":219}," ValueError",[138,2579,442],{"class":199},[138,2581,2041],{"class":148},[138,2583,2044],{"class":199},[138,2585,2586],{"class":140,"line":295},[138,2587,2588],{"class":199},"            succeeded.append((unit, output))\n",[138,2590,2591,2594,2597,2599,2602,2604,2606],{"class":140,"line":321},[138,2592,2593],{"class":199},"            log.info(",[138,2595,2596],{"class":148},"\"unit=",[138,2598,764],{"class":307},[138,2600,2601],{"class":148}," ok output=",[138,2603,764],{"class":307},[138,2605,316],{"class":148},[138,2607,2608],{"class":199},", unit, output)\n",[138,2610,2611,2613,2615,2617,2620],{"class":140,"line":341},[138,2612,1127],{"class":195},[138,2614,2411],{"class":219},[138,2616,1133],{"class":195},[138,2618,2619],{"class":199}," exc:                      ",[138,2621,2622],{"class":1586},"# noqa: BLE001 - boundary is deliberate\n",[138,2624,2625,2628,2630,2632,2634,2637,2640,2643,2645,2648,2650,2653,2655,2657],{"class":140,"line":361},[138,2626,2627],{"class":199},"            failed.append((unit, ",[138,2629,301],{"class":195},[138,2631,316],{"class":148},[138,2633,308],{"class":307},[138,2635,2636],{"class":219},"type",[138,2638,2639],{"class":199},"(exc).",[138,2641,2642],{"class":219},"__name__",[138,2644,313],{"class":307},[138,2646,2647],{"class":148},": ",[138,2649,308],{"class":307},[138,2651,2652],{"class":199},"exc",[138,2654,313],{"class":307},[138,2656,316],{"class":148},[138,2658,1212],{"class":199},[138,2660,2661,2664,2666,2668,2671],{"class":140,"line":373},[138,2662,2663],{"class":199},"            log.exception(",[138,2665,2596],{"class":148},[138,2667,764],{"class":307},[138,2669,2670],{"class":148}," failed\"",[138,2672,2673],{"class":199},", unit)\n",[138,2675,2676],{"class":140,"line":393},[138,2677,213],{"emptyLinePlaceholder":212},[138,2679,2680,2683,2686,2688,2691,2693,2695,2697,2699,2702,2704],{"class":140,"line":405},[138,2681,2682],{"class":199},"    log.info(",[138,2684,2685],{"class":148},"\"batch complete ok=",[138,2687,642],{"class":307},[138,2689,2690],{"class":148}," failed=",[138,2692,642],{"class":307},[138,2694,316],{"class":148},[138,2696,162],{"class":199},[138,2698,849],{"class":219},[138,2700,2701],{"class":199},"(succeeded), ",[138,2703,849],{"class":219},[138,2705,2706],{"class":199},"(failed))\n",[138,2708,2709,2711],{"class":140,"line":425},[138,2710,283],{"class":195},[138,2712,2713],{"class":199}," failed:\n",[138,2715,2716],{"class":140,"line":433},[138,2717,2718],{"class":199},"        alert(\n",[138,2720,2721,2724,2726,2728,2730,2733,2735,2738,2740,2742,2745,2747,2750],{"class":140,"line":467},[138,2722,2723],{"class":195},"            f",[138,2725,316],{"class":148},[138,2727,308],{"class":307},[138,2729,849],{"class":219},[138,2731,2732],{"class":199},"(failed)",[138,2734,313],{"class":307},[138,2736,2737],{"class":148}," of ",[138,2739,308],{"class":307},[138,2741,849],{"class":219},[138,2743,2744],{"class":199},"(units)",[138,2746,313],{"class":307},[138,2748,2749],{"class":148}," reports failed\"",[138,2751,629],{"class":199},[138,2753,2754,2757,2759,2761,2764,2766,2768,2770,2773,2775,2777,2779,2782,2784,2786,2789,2792,2794],{"class":140,"line":472},[138,2755,2756],{"class":148},"            \"",[138,2758,448],{"class":307},[138,2760,316],{"class":148},[138,2762,2763],{"class":199},".join(",[138,2765,301],{"class":195},[138,2767,316],{"class":148},[138,2769,308],{"class":307},[138,2771,2772],{"class":199},"unit",[138,2774,313],{"class":307},[138,2776,2647],{"class":148},[138,2778,308],{"class":307},[138,2780,2781],{"class":199},"reason",[138,2783,313],{"class":307},[138,2785,316],{"class":148},[138,2787,2788],{"class":195}," for",[138,2790,2791],{"class":199}," unit, reason ",[138,2793,1088],{"class":195},[138,2795,2796],{"class":199}," failed),\n",[138,2798,2799],{"class":140,"line":724},[138,2800,2801],{"class":199},"        )\n",[138,2803,2804,2806],{"class":140,"line":730},[138,2805,733],{"class":195},[138,2807,2808],{"class":199}," succeeded, failed\n",[10,2810,2811,2812,2815],{},"The broad ",[135,2813,2814],{},"except"," here is a deliberate boundary rather than laziness — it exists precisely so that one region's malformed file does not deprive the other eleven of their reports. Everything inside it is still logged with a full traceback, and the alert names each failing unit with its reason, so the follow-up is targeted rather than a re-run of the whole batch.",[10,2817,2818,2819,2821],{},"One rule keeps this honest: a batch that fails entirely should exit non-zero even though every individual failure was handled. Twelve failures out of twelve is not a partial success, and a scheduler that sees exit code ",[135,2820,41],{}," will never tell anyone.",[123,2823,2825],{"id":2824},"make-the-job-safe-to-re-run","Make the job safe to re-run",[10,2827,2828],{},"Anything scheduled will eventually be run twice — by a retry, by a person, or by two schedulers that both think they own it. Design for it:",[128,2830,2832],{"className":186,"code":2831,"language":188,"meta":133,"style":133},"from pathlib import Path\n\ndef already_done(marker_dir=\"state\", key=\"2026-03\"):\n    marker = Path(marker_dir) \u002F f\"{key}.done\"\n    if marker.exists():\n        log.info(\"run for %s already completed at %s\", key, marker.read_text().strip())\n        return True\n    return False\n\ndef mark_done(marker_dir=\"state\", key=\"2026-03\"):\n    from datetime import datetime, timezone\n    Path(marker_dir).mkdir(exist_ok=True)\n    (Path(marker_dir) \u002F f\"{key}.done\").write_text(datetime.now(timezone.utc).isoformat())\n",[135,2833,2834,2844,2848,2873,2899,2906,2925,2932,2939,2943,2964,2977,2990],{"__ignoreMap":133},[138,2835,2836,2838,2840,2842],{"class":140,"line":141},[138,2837,196],{"class":195},[138,2839,200],{"class":199},[138,2841,203],{"class":195},[138,2843,206],{"class":199},[138,2845,2846],{"class":140,"line":209},[138,2847,213],{"emptyLinePlaceholder":212},[138,2849,2850,2852,2855,2858,2860,2863,2866,2868,2871],{"class":140,"line":216},[138,2851,258],{"class":195},[138,2853,2854],{"class":261}," already_done",[138,2856,2857],{"class":199},"(marker_dir",[138,2859,274],{"class":195},[138,2861,2862],{"class":148},"\"state\"",[138,2864,2865],{"class":199},", key",[138,2867,274],{"class":195},[138,2869,2870],{"class":148},"\"2026-03\"",[138,2872,548],{"class":199},[138,2874,2875,2878,2880,2883,2885,2887,2889,2891,2894,2896],{"class":140,"line":235},[138,2876,2877],{"class":199},"    marker ",[138,2879,274],{"class":195},[138,2881,2882],{"class":199}," Path(marker_dir) ",[138,2884,665],{"class":195},[138,2886,668],{"class":195},[138,2888,316],{"class":148},[138,2890,308],{"class":307},[138,2892,2893],{"class":199},"key",[138,2895,313],{"class":307},[138,2897,2898],{"class":148},".done\"\n",[138,2900,2901,2903],{"class":140,"line":250},[138,2902,283],{"class":195},[138,2904,2905],{"class":199}," marker.exists():\n",[138,2907,2908,2910,2913,2915,2918,2920,2922],{"class":140,"line":255},[138,2909,2370],{"class":199},[138,2911,2912],{"class":148},"\"run for ",[138,2914,764],{"class":307},[138,2916,2917],{"class":148}," already completed at ",[138,2919,764],{"class":307},[138,2921,316],{"class":148},[138,2923,2924],{"class":199},", key, marker.read_text().strip())\n",[138,2926,2927,2929],{"class":140,"line":268},[138,2928,1797],{"class":195},[138,2930,2931],{"class":219}," True\n",[138,2933,2934,2936],{"class":140,"line":280},[138,2935,733],{"class":195},[138,2937,2938],{"class":219}," False\n",[138,2940,2941],{"class":140,"line":295},[138,2942,213],{"emptyLinePlaceholder":212},[138,2944,2945,2947,2950,2952,2954,2956,2958,2960,2962],{"class":140,"line":321},[138,2946,258],{"class":195},[138,2948,2949],{"class":261}," mark_done",[138,2951,2857],{"class":199},[138,2953,274],{"class":195},[138,2955,2862],{"class":148},[138,2957,2865],{"class":199},[138,2959,274],{"class":195},[138,2961,2870],{"class":148},[138,2963,548],{"class":199},[138,2965,2966,2969,2972,2974],{"class":140,"line":341},[138,2967,2968],{"class":195},"    from",[138,2970,2971],{"class":199}," datetime ",[138,2973,203],{"class":195},[138,2975,2976],{"class":199}," datetime, timezone\n",[138,2978,2979,2982,2984,2986,2988],{"class":140,"line":361},[138,2980,2981],{"class":199},"    Path(marker_dir).mkdir(",[138,2983,567],{"class":556},[138,2985,274],{"class":195},[138,2987,562],{"class":219},[138,2989,232],{"class":199},[138,2991,2992,2995,2997,2999,3001,3003,3005,3007,3010],{"class":140,"line":373},[138,2993,2994],{"class":199},"    (Path(marker_dir) ",[138,2996,665],{"class":195},[138,2998,668],{"class":195},[138,3000,316],{"class":148},[138,3002,308],{"class":307},[138,3004,2893],{"class":199},[138,3006,313],{"class":307},[138,3008,3009],{"class":148},".done\"",[138,3011,3012],{"class":199},").write_text(datetime.now(timezone.utc).isoformat())\n",[10,3014,3015,3016,3019,3020,3023],{},"Deterministic output filenames help too: writing ",[135,3017,3018],{},"march_2026.xlsx"," rather than ",[135,3021,3022],{},"report_20260302_060411.xlsx"," means a second run replaces the file instead of adding a near-duplicate that someone will later email by mistake. Where a timestamped archive is genuinely wanted, keep both — a stable current name and a dated copy.",[123,3025,3027],{"id":3026},"keep-the-logs-from-becoming-the-problem","Keep the logs from becoming the problem",[10,3029,3030],{},"A daily job left running for two years produces a log file nobody can open. Rotation solves it, and choosing the right rotation says something about how you investigate:",[128,3032,3034],{"className":186,"code":3033,"language":188,"meta":133,"style":133},"import logging\nfrom logging.handlers import RotatingFileHandler, TimedRotatingFileHandler\nfrom pathlib import Path\n\ndef add_rotating_file(logger, path=\"logs\u002Freport.log\", by=\"size\"):\n    Path(path).parent.mkdir(parents=True, exist_ok=True)\n    if by == \"size\":\n        handler = RotatingFileHandler(path, maxBytes=5_000_000, backupCount=5, encoding=\"utf-8\")\n    else:\n        handler = TimedRotatingFileHandler(path, when=\"midnight\", backupCount=30, encoding=\"utf-8\")\n    handler.setFormatter(\n        logging.Formatter(\"%(asctime)s %(levelname)-8s %(message)s\", \"%Y-%m-%d %H:%M:%S\")\n    )\n    logger.addHandler(handler)\n    return logger\n",[135,3035,3036,3042,3054,3064,3068,3093,3114,3128,3166,3173,3208,3213,3238,3242,3247],{"__ignoreMap":133},[138,3037,3038,3040],{"class":140,"line":141},[138,3039,203],{"class":195},[138,3041,497],{"class":199},[138,3043,3044,3046,3049,3051],{"class":140,"line":209},[138,3045,196],{"class":195},[138,3047,3048],{"class":199}," logging.handlers ",[138,3050,203],{"class":195},[138,3052,3053],{"class":199}," RotatingFileHandler, TimedRotatingFileHandler\n",[138,3055,3056,3058,3060,3062],{"class":140,"line":216},[138,3057,196],{"class":195},[138,3059,200],{"class":199},[138,3061,203],{"class":195},[138,3063,206],{"class":199},[138,3065,3066],{"class":140,"line":235},[138,3067,213],{"emptyLinePlaceholder":212},[138,3069,3070,3072,3075,3078,3080,3083,3086,3088,3091],{"class":140,"line":250},[138,3071,258],{"class":195},[138,3073,3074],{"class":261}," add_rotating_file",[138,3076,3077],{"class":199},"(logger, path",[138,3079,274],{"class":195},[138,3081,3082],{"class":148},"\"logs\u002Freport.log\"",[138,3084,3085],{"class":199},", by",[138,3087,274],{"class":195},[138,3089,3090],{"class":148},"\"size\"",[138,3092,548],{"class":199},[138,3094,3095,3098,3100,3102,3104,3106,3108,3110,3112],{"class":140,"line":255},[138,3096,3097],{"class":199},"    Path(path).parent.mkdir(",[138,3099,557],{"class":556},[138,3101,274],{"class":195},[138,3103,562],{"class":219},[138,3105,162],{"class":199},[138,3107,567],{"class":556},[138,3109,274],{"class":195},[138,3111,562],{"class":219},[138,3113,232],{"class":199},[138,3115,3116,3118,3121,3123,3126],{"class":140,"line":268},[138,3117,283],{"class":195},[138,3119,3120],{"class":199}," by ",[138,3122,332],{"class":195},[138,3124,3125],{"class":148}," \"size\"",[138,3127,338],{"class":199},[138,3129,3130,3133,3135,3138,3141,3143,3146,3148,3151,3153,3156,3158,3160,3162,3164],{"class":140,"line":280},[138,3131,3132],{"class":199},"        handler ",[138,3134,274],{"class":195},[138,3136,3137],{"class":199}," RotatingFileHandler(path, ",[138,3139,3140],{"class":556},"maxBytes",[138,3142,274],{"class":195},[138,3144,3145],{"class":219},"5_000_000",[138,3147,162],{"class":199},[138,3149,3150],{"class":556},"backupCount",[138,3152,274],{"class":195},[138,3154,3155],{"class":219},"5",[138,3157,162],{"class":199},[138,3159,685],{"class":556},[138,3161,274],{"class":195},[138,3163,690],{"class":148},[138,3165,232],{"class":199},[138,3167,3168,3171],{"class":140,"line":295},[138,3169,3170],{"class":195},"    else",[138,3172,338],{"class":199},[138,3174,3175,3177,3179,3182,3185,3187,3190,3192,3194,3196,3198,3200,3202,3204,3206],{"class":140,"line":321},[138,3176,3132],{"class":199},[138,3178,274],{"class":195},[138,3180,3181],{"class":199}," TimedRotatingFileHandler(path, ",[138,3183,3184],{"class":556},"when",[138,3186,274],{"class":195},[138,3188,3189],{"class":148},"\"midnight\"",[138,3191,162],{"class":199},[138,3193,3150],{"class":556},[138,3195,274],{"class":195},[138,3197,48],{"class":219},[138,3199,162],{"class":199},[138,3201,685],{"class":556},[138,3203,274],{"class":195},[138,3205,690],{"class":148},[138,3207,232],{"class":199},[138,3209,3210],{"class":140,"line":341},[138,3211,3212],{"class":199},"    handler.setFormatter(\n",[138,3214,3215,3218,3220,3222,3224,3226,3228,3230,3232,3234,3236],{"class":140,"line":361},[138,3216,3217],{"class":199},"        logging.Formatter(",[138,3219,316],{"class":148},[138,3221,615],{"class":307},[138,3223,618],{"class":307},[138,3225,624],{"class":307},[138,3227,316],{"class":148},[138,3229,162],{"class":199},[138,3231,639],{"class":148},[138,3233,642],{"class":307},[138,3235,645],{"class":148},[138,3237,232],{"class":199},[138,3239,3240],{"class":140,"line":373},[138,3241,652],{"class":199},[138,3243,3244],{"class":140,"line":393},[138,3245,3246],{"class":199},"    logger.addHandler(handler)\n",[138,3248,3249,3251],{"class":140,"line":405},[138,3250,733],{"class":195},[138,3252,736],{"class":199},[10,3254,3255],{},"Size-based rotation suits a job whose output volume varies; time-based rotation suits a scheduled report, because \"the log for the 3rd of March\" is exactly how you will think about it when someone asks why that day's numbers looked odd. Thirty daily files is a month of history for a few megabytes, which is almost always the right trade.",[10,3257,3258,3259,3261,3262,3265],{},"Two habits keep the content useful as well as the size manageable. Log at ",[135,3260,545],{}," for the steps a reader would want to reconstruct and at ",[135,3263,3264],{},"DEBUG"," for anything voluminous, so raising the level in an emergency gives you more detail rather than a different program. And never log the contents of the data — a row of customer records in a log file is a data-protection problem that outlives the report it came from.",[123,3267,3269],{"id":3268},"record-a-run-summary-someone-can-scan","Record a run summary someone can scan",[10,3271,3272],{},"A log is a narrative; a summary is a fact sheet. Writing one row per run into a small CSV gives you a history that answers \"has this been getting slower?\" without reading anything:",[128,3274,3276],{"className":186,"code":3275,"language":188,"meta":133,"style":133},"import csv\nimport time\nfrom datetime import datetime, timezone\nfrom pathlib import Path\n\ndef record_run(status, rows, output, started, path=\"logs\u002Fruns.csv\"):\n    Path(path).parent.mkdir(parents=True, exist_ok=True)\n    new = not Path(path).exists()\n    with open(path, \"a\", newline=\"\", encoding=\"utf-8\") as handle:\n        writer = csv.writer(handle)\n        if new:\n            writer.writerow([\"finished_utc\", \"status\", \"rows\", \"seconds\", \"output\"])\n        writer.writerow([\n            datetime.now(timezone.utc).isoformat(timespec=\"seconds\"),\n            status, rows, round(time.perf_counter() - started, 1), output,\n        ])\n\nstarted = time.perf_counter()\nrecord_run(\"ok\", 48_211, \"reports\u002Fmarch.xlsx\", started)\n",[135,3277,3278,3285,3291,3301,3311,3315,3332,3352,3364,3402,3412,3419,3449,3454,3469,3490,3495,3499,3509],{"__ignoreMap":133},[138,3279,3280,3282],{"class":140,"line":141},[138,3281,203],{"class":195},[138,3283,3284],{"class":199}," csv\n",[138,3286,3287,3289],{"class":140,"line":209},[138,3288,203],{"class":195},[138,3290,1006],{"class":199},[138,3292,3293,3295,3297,3299],{"class":140,"line":216},[138,3294,196],{"class":195},[138,3296,2971],{"class":199},[138,3298,203],{"class":195},[138,3300,2976],{"class":199},[138,3302,3303,3305,3307,3309],{"class":140,"line":235},[138,3304,196],{"class":195},[138,3306,200],{"class":199},[138,3308,203],{"class":195},[138,3310,206],{"class":199},[138,3312,3313],{"class":140,"line":250},[138,3314,213],{"emptyLinePlaceholder":212},[138,3316,3317,3319,3322,3325,3327,3330],{"class":140,"line":255},[138,3318,258],{"class":195},[138,3320,3321],{"class":261}," record_run",[138,3323,3324],{"class":199},"(status, rows, output, started, path",[138,3326,274],{"class":195},[138,3328,3329],{"class":148},"\"logs\u002Fruns.csv\"",[138,3331,548],{"class":199},[138,3333,3334,3336,3338,3340,3342,3344,3346,3348,3350],{"class":140,"line":268},[138,3335,3097],{"class":199},[138,3337,557],{"class":556},[138,3339,274],{"class":195},[138,3341,562],{"class":219},[138,3343,162],{"class":199},[138,3345,567],{"class":556},[138,3347,274],{"class":195},[138,3349,562],{"class":219},[138,3351,232],{"class":199},[138,3353,3354,3357,3359,3361],{"class":140,"line":280},[138,3355,3356],{"class":199},"    new ",[138,3358,274],{"class":195},[138,3360,286],{"class":195},[138,3362,3363],{"class":199}," Path(path).exists()\n",[138,3365,3366,3368,3371,3374,3377,3379,3382,3384,3387,3389,3391,3393,3395,3397,3399],{"class":140,"line":295},[138,3367,2234],{"class":195},[138,3369,3370],{"class":219}," open",[138,3372,3373],{"class":199},"(path, ",[138,3375,3376],{"class":148},"\"a\"",[138,3378,162],{"class":199},[138,3380,3381],{"class":556},"newline",[138,3383,274],{"class":195},[138,3385,3386],{"class":148},"\"\"",[138,3388,162],{"class":199},[138,3390,685],{"class":556},[138,3392,274],{"class":195},[138,3394,690],{"class":148},[138,3396,2252],{"class":199},[138,3398,797],{"class":195},[138,3400,3401],{"class":199}," handle:\n",[138,3403,3404,3407,3409],{"class":140,"line":321},[138,3405,3406],{"class":199},"        writer ",[138,3408,274],{"class":195},[138,3410,3411],{"class":199}," csv.writer(handle)\n",[138,3413,3414,3416],{"class":140,"line":341},[138,3415,1882],{"class":195},[138,3417,3418],{"class":199}," new:\n",[138,3420,3421,3424,3427,3429,3432,3434,3437,3439,3442,3444,3447],{"class":140,"line":361},[138,3422,3423],{"class":199},"            writer.writerow([",[138,3425,3426],{"class":148},"\"finished_utc\"",[138,3428,162],{"class":199},[138,3430,3431],{"class":148},"\"status\"",[138,3433,162],{"class":199},[138,3435,3436],{"class":148},"\"rows\"",[138,3438,162],{"class":199},[138,3440,3441],{"class":148},"\"seconds\"",[138,3443,162],{"class":199},[138,3445,3446],{"class":148},"\"output\"",[138,3448,885],{"class":199},[138,3450,3451],{"class":140,"line":373},[138,3452,3453],{"class":199},"        writer.writerow([\n",[138,3455,3456,3459,3462,3464,3466],{"class":140,"line":393},[138,3457,3458],{"class":199},"            datetime.now(timezone.utc).isoformat(",[138,3460,3461],{"class":556},"timespec",[138,3463,274],{"class":195},[138,3465,3441],{"class":148},[138,3467,3468],{"class":199},"),\n",[138,3470,3471,3474,3477,3480,3482,3485,3487],{"class":140,"line":405},[138,3472,3473],{"class":199},"            status, rows, ",[138,3475,3476],{"class":219},"round",[138,3478,3479],{"class":199},"(time.perf_counter() ",[138,3481,916],{"class":195},[138,3483,3484],{"class":199}," started, ",[138,3486,1096],{"class":219},[138,3488,3489],{"class":199},"), output,\n",[138,3491,3492],{"class":140,"line":425},[138,3493,3494],{"class":199},"        ])\n",[138,3496,3497],{"class":140,"line":433},[138,3498,213],{"emptyLinePlaceholder":212},[138,3500,3501,3504,3506],{"class":140,"line":467},[138,3502,3503],{"class":199},"started ",[138,3505,274],{"class":195},[138,3507,3508],{"class":199}," time.perf_counter()\n",[138,3510,3511,3514,3517,3519,3522,3524,3526],{"class":140,"line":472},[138,3512,3513],{"class":199},"record_run(",[138,3515,3516],{"class":148},"\"ok\"",[138,3518,162],{"class":199},[138,3520,3521],{"class":219},"48_211",[138,3523,162],{"class":199},[138,3525,2013],{"class":148},[138,3527,3528],{"class":199},", started)\n",[10,3530,3531],{},"Five columns is enough. When the row count halves or the runtime triples, the trend is visible in a spreadsheet — which is, appropriately, the tool your readers already have open.",[123,3533,3535],{"id":3534},"common-failures-and-how-to-respond","Common failures and how to respond",[3537,3538,3539,3555],"table",{},[3540,3541,3542],"thead",{},[3543,3544,3545,3549,3552],"tr",{},[3546,3547,3548],"th",{},"Failure",[3546,3550,3551],{},"Class",[3546,3553,3554],{},"Response",[3556,3557,3558,3572,3586,3600,3611,3621,3631],"tbody",{},[3543,3559,3560,3566,3569],{},[3561,3562,3563,3565],"td",{},[135,3564,1023],{}," on save",[3561,3567,3568],{},"Transient",[3561,3570,3571],{},"Retry with backoff; report the lock file if present",[3543,3573,3574,3580,3583],{},[3561,3575,3576,3579],{},[135,3577,3578],{},"FileNotFoundError"," on the source",[3561,3581,3582],{},"Environment",[3561,3584,3585],{},"Fail in preflight with the exact path",[3543,3587,3588,3594,3597],{},[3561,3589,3590,3593],{},[135,3591,3592],{},"KeyError"," on a column",[3561,3595,3596],{},"Data",[3561,3598,3599],{},"Stop; validate structure at the boundary instead",[3543,3601,3602,3605,3608],{},[3561,3603,3604],{},"Empty result set",[3561,3606,3607],{},"Output",[3561,3609,3610],{},"Block delivery, keep the previous file",[3543,3612,3613,3616,3618],{},[3561,3614,3615],{},"SMTP timeout",[3561,3617,3568],{},[3561,3619,3620],{},"Retry; if it persists, write the report and alert separately",[3543,3622,3623,3626,3628],{},[3561,3624,3625],{},"Corrupt workbook",[3561,3627,3596],{},[3561,3629,3630],{},"Quarantine the file, continue with the rest of the batch",[3543,3632,3633,3636,3638],{},[3561,3634,3635],{},"Disk full",[3561,3637,3582],{},[3561,3639,3640],{},"Fail fast; atomic write means the old file survives",[123,3642,3644],{"id":3643},"test-the-failure-paths","Test the failure paths",[10,3646,3647],{},"Error handling is the code least likely to have been exercised, because the failures it handles are rare by definition. Simulating them takes a few lines and is the difference between a handler that works and one that has never run:",[128,3649,3651],{"className":186,"code":3650,"language":188,"meta":133,"style":133},"import pytest\n\n\ndef test_preflight_names_the_missing_file(tmp_path, monkeypatch):\n    monkeypatch.setattr(\"report.SOURCE\", tmp_path \u002F \"nope.xlsx\")\n    with pytest.raises(SystemExit) as excinfo:\n        preflight()\n    assert \"source file not found\" in str(excinfo.value)\n\n\ndef test_retry_gives_up_and_reraises():\n    calls = {\"n\": 0}\n\n    def always_locked():\n        calls[\"n\"] += 1\n        raise PermissionError(\"locked\")\n\n    with pytest.raises(PermissionError):\n        with_retries(always_locked, attempts=3, base_delay=0)\n    assert calls[\"n\"] == 3\n\n\ndef test_validation_blocks_an_empty_report(tmp_path):\n    import pandas as pd\n    path = tmp_path \u002F \"empty.xlsx\"\n    pd.DataFrame(columns=[\"Region\"]).to_excel(path, sheet_name=\"Summary\", index=False)\n    assert validate_output(path, min_rows=1, min_bytes=0)\n",[135,3652,3653,3660,3664,3668,3678,3696,3713,3717,3733,3737,3741,3750,3770,3774,3784,3799,3812,3816,3826,3850,3866,3870,3874,3884,3895,3909,3945],{"__ignoreMap":133},[138,3654,3655,3657],{"class":140,"line":141},[138,3656,203],{"class":195},[138,3658,3659],{"class":199}," pytest\n",[138,3661,3662],{"class":140,"line":209},[138,3663,213],{"emptyLinePlaceholder":212},[138,3665,3666],{"class":140,"line":216},[138,3667,213],{"emptyLinePlaceholder":212},[138,3669,3670,3672,3675],{"class":140,"line":235},[138,3671,258],{"class":195},[138,3673,3674],{"class":261}," test_preflight_names_the_missing_file",[138,3676,3677],{"class":199},"(tmp_path, monkeypatch):\n",[138,3679,3680,3683,3686,3689,3691,3694],{"class":140,"line":250},[138,3681,3682],{"class":199},"    monkeypatch.setattr(",[138,3684,3685],{"class":148},"\"report.SOURCE\"",[138,3687,3688],{"class":199},", tmp_path ",[138,3690,665],{"class":195},[138,3692,3693],{"class":148}," \"nope.xlsx\"",[138,3695,232],{"class":199},[138,3697,3698,3700,3703,3706,3708,3710],{"class":140,"line":255},[138,3699,2234],{"class":195},[138,3701,3702],{"class":199}," pytest.raises(",[138,3704,3705],{"class":219},"SystemExit",[138,3707,2252],{"class":199},[138,3709,797],{"class":195},[138,3711,3712],{"class":199}," excinfo:\n",[138,3714,3715],{"class":140,"line":268},[138,3716,2291],{"class":199},[138,3718,3719,3722,3725,3727,3730],{"class":140,"line":280},[138,3720,3721],{"class":195},"    assert",[138,3723,3724],{"class":148}," \"source file not found\"",[138,3726,1890],{"class":195},[138,3728,3729],{"class":219}," str",[138,3731,3732],{"class":199},"(excinfo.value)\n",[138,3734,3735],{"class":140,"line":295},[138,3736,213],{"emptyLinePlaceholder":212},[138,3738,3739],{"class":140,"line":321},[138,3740,213],{"emptyLinePlaceholder":212},[138,3742,3743,3745,3748],{"class":140,"line":341},[138,3744,258],{"class":195},[138,3746,3747],{"class":261}," test_retry_gives_up_and_reraises",[138,3749,265],{"class":199},[138,3751,3752,3755,3757,3760,3763,3765,3767],{"class":140,"line":361},[138,3753,3754],{"class":199},"    calls ",[138,3756,274],{"class":195},[138,3758,3759],{"class":199}," {",[138,3761,3762],{"class":148},"\"n\"",[138,3764,2647],{"class":199},[138,3766,41],{"class":219},[138,3768,3769],{"class":199},"}\n",[138,3771,3772],{"class":140,"line":373},[138,3773,213],{"emptyLinePlaceholder":212},[138,3775,3776,3779,3782],{"class":140,"line":393},[138,3777,3778],{"class":195},"    def",[138,3780,3781],{"class":261}," always_locked",[138,3783,265],{"class":199},[138,3785,3786,3789,3791,3793,3796],{"class":140,"line":405},[138,3787,3788],{"class":199},"        calls[",[138,3790,3762],{"class":148},[138,3792,2191],{"class":199},[138,3794,3795],{"class":195},"+=",[138,3797,3798],{"class":219}," 1\n",[138,3800,3801,3803,3805,3807,3810],{"class":140,"line":425},[138,3802,436],{"class":195},[138,3804,1388],{"class":219},[138,3806,442],{"class":199},[138,3808,3809],{"class":148},"\"locked\"",[138,3811,232],{"class":199},[138,3813,3814],{"class":140,"line":433},[138,3815,213],{"emptyLinePlaceholder":212},[138,3817,3818,3820,3822,3824],{"class":140,"line":467},[138,3819,2234],{"class":195},[138,3821,3702],{"class":199},[138,3823,1023],{"class":219},[138,3825,548],{"class":199},[138,3827,3828,3831,3834,3836,3839,3841,3844,3846,3848],{"class":140,"line":472},[138,3829,3830],{"class":199},"        with_retries(always_locked, ",[138,3832,3833],{"class":556},"attempts",[138,3835,274],{"class":195},[138,3837,3838],{"class":219},"3",[138,3840,162],{"class":199},[138,3842,3843],{"class":556},"base_delay",[138,3845,274],{"class":195},[138,3847,41],{"class":219},[138,3849,232],{"class":199},[138,3851,3852,3854,3857,3859,3861,3863],{"class":140,"line":724},[138,3853,3721],{"class":195},[138,3855,3856],{"class":199}," calls[",[138,3858,3762],{"class":148},[138,3860,2191],{"class":199},[138,3862,332],{"class":195},[138,3864,3865],{"class":219}," 3\n",[138,3867,3868],{"class":140,"line":730},[138,3869,213],{"emptyLinePlaceholder":212},[138,3871,3872],{"class":140,"line":739},[138,3873,213],{"emptyLinePlaceholder":212},[138,3875,3876,3878,3881],{"class":140,"line":744},[138,3877,258],{"class":195},[138,3879,3880],{"class":261}," test_validation_blocks_an_empty_report",[138,3882,3883],{"class":199},"(tmp_path):\n",[138,3885,3886,3889,3891,3893],{"class":140,"line":755},[138,3887,3888],{"class":195},"    import",[138,3890,794],{"class":199},[138,3892,797],{"class":195},[138,3894,800],{"class":199},[138,3896,3897,3899,3901,3904,3906],{"class":140,"line":2026},[138,3898,1770],{"class":199},[138,3900,274],{"class":195},[138,3902,3903],{"class":199}," tmp_path ",[138,3905,665],{"class":195},[138,3907,3908],{"class":148}," \"empty.xlsx\"\n",[138,3910,3911,3914,3917,3919,3921,3924,3927,3929,3931,3933,3935,3938,3940,3943],{"class":140,"line":2047},[138,3912,3913],{"class":199},"    pd.DataFrame(",[138,3915,3916],{"class":556},"columns",[138,3918,274],{"class":195},[138,3920,879],{"class":199},[138,3922,3923],{"class":148},"\"Region\"",[138,3925,3926],{"class":199},"]).to_excel(path, ",[138,3928,821],{"class":556},[138,3930,274],{"class":195},[138,3932,1762],{"class":148},[138,3934,162],{"class":199},[138,3936,3937],{"class":556},"index",[138,3939,274],{"class":195},[138,3941,3942],{"class":219},"False",[138,3944,232],{"class":199},[138,3946,3947,3949,3952,3954,3956,3958,3960,3963,3965,3967],{"class":140,"line":2061},[138,3948,3721],{"class":195},[138,3950,3951],{"class":199}," validate_output(path, ",[138,3953,1983],{"class":556},[138,3955,274],{"class":195},[138,3957,1096],{"class":219},[138,3959,162],{"class":199},[138,3961,3962],{"class":556},"min_bytes",[138,3964,274],{"class":195},[138,3966,41],{"class":219},[138,3968,232],{"class":199},[10,3970,3971,3972,3975],{},"Passing ",[135,3973,3974],{},"base_delay=0"," keeps the retry test instant while still exercising the loop. These three tests cover the paths that actually fire at 6am on a bad morning, and they run in milliseconds — there is no reason for a scheduled job not to have them.",[123,3977,3979],{"id":3978},"key-takeaways","Key takeaways",[3981,3982,3983,3987,3990,3993,4002,4008,4011],"ul",{},[3984,3985,3986],"li",{},"Check the environment before doing any work, and name the exact path or credential that is missing.",[3984,3988,3989],{},"Log one line per step with counts and identifiers; \"error occurred\" costs someone an hour.",[3984,3991,3992],{},"Retry transient failures with exponential backoff, and never retry a data problem.",[3984,3994,3995,3996,3998,3999,4001],{},"Treat ",[135,3997,1023],{}," as \"someone has it open\" and say so, checking for the ",[135,4000,1501],{}," lock file.",[3984,4003,4004,4005,4007],{},"Write to a temporary file in the same directory and ",[135,4006,1607],{}," it into place, so a crash never destroys the last good report.",[3984,4009,4010],{},"Validate the output — sheets present, rows above a floor, size sane — before anything is delivered.",[3984,4012,4013],{},"Use distinct exit codes, alert on outcomes rather than tracebacks, and make the whole job safe to run twice.",[123,4015,4017],{"id":4016},"frequently-asked-questions","Frequently asked questions",[10,4019,4020,4024],{},[4021,4022,4023],"strong",{},"Should a report job retry automatically?","\nRetry transient failures — a locked file, a network share that blinked, an SMTP timeout. Never retry a validation failure or a missing column, because the second attempt fails identically and hides the real problem.",[10,4026,4027,4030],{},[4021,4028,4029],{},"What belongs in a log line?","\nWhat the job was doing, which file, how many rows, and what happened. A line that says \"error occurred\" tells the person reading it at 7am nothing they can act on.",[10,4032,4033,4036],{},[4021,4034,4035],{},"Why write to a temporary file first?","\nSo a crash halfway through cannot leave a truncated workbook where the last good one was. Write to a temp name in the same directory, then rename it into place.",[10,4038,4039,4042],{},[4021,4040,4041],{},"How do I stop alerts being ignored?","\nAlert on outcomes, not on exceptions. One message saying \"March report not sent — source file missing\" is read; forty tracebacks a week are filtered into a folder nobody opens.",[123,4044,4046],{"id":4045},"related","Related",[10,4048,4049],{},"Up to the parent guide:",[3981,4051,4052],{},[3984,4053,4054,4056],{},[14,4055,17],{"href":16}," — the wider pipeline these safeguards protect.",[10,4058,4059],{},"Go deeper here:",[3981,4061,4062,4069,4076,4083],{},[3984,4063,4064,4068],{},[14,4065,4067],{"href":4066},"\u002Fautomating-reporting-workflows\u002Ferror-handling-and-logging-in-excel-automation\u002Fhandle-permission-denied-when-writing-excel-in-python\u002F","Handle Permission Denied When Writing Excel in Python"," — the most common scheduled-job failure.",[3984,4070,4071,4075],{},[14,4072,4074],{"href":4073},"\u002Fautomating-reporting-workflows\u002Ferror-handling-and-logging-in-excel-automation\u002Flog-python-excel-script-output-to-a-file\u002F","Log Python Excel Script Output to a File"," — handlers, rotation and what to record.",[3984,4077,4078,4082],{},[14,4079,4081],{"href":4080},"\u002Fautomating-reporting-workflows\u002Ferror-handling-and-logging-in-excel-automation\u002Fretry-a-failed-excel-report-job-in-python\u002F","Retry a Failed Excel Report Job in Python"," — backoff, jitter and which errors qualify.",[3984,4084,4085,4089],{},[14,4086,4088],{"href":4087},"\u002Fautomating-reporting-workflows\u002Ferror-handling-and-logging-in-excel-automation\u002Fvalidate-an-excel-report-before-sending-it\u002F","Validate an Excel Report Before Sending It"," — the last gate before delivery.",[10,4091,4092],{},"Related areas:",[3981,4094,4095,4102,4108],{},[3984,4096,4097,4101],{},[14,4098,4100],{"href":4099},"\u002Fautomating-reporting-workflows\u002Fscheduling-python-excel-scripts-with-cron\u002F","Scheduling Python Excel Scripts with cron"," — where these exit codes are interpreted.",[3984,4103,4104,4107],{},[14,4105,4106],{"href":2488},"Emailing Excel Reports with smtplib"," — delivery, and alerting when it fails.",[3984,4109,4110,4114],{},[14,4111,4113],{"href":4112},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002F","Validating Excel Data with Python"," — the input-side checks that prevent data failures.",[4116,4117,4118],"style",{},"html pre.shiki code .sMTad, html code.shiki .sMTad{--shiki-default:#6F42C1;--shiki-dark:#FFB757}html pre.shiki code .srMev, html code.shiki .srMev{--shiki-default:#032F62;--shiki-dark:#ADDCFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .s-kum, html code.shiki .s-kum{--shiki-default:#D73A49;--shiki-dark:#FF9492}html pre.shiki code .skGVy, html code.shiki .skGVy{--shiki-default:#24292E;--shiki-dark:#F0F3F6}html pre.shiki code .sP0c6, html code.shiki .sP0c6{--shiki-default:#005CC5;--shiki-dark:#91CBFF}html pre.shiki code .s_Opv, html code.shiki .s_Opv{--shiki-default:#6F42C1;--shiki-dark:#DBB7FF}html pre.shiki code .sSjpA, html code.shiki .sSjpA{--shiki-default:#005CC5;--shiki-dark:#FF9492}html pre.shiki code .sa561, html code.shiki .sa561{--shiki-default:#E36209;--shiki-dark:#FFB757}html pre.shiki code .s-wDw, html code.shiki .s-wDw{--shiki-default:#6A737D;--shiki-dark:#BDC4CC}",{"title":133,"searchDepth":209,"depth":209,"links":4120},[4121,4122,4123,4124,4125,4126,4127,4128,4129,4130,4131,4132,4133,4134,4135,4136,4137],{"id":125,"depth":209,"text":126},{"id":179,"depth":209,"text":180},{"id":481,"depth":209,"text":482},{"id":990,"depth":209,"text":991},{"id":1318,"depth":209,"text":1319},{"id":1505,"depth":209,"text":1506},{"id":1693,"depth":209,"text":1694},{"id":2074,"depth":209,"text":2075},{"id":2493,"depth":209,"text":2494},{"id":2824,"depth":209,"text":2825},{"id":3026,"depth":209,"text":3027},{"id":3268,"depth":209,"text":3269},{"id":3534,"depth":209,"text":3535},{"id":3643,"depth":209,"text":3644},{"id":3978,"depth":209,"text":3979},{"id":4016,"depth":209,"text":4017},{"id":4045,"depth":209,"text":4046},"2026-08-01","Make scheduled Excel jobs survivable — structured logging, retries with backoff, atomic writes, permission and lock errors, output validation, exit codes and alerts that mean something.","md",[4142,4144,4146,4148],{"q":4023,"a":4143},"Retry transient failures — a locked file, a network share that blinked, an SMTP timeout. Never retry a validation failure or a missing column, because the second attempt fails identically and hides the real problem.",{"q":4029,"a":4145},"What the job was doing, which file, how many rows, and what happened. A line that says \"error occurred\" tells the person reading it at 7am nothing they can act on.",{"q":4035,"a":4147},"So a crash halfway through cannot leave a truncated workbook where the last good one was. Write to a temp name in the same directory, then rename it into place.",{"q":4041,"a":4149},"Alert on outcomes, not on exceptions. One message saying \"March report not sent — source file missing\" is read; forty tracebacks a week are filtered into a folder nobody opens.",{"breadcrumb":4151},[4152,4154,4155],{"name":4153,"item":665},"Home",{"name":17,"item":16},{"name":5,"item":4156},"\u002Fautomating-reporting-workflows\u002Ferror-handling-and-logging-in-excel-automation\u002F","\u002Fautomating-reporting-workflows\u002Ferror-handling-and-logging-in-excel-automation",{"title":5,"description":4159},"Patterns for unattended Excel reporting jobs: what fails and why, logging that answers questions, retry with backoff, atomic file writes, output checks and alerting.","error-handling-and-logging-in-excel-automation","automating-reporting-workflows\u002Ferror-handling-and-logging-in-excel-automation\u002Findex","guide","Y0l_J2nlBCWM-bb0AVTCDMI11KS0m9yZ71OnQ0v8XE8",[4165,4169],{"title":4166,"path":4167,"stem":4168,"children":-1},"Send an Excel Report to Multiple Recipients in Python","\u002Fautomating-reporting-workflows\u002Femailing-excel-reports-with-smtplib\u002Fsend-excel-report-to-multiple-recipients-python","automating-reporting-workflows\u002Femailing-excel-reports-with-smtplib\u002Fsend-excel-report-to-multiple-recipients-python\u002Findex",{"title":4067,"path":4170,"stem":4171,"children":-1},"\u002Fautomating-reporting-workflows\u002Ferror-handling-and-logging-in-excel-automation\u002Fhandle-permission-denied-when-writing-excel-in-python","automating-reporting-workflows\u002Ferror-handling-and-logging-in-excel-automation\u002Fhandle-permission-denied-when-writing-excel-in-python\u002Findex",1785584462253]