[{"data":1,"prerenderedAt":2208},["ShallowReactive",2],{"doc:\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fvalidate-excel-data-with-pandera-schemas":3,"surround:\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fvalidate-excel-data-with-pandera-schemas":2199},{"id":4,"title":5,"body":6,"dateModified":2176,"datePublished":2176,"description":2177,"extension":2178,"faq":2179,"meta":2190,"navigation":247,"path":2191,"seo":2192,"slug":2195,"stem":2196,"type":2197,"__hash__":2198},"docs\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fvalidate-excel-data-with-pandera-schemas\u002Findex.md","Validate Excel Data with pandera Schemas",{"type":7,"value":8,"toc":2160},"minimark",[9,33,157,162,192,196,199,480,499,503,760,770,777,781,784,1081,1093,1164,1168,1171,1310,1317,1321,1324,1504,1507,1511,1514,1714,1717,1721,1724,1799,1802,1903,1906,1910,1913,1986,1989,1993,2044,2048,2062,2066,2072,2076,2082,2091,2104,2110,2116,2120,2156],[10,11,12,13,17,18,22,23,26,27,32],"p",{},"Spreadsheets arrive imperfect: a blank in a required column, a negative quantity, a date typed as text, a duplicate reference. Catching those with scattered ",[14,15,16],"code",{},"assert"," statements works until there are five of them and someone needs to know ",[19,20,21],"em",{},"which rows"," failed. ",[14,24,25],{},"pandera"," replaces that with a declared schema — types, ranges, patterns, uniqueness — validated in one call that reports every violation at once. This guide builds a schema for a real import, produces an error report a colleague can act on, and wires it into an ingest job. It belongs to ",[28,29,31],"a",{"href":30},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002F","Validating Excel Data with Python",".",[34,35,43,44,43,48,43,52,43,59,43,66,43,74,43,80,43,85,43,93,43,98,43,103,43,108,43,111,43,118,43,122,43,127,43,131,43,139,43,145,43,149,43,153],"svg",{"viewBox":36,"role":37,"ariaLabelledBy":38,"xmlns":41,"style":42},"0 0 760 224","img",[39,40],"pa-t","pa-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;max-width:760px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif","\n  ",[45,46,47],"title",{"id":39},"A schema sitting between the sheet and the pipeline",[49,50,51],"desc",{"id":40},"The workbook is read into a frame, validated against a declared schema, and either continues into the pipeline or produces a report naming every failing row and rule.",[53,54],"rect",{"x":55,"y":55,"width":56,"height":57,"fill":58},"0","760","224","#ffffff",[60,61,65],"text",{"x":62,"y":63,"style":64},"380","26","font-size:13px;font-weight:600;fill:var(--muted,#5b6780);text-anchor:middle","One gate, every rule, one report",[53,67],{"x":63,"y":68,"width":69,"height":70,"rx":71,"fill":72,"stroke":73},"82","146","60","11","#f0f2f5","var(--line,#cdd5e6)",[60,75,79],{"x":76,"y":77,"style":78},"99","110","font-size:12px;font-weight:700;fill:var(--text,#172033);text-anchor:middle","orders.xlsx",[60,81,84],{"x":76,"y":82,"style":83},"130","font-size:11.5px;fill:var(--muted,#5b6780);text-anchor:middle","from a colleague",[86,87],"line",{"x1":88,"y1":89,"x2":90,"y2":89,"stroke":91,"style":92},"172","112","206","var(--brand,#5b5cf0)","stroke-width:2px",[94,95],"polygon",{"points":96,"fill":97},"206,112 196,107 196,117","#5b5cf0",[53,99],{"x":100,"y":68,"width":101,"height":70,"rx":71,"fill":102,"stroke":73},"210","160","#ebebfd",[60,104,107],{"x":105,"y":77,"style":106},"290","font-size:12px;font-weight:700;fill:var(--brand-strong,#4338ca);text-anchor:middle","DataFrameSchema",[60,109,110],{"x":105,"y":82,"style":83},"types + checks",[86,112],{"x1":113,"y1":114,"x2":115,"y2":116,"stroke":117,"style":92},"370","98","404","70","var(--teal,#0f9488)",[94,119],{"points":120,"fill":121},"404,70 397,81 392,73","#0f766e",[86,123],{"x1":113,"y1":124,"x2":115,"y2":125,"stroke":126,"style":92},"126","154","var(--accent,#f43f8f)",[94,128],{"points":129,"fill":130},"404,154 392,151 397,143","#be185d",[53,132],{"x":133,"y":134,"width":135,"height":136,"rx":137,"fill":138,"stroke":73},"408","44","322","52","10","#d9f4f1",[60,140,144],{"x":141,"y":142,"style":143},"569","76","font-size:12px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","valid: continue, types coerced",[53,146],{"x":133,"y":147,"width":135,"height":136,"rx":137,"fill":148,"stroke":73},"128","#fee8f2",[60,150,152],{"x":141,"y":125,"style":151},"font-size:12px;font-weight:700;fill:var(--accent-ink,#be185d);text-anchor:middle","invalid: every failing row and rule",[60,154,156],{"x":141,"y":88,"style":155},"font-size:11px;fill:var(--accent-ink,#be185d);text-anchor:middle","as a frame you can export",[158,159,161],"h2",{"id":160},"prerequisites","Prerequisites",[163,164,169],"pre",{"className":165,"code":166,"language":167,"meta":168,"style":168},"language-bash shiki shiki-themes github-light github-dark-high-contrast","pip install pandera pandas openpyxl\n","bash","",[14,170,171],{"__ignoreMap":168},[172,173,175,179,183,186,189],"span",{"class":86,"line":174},1,[172,176,178],{"class":177},"sMTad","pip",[172,180,182],{"class":181},"srMev"," install",[172,184,185],{"class":181}," pandera",[172,187,188],{"class":181}," pandas",[172,190,191],{"class":181}," openpyxl\n",[158,193,195],{"id":194},"declare-the-schema","Declare the schema",[10,197,198],{},"A schema is a dictionary of columns, each with a type and any number of checks:",[163,200,204],{"className":201,"code":202,"language":203,"meta":168,"style":168},"language-python shiki shiki-themes github-light github-dark-high-contrast","\"\"\"schema.py — what a valid orders sheet looks like.\"\"\"\nimport pandera.pandas as pa\nfrom pandera import Check, Column, DataFrameSchema\n\nORDERS = DataFrameSchema(\n    {\n        \"order_id\": Column(str, Check.str_matches(r\"^\\d{5}$\"), unique=True),\n        \"ordered\": Column(\"datetime64[ns]\", Check.le(pa.Timestamp(\"2030-01-01\"))),\n        \"region\": Column(str, Check.isin([\"North\", \"South\", \"East\", \"West\"])),\n        \"quantity\": Column(int, Check.gt(0)),\n        \"unit_price\": Column(float, Check.in_range(0, 10_000)),\n        \"note\": Column(str, nullable=True),\n    },\n    strict=False,        # extra columns allowed; set True to reject them\n    coerce=True,         # cast to the declared types during validation\n)\n","python",[14,205,206,211,228,242,249,262,268,316,336,371,390,413,434,440,458,474],{"__ignoreMap":168},[172,207,208],{"class":86,"line":174},[172,209,210],{"class":181},"\"\"\"schema.py — what a valid orders sheet looks like.\"\"\"\n",[172,212,214,218,222,225],{"class":86,"line":213},2,[172,215,217],{"class":216},"s-kum","import",[172,219,221],{"class":220},"skGVy"," pandera.pandas ",[172,223,224],{"class":216},"as",[172,226,227],{"class":220}," pa\n",[172,229,231,234,237,239],{"class":86,"line":230},3,[172,232,233],{"class":216},"from",[172,235,236],{"class":220}," pandera ",[172,238,217],{"class":216},[172,240,241],{"class":220}," Check, Column, DataFrameSchema\n",[172,243,245],{"class":86,"line":244},4,[172,246,248],{"emptyLinePlaceholder":247},true,"\n",[172,250,252,256,259],{"class":86,"line":251},5,[172,253,255],{"class":254},"sP0c6","ORDERS",[172,257,258],{"class":216}," =",[172,260,261],{"class":220}," DataFrameSchema(\n",[172,263,265],{"class":86,"line":264},6,[172,266,267],{"class":220},"    {\n",[172,269,271,274,277,280,283,286,289,292,295,298,300,303,307,310,313],{"class":86,"line":270},7,[172,272,273],{"class":181},"        \"order_id\"",[172,275,276],{"class":220},": Column(",[172,278,279],{"class":254},"str",[172,281,282],{"class":220},", Check.str_matches(",[172,284,285],{"class":216},"r",[172,287,288],{"class":181},"\"",[172,290,291],{"class":254},"^\\d",[172,293,294],{"class":216},"{5}",[172,296,297],{"class":254},"$",[172,299,288],{"class":181},[172,301,302],{"class":220},"), ",[172,304,306],{"class":305},"sa561","unique",[172,308,309],{"class":216},"=",[172,311,312],{"class":254},"True",[172,314,315],{"class":220},"),\n",[172,317,319,322,324,327,330,333],{"class":86,"line":318},8,[172,320,321],{"class":181},"        \"ordered\"",[172,323,276],{"class":220},[172,325,326],{"class":181},"\"datetime64[ns]\"",[172,328,329],{"class":220},", Check.le(pa.Timestamp(",[172,331,332],{"class":181},"\"2030-01-01\"",[172,334,335],{"class":220},"))),\n",[172,337,339,342,344,346,349,352,355,358,360,363,365,368],{"class":86,"line":338},9,[172,340,341],{"class":181},"        \"region\"",[172,343,276],{"class":220},[172,345,279],{"class":254},[172,347,348],{"class":220},", Check.isin([",[172,350,351],{"class":181},"\"North\"",[172,353,354],{"class":220},", ",[172,356,357],{"class":181},"\"South\"",[172,359,354],{"class":220},[172,361,362],{"class":181},"\"East\"",[172,364,354],{"class":220},[172,366,367],{"class":181},"\"West\"",[172,369,370],{"class":220},"])),\n",[172,372,374,377,379,382,385,387],{"class":86,"line":373},10,[172,375,376],{"class":181},"        \"quantity\"",[172,378,276],{"class":220},[172,380,381],{"class":254},"int",[172,383,384],{"class":220},", Check.gt(",[172,386,55],{"class":254},[172,388,389],{"class":220},")),\n",[172,391,393,396,398,401,404,406,408,411],{"class":86,"line":392},11,[172,394,395],{"class":181},"        \"unit_price\"",[172,397,276],{"class":220},[172,399,400],{"class":254},"float",[172,402,403],{"class":220},", Check.in_range(",[172,405,55],{"class":254},[172,407,354],{"class":220},[172,409,410],{"class":254},"10_000",[172,412,389],{"class":220},[172,414,416,419,421,423,425,428,430,432],{"class":86,"line":415},12,[172,417,418],{"class":181},"        \"note\"",[172,420,276],{"class":220},[172,422,279],{"class":254},[172,424,354],{"class":220},[172,426,427],{"class":305},"nullable",[172,429,309],{"class":216},[172,431,312],{"class":254},[172,433,315],{"class":220},[172,435,437],{"class":86,"line":436},13,[172,438,439],{"class":220},"    },\n",[172,441,443,446,448,451,454],{"class":86,"line":442},14,[172,444,445],{"class":305},"    strict",[172,447,309],{"class":216},[172,449,450],{"class":254},"False",[172,452,453],{"class":220},",        ",[172,455,457],{"class":456},"s-wDw","# extra columns allowed; set True to reject them\n",[172,459,461,464,466,468,471],{"class":86,"line":460},15,[172,462,463],{"class":305},"    coerce",[172,465,309],{"class":216},[172,467,312],{"class":254},[172,469,470],{"class":220},",         ",[172,472,473],{"class":456},"# cast to the declared types during validation\n",[172,475,477],{"class":86,"line":476},16,[172,478,479],{"class":220},")\n",[10,481,482,483,486,487,490,491,494,495,498],{},"Each argument earns its place. ",[14,484,485],{},"unique=True"," catches the duplicate reference that would double-count a sale. ",[14,488,489],{},"Check.isin"," catches the region typed as \"Norht\". ",[14,492,493],{},"nullable=True"," says a blank note is fine, which by contrast means every other column must be populated. And ",[14,496,497],{},"coerce=True"," makes validation do the type conversion too, so the frame that comes out is correctly typed rather than merely approved.",[158,500,502],{"id":501},"validate-and-read-the-report","Validate and read the report",[163,504,506],{"className":201,"code":505,"language":203,"meta":168,"style":168},"import pandas as pd\nimport pandera.errors\n\nfrom schema import ORDERS\n\ndf = pd.read_excel(\"orders.xlsx\", dtype={\"order_id\": \"string\"}, parse_dates=[\"ordered\"])\n\ntry:\n    clean = ORDERS.validate(df, lazy=True)\n    print(f\"{len(clean):,} rows validated\")\nexcept pandera.errors.SchemaErrors as exc:\n    failures = exc.failure_cases\n    failures[\"excel_row\"] = failures[\"index\"].astype(\"Int64\") + 2   # header offset\n    print(failures[[\"excel_row\", \"column\", \"check\", \"failure_case\"]].to_string(index=False))\n",[14,507,508,520,527,531,543,547,596,600,608,630,663,676,686,723],{"__ignoreMap":168},[172,509,510,512,515,517],{"class":86,"line":174},[172,511,217],{"class":216},[172,513,514],{"class":220}," pandas ",[172,516,224],{"class":216},[172,518,519],{"class":220}," pd\n",[172,521,522,524],{"class":86,"line":213},[172,523,217],{"class":216},[172,525,526],{"class":220}," pandera.errors\n",[172,528,529],{"class":86,"line":230},[172,530,248],{"emptyLinePlaceholder":247},[172,532,533,535,538,540],{"class":86,"line":244},[172,534,233],{"class":216},[172,536,537],{"class":220}," schema ",[172,539,217],{"class":216},[172,541,542],{"class":254}," ORDERS\n",[172,544,545],{"class":86,"line":251},[172,546,248],{"emptyLinePlaceholder":247},[172,548,549,552,554,557,560,562,565,567,570,573,576,579,582,585,587,590,593],{"class":86,"line":264},[172,550,551],{"class":220},"df ",[172,553,309],{"class":216},[172,555,556],{"class":220}," pd.read_excel(",[172,558,559],{"class":181},"\"orders.xlsx\"",[172,561,354],{"class":220},[172,563,564],{"class":305},"dtype",[172,566,309],{"class":216},[172,568,569],{"class":220},"{",[172,571,572],{"class":181},"\"order_id\"",[172,574,575],{"class":220},": ",[172,577,578],{"class":181},"\"string\"",[172,580,581],{"class":220},"}, ",[172,583,584],{"class":305},"parse_dates",[172,586,309],{"class":216},[172,588,589],{"class":220},"[",[172,591,592],{"class":181},"\"ordered\"",[172,594,595],{"class":220},"])\n",[172,597,598],{"class":86,"line":270},[172,599,248],{"emptyLinePlaceholder":247},[172,601,602,605],{"class":86,"line":318},[172,603,604],{"class":216},"try",[172,606,607],{"class":220},":\n",[172,609,610,613,615,618,621,624,626,628],{"class":86,"line":338},[172,611,612],{"class":220},"    clean ",[172,614,309],{"class":216},[172,616,617],{"class":254}," ORDERS",[172,619,620],{"class":220},".validate(df, ",[172,622,623],{"class":305},"lazy",[172,625,309],{"class":216},[172,627,312],{"class":254},[172,629,479],{"class":220},[172,631,632,635,638,641,643,646,649,652,655,658,661],{"class":86,"line":373},[172,633,634],{"class":254},"    print",[172,636,637],{"class":220},"(",[172,639,640],{"class":216},"f",[172,642,288],{"class":181},[172,644,569],{"class":645},"sSjpA",[172,647,648],{"class":254},"len",[172,650,651],{"class":220},"(clean)",[172,653,654],{"class":216},":,",[172,656,657],{"class":645},"}",[172,659,660],{"class":181}," rows validated\"",[172,662,479],{"class":220},[172,664,665,668,671,673],{"class":86,"line":392},[172,666,667],{"class":216},"except",[172,669,670],{"class":220}," pandera.errors.SchemaErrors ",[172,672,224],{"class":216},[172,674,675],{"class":220}," exc:\n",[172,677,678,681,683],{"class":86,"line":415},[172,679,680],{"class":220},"    failures ",[172,682,309],{"class":216},[172,684,685],{"class":220}," exc.failure_cases\n",[172,687,688,691,694,697,699,702,705,708,711,714,717,720],{"class":86,"line":436},[172,689,690],{"class":220},"    failures[",[172,692,693],{"class":181},"\"excel_row\"",[172,695,696],{"class":220},"] ",[172,698,309],{"class":216},[172,700,701],{"class":220}," failures[",[172,703,704],{"class":181},"\"index\"",[172,706,707],{"class":220},"].astype(",[172,709,710],{"class":181},"\"Int64\"",[172,712,713],{"class":220},") ",[172,715,716],{"class":216},"+",[172,718,719],{"class":254}," 2",[172,721,722],{"class":456},"   # header offset\n",[172,724,725,727,730,732,734,737,739,742,744,747,750,753,755,757],{"class":86,"line":442},[172,726,634],{"class":254},[172,728,729],{"class":220},"(failures[[",[172,731,693],{"class":181},[172,733,354],{"class":220},[172,735,736],{"class":181},"\"column\"",[172,738,354],{"class":220},[172,740,741],{"class":181},"\"check\"",[172,743,354],{"class":220},[172,745,746],{"class":181},"\"failure_case\"",[172,748,749],{"class":220},"]].to_string(",[172,751,752],{"class":305},"index",[172,754,309],{"class":216},[172,756,450],{"class":254},[172,758,759],{"class":220},"))\n",[10,761,762,765,766,769],{},[14,763,764],{},"lazy=True"," is the argument that changes the experience. Without it, validation stops at the first failing check; with it, every rule runs and ",[14,767,768],{},"failure_cases"," holds one row per violation — the column, the rule that failed, the offending value and its position. Adding two to the index converts a zero-based DataFrame position into the row number the reader will see in Excel.",[163,771,775],{"className":772,"code":774,"language":60,"meta":168},[773],"language-text"," excel_row     column                    check failure_case\n        14   quantity                greater_than           -3\n        27     region     isin(['North', ...])        Norht\n        41   order_id                  unique         00417\n",[14,776,774],{"__ignoreMap":168},[158,778,780],{"id":779},"send-the-failures-back-as-a-workbook","Send the failures back as a workbook",[10,782,783],{},"A list in a log helps you; a spreadsheet helps the person who has to fix it:",[163,785,787],{"className":201,"code":786,"language":203,"meta":168,"style":168},"import pandas as pd\n\ndef write_error_report(failures: pd.DataFrame, path: str) -> None:\n    report = failures.assign(excel_row=failures[\"index\"].astype(\"Int64\") + 2)[\n        [\"excel_row\", \"column\", \"check\", \"failure_case\"]\n    ].rename(columns={\n        \"excel_row\": \"Row\", \"column\": \"Column\",\n        \"check\": \"Rule broken\", \"failure_case\": \"Value found\",\n    })\n    with pd.ExcelWriter(path, engine=\"xlsxwriter\") as writer:\n        report.to_excel(writer, index=False, sheet_name=\"Problems\")\n        ws = writer.sheets[\"Problems\"]\n        ws.freeze_panes(1, 0)\n        ws.autofilter(0, 0, len(report), len(report.columns) - 1)\n        ws.set_column(0, 0, 8)\n        ws.set_column(1, 3, 26)\n",[14,788,789,799,803,825,858,880,893,915,936,941,964,987,1001,1015,1046,1064],{"__ignoreMap":168},[172,790,791,793,795,797],{"class":86,"line":174},[172,792,217],{"class":216},[172,794,514],{"class":220},[172,796,224],{"class":216},[172,798,519],{"class":220},[172,800,801],{"class":86,"line":213},[172,802,248],{"emptyLinePlaceholder":247},[172,804,805,808,812,815,817,820,823],{"class":86,"line":230},[172,806,807],{"class":216},"def",[172,809,811],{"class":810},"s_Opv"," write_error_report",[172,813,814],{"class":220},"(failures: pd.DataFrame, path: ",[172,816,279],{"class":254},[172,818,819],{"class":220},") -> ",[172,821,822],{"class":254},"None",[172,824,607],{"class":220},[172,826,827,830,832,835,838,840,843,845,847,849,851,853,855],{"class":86,"line":244},[172,828,829],{"class":220},"    report ",[172,831,309],{"class":216},[172,833,834],{"class":220}," failures.assign(",[172,836,837],{"class":305},"excel_row",[172,839,309],{"class":216},[172,841,842],{"class":220},"failures[",[172,844,704],{"class":181},[172,846,707],{"class":220},[172,848,710],{"class":181},[172,850,713],{"class":220},[172,852,716],{"class":216},[172,854,719],{"class":254},[172,856,857],{"class":220},")[\n",[172,859,860,863,865,867,869,871,873,875,877],{"class":86,"line":251},[172,861,862],{"class":220},"        [",[172,864,693],{"class":181},[172,866,354],{"class":220},[172,868,736],{"class":181},[172,870,354],{"class":220},[172,872,741],{"class":181},[172,874,354],{"class":220},[172,876,746],{"class":181},[172,878,879],{"class":220},"]\n",[172,881,882,885,888,890],{"class":86,"line":264},[172,883,884],{"class":220},"    ].rename(",[172,886,887],{"class":305},"columns",[172,889,309],{"class":216},[172,891,892],{"class":220},"{\n",[172,894,895,898,900,903,905,907,909,912],{"class":86,"line":270},[172,896,897],{"class":181},"        \"excel_row\"",[172,899,575],{"class":220},[172,901,902],{"class":181},"\"Row\"",[172,904,354],{"class":220},[172,906,736],{"class":181},[172,908,575],{"class":220},[172,910,911],{"class":181},"\"Column\"",[172,913,914],{"class":220},",\n",[172,916,917,920,922,925,927,929,931,934],{"class":86,"line":318},[172,918,919],{"class":181},"        \"check\"",[172,921,575],{"class":220},[172,923,924],{"class":181},"\"Rule broken\"",[172,926,354],{"class":220},[172,928,746],{"class":181},[172,930,575],{"class":220},[172,932,933],{"class":181},"\"Value found\"",[172,935,914],{"class":220},[172,937,938],{"class":86,"line":338},[172,939,940],{"class":220},"    })\n",[172,942,943,946,949,952,954,957,959,961],{"class":86,"line":373},[172,944,945],{"class":216},"    with",[172,947,948],{"class":220}," pd.ExcelWriter(path, ",[172,950,951],{"class":305},"engine",[172,953,309],{"class":216},[172,955,956],{"class":181},"\"xlsxwriter\"",[172,958,713],{"class":220},[172,960,224],{"class":216},[172,962,963],{"class":220}," writer:\n",[172,965,966,969,971,973,975,977,980,982,985],{"class":86,"line":392},[172,967,968],{"class":220},"        report.to_excel(writer, ",[172,970,752],{"class":305},[172,972,309],{"class":216},[172,974,450],{"class":254},[172,976,354],{"class":220},[172,978,979],{"class":305},"sheet_name",[172,981,309],{"class":216},[172,983,984],{"class":181},"\"Problems\"",[172,986,479],{"class":220},[172,988,989,992,994,997,999],{"class":86,"line":415},[172,990,991],{"class":220},"        ws ",[172,993,309],{"class":216},[172,995,996],{"class":220}," writer.sheets[",[172,998,984],{"class":181},[172,1000,879],{"class":220},[172,1002,1003,1006,1009,1011,1013],{"class":86,"line":436},[172,1004,1005],{"class":220},"        ws.freeze_panes(",[172,1007,1008],{"class":254},"1",[172,1010,354],{"class":220},[172,1012,55],{"class":254},[172,1014,479],{"class":220},[172,1016,1017,1020,1022,1024,1026,1028,1030,1033,1035,1038,1041,1044],{"class":86,"line":442},[172,1018,1019],{"class":220},"        ws.autofilter(",[172,1021,55],{"class":254},[172,1023,354],{"class":220},[172,1025,55],{"class":254},[172,1027,354],{"class":220},[172,1029,648],{"class":254},[172,1031,1032],{"class":220},"(report), ",[172,1034,648],{"class":254},[172,1036,1037],{"class":220},"(report.columns) ",[172,1039,1040],{"class":216},"-",[172,1042,1043],{"class":254}," 1",[172,1045,479],{"class":220},[172,1047,1048,1051,1053,1055,1057,1059,1062],{"class":86,"line":460},[172,1049,1050],{"class":220},"        ws.set_column(",[172,1052,55],{"class":254},[172,1054,354],{"class":220},[172,1056,55],{"class":254},[172,1058,354],{"class":220},[172,1060,1061],{"class":254},"8",[172,1063,479],{"class":220},[172,1065,1066,1068,1070,1072,1075,1077,1079],{"class":86,"line":476},[172,1067,1050],{"class":220},[172,1069,1008],{"class":254},[172,1071,354],{"class":220},[172,1073,1074],{"class":254},"3",[172,1076,354],{"class":220},[172,1078,63],{"class":254},[172,1080,479],{"class":220},[10,1082,1083,1084,1088,1089,32],{},"Emailing that back to the sender closes the loop without anybody reading a stack trace — the delivery step is in ",[28,1085,1087],{"href":1086},"\u002Fautomating-reporting-workflows\u002Femailing-excel-reports-with-smtplib\u002F","Emailing Excel Reports with smtplib",", and highlighting the offending cells in the original file is covered in ",[28,1090,1092],{"href":1091},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fhighlight-invalid-cells-in-excel-with-python\u002F","Highlight invalid cells in Excel with Python",[34,1094,43,1099,43,1102,43,1105,43,1108,43,1114,43,1118,43,1122,43,1127,43,1132,43,1135,43,1139,43,1142,43,1146,43,1150,43,1155,43,1160],{"viewBox":1095,"role":37,"ariaLabelledBy":1096,"xmlns":41,"style":42},"0 0 760 216",[1097,1098],"pa2-t","pa2-d",[45,1100,1101],{"id":1097},"Eager validation versus lazy validation",[49,1103,1104],{"id":1098},"Eager validation stops at the first failure, so problems are discovered one per round trip, while lazy validation reports every violation in a single pass.",[53,1106],{"x":55,"y":55,"width":56,"height":1107,"fill":58},"216",[60,1109,1113],{"x":1110,"y":1111,"style":1112},"196","28","font-size:12.5px;font-weight:700;fill:var(--accent-ink,#be185d);text-anchor:middle","lazy=False",[60,1115,764],{"x":1116,"y":1111,"style":1117},"566","font-size:12.5px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle",[86,1119],{"x1":62,"y1":1120,"x2":62,"y2":1110,"stroke":73,"style":1121},"40","stroke-width:1px",[53,1123],{"x":1124,"y":1125,"width":1126,"height":1120,"rx":1061,"fill":148,"stroke":73},"30","46","332",[60,1128,1131],{"x":1110,"y":1129,"style":1130},"71","font-size:12px;fill:var(--accent-ink,#be185d);text-anchor:middle","run 1: quantity is negative — stop",[53,1133],{"x":1124,"y":1134,"width":1126,"height":1120,"rx":1061,"fill":148,"stroke":73},"94",[60,1136,1138],{"x":1110,"y":1137,"style":1130},"119","run 2: region misspelled — stop",[53,1140],{"x":1124,"y":1141,"width":1126,"height":1120,"rx":1061,"fill":148,"stroke":73},"142",[60,1143,1145],{"x":1110,"y":1144,"style":1130},"167","run 3: duplicate id — stop",[53,1147],{"x":1148,"y":1125,"width":1126,"height":1149,"rx":137,"fill":138,"stroke":73},"398","136",[60,1151,1154],{"x":1152,"y":1153,"style":1117},"564","86","one run",[60,1156,1159],{"x":1152,"y":1157,"style":1158},"118","font-size:12px;fill:var(--teal-ink,#0b6157);text-anchor:middle","3 violations, with rows and rules",[60,1161,1163],{"x":1152,"y":1162,"style":1158},"148","one message back to the sender",[158,1165,1167],{"id":1166},"add-rules-that-span-columns","Add rules that span columns",[10,1169,1170],{},"Some rules are about the row, not the cell. A schema-level check receives the whole frame:",[163,1172,1174],{"className":201,"code":1173,"language":203,"meta":168,"style":168},"from pandera import Check, DataFrameSchema\n\nORDERS_WITH_ROW_RULES = ORDERS.add_checks([\n    Check(lambda df: df[\"shipped\"].isna() | (df[\"shipped\"] >= df[\"ordered\"]),\n          error=\"shipped date is before ordered date\"),\n    Check(lambda df: (df[\"quantity\"] * df[\"unit_price\"] - df[\"total\"]).abs() \u003C 0.01,\n          error=\"total does not equal quantity x unit price\"),\n])\n",[14,1175,1176,1187,1191,1203,1241,1253,1295,1306],{"__ignoreMap":168},[172,1177,1178,1180,1182,1184],{"class":86,"line":174},[172,1179,233],{"class":216},[172,1181,236],{"class":220},[172,1183,217],{"class":216},[172,1185,1186],{"class":220}," Check, DataFrameSchema\n",[172,1188,1189],{"class":86,"line":213},[172,1190,248],{"emptyLinePlaceholder":247},[172,1192,1193,1196,1198,1200],{"class":86,"line":230},[172,1194,1195],{"class":254},"ORDERS_WITH_ROW_RULES",[172,1197,258],{"class":216},[172,1199,617],{"class":254},[172,1201,1202],{"class":220},".add_checks([\n",[172,1204,1205,1208,1211,1214,1217,1220,1223,1226,1228,1230,1233,1236,1238],{"class":86,"line":244},[172,1206,1207],{"class":220},"    Check(",[172,1209,1210],{"class":216},"lambda",[172,1212,1213],{"class":220}," df: df[",[172,1215,1216],{"class":181},"\"shipped\"",[172,1218,1219],{"class":220},"].isna() ",[172,1221,1222],{"class":216},"|",[172,1224,1225],{"class":220}," (df[",[172,1227,1216],{"class":181},[172,1229,696],{"class":220},[172,1231,1232],{"class":216},">=",[172,1234,1235],{"class":220}," df[",[172,1237,592],{"class":181},[172,1239,1240],{"class":220},"]),\n",[172,1242,1243,1246,1248,1251],{"class":86,"line":251},[172,1244,1245],{"class":305},"          error",[172,1247,309],{"class":216},[172,1249,1250],{"class":181},"\"shipped date is before ordered date\"",[172,1252,315],{"class":220},[172,1254,1255,1257,1259,1262,1265,1267,1270,1272,1275,1277,1279,1281,1284,1287,1290,1293],{"class":86,"line":264},[172,1256,1207],{"class":220},[172,1258,1210],{"class":216},[172,1260,1261],{"class":220}," df: (df[",[172,1263,1264],{"class":181},"\"quantity\"",[172,1266,696],{"class":220},[172,1268,1269],{"class":216},"*",[172,1271,1235],{"class":220},[172,1273,1274],{"class":181},"\"unit_price\"",[172,1276,696],{"class":220},[172,1278,1040],{"class":216},[172,1280,1235],{"class":220},[172,1282,1283],{"class":181},"\"total\"",[172,1285,1286],{"class":220},"]).abs() ",[172,1288,1289],{"class":216},"\u003C",[172,1291,1292],{"class":254}," 0.01",[172,1294,914],{"class":220},[172,1296,1297,1299,1301,1304],{"class":86,"line":270},[172,1298,1245],{"class":305},[172,1300,309],{"class":216},[172,1302,1303],{"class":181},"\"total does not equal quantity x unit price\"",[172,1305,315],{"class":220},[172,1307,1308],{"class":86,"line":318},[172,1309,595],{"class":220},[10,1311,1312,1313,1316],{},"The second check is the one that catches manual edits: somebody overtyped a total in the spreadsheet and the arithmetic no longer holds. Giving each check an ",[14,1314,1315],{},"error"," string means the report says what is wrong in words rather than showing a lambda.",[158,1318,1320],{"id":1319},"reuse-the-schema-as-documentation-and-as-a-test","Reuse the schema as documentation and as a test",[10,1322,1323],{},"A schema is also the clearest description of the file format, so it belongs in version control next to the ingest code and in the test suite:",[163,1325,1327],{"className":201,"code":1326,"language":203,"meta":168,"style":168},"\"\"\"tests\u002Ftest_schema.py\"\"\"\nimport pandas as pd\nimport pytest\nimport pandera.errors\n\nfrom schema import ORDERS\n\ndef test_good_sheet_validates():\n    df = pd.read_excel(\"tests\u002Ffixtures\u002Forders_good.xlsx\", parse_dates=[\"ordered\"])\n    ORDERS.validate(df, lazy=True)\n\ndef test_bad_sheet_reports_every_problem():\n    df = pd.read_excel(\"tests\u002Ffixtures\u002Forders_bad.xlsx\", parse_dates=[\"ordered\"])\n    with pytest.raises(pandera.errors.SchemaErrors) as excinfo:\n        ORDERS.validate(df, lazy=True)\n    assert len(excinfo.value.failure_cases) == 3\n",[14,1328,1329,1334,1344,1351,1357,1361,1371,1375,1385,1409,1424,1428,1437,1460,1472,1487],{"__ignoreMap":168},[172,1330,1331],{"class":86,"line":174},[172,1332,1333],{"class":181},"\"\"\"tests\u002Ftest_schema.py\"\"\"\n",[172,1335,1336,1338,1340,1342],{"class":86,"line":213},[172,1337,217],{"class":216},[172,1339,514],{"class":220},[172,1341,224],{"class":216},[172,1343,519],{"class":220},[172,1345,1346,1348],{"class":86,"line":230},[172,1347,217],{"class":216},[172,1349,1350],{"class":220}," pytest\n",[172,1352,1353,1355],{"class":86,"line":244},[172,1354,217],{"class":216},[172,1356,526],{"class":220},[172,1358,1359],{"class":86,"line":251},[172,1360,248],{"emptyLinePlaceholder":247},[172,1362,1363,1365,1367,1369],{"class":86,"line":264},[172,1364,233],{"class":216},[172,1366,537],{"class":220},[172,1368,217],{"class":216},[172,1370,542],{"class":254},[172,1372,1373],{"class":86,"line":270},[172,1374,248],{"emptyLinePlaceholder":247},[172,1376,1377,1379,1382],{"class":86,"line":318},[172,1378,807],{"class":216},[172,1380,1381],{"class":810}," test_good_sheet_validates",[172,1383,1384],{"class":220},"():\n",[172,1386,1387,1390,1392,1394,1397,1399,1401,1403,1405,1407],{"class":86,"line":338},[172,1388,1389],{"class":220},"    df ",[172,1391,309],{"class":216},[172,1393,556],{"class":220},[172,1395,1396],{"class":181},"\"tests\u002Ffixtures\u002Forders_good.xlsx\"",[172,1398,354],{"class":220},[172,1400,584],{"class":305},[172,1402,309],{"class":216},[172,1404,589],{"class":220},[172,1406,592],{"class":181},[172,1408,595],{"class":220},[172,1410,1411,1414,1416,1418,1420,1422],{"class":86,"line":373},[172,1412,1413],{"class":254},"    ORDERS",[172,1415,620],{"class":220},[172,1417,623],{"class":305},[172,1419,309],{"class":216},[172,1421,312],{"class":254},[172,1423,479],{"class":220},[172,1425,1426],{"class":86,"line":392},[172,1427,248],{"emptyLinePlaceholder":247},[172,1429,1430,1432,1435],{"class":86,"line":415},[172,1431,807],{"class":216},[172,1433,1434],{"class":810}," test_bad_sheet_reports_every_problem",[172,1436,1384],{"class":220},[172,1438,1439,1441,1443,1445,1448,1450,1452,1454,1456,1458],{"class":86,"line":436},[172,1440,1389],{"class":220},[172,1442,309],{"class":216},[172,1444,556],{"class":220},[172,1446,1447],{"class":181},"\"tests\u002Ffixtures\u002Forders_bad.xlsx\"",[172,1449,354],{"class":220},[172,1451,584],{"class":305},[172,1453,309],{"class":216},[172,1455,589],{"class":220},[172,1457,592],{"class":181},[172,1459,595],{"class":220},[172,1461,1462,1464,1467,1469],{"class":86,"line":442},[172,1463,945],{"class":216},[172,1465,1466],{"class":220}," pytest.raises(pandera.errors.SchemaErrors) ",[172,1468,224],{"class":216},[172,1470,1471],{"class":220}," excinfo:\n",[172,1473,1474,1477,1479,1481,1483,1485],{"class":86,"line":460},[172,1475,1476],{"class":254},"        ORDERS",[172,1478,620],{"class":220},[172,1480,623],{"class":305},[172,1482,309],{"class":216},[172,1484,312],{"class":254},[172,1486,479],{"class":220},[172,1488,1489,1492,1495,1498,1501],{"class":86,"line":476},[172,1490,1491],{"class":216},"    assert",[172,1493,1494],{"class":254}," len",[172,1496,1497],{"class":220},"(excinfo.value.failure_cases) ",[172,1499,1500],{"class":216},"==",[172,1502,1503],{"class":254}," 3\n",[10,1505,1506],{},"When the upstream format changes, the schema and its fixtures change together, and the diff shows exactly what was renegotiated.",[158,1508,1510],{"id":1509},"wire-it-into-the-job","Wire it into the job",[10,1512,1513],{},"Validation belongs immediately after the read and before anything computes:",[163,1515,1517],{"className":201,"code":1516,"language":203,"meta":168,"style":168},"import sys\n\nimport pandas as pd\nimport pandera.errors\n\nfrom schema import ORDERS\n\ndef load_orders(path: str) -> pd.DataFrame:\n    df = pd.read_excel(path, dtype={\"order_id\": \"string\"}, parse_dates=[\"ordered\"])\n    try:\n        return ORDERS.validate(df, lazy=True)\n    except pandera.errors.SchemaErrors as exc:\n        write_error_report(exc.failure_cases, \"orders_problems.xlsx\")\n        print(f\"{len(exc.failure_cases)} problem(s) in {path}\", file=sys.stderr)\n        raise SystemExit(2)\n",[14,1518,1519,1526,1530,1540,1546,1550,1560,1564,1579,1612,1619,1636,1647,1657,1699],{"__ignoreMap":168},[172,1520,1521,1523],{"class":86,"line":174},[172,1522,217],{"class":216},[172,1524,1525],{"class":220}," sys\n",[172,1527,1528],{"class":86,"line":213},[172,1529,248],{"emptyLinePlaceholder":247},[172,1531,1532,1534,1536,1538],{"class":86,"line":230},[172,1533,217],{"class":216},[172,1535,514],{"class":220},[172,1537,224],{"class":216},[172,1539,519],{"class":220},[172,1541,1542,1544],{"class":86,"line":244},[172,1543,217],{"class":216},[172,1545,526],{"class":220},[172,1547,1548],{"class":86,"line":251},[172,1549,248],{"emptyLinePlaceholder":247},[172,1551,1552,1554,1556,1558],{"class":86,"line":264},[172,1553,233],{"class":216},[172,1555,537],{"class":220},[172,1557,217],{"class":216},[172,1559,542],{"class":254},[172,1561,1562],{"class":86,"line":270},[172,1563,248],{"emptyLinePlaceholder":247},[172,1565,1566,1568,1571,1574,1576],{"class":86,"line":318},[172,1567,807],{"class":216},[172,1569,1570],{"class":810}," load_orders",[172,1572,1573],{"class":220},"(path: ",[172,1575,279],{"class":254},[172,1577,1578],{"class":220},") -> pd.DataFrame:\n",[172,1580,1581,1583,1585,1588,1590,1592,1594,1596,1598,1600,1602,1604,1606,1608,1610],{"class":86,"line":338},[172,1582,1389],{"class":220},[172,1584,309],{"class":216},[172,1586,1587],{"class":220}," pd.read_excel(path, ",[172,1589,564],{"class":305},[172,1591,309],{"class":216},[172,1593,569],{"class":220},[172,1595,572],{"class":181},[172,1597,575],{"class":220},[172,1599,578],{"class":181},[172,1601,581],{"class":220},[172,1603,584],{"class":305},[172,1605,309],{"class":216},[172,1607,589],{"class":220},[172,1609,592],{"class":181},[172,1611,595],{"class":220},[172,1613,1614,1617],{"class":86,"line":373},[172,1615,1616],{"class":216},"    try",[172,1618,607],{"class":220},[172,1620,1621,1624,1626,1628,1630,1632,1634],{"class":86,"line":392},[172,1622,1623],{"class":216},"        return",[172,1625,617],{"class":254},[172,1627,620],{"class":220},[172,1629,623],{"class":305},[172,1631,309],{"class":216},[172,1633,312],{"class":254},[172,1635,479],{"class":220},[172,1637,1638,1641,1643,1645],{"class":86,"line":415},[172,1639,1640],{"class":216},"    except",[172,1642,670],{"class":220},[172,1644,224],{"class":216},[172,1646,675],{"class":220},[172,1648,1649,1652,1655],{"class":86,"line":436},[172,1650,1651],{"class":220},"        write_error_report(exc.failure_cases, ",[172,1653,1654],{"class":181},"\"orders_problems.xlsx\"",[172,1656,479],{"class":220},[172,1658,1659,1662,1664,1666,1668,1670,1672,1675,1677,1680,1682,1685,1687,1689,1691,1694,1696],{"class":86,"line":442},[172,1660,1661],{"class":254},"        print",[172,1663,637],{"class":220},[172,1665,640],{"class":216},[172,1667,288],{"class":181},[172,1669,569],{"class":645},[172,1671,648],{"class":254},[172,1673,1674],{"class":220},"(exc.failure_cases)",[172,1676,657],{"class":645},[172,1678,1679],{"class":181}," problem(s) in ",[172,1681,569],{"class":645},[172,1683,1684],{"class":220},"path",[172,1686,657],{"class":645},[172,1688,288],{"class":181},[172,1690,354],{"class":220},[172,1692,1693],{"class":305},"file",[172,1695,309],{"class":216},[172,1697,1698],{"class":220},"sys.stderr)\n",[172,1700,1701,1704,1707,1709,1712],{"class":86,"line":460},[172,1702,1703],{"class":216},"        raise",[172,1705,1706],{"class":254}," SystemExit",[172,1708,637],{"class":220},[172,1710,1711],{"class":254},"2",[172,1713,479],{"class":220},[10,1715,1716],{},"Exiting non-zero matters: a scheduled job that validated its input and then carried on regardless has only added a log line, not a safeguard. Fail the run, keep the report, and let the retry happen after the file is fixed.",[158,1718,1720],{"id":1719},"decide-what-a-failure-should-do","Decide what a failure should do",[10,1722,1723],{},"Not every violation deserves the same response, and the schema is a good place to make that explicit. Three tiers cover most imports:",[34,1725,43,1730,43,1733,43,1736,43,1739,43,1742,43,1746,43,1750,43,1755,43,1758,43,1762,43,1766,43,1771,43,1774,43,1777,43,1781,43,1784,43,1789,43,1792,43,1795],{"viewBox":1726,"role":37,"ariaLabelledBy":1727,"xmlns":41,"style":42},"0 0 760 218",[1728,1729],"pa3-t","pa3-d",[45,1731,1732],{"id":1728},"Three severities of validation failure",[49,1734,1735],{"id":1729},"A structural failure stops the job, a row-level failure quarantines the offending rows and continues, and a warning is recorded for review without affecting the run.",[53,1737],{"x":55,"y":55,"width":56,"height":1738,"fill":58},"218",[60,1740,1741],{"x":62,"y":63,"style":64},"What each kind of problem should cost",[53,1743],{"x":1111,"y":1125,"width":1744,"height":1141,"rx":1745,"fill":148,"stroke":73},"226","12",[60,1747,1749],{"x":1748,"y":142,"style":1112},"141","structural",[60,1751,1754],{"x":1748,"y":1752,"style":1753},"106","font-size:11.5px;fill:var(--text,#172033);text-anchor:middle","a column is missing",[60,1756,1757],{"x":1748,"y":82,"style":1753},"a type cannot coerce",[60,1759,1761],{"x":1748,"y":101,"style":1760},"font-size:11.5px;fill:var(--accent-ink,#be185d);text-anchor:middle","stop the job",[53,1763],{"x":1764,"y":1125,"width":1744,"height":1141,"rx":1745,"fill":1765,"stroke":73},"268","#fdefd8",[60,1767,1770],{"x":1768,"y":142,"style":1769},"381","font-size:12.5px;font-weight:700;fill:var(--gold-ink,#7a4e06);text-anchor:middle","row-level",[60,1772,1773],{"x":1768,"y":1752,"style":1753},"a negative quantity",[60,1775,1776],{"x":1768,"y":82,"style":1753},"an unknown region",[60,1778,1780],{"x":1768,"y":101,"style":1779},"font-size:11.5px;fill:var(--gold-ink,#7a4e06);text-anchor:middle","quarantine those rows",[53,1782],{"x":1783,"y":1125,"width":57,"height":1141,"rx":1745,"fill":102,"stroke":73},"508",[60,1785,1788],{"x":1786,"y":142,"style":1787},"620","font-size:12.5px;font-weight:700;fill:var(--brand-strong,#4338ca);text-anchor:middle","advisory",[60,1790,1791],{"x":1786,"y":1752,"style":1753},"an empty note field",[60,1793,1794],{"x":1786,"y":82,"style":1753},"an unusual outlier",[60,1796,1798],{"x":1786,"y":101,"style":1797},"font-size:11.5px;fill:var(--brand-strong,#4338ca);text-anchor:middle","record and continue",[10,1800,1801],{},"The middle column is the one worth implementing deliberately, because \"quarantine the bad rows and process the rest\" is what most operations teams actually want. pandera makes it a two-line split once you have the failure cases:",[163,1803,1805],{"className":201,"code":1804,"language":203,"meta":168,"style":168},"bad_rows = set(exc.failure_cases[\"index\"].dropna().astype(int))\nusable = df.drop(index=bad_rows)\nquarantined = df.loc[sorted(bad_rows)]\nprint(f\"processing {len(usable):,} rows, quarantining {len(quarantined):,}\")\n",[14,1806,1807,1829,1846,1862],{"__ignoreMap":168},[172,1808,1809,1812,1814,1817,1820,1822,1825,1827],{"class":86,"line":174},[172,1810,1811],{"class":220},"bad_rows ",[172,1813,309],{"class":216},[172,1815,1816],{"class":254}," set",[172,1818,1819],{"class":220},"(exc.failure_cases[",[172,1821,704],{"class":181},[172,1823,1824],{"class":220},"].dropna().astype(",[172,1826,381],{"class":254},[172,1828,759],{"class":220},[172,1830,1831,1834,1836,1839,1841,1843],{"class":86,"line":213},[172,1832,1833],{"class":220},"usable ",[172,1835,309],{"class":216},[172,1837,1838],{"class":220}," df.drop(",[172,1840,752],{"class":305},[172,1842,309],{"class":216},[172,1844,1845],{"class":220},"bad_rows)\n",[172,1847,1848,1851,1853,1856,1859],{"class":86,"line":230},[172,1849,1850],{"class":220},"quarantined ",[172,1852,309],{"class":216},[172,1854,1855],{"class":220}," df.loc[",[172,1857,1858],{"class":254},"sorted",[172,1860,1861],{"class":220},"(bad_rows)]\n",[172,1863,1864,1867,1869,1871,1874,1876,1878,1881,1883,1885,1888,1890,1892,1895,1897,1899,1901],{"class":86,"line":244},[172,1865,1866],{"class":254},"print",[172,1868,637],{"class":220},[172,1870,640],{"class":216},[172,1872,1873],{"class":181},"\"processing ",[172,1875,569],{"class":645},[172,1877,648],{"class":254},[172,1879,1880],{"class":220},"(usable)",[172,1882,654],{"class":216},[172,1884,657],{"class":645},[172,1886,1887],{"class":181}," rows, quarantining ",[172,1889,569],{"class":645},[172,1891,648],{"class":254},[172,1893,1894],{"class":220},"(quarantined)",[172,1896,654],{"class":216},[172,1898,657],{"class":645},[172,1900,288],{"class":181},[172,1902,479],{"class":220},[10,1904,1905],{},"Write the quarantined rows out beside the error report so the sender can correct and resubmit just those, rather than the whole file.",[158,1907,1909],{"id":1908},"generate-the-first-draft-of-a-schema","Generate the first draft of a schema",[10,1911,1912],{},"Writing a schema for a forty-column export by hand is tedious, and pandera will infer one from a known-good file to start from:",[163,1914,1916],{"className":201,"code":1915,"language":203,"meta":168,"style":168},"import pandas as pd\nimport pandera.pandas as pa\n\ngood = pd.read_excel(\"orders_known_good.xlsx\", parse_dates=[\"ordered\"])\ninferred = pa.infer_schema(good)\nprint(inferred.to_script())      # a Python module you can edit and commit\n",[14,1917,1918,1928,1938,1942,1966,1976],{"__ignoreMap":168},[172,1919,1920,1922,1924,1926],{"class":86,"line":174},[172,1921,217],{"class":216},[172,1923,514],{"class":220},[172,1925,224],{"class":216},[172,1927,519],{"class":220},[172,1929,1930,1932,1934,1936],{"class":86,"line":213},[172,1931,217],{"class":216},[172,1933,221],{"class":220},[172,1935,224],{"class":216},[172,1937,227],{"class":220},[172,1939,1940],{"class":86,"line":230},[172,1941,248],{"emptyLinePlaceholder":247},[172,1943,1944,1947,1949,1951,1954,1956,1958,1960,1962,1964],{"class":86,"line":244},[172,1945,1946],{"class":220},"good ",[172,1948,309],{"class":216},[172,1950,556],{"class":220},[172,1952,1953],{"class":181},"\"orders_known_good.xlsx\"",[172,1955,354],{"class":220},[172,1957,584],{"class":305},[172,1959,309],{"class":216},[172,1961,589],{"class":220},[172,1963,592],{"class":181},[172,1965,595],{"class":220},[172,1967,1968,1971,1973],{"class":86,"line":251},[172,1969,1970],{"class":220},"inferred ",[172,1972,309],{"class":216},[172,1974,1975],{"class":220}," pa.infer_schema(good)\n",[172,1977,1978,1980,1983],{"class":86,"line":264},[172,1979,1866],{"class":254},[172,1981,1982],{"class":220},"(inferred.to_script())      ",[172,1984,1985],{"class":456},"# a Python module you can edit and commit\n",[10,1987,1988],{},"Treat the output as a draft, not an answer. Inference sees the ranges that happen to exist in one file, so it will propose a maximum quantity of 47 simply because that was the largest value present. Keep the column names and types, replace the invented bounds with the real business rules, and commit the edited version.",[158,1990,1992],{"id":1991},"common-pitfalls-and-gotchas","Common pitfalls and gotchas",[1994,1995,1996,2006,2012,2022,2028],"ul",{},[1997,1998,1999,2005],"li",{},[2000,2001,2002,2003,32],"strong",{},"Forgetting ",[14,2004,764],{}," Eager validation makes fixing a sheet an iterative guessing game.",[1997,2007,2008,2011],{},[2000,2009,2010],{},"Off-by-two row numbers."," A DataFrame index is zero-based and the sheet has a header, so add two before showing a row number to a person.",[1997,2013,2014,2017,2018,2021],{},[2000,2015,2016],{},"Over-strict schemas."," ",[14,2019,2020],{},"strict=True"," rejects extra columns, which is right for a controlled feed and wrong for a file people also use for their own notes.",[1997,2023,2024,2027],{},[2000,2025,2026],{},"Validating after transforming."," Validate the raw import; a rule that runs after cleaning tells you about your code, not about the file.",[1997,2029,2030,2017,2033,2035,2036,2039,2040,2043],{},[2000,2031,2032],{},"Silent coercion surprises.",[14,2034,497],{}," will turn ",[14,2037,2038],{},"\"5\""," into ",[14,2041,2042],{},"5","; that is usually welcome, but be deliberate about identifier columns.",[158,2045,2047],{"id":2046},"performance-and-scale-notes","Performance and scale notes",[10,2049,2050,2051,2054,2055,2057,2058,32],{},"Validation is vectorised, so cost scales with rows in the same way a ",[14,2052,2053],{},"groupby"," does — negligible next to the Excel parse for a report-sized file. The exceptions are ",[14,2056,485],{}," on a very wide string column and custom lambdas that fall back to element-wise evaluation; write checks as vectorised expressions over the frame wherever possible. For genuinely large imports, validate after converting to Parquet rather than on every Excel read, so the parse happens once — the conversion is covered in ",[28,2059,2061],{"href":2060},"\u002Fadvanced-data-transformation-and-cleaning\u002Freading-excel-with-polars-and-arrow\u002Fconvert-excel-files-to-parquet-with-python\u002F","Convert Excel files to Parquet with Python",[158,2063,2065],{"id":2064},"conclusion","Conclusion",[10,2067,2068,2069,2071],{},"A schema turns \"check the spreadsheet\" into an executable, reviewable definition of what the spreadsheet must contain. Declare types and checks once, validate with ",[14,2070,764],{}," so every problem surfaces together, translate the failures into Excel row numbers, and hand the sender a workbook naming what to fix. Then fail the job on invalid input, so bad data never becomes a plausible-looking report.",[158,2073,2075],{"id":2074},"frequently-asked-questions","Frequently asked questions",[10,2077,2078,2081],{},[2000,2079,2080],{},"Why use pandera instead of a few assert statements?","\nA schema is declarative, reusable and self-documenting, and lazy validation reports every failure at once instead of stopping at the first. Assertions are fine for one script; a schema is what you want when several jobs read the same file shape.",[10,2083,2084,2087,2088,2090],{},[2000,2085,2086],{},"Does pandera change my data?","\nOnly if you ask it to. With ",[14,2089,497],{}," it casts columns to the declared types as part of validation; otherwise it checks and reports without modifying anything.",[10,2092,2093,2096,2097,2100,2101,2103],{},[2000,2094,2095],{},"How do I report which spreadsheet rows failed?","\nCatch ",[14,2098,2099],{},"SchemaErrors"," and read its ",[14,2102,768],{}," frame. Add two to the index to get the Excel row number, since a DataFrame is zero-based and the sheet has a header row.",[10,2105,2106,2109],{},[2000,2107,2108],{},"Can it validate that two columns agree?","\nYes. A schema-level check receives the whole frame, so a rule like \"end date is not before start date\" is a single lambda.",[10,2111,2112,2115],{},[2000,2113,2114],{},"Does it work with Polars?","\nYes, recent pandera versions support Polars alongside pandas, with the same schema vocabulary — useful if the read side has already moved.",[158,2117,2119],{"id":2118},"related","Related",[1994,2121,2122,2130,2137,2142,2149],{},[1997,2123,2124,2017,2127,2129],{},[2000,2125,2126],{},"Up:",[28,2128,31],{"href":30}," — the wider set of checks an import deserves.",[1997,2131,2132,2136],{},[28,2133,2135],{"href":2134},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fvalidate-excel-columns-before-import-with-pandas\u002F","Validate Excel columns before import with pandas"," — the hand-rolled version, and when it is enough.",[1997,2138,2139,2141],{},[28,2140,1092],{"href":1091}," — marking the failures in the original workbook.",[1997,2143,2144,2148],{},[28,2145,2147],{"href":2146},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Ffind-duplicate-rows-in-excel-with-python\u002F","Find duplicate rows in Excel with Python"," — the uniqueness rule, examined on its own.",[1997,2150,2151,2155],{},[28,2152,2154],{"href":2153},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fcheck-excel-data-types-with-pandas\u002F","Check Excel data types with pandas"," — the type half of a schema, without the framework.",[2157,2158,2159],"style",{},"html pre.shiki code .sMTad, html code.shiki .sMTad{--shiki-default:#6F42C1;--shiki-dark:#FFB757}html pre.shiki code .srMev, html code.shiki .srMev{--shiki-default:#032F62;--shiki-dark:#ADDCFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .s-kum, html code.shiki .s-kum{--shiki-default:#D73A49;--shiki-dark:#FF9492}html pre.shiki code .skGVy, html code.shiki .skGVy{--shiki-default:#24292E;--shiki-dark:#F0F3F6}html pre.shiki code .sP0c6, html code.shiki .sP0c6{--shiki-default:#005CC5;--shiki-dark:#91CBFF}html pre.shiki code .sa561, html code.shiki .sa561{--shiki-default:#E36209;--shiki-dark:#FFB757}html pre.shiki code .s-wDw, html code.shiki .s-wDw{--shiki-default:#6A737D;--shiki-dark:#BDC4CC}html pre.shiki code .sSjpA, html code.shiki .sSjpA{--shiki-default:#005CC5;--shiki-dark:#FF9492}html pre.shiki code .s_Opv, html code.shiki .s_Opv{--shiki-default:#6F42C1;--shiki-dark:#DBB7FF}",{"title":168,"searchDepth":213,"depth":213,"links":2161},[2162,2163,2164,2165,2166,2167,2168,2169,2170,2171,2172,2173,2174,2175],{"id":160,"depth":213,"text":161},{"id":194,"depth":213,"text":195},{"id":501,"depth":213,"text":502},{"id":779,"depth":213,"text":780},{"id":1166,"depth":213,"text":1167},{"id":1319,"depth":213,"text":1320},{"id":1509,"depth":213,"text":1510},{"id":1719,"depth":213,"text":1720},{"id":1908,"depth":213,"text":1909},{"id":1991,"depth":213,"text":1992},{"id":2046,"depth":213,"text":2047},{"id":2064,"depth":213,"text":2065},{"id":2074,"depth":213,"text":2075},{"id":2118,"depth":213,"text":2119},"2026-08-27","Declare a schema for a spreadsheet and enforce it: column types, ranges, regex checks, uniqueness, cross-column rules, lazy validation and a readable error report.","md",[2180,2182,2184,2186,2188],{"q":2080,"a":2181},"A schema is declarative, reusable and self-documenting, and lazy validation reports every failure at once instead of stopping at the first. Assertions are fine for one script; a schema is what you want when several jobs read the same file shape.",{"q":2086,"a":2183},"Only if you ask it to. With coerce=True it casts columns to the declared types as part of validation; otherwise it checks and reports without modifying anything.",{"q":2095,"a":2185},"Catch SchemaErrors and read its failure_cases frame. Add two to the index to get the Excel row number, since a DataFrame is zero-based and the sheet has a header row.",{"q":2108,"a":2187},"Yes. A DataFrameSchema-level check receives the whole frame, so a rule like \"end date is not before start date\" is a single lambda.",{"q":2114,"a":2189},"Yes, recent pandera versions support Polars alongside pandas, with the same schema vocabulary — useful if the read side has already moved.",{},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fvalidate-excel-data-with-pandera-schemas",{"title":2193,"description":2194},"Validate Excel Data with pandera","Use pandera to validate Excel imports: DataFrameSchema, typed columns, value checks, nullable and unique rules, lazy=True error reports, and coercion at the boundary.","validate-excel-data-with-pandera-schemas","advanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fvalidate-excel-data-with-pandera-schemas\u002Findex","how-to","s6ToUG28SQ8Dor_vVPfyXeGMH426iqft-AkZMtZHf4o",[2200,2204],{"title":2201,"path":2202,"stem":2203,"children":-1},"Validate Excel Columns Before Import with pandas","\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fvalidate-excel-columns-before-import-with-pandas","advanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fvalidate-excel-columns-before-import-with-pandas\u002Findex",{"title":2205,"path":2206,"stem":2207,"children":-1},"Working with Dates and Times in Excel Data","\u002Fadvanced-data-transformation-and-cleaning\u002Fworking-with-dates-and-times-in-excel-data","advanced-data-transformation-and-cleaning\u002Fworking-with-dates-and-times-in-excel-data\u002Findex",1788710158442]