[{"data":1,"prerenderedAt":2213},["ShallowReactive",2],{"doc:\u002Fadvanced-data-transformation-and-cleaning\u002Fhandling-missing-data-in-excel-reports\u002Fdrop-rows-with-missing-required-fields-in-pandas":3,"surround:\u002Fadvanced-data-transformation-and-cleaning\u002Fhandling-missing-data-in-excel-reports\u002Fdrop-rows-with-missing-required-fields-in-pandas":2204},{"id":4,"title":5,"body":6,"dateModified":2177,"datePublished":2177,"description":2178,"extension":2179,"faq":2180,"meta":2189,"navigation":219,"path":2197,"seo":2198,"slug":2200,"stem":2201,"type":2202,"__hash__":2203},"docs\u002Fadvanced-data-transformation-and-cleaning\u002Fhandling-missing-data-in-excel-reports\u002Fdrop-rows-with-missing-required-fields-in-pandas\u002Findex.md","Drop Rows with Missing Required Fields in pandas",{"type":7,"value":8,"toc":2160},"minimark",[9,24,146,151,178,370,387,391,414,423,427,524,532,536,636,659,869,879,883,886,1071,1077,1169,1173,1180,1238,1246,1250,1253,1308,1315,1319,1322,1672,1679,1683,1686,1780,1791,1795,1899,1903,1906,1918,1922,2003,2008,2076,2079,2083,2089,2093,2100,2106,2112,2118,2122,2156],[10,11,12,13,17,18,23],"p",{},"Dropping rows is the easiest data-quality decision to get wrong, because ",[14,15,16],"code",{},"dropna()"," with no arguments\ndoes something almost nobody wants: it removes any row with a blank in any column, which on a real\nexport means most of them. The useful version names the columns that genuinely must be present,\ncounts what it removed, and keeps the removed rows where somebody can look at them. This guide is\npart of\n",[19,20,22],"a",{"href":21},"\u002Fadvanced-data-transformation-and-cleaning\u002Fhandling-missing-data-in-excel-reports\u002F","Handling Missing Data in Excel Reports",".",[25,26,34,35,34,39,34,43,34,50,34,60,34,66,34,73,34,78,34,82,34,86,34,91,34,96,34,100,34,103,34,106,34,109,34,118,34,124,34,129,34,134,34,138,34,141],"svg",{"viewBox":27,"role":28,"ariaLabelledBy":29,"xmlns":32,"style":33},"0 0 760 201","img",[30,31],"dr-default-t","dr-default-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;max-width:760px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif","\n  ",[36,37,38],"title",{"id":30},"What dropna does by default, and what you meant",[40,41,42],"desc",{"id":31},"Calling dropna with no arguments removes any row with a blank in any column, so an empty optional note discards the record; passing a subset limits the test to the columns that must be present.",[44,45],"rect",{"x":46,"y":46,"width":47,"height":48,"fill":49},"0","760","201","#ffffff",[44,51],{"x":52,"y":53,"width":54,"height":55,"rx":56,"fill":57,"stroke":58,"style":59},"20","28","270.0","139","14","#fee8f2","var(--accent,#d81b73)","stroke-width:2px",[61,62,16],"text",{"x":63,"y":64,"style":65},"155.0","54","font-size:13px;font-weight:700;fill:var(--accent,#d81b73);text-anchor:middle",[67,68],"line",{"x1":69,"y1":70,"x2":71,"y2":70,"stroke":58,"style":72},"36","64","274.0","stroke-width:1px",[61,74,77],{"x":63,"y":75,"style":76},"86","font-size:11.5px;font-weight:400;fill:var(--text,#172033);text-anchor:middle","tests every column",[61,79,81],{"x":63,"y":80,"style":76},"109","an optional blank drops the row",[61,83,85],{"x":63,"y":84,"style":76},"132","2 of 5 rows survive",[44,87],{"x":88,"y":53,"width":54,"height":55,"rx":56,"fill":89,"stroke":90,"style":59},"470.0","#d9f4f1","var(--teal,#0f9488)",[61,92,95],{"x":93,"y":64,"style":94},"605.0","font-size:13px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","dropna(subset=…)",[67,97],{"x1":98,"y1":70,"x2":99,"y2":70,"stroke":90,"style":72},"486.0","724.0",[61,101,102],{"x":93,"y":75,"style":76},"tests the required columns",[61,104,105],{"x":93,"y":80,"style":76},"optional blanks ignored",[61,107,108],{"x":93,"y":84,"style":76},"4 of 5 rows survive",[44,110],{"x":111,"y":112,"width":113,"height":114,"rx":115,"fill":116,"stroke":117},"316.0","78.5","128","38","19","#ebebfd","var(--brand,#5b5cf0)",[61,119,123],{"x":120,"y":121,"style":122},"380.0","102.5","font-size:12.5px;font-weight:700;fill:var(--brand-strong,#4338ca);text-anchor:middle","subset",[67,125],{"x1":126,"y1":127,"x2":128,"y2":127,"stroke":117,"style":59},"295.0","97.5","309.0",[130,131],"polygon",{"points":132,"fill":133},"309.0,97.5 300.0,92.5 300.0,102.5","#5b5cf0",[67,135],{"x1":136,"y1":127,"x2":137,"y2":127,"stroke":117,"style":59},"449.0","463.0",[130,139],{"points":140,"fill":133},"463.0,97.5 454.0,92.5 454.0,102.5",[61,142,145],{"x":120,"y":143,"style":144},"187","font-size:12.5px;font-weight:400;fill:var(--muted,#5b6780);text-anchor:middle","the default is almost never the intended rule",[147,148,150],"h2",{"id":149},"prerequisites","Prerequisites",[152,153,158],"pre",{"className":154,"code":155,"language":156,"meta":157,"style":157},"language-bash shiki shiki-themes github-light github-dark-high-contrast","pip install pandas openpyxl\n","bash","",[14,159,160],{"__ignoreMap":157},[161,162,164,168,172,175],"span",{"class":67,"line":163},1,[161,165,167],{"class":166},"sMTad","pip",[161,169,171],{"class":170},"srMev"," install",[161,173,174],{"class":170}," pandas",[161,176,177],{"class":170}," openpyxl\n",[152,179,183],{"className":180,"code":181,"language":182,"meta":157,"style":157},"language-python shiki shiki-themes github-light github-dark-high-contrast","import numpy as np\nimport pandas as pd\n\norders = pd.DataFrame({\n    \"Order_ID\": [1001, 1002, None, 1004, 1005],\n    \"Region\":   [\"North\", \"  \", \"West\", \"South\", \"North\"],\n    \"Revenue\":  [12400.0, 9800.5, 15320.25, np.nan, 7010.0],\n    \"Notes\":    [\"rush\", None, None, None, \"backorder\"],\n})\n","python",[14,184,185,201,214,221,233,270,303,333,364],{"__ignoreMap":157},[161,186,187,191,195,198],{"class":67,"line":163},[161,188,190],{"class":189},"s-kum","import",[161,192,194],{"class":193},"skGVy"," numpy ",[161,196,197],{"class":189},"as",[161,199,200],{"class":193}," np\n",[161,202,204,206,209,211],{"class":67,"line":203},2,[161,205,190],{"class":189},[161,207,208],{"class":193}," pandas ",[161,210,197],{"class":189},[161,212,213],{"class":193}," pd\n",[161,215,217],{"class":67,"line":216},3,[161,218,220],{"emptyLinePlaceholder":219},true,"\n",[161,222,224,227,230],{"class":67,"line":223},4,[161,225,226],{"class":193},"orders ",[161,228,229],{"class":189},"=",[161,231,232],{"class":193}," pd.DataFrame({\n",[161,234,236,239,242,246,249,252,254,257,259,262,264,267],{"class":67,"line":235},5,[161,237,238],{"class":170},"    \"Order_ID\"",[161,240,241],{"class":193},": [",[161,243,245],{"class":244},"sP0c6","1001",[161,247,248],{"class":193},", ",[161,250,251],{"class":244},"1002",[161,253,248],{"class":193},[161,255,256],{"class":244},"None",[161,258,248],{"class":193},[161,260,261],{"class":244},"1004",[161,263,248],{"class":193},[161,265,266],{"class":244},"1005",[161,268,269],{"class":193},"],\n",[161,271,273,276,279,282,284,287,289,292,294,297,299,301],{"class":67,"line":272},6,[161,274,275],{"class":170},"    \"Region\"",[161,277,278],{"class":193},":   [",[161,280,281],{"class":170},"\"North\"",[161,283,248],{"class":193},[161,285,286],{"class":170},"\"  \"",[161,288,248],{"class":193},[161,290,291],{"class":170},"\"West\"",[161,293,248],{"class":193},[161,295,296],{"class":170},"\"South\"",[161,298,248],{"class":193},[161,300,281],{"class":170},[161,302,269],{"class":193},[161,304,306,309,312,315,317,320,322,325,328,331],{"class":67,"line":305},7,[161,307,308],{"class":170},"    \"Revenue\"",[161,310,311],{"class":193},":  [",[161,313,314],{"class":244},"12400.0",[161,316,248],{"class":193},[161,318,319],{"class":244},"9800.5",[161,321,248],{"class":193},[161,323,324],{"class":244},"15320.25",[161,326,327],{"class":193},", np.nan, ",[161,329,330],{"class":244},"7010.0",[161,332,269],{"class":193},[161,334,336,339,342,345,347,349,351,353,355,357,359,362],{"class":67,"line":335},8,[161,337,338],{"class":170},"    \"Notes\"",[161,340,341],{"class":193},":    [",[161,343,344],{"class":170},"\"rush\"",[161,346,248],{"class":193},[161,348,256],{"class":244},[161,350,248],{"class":193},[161,352,256],{"class":244},[161,354,248],{"class":193},[161,356,256],{"class":244},[161,358,248],{"class":193},[161,360,361],{"class":170},"\"backorder\"",[161,363,269],{"class":193},[161,365,367],{"class":67,"line":366},9,[161,368,369],{"class":193},"})\n",[10,371,372,375,376,248,379,382,383,386],{},[14,373,374],{},"Notes"," is empty for three rows and that is fine — it is an optional column. ",[14,377,378],{},"Order_ID",[14,380,381],{},"Region"," and\n",[14,384,385],{},"Revenue"," are not.",[147,388,390],{"id":389},"why-the-default-is-wrong","Why the default is wrong",[152,392,394],{"className":180,"code":393,"language":182,"meta":157,"style":157},"print(len(orders.dropna()))          # 2 of 5 — the Notes column decided\n",[14,395,396],{"__ignoreMap":157},[161,397,398,401,404,407,410],{"class":67,"line":163},[161,399,400],{"class":244},"print",[161,402,403],{"class":193},"(",[161,405,406],{"class":244},"len",[161,408,409],{"class":193},"(orders.dropna()))          ",[161,411,413],{"class":412},"s-wDw","# 2 of 5 — the Notes column decided\n",[10,415,416,418,419,422],{},[14,417,16],{}," defaults to ",[14,420,421],{},"how=\"any\""," across every column, so an optional note being blank removes the\nrow. On a forty-column export with a handful of optional fields, that routinely discards the majority\nof the data and looks, from the outside, like the source file was nearly empty.",[147,424,426],{"id":425},"name-the-columns-that-matter","Name the columns that matter",[152,428,430],{"className":180,"code":429,"language":182,"meta":157,"style":157},"REQUIRED = [\"Order_ID\", \"Region\", \"Revenue\"]\n\nclean = orders.dropna(subset=REQUIRED)\nprint(f\"{len(clean)} of {len(orders)} rows kept\")\n",[14,431,432,459,463,483],{"__ignoreMap":157},[161,433,434,437,440,443,446,448,451,453,456],{"class":67,"line":163},[161,435,436],{"class":244},"REQUIRED",[161,438,439],{"class":189}," =",[161,441,442],{"class":193}," [",[161,444,445],{"class":170},"\"Order_ID\"",[161,447,248],{"class":193},[161,449,450],{"class":170},"\"Region\"",[161,452,248],{"class":193},[161,454,455],{"class":170},"\"Revenue\"",[161,457,458],{"class":193},"]\n",[161,460,461],{"class":67,"line":203},[161,462,220],{"emptyLinePlaceholder":219},[161,464,465,468,470,473,476,478,480],{"class":67,"line":216},[161,466,467],{"class":193},"clean ",[161,469,229],{"class":189},[161,471,472],{"class":193}," orders.dropna(",[161,474,123],{"class":475},"sa561",[161,477,229],{"class":189},[161,479,436],{"class":244},[161,481,482],{"class":193},")\n",[161,484,485,487,489,492,495,499,501,504,507,510,512,514,517,519,522],{"class":67,"line":223},[161,486,400],{"class":244},[161,488,403],{"class":193},[161,490,491],{"class":189},"f",[161,493,494],{"class":170},"\"",[161,496,498],{"class":497},"sSjpA","{",[161,500,406],{"class":244},[161,502,503],{"class":193},"(clean)",[161,505,506],{"class":497},"}",[161,508,509],{"class":170}," of ",[161,511,498],{"class":497},[161,513,406],{"class":244},[161,515,516],{"class":193},"(orders)",[161,518,506],{"class":497},[161,520,521],{"class":170}," rows kept\"",[161,523,482],{"class":193},[10,525,526,527,531],{},"Declaring the required columns as a constant is worth more than the line it saves. It documents the\ncontract, it can be asserted against the incoming file before anything else happens, and it is the\nsame list the validation in\n",[19,528,530],{"href":529},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fvalidate-excel-columns-before-import-with-pandas\u002F","Validate Excel Columns Before Import with Pandas","\nchecks for existence.",[147,533,535],{"id":534},"blank-is-not-always-nan","Blank is not always NaN",[25,537,34,542,34,545,34,548,34,551,34,557,34,562,34,567,34,573,34,578,34,581,34,584,34,588,34,592,34,596,34,599,34,602,34,606,34,610,34,614,34,617,34,620,34,624,34,628,34,632],{"viewBox":538,"role":28,"ariaLabelledBy":539,"xmlns":32,"style":33},"0 0 760 308",[540,541],"dr-blank-t","dr-blank-d",[36,543,544],{"id":540},"Making blank mean blank before you test for it",[40,546,547],{"id":541},"Strip whitespace, map the common text placeholders to missing, then test — otherwise a cell containing two spaces or the letters N slash A counts as a value and survives every check.",[44,549],{"x":46,"y":46,"width":47,"height":550,"fill":49},"308",[44,552],{"x":553,"y":53,"width":554,"height":555,"rx":556,"fill":89,"stroke":90},"22","716","52","10",[558,559],"circle",{"cx":555,"cy":560,"r":561,"fill":133},"54.0","15",[61,563,566],{"x":555,"y":564,"style":565},"59.0","font-size:13px;font-weight:700;fill:#ffffff;text-anchor:middle","1",[61,568,572],{"x":569,"y":570,"style":571},"82","50","font-size:13px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:start","Strip whitespace",[61,574,577],{"x":569,"y":575,"style":576},"68","font-size:11.5px;font-weight:400;fill:var(--muted,#5b6780);text-anchor:start","two spaces is a value, not a blank",[44,579],{"x":553,"y":580,"width":554,"height":555,"rx":556,"fill":89,"stroke":90},"90",[558,582],{"cx":555,"cy":583,"r":561,"fill":133},"116.0",[61,585,587],{"x":555,"y":586,"style":565},"121.0","2",[61,589,591],{"x":569,"y":590,"style":571},"112","Map placeholders to NaN",[61,593,595],{"x":569,"y":594,"style":576},"130","N\u002FA, -, null, empty string",[44,597],{"x":553,"y":598,"width":554,"height":555,"rx":556,"fill":89,"stroke":90},"152",[558,600],{"cx":555,"cy":601,"r":561,"fill":133},"178.0",[61,603,605],{"x":555,"y":604,"style":565},"183.0","3",[61,607,609],{"x":569,"y":608,"style":571},"174","Then test for missing",[61,611,613],{"x":569,"y":612,"style":576},"192","isna now agrees with what a person sees",[44,615],{"x":553,"y":616,"width":554,"height":555,"rx":556,"fill":89,"stroke":90},"214",[558,618],{"cx":555,"cy":619,"r":561,"fill":133},"240.0",[61,621,623],{"x":555,"y":622,"style":565},"245.0","4",[61,625,627],{"x":569,"y":626,"style":571},"236","Limit it to the columns you need",[61,629,631],{"x":569,"y":630,"style":576},"254","normalising thirty text columns is real work",[61,633,635],{"x":120,"y":634,"style":144},"296","the check is cheap; making the data honest first is the work",[10,637,638,639,641,642,645,646,648,649,248,652,248,655,658],{},"The ",[14,640,381],{}," value on row two is two spaces. It is not NaN, so ",[14,643,644],{},"dropna"," keeps it — and every\nsubsequent group-by treats ",[14,647,286],{}," as its own region. Excel exports produce this constantly, along\nwith ",[14,650,651],{},"\"N\u002FA\"",[14,653,654],{},"\"-\"",[14,656,657],{},"\"null\""," and empty strings.",[152,660,662],{"className":180,"code":661,"language":182,"meta":157,"style":157},"PLACEHOLDERS = {\"\", \" \", \"-\", \"--\", \"n\u002Fa\", \"N\u002FA\", \"na\", \"null\", \"NULL\", \"#N\u002FA\"}\n\ndef normalise_blanks(frame: pd.DataFrame) -> pd.DataFrame:\n    frame = frame.copy()\n    for column in frame.select_dtypes(include=[\"object\", \"string\"]).columns:\n        stripped = frame[column].astype(\"string\").str.strip()\n        frame[column] = stripped.mask(stripped.str.casefold().isin(\n            {p.casefold() for p in PLACEHOLDERS}\n        ))\n    return frame\n\norders = normalise_blanks(orders)\nprint(orders[\"Region\"].isna().sum())          # now 1, as expected\n",[14,663,664,722,726,738,748,781,796,806,824,829,838,843,853],{"__ignoreMap":157},[161,665,666,669,671,674,677,679,682,684,686,688,691,693,696,698,700,702,705,707,709,711,714,716,719],{"class":67,"line":163},[161,667,668],{"class":244},"PLACEHOLDERS",[161,670,439],{"class":189},[161,672,673],{"class":193}," {",[161,675,676],{"class":170},"\"\"",[161,678,248],{"class":193},[161,680,681],{"class":170},"\" \"",[161,683,248],{"class":193},[161,685,654],{"class":170},[161,687,248],{"class":193},[161,689,690],{"class":170},"\"--\"",[161,692,248],{"class":193},[161,694,695],{"class":170},"\"n\u002Fa\"",[161,697,248],{"class":193},[161,699,651],{"class":170},[161,701,248],{"class":193},[161,703,704],{"class":170},"\"na\"",[161,706,248],{"class":193},[161,708,657],{"class":170},[161,710,248],{"class":193},[161,712,713],{"class":170},"\"NULL\"",[161,715,248],{"class":193},[161,717,718],{"class":170},"\"#N\u002FA\"",[161,720,721],{"class":193},"}\n",[161,723,724],{"class":67,"line":203},[161,725,220],{"emptyLinePlaceholder":219},[161,727,728,731,735],{"class":67,"line":216},[161,729,730],{"class":189},"def",[161,732,734],{"class":733},"s_Opv"," normalise_blanks",[161,736,737],{"class":193},"(frame: pd.DataFrame) -> pd.DataFrame:\n",[161,739,740,743,745],{"class":67,"line":223},[161,741,742],{"class":193},"    frame ",[161,744,229],{"class":189},[161,746,747],{"class":193}," frame.copy()\n",[161,749,750,753,756,759,762,765,767,770,773,775,778],{"class":67,"line":235},[161,751,752],{"class":189},"    for",[161,754,755],{"class":193}," column ",[161,757,758],{"class":189},"in",[161,760,761],{"class":193}," frame.select_dtypes(",[161,763,764],{"class":475},"include",[161,766,229],{"class":189},[161,768,769],{"class":193},"[",[161,771,772],{"class":170},"\"object\"",[161,774,248],{"class":193},[161,776,777],{"class":170},"\"string\"",[161,779,780],{"class":193},"]).columns:\n",[161,782,783,786,788,791,793],{"class":67,"line":272},[161,784,785],{"class":193},"        stripped ",[161,787,229],{"class":189},[161,789,790],{"class":193}," frame[column].astype(",[161,792,777],{"class":170},[161,794,795],{"class":193},").str.strip()\n",[161,797,798,801,803],{"class":67,"line":305},[161,799,800],{"class":193},"        frame[column] ",[161,802,229],{"class":189},[161,804,805],{"class":193}," stripped.mask(stripped.str.casefold().isin(\n",[161,807,808,811,814,817,819,822],{"class":67,"line":335},[161,809,810],{"class":193},"            {p.casefold() ",[161,812,813],{"class":189},"for",[161,815,816],{"class":193}," p ",[161,818,758],{"class":189},[161,820,821],{"class":244}," PLACEHOLDERS",[161,823,721],{"class":193},[161,825,826],{"class":67,"line":366},[161,827,828],{"class":193},"        ))\n",[161,830,832,835],{"class":67,"line":831},10,[161,833,834],{"class":189},"    return",[161,836,837],{"class":193}," frame\n",[161,839,841],{"class":67,"line":840},11,[161,842,220],{"emptyLinePlaceholder":219},[161,844,846,848,850],{"class":67,"line":845},12,[161,847,226],{"class":193},[161,849,229],{"class":189},[161,851,852],{"class":193}," normalise_blanks(orders)\n",[161,854,856,858,861,863,866],{"class":67,"line":855},13,[161,857,400],{"class":244},[161,859,860],{"class":193},"(orders[",[161,862,450],{"class":170},[161,864,865],{"class":193},"].isna().sum())          ",[161,867,868],{"class":412},"# now 1, as expected\n",[10,870,871,872,874,875,878],{},"Running that before any missing-data logic is what makes the rest of it behave predictably. Without\nit, ",[14,873,644],{}," and ",[14,876,877],{},"isna()"," both report numbers that disagree with what a person sees on the sheet,\nwhich is the most confusing kind of disagreement to debug.",[147,880,882],{"id":881},"keep-what-you-dropped","Keep what you dropped",[10,884,885],{},"The single most important habit in this whole area: never discard rows without keeping them\nsomewhere.",[152,887,889],{"className":180,"code":888,"language":182,"meta":157,"style":157},"required_missing = orders[REQUIRED].isna().any(axis=1)\nclean = orders[~required_missing].copy()\nrejects = orders[required_missing].copy()\n\nrejects[\"Reason\"] = (\n    orders.loc[required_missing, REQUIRED]\n          .isna()\n          .apply(lambda row: \"missing \" + \", \".join(row.index[row]), axis=1)\n)\n\nprint(f\"kept {len(clean)}, rejected {len(rejects)}\")\nprint(rejects[[\"Order_ID\", \"Region\", \"Revenue\", \"Reason\"]])\n",[14,890,891,915,929,939,943,959,968,973,1004,1008,1012,1047],{"__ignoreMap":157},[161,892,893,896,898,901,903,906,909,911,913],{"class":67,"line":163},[161,894,895],{"class":193},"required_missing ",[161,897,229],{"class":189},[161,899,900],{"class":193}," orders[",[161,902,436],{"class":244},[161,904,905],{"class":193},"].isna().any(",[161,907,908],{"class":475},"axis",[161,910,229],{"class":189},[161,912,566],{"class":244},[161,914,482],{"class":193},[161,916,917,919,921,923,926],{"class":67,"line":203},[161,918,467],{"class":193},[161,920,229],{"class":189},[161,922,900],{"class":193},[161,924,925],{"class":189},"~",[161,927,928],{"class":193},"required_missing].copy()\n",[161,930,931,934,936],{"class":67,"line":216},[161,932,933],{"class":193},"rejects ",[161,935,229],{"class":189},[161,937,938],{"class":193}," orders[required_missing].copy()\n",[161,940,941],{"class":67,"line":223},[161,942,220],{"emptyLinePlaceholder":219},[161,944,945,948,951,954,956],{"class":67,"line":235},[161,946,947],{"class":193},"rejects[",[161,949,950],{"class":170},"\"Reason\"",[161,952,953],{"class":193},"] ",[161,955,229],{"class":189},[161,957,958],{"class":193}," (\n",[161,960,961,964,966],{"class":67,"line":272},[161,962,963],{"class":193},"    orders.loc[required_missing, ",[161,965,436],{"class":244},[161,967,458],{"class":193},[161,969,970],{"class":67,"line":305},[161,971,972],{"class":193},"          .isna()\n",[161,974,975,978,981,984,987,990,993,996,998,1000,1002],{"class":67,"line":335},[161,976,977],{"class":193},"          .apply(",[161,979,980],{"class":189},"lambda",[161,982,983],{"class":193}," row: ",[161,985,986],{"class":170},"\"missing \"",[161,988,989],{"class":189}," +",[161,991,992],{"class":170}," \", \"",[161,994,995],{"class":193},".join(row.index[row]), ",[161,997,908],{"class":475},[161,999,229],{"class":189},[161,1001,566],{"class":244},[161,1003,482],{"class":193},[161,1005,1006],{"class":67,"line":366},[161,1007,482],{"class":193},[161,1009,1010],{"class":67,"line":831},[161,1011,220],{"emptyLinePlaceholder":219},[161,1013,1014,1016,1018,1020,1023,1025,1027,1029,1031,1034,1036,1038,1041,1043,1045],{"class":67,"line":840},[161,1015,400],{"class":244},[161,1017,403],{"class":193},[161,1019,491],{"class":189},[161,1021,1022],{"class":170},"\"kept ",[161,1024,498],{"class":497},[161,1026,406],{"class":244},[161,1028,503],{"class":193},[161,1030,506],{"class":497},[161,1032,1033],{"class":170},", rejected ",[161,1035,498],{"class":497},[161,1037,406],{"class":244},[161,1039,1040],{"class":193},"(rejects)",[161,1042,506],{"class":497},[161,1044,494],{"class":170},[161,1046,482],{"class":193},[161,1048,1049,1051,1054,1056,1058,1060,1062,1064,1066,1068],{"class":67,"line":845},[161,1050,400],{"class":244},[161,1052,1053],{"class":193},"(rejects[[",[161,1055,445],{"class":170},[161,1057,248],{"class":193},[161,1059,450],{"class":170},[161,1061,248],{"class":193},[161,1063,455],{"class":170},[161,1065,248],{"class":193},[161,1067,950],{"class":170},[161,1069,1070],{"class":193},"]])\n",[10,1072,638,1073,1076],{},[14,1074,1075],{},"Reason"," column is what makes the rejects actionable — \"missing Revenue\" and \"missing Order_ID,\nRegion\" are different problems with different owners. Writing that frame to a second sheet of the\noutput workbook costs one line and turns an invisible exclusion into something a recipient can fix.",[152,1078,1080],{"className":180,"code":1079,"language":182,"meta":157,"style":157},"with pd.ExcelWriter(\"orders-clean.xlsx\", engine=\"xlsxwriter\") as writer:\n    clean.to_excel(writer, sheet_name=\"Data\", index=False)\n    if not rejects.empty:\n        rejects.to_excel(writer, sheet_name=\"Excluded rows\", index=False)\n",[14,1081,1082,1111,1136,1147],{"__ignoreMap":157},[161,1083,1084,1087,1090,1093,1095,1098,1100,1103,1106,1108],{"class":67,"line":163},[161,1085,1086],{"class":189},"with",[161,1088,1089],{"class":193}," pd.ExcelWriter(",[161,1091,1092],{"class":170},"\"orders-clean.xlsx\"",[161,1094,248],{"class":193},[161,1096,1097],{"class":475},"engine",[161,1099,229],{"class":189},[161,1101,1102],{"class":170},"\"xlsxwriter\"",[161,1104,1105],{"class":193},") ",[161,1107,197],{"class":189},[161,1109,1110],{"class":193}," writer:\n",[161,1112,1113,1116,1119,1121,1124,1126,1129,1131,1134],{"class":67,"line":203},[161,1114,1115],{"class":193},"    clean.to_excel(writer, ",[161,1117,1118],{"class":475},"sheet_name",[161,1120,229],{"class":189},[161,1122,1123],{"class":170},"\"Data\"",[161,1125,248],{"class":193},[161,1127,1128],{"class":475},"index",[161,1130,229],{"class":189},[161,1132,1133],{"class":244},"False",[161,1135,482],{"class":193},[161,1137,1138,1141,1144],{"class":67,"line":216},[161,1139,1140],{"class":189},"    if",[161,1142,1143],{"class":189}," not",[161,1145,1146],{"class":193}," rejects.empty:\n",[161,1148,1149,1152,1154,1156,1159,1161,1163,1165,1167],{"class":67,"line":223},[161,1150,1151],{"class":193},"        rejects.to_excel(writer, ",[161,1153,1118],{"class":475},[161,1155,229],{"class":189},[161,1157,1158],{"class":170},"\"Excluded rows\"",[161,1160,248],{"class":193},[161,1162,1128],{"class":475},[161,1164,229],{"class":189},[161,1166,1133],{"class":244},[161,1168,482],{"class":193},[147,1170,1172],{"id":1171},"thresholds-and-dropping-by-count","Thresholds, and dropping by count",[10,1174,1175,1176,1179],{},"Sometimes the rule is not \"these columns\" but \"enough columns\". ",[14,1177,1178],{},"thresh"," keeps rows with at least a\ngiven number of non-null values.",[152,1181,1183],{"className":180,"code":1182,"language":182,"meta":157,"style":157},"print(len(orders.dropna(thresh=3)))                       # at least 3 populated columns\nprint(len(orders.dropna(subset=REQUIRED, thresh=2)))      # at least 2 of the required three\n",[14,1184,1185,1208],{"__ignoreMap":157},[161,1186,1187,1189,1191,1193,1196,1198,1200,1202,1205],{"class":67,"line":163},[161,1188,400],{"class":244},[161,1190,403],{"class":193},[161,1192,406],{"class":244},[161,1194,1195],{"class":193},"(orders.dropna(",[161,1197,1178],{"class":475},[161,1199,229],{"class":189},[161,1201,605],{"class":244},[161,1203,1204],{"class":193},")))                       ",[161,1206,1207],{"class":412},"# at least 3 populated columns\n",[161,1209,1210,1212,1214,1216,1218,1220,1222,1224,1226,1228,1230,1232,1235],{"class":67,"line":203},[161,1211,400],{"class":244},[161,1213,403],{"class":193},[161,1215,406],{"class":244},[161,1217,1195],{"class":193},[161,1219,123],{"class":475},[161,1221,229],{"class":189},[161,1223,436],{"class":244},[161,1225,248],{"class":193},[161,1227,1178],{"class":475},[161,1229,229],{"class":189},[161,1231,587],{"class":244},[161,1233,1234],{"class":193},")))      ",[161,1236,1237],{"class":412},"# at least 2 of the required three\n",[10,1239,1240,1241,248,1243,1245],{},"The second form is the more useful one and the less known: combined with ",[14,1242,123],{},[14,1244,1178],{}," counts\nonly within those columns. It suits the case where a record is usable with partial information —\nan order with a region and a value but no identifier might still count towards a regional total, and\nthat is a business decision worth expressing explicitly rather than by omission.",[147,1247,1249],{"id":1248},"row-numbers-the-source-file-recognises","Row numbers the source file recognises",[10,1251,1252],{},"A rejects sheet is only useful if the row numbers match the spreadsheet the user is looking at.\npandas indexes from zero and the header takes row one, so the sheet row is the index plus two.",[152,1254,1256],{"className":180,"code":1255,"language":182,"meta":157,"style":157},"rejects = rejects.reset_index().rename(columns={\"index\": \"Source_row\"})\nrejects[\"Source_row\"] = rejects[\"Source_row\"] + 2\n",[14,1257,1258,1285],{"__ignoreMap":157},[161,1259,1260,1262,1264,1267,1270,1272,1274,1277,1280,1283],{"class":67,"line":163},[161,1261,933],{"class":193},[161,1263,229],{"class":189},[161,1265,1266],{"class":193}," rejects.reset_index().rename(",[161,1268,1269],{"class":475},"columns",[161,1271,229],{"class":189},[161,1273,498],{"class":193},[161,1275,1276],{"class":170},"\"index\"",[161,1278,1279],{"class":193},": ",[161,1281,1282],{"class":170},"\"Source_row\"",[161,1284,369],{"class":193},[161,1286,1287,1289,1291,1293,1295,1298,1300,1302,1305],{"class":67,"line":203},[161,1288,947],{"class":193},[161,1290,1282],{"class":170},[161,1292,953],{"class":193},[161,1294,229],{"class":189},[161,1296,1297],{"class":193}," rejects[",[161,1299,1282],{"class":170},[161,1301,953],{"class":193},[161,1303,1304],{"class":189},"+",[161,1306,1307],{"class":244}," 2\n",[10,1309,1310,1311,23],{},"Doing that before writing the sheet is the difference between a rejects report somebody uses and one\nthey give up on. The same adjustment appears wherever a validation result has to point back at a\nworkbook, including\n",[19,1312,1314],{"href":1313},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fhighlight-invalid-cells-in-excel-with-python\u002F","Highlight Invalid Cells in Excel with Python",[147,1316,1318],{"id":1317},"making-the-drop-a-checked-step","Making the drop a checked step",[10,1320,1321],{},"A row count that changes between the read and the report is worth asserting on rather than\ndiscovering. Wrapping the whole operation in a small function that returns both frames and a summary\nmakes it testable and gives the log something specific to say.",[152,1323,1325],{"className":180,"code":1324,"language":182,"meta":157,"style":157},"from dataclasses import dataclass\n\n@dataclass\nclass DropResult:\n    kept: pd.DataFrame\n    rejected: pd.DataFrame\n    def summary(self) -> str:\n        total = len(self.kept) + len(self.rejected)\n        share = len(self.rejected) \u002F total if total else 0.0\n        return f\"kept {len(self.kept):,} of {total:,} rows ({share:.1%} rejected)\"\n\ndef drop_incomplete(frame: pd.DataFrame, required: list[str],\n                    max_rejected_share: float = 0.05) -> DropResult:\n    missing = frame[required].isna().any(axis=1)\n    result = DropResult(kept=frame[~missing].copy(), rejected=frame[missing].copy())\n    total = len(frame)\n    if total and len(result.rejected) \u002F total > max_rejected_share:\n        raise ValueError(f\"{result.summary()} — above the {max_rejected_share:.0%} threshold\")\n    return result\n",[14,1326,1327,1340,1344,1349,1360,1365,1370,1386,1415,1448,1501,1505,1519,1535,1554,1586,1599,1624,1664],{"__ignoreMap":157},[161,1328,1329,1332,1335,1337],{"class":67,"line":163},[161,1330,1331],{"class":189},"from",[161,1333,1334],{"class":193}," dataclasses ",[161,1336,190],{"class":189},[161,1338,1339],{"class":193}," dataclass\n",[161,1341,1342],{"class":67,"line":203},[161,1343,220],{"emptyLinePlaceholder":219},[161,1345,1346],{"class":67,"line":216},[161,1347,1348],{"class":733},"@dataclass\n",[161,1350,1351,1354,1357],{"class":67,"line":223},[161,1352,1353],{"class":189},"class",[161,1355,1356],{"class":166}," DropResult",[161,1358,1359],{"class":193},":\n",[161,1361,1362],{"class":67,"line":235},[161,1363,1364],{"class":193},"    kept: pd.DataFrame\n",[161,1366,1367],{"class":67,"line":272},[161,1368,1369],{"class":193},"    rejected: pd.DataFrame\n",[161,1371,1372,1375,1378,1381,1384],{"class":67,"line":305},[161,1373,1374],{"class":189},"    def",[161,1376,1377],{"class":733}," summary",[161,1379,1380],{"class":193},"(self) -> ",[161,1382,1383],{"class":244},"str",[161,1385,1359],{"class":193},[161,1387,1388,1391,1393,1396,1398,1401,1404,1406,1408,1410,1412],{"class":67,"line":335},[161,1389,1390],{"class":193},"        total ",[161,1392,229],{"class":189},[161,1394,1395],{"class":244}," len",[161,1397,403],{"class":193},[161,1399,1400],{"class":244},"self",[161,1402,1403],{"class":193},".kept) ",[161,1405,1304],{"class":189},[161,1407,1395],{"class":244},[161,1409,403],{"class":193},[161,1411,1400],{"class":244},[161,1413,1414],{"class":193},".rejected)\n",[161,1416,1417,1420,1422,1424,1426,1428,1431,1434,1437,1440,1442,1445],{"class":67,"line":366},[161,1418,1419],{"class":193},"        share ",[161,1421,229],{"class":189},[161,1423,1395],{"class":244},[161,1425,403],{"class":193},[161,1427,1400],{"class":244},[161,1429,1430],{"class":193},".rejected) ",[161,1432,1433],{"class":189},"\u002F",[161,1435,1436],{"class":193}," total ",[161,1438,1439],{"class":189},"if",[161,1441,1436],{"class":193},[161,1443,1444],{"class":189},"else",[161,1446,1447],{"class":244}," 0.0\n",[161,1449,1450,1453,1456,1458,1460,1462,1464,1466,1469,1472,1474,1476,1478,1481,1483,1485,1488,1490,1493,1496,1498],{"class":67,"line":831},[161,1451,1452],{"class":189},"        return",[161,1454,1455],{"class":189}," f",[161,1457,1022],{"class":170},[161,1459,498],{"class":497},[161,1461,406],{"class":244},[161,1463,403],{"class":193},[161,1465,1400],{"class":244},[161,1467,1468],{"class":193},".kept)",[161,1470,1471],{"class":189},":,",[161,1473,506],{"class":497},[161,1475,509],{"class":170},[161,1477,498],{"class":497},[161,1479,1480],{"class":193},"total",[161,1482,1471],{"class":189},[161,1484,506],{"class":497},[161,1486,1487],{"class":170}," rows (",[161,1489,498],{"class":497},[161,1491,1492],{"class":193},"share",[161,1494,1495],{"class":189},":.1%",[161,1497,506],{"class":497},[161,1499,1500],{"class":170}," rejected)\"\n",[161,1502,1503],{"class":67,"line":840},[161,1504,220],{"emptyLinePlaceholder":219},[161,1506,1507,1509,1512,1515,1517],{"class":67,"line":845},[161,1508,730],{"class":189},[161,1510,1511],{"class":733}," drop_incomplete",[161,1513,1514],{"class":193},"(frame: pd.DataFrame, required: list[",[161,1516,1383],{"class":244},[161,1518,269],{"class":193},[161,1520,1521,1524,1527,1529,1532],{"class":67,"line":855},[161,1522,1523],{"class":193},"                    max_rejected_share: ",[161,1525,1526],{"class":244},"float",[161,1528,439],{"class":189},[161,1530,1531],{"class":244}," 0.05",[161,1533,1534],{"class":193},") -> DropResult:\n",[161,1536,1538,1541,1543,1546,1548,1550,1552],{"class":67,"line":1537},14,[161,1539,1540],{"class":193},"    missing ",[161,1542,229],{"class":189},[161,1544,1545],{"class":193}," frame[required].isna().any(",[161,1547,908],{"class":475},[161,1549,229],{"class":189},[161,1551,566],{"class":244},[161,1553,482],{"class":193},[161,1555,1557,1560,1562,1565,1568,1570,1573,1575,1578,1581,1583],{"class":67,"line":1556},15,[161,1558,1559],{"class":193},"    result ",[161,1561,229],{"class":189},[161,1563,1564],{"class":193}," DropResult(",[161,1566,1567],{"class":475},"kept",[161,1569,229],{"class":189},[161,1571,1572],{"class":193},"frame[",[161,1574,925],{"class":189},[161,1576,1577],{"class":193},"missing].copy(), ",[161,1579,1580],{"class":475},"rejected",[161,1582,229],{"class":189},[161,1584,1585],{"class":193},"frame[missing].copy())\n",[161,1587,1589,1592,1594,1596],{"class":67,"line":1588},16,[161,1590,1591],{"class":193},"    total ",[161,1593,229],{"class":189},[161,1595,1395],{"class":244},[161,1597,1598],{"class":193},"(frame)\n",[161,1600,1602,1604,1606,1609,1611,1614,1616,1618,1621],{"class":67,"line":1601},17,[161,1603,1140],{"class":189},[161,1605,1436],{"class":193},[161,1607,1608],{"class":189},"and",[161,1610,1395],{"class":244},[161,1612,1613],{"class":193},"(result.rejected) ",[161,1615,1433],{"class":189},[161,1617,1436],{"class":193},[161,1619,1620],{"class":189},">",[161,1622,1623],{"class":193}," max_rejected_share:\n",[161,1625,1627,1630,1633,1635,1637,1639,1641,1644,1646,1649,1651,1654,1657,1659,1662],{"class":67,"line":1626},18,[161,1628,1629],{"class":189},"        raise",[161,1631,1632],{"class":244}," ValueError",[161,1634,403],{"class":193},[161,1636,491],{"class":189},[161,1638,494],{"class":170},[161,1640,498],{"class":497},[161,1642,1643],{"class":193},"result.summary()",[161,1645,506],{"class":497},[161,1647,1648],{"class":170}," — above the ",[161,1650,498],{"class":497},[161,1652,1653],{"class":193},"max_rejected_share",[161,1655,1656],{"class":189},":.0%",[161,1658,506],{"class":497},[161,1660,1661],{"class":170}," threshold\"",[161,1663,482],{"class":193},[161,1665,1667,1669],{"class":67,"line":1666},19,[161,1668,834],{"class":189},[161,1670,1671],{"class":193}," result\n",[10,1673,1674,1675,23],{},"The threshold is the useful part. Losing one row in a thousand is normal; losing one in three means\nthe source file changed shape, and continuing produces a report that is confidently wrong. Failing\nloudly at that point is the behaviour that catches an upstream schema change on the day it happens\nrather than at the end of the quarter, and it fits the pre-send checks in\n",[19,1676,1678],{"href":1677},"\u002Fautomating-reporting-workflows\u002Ferror-handling-and-logging-in-excel-automation\u002Fvalidate-an-excel-report-before-sending-it\u002F","Validate an Excel Report Before Sending It",[147,1680,1682],{"id":1681},"duplicates-hide-behind-missing-values","Duplicates hide behind missing values",[10,1684,1685],{},"A related failure worth checking in the same pass: rows that are not missing anything but are\nduplicates of each other, often because a partially-blank row was re-entered rather than corrected.",[152,1687,1689],{"className":180,"code":1688,"language":182,"meta":157,"style":157},"duplicates = clean[clean.duplicated(subset=[\"Order_ID\"], keep=False)]\nif not duplicates.empty:\n    print(f\"{len(duplicates)} row(s) share an Order_ID:\")\n    print(duplicates.sort_values(\"Order_ID\")[[\"Order_ID\", \"Region\", \"Revenue\"]])\n",[14,1690,1691,1722,1731,1756],{"__ignoreMap":157},[161,1692,1693,1696,1698,1701,1703,1705,1707,1709,1712,1715,1717,1719],{"class":67,"line":163},[161,1694,1695],{"class":193},"duplicates ",[161,1697,229],{"class":189},[161,1699,1700],{"class":193}," clean[clean.duplicated(",[161,1702,123],{"class":475},[161,1704,229],{"class":189},[161,1706,769],{"class":193},[161,1708,445],{"class":170},[161,1710,1711],{"class":193},"], ",[161,1713,1714],{"class":475},"keep",[161,1716,229],{"class":189},[161,1718,1133],{"class":244},[161,1720,1721],{"class":193},")]\n",[161,1723,1724,1726,1728],{"class":67,"line":203},[161,1725,1439],{"class":189},[161,1727,1143],{"class":189},[161,1729,1730],{"class":193}," duplicates.empty:\n",[161,1732,1733,1736,1738,1740,1742,1744,1746,1749,1751,1754],{"class":67,"line":216},[161,1734,1735],{"class":244},"    print",[161,1737,403],{"class":193},[161,1739,491],{"class":189},[161,1741,494],{"class":170},[161,1743,498],{"class":497},[161,1745,406],{"class":244},[161,1747,1748],{"class":193},"(duplicates)",[161,1750,506],{"class":497},[161,1752,1753],{"class":170}," row(s) share an Order_ID:\"",[161,1755,482],{"class":193},[161,1757,1758,1760,1763,1765,1768,1770,1772,1774,1776,1778],{"class":67,"line":223},[161,1759,1735],{"class":244},[161,1761,1762],{"class":193},"(duplicates.sort_values(",[161,1764,445],{"class":170},[161,1766,1767],{"class":193},")[[",[161,1769,445],{"class":170},[161,1771,248],{"class":193},[161,1773,450],{"class":170},[161,1775,248],{"class":193},[161,1777,455],{"class":170},[161,1779,1070],{"class":193},[10,1781,1782,1785,1786,1790],{},[14,1783,1784],{},"keep=False"," marks every member of each duplicate group rather than just the repeats, which is what\nyou want when reporting them — the person fixing the file needs to see both rows to decide which is\ncorrect. Handling them properly is covered in\n",[19,1787,1789],{"href":1788},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Ffind-duplicate-rows-in-excel-with-python\u002F","Find Duplicate Rows in Excel with Python",",\nand running both checks together is worth the extra line because they usually have the same cause.",[147,1792,1794],{"id":1793},"common-pitfalls","Common pitfalls",[1796,1797,1798,1814],"table",{},[1799,1800,1801],"thead",{},[1802,1803,1804,1808,1811],"tr",{},[1805,1806,1807],"th",{},"Symptom",[1805,1809,1810],{},"Cause",[1805,1812,1813],{},"Fix",[1815,1816,1817,1834,1848,1859,1877,1888],"tbody",{},[1802,1818,1819,1823,1828],{},[1820,1821,1822],"td",{},"Almost every row is dropped",[1820,1824,1825,1827],{},[14,1826,16],{}," with no subset",[1820,1829,1830,1831],{},"Pass ",[14,1832,1833],{},"subset=REQUIRED",[1802,1835,1836,1839,1845],{},[1820,1837,1838],{},"A blank-looking value is kept",[1820,1840,1841,1842,1844],{},"It is a space or ",[14,1843,651],{},", not NaN",[1820,1846,1847],{},"Normalise placeholders to NaN first",[1802,1849,1850,1853,1856],{},[1820,1851,1852],{},"Totals do not match the source",[1820,1854,1855],{},"Rows were dropped silently",[1820,1857,1858],{},"Keep and report a rejects frame",[1802,1860,1861,1867,1870],{},[1820,1862,1863,1866],{},[14,1864,1865],{},"SettingWithCopyWarning"," after filtering",[1820,1868,1869],{},"The filtered frame is a view",[1820,1871,1872,1873,1876],{},"Add ",[14,1874,1875],{},".copy()"," after the boolean selection",[1802,1878,1879,1882,1885],{},[1820,1880,1881],{},"Row numbers in the report do not match",[1820,1883,1884],{},"Zero-based index reported",[1820,1886,1887],{},"Add two for the header and one-based rows",[1802,1889,1890,1893,1896],{},[1820,1891,1892],{},"Numeric column full of NaN after cleaning",[1820,1894,1895],{},"Numbers stored as text, coerced to NaN",[1820,1897,1898],{},"Check the dtype before assuming data loss",[147,1900,1902],{"id":1901},"deciding-between-dropping-and-filling","Deciding between dropping and filling",[10,1904,1905],{},"Dropping is not always the right answer, and the choice comes down to what the missing value means.\nA missing identifier makes the row unusable and it should be dropped. A missing optional note means\nnothing and should be left alone. A missing measurement is the interesting case: dropping it biases\nthe average, filling it with zero biases it differently, and interpolating asserts something about\nthe data that may not be true.",[10,1907,1908,1909,1913,1914,23],{},"The rule that holds up is to drop only what makes a row unusable, and to make every other decision\nvisible in the output — a count of imputed values, a flag column, a note in the report. The\nalternatives are covered in\n",[19,1910,1912],{"href":1911},"\u002Fadvanced-data-transformation-and-cleaning\u002Fhandling-missing-data-in-excel-reports\u002Ffill-missing-values-in-excel-with-pandas-fillna\u002F","Fill Missing Values in Excel with pandas fillna","\nand\n",[19,1915,1917],{"href":1916},"\u002Fadvanced-data-transformation-and-cleaning\u002Fhandling-missing-data-in-excel-reports\u002Finterpolate-missing-numeric-values-in-excel-data\u002F","Interpolate Missing Numeric Values in Excel Data",[147,1919,1921],{"id":1920},"performance-and-scale","Performance and scale",[25,1923,34,1928,34,1931,34,1934,34,1937,34,1942,34,1951,34,1957,34,1962,34,1967,34,1970,34,1974,34,1978,34,1981,34,1985,34,1988,34,1991,34,1996,34,1999],{"viewBox":1924,"role":28,"ariaLabelledBy":1925,"xmlns":32,"style":33},"0 0 760 232",[1926,1927],"dr-split-t","dr-split-d",[36,1929,1930],{"id":1926},"Split, do not filter",[40,1932,1933],{"id":1927},"Rather than removing unusable rows in place, partition the frame into the rows that pass and the rows that do not, annotate the rejects with a reason, and ship both.",[44,1935],{"x":46,"y":46,"width":47,"height":1936,"fill":49},"232",[61,1938,1941],{"x":120,"y":1939,"style":1940},"32","font-size:13px;font-weight:600;fill:var(--muted,#5b6780);text-anchor:middle","keep what you drop",[44,1943],{"x":1944,"y":1945,"width":1946,"height":1947,"rx":1948,"fill":1949,"stroke":1950,"style":59},"24.0","74","208.0","96","12","#e7ebef","var(--line,#cdd5e6)",[61,1952,1956],{"x":1953,"y":1954,"style":1955},"128.0","114","font-size:14px;font-weight:700;fill:var(--muted,#5b6780);text-anchor:middle","one frame in",[61,1958,1961],{"x":1953,"y":1959,"style":1960},"136","font-size:11.5px;font-weight:400;fill:var(--muted,#5b6780);text-anchor:middle","as read from the sheet",[67,1963],{"x1":1964,"y1":1965,"x2":1966,"y2":1965,"stroke":117,"style":59},"237.0","122.0","269.0",[130,1968],{"points":1969,"fill":133},"269.0,122.0 260.0,117.0 260.0,127.0",[44,1971],{"x":1972,"y":1945,"width":1946,"height":1947,"rx":1948,"fill":1973,"stroke":117,"style":59},"276.0","#f0f4ff",[61,1975,1977],{"x":120,"y":1954,"style":1976},"font-size:14px;font-weight:700;fill:var(--brand-strong,#4338ca);text-anchor:middle","split on the mask",[61,1979,1980],{"x":120,"y":1959,"style":1960},"kept and rejected",[67,1982],{"x1":1983,"y1":1965,"x2":1984,"y2":1965,"stroke":117,"style":59},"489.0","521.0",[130,1986],{"points":1987,"fill":133},"521.0,122.0 512.0,117.0 512.0,127.0",[44,1989],{"x":1990,"y":1945,"width":1946,"height":1947,"rx":1948,"fill":89,"stroke":90,"style":59},"528.0",[61,1992,1995],{"x":1993,"y":1954,"style":1994},"632.0","font-size:14px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","two sheets out",[61,1997,1998],{"x":1993,"y":1959,"style":1960},"data, and excluded rows",[61,2000,2002],{"x":120,"y":2001,"style":144},"210","a report that silently loses rows is the failure this prevents",[10,2004,2005,2007],{},[14,2006,644],{}," is a single pass and costs almost nothing. What does cost is the placeholder normalisation,\nbecause it touches every string cell — on a wide export with thirty text columns that is a real\namount of work, and it is worth limiting to the columns that matter.",[152,2009,2011],{"className":180,"code":2010,"language":182,"meta":157,"style":157},"# Normalise only the columns the pipeline depends on\nfor column in REQUIRED:\n    if orders[column].dtype == \"object\":\n        stripped = orders[column].astype(\"string\").str.strip()\n        orders[column] = stripped.mask(stripped == \"\")\n",[14,2012,2013,2018,2031,2046,2059],{"__ignoreMap":157},[161,2014,2015],{"class":67,"line":163},[161,2016,2017],{"class":412},"# Normalise only the columns the pipeline depends on\n",[161,2019,2020,2022,2024,2026,2029],{"class":67,"line":203},[161,2021,813],{"class":189},[161,2023,755],{"class":193},[161,2025,758],{"class":189},[161,2027,2028],{"class":244}," REQUIRED",[161,2030,1359],{"class":193},[161,2032,2033,2035,2038,2041,2044],{"class":67,"line":216},[161,2034,1140],{"class":189},[161,2036,2037],{"class":193}," orders[column].dtype ",[161,2039,2040],{"class":189},"==",[161,2042,2043],{"class":170}," \"object\"",[161,2045,1359],{"class":193},[161,2047,2048,2050,2052,2055,2057],{"class":67,"line":223},[161,2049,785],{"class":193},[161,2051,229],{"class":189},[161,2053,2054],{"class":193}," orders[column].astype(",[161,2056,777],{"class":170},[161,2058,795],{"class":193},[161,2060,2061,2064,2066,2069,2071,2074],{"class":67,"line":235},[161,2062,2063],{"class":193},"        orders[column] ",[161,2065,229],{"class":189},[161,2067,2068],{"class":193}," stripped.mask(stripped ",[161,2070,2040],{"class":189},[161,2072,2073],{"class":170}," \"\"",[161,2075,482],{"class":193},[10,2077,2078],{},"The other scale consideration is where in the pipeline the drop happens. Doing it immediately after\nthe read means every later step works on fewer rows, and it means the rejects are captured before any\ntransformation has had a chance to obscure why a row was unusable.",[147,2080,2082],{"id":2081},"conclusion","Conclusion",[10,2084,2085,2086,2088],{},"Never call ",[14,2087,16],{}," bare. Name the required columns in a constant, normalise blank-looking\nplaceholders to NaN first so the check sees what a person sees, and split the frame into kept and\nrejected rather than filtering in place. Write the rejects to a second sheet with a reason and a\nsource row number, and reserve dropping for rows that are genuinely unusable — everything else is a\nfill decision that should be visible in the output.",[147,2090,2092],{"id":2091},"frequently-asked-questions","Frequently asked questions",[10,2094,2095,2099],{},[2096,2097,2098],"strong",{},"What is the difference between dropna(how='any') and a subset?","\nhow='any' drops a row if any column is missing, which on a wide export removes almost everything because one optional note column is blank. A subset limits the check to the columns that genuinely must be present, which is nearly always what you want.",[10,2101,2102,2105],{},[2096,2103,2104],{},"Do empty Excel cells always become NaN?","\nNo. A truly empty cell does, but a cell containing a space, an empty string, or a text placeholder like 'N\u002FA' or '-' does not — it becomes a value. Normalising those to NaN before the check is what makes dropna behave the way people expect.",[10,2107,2108,2111],{},[2096,2109,2110],{},"Should I drop the rows or report them?","\nBoth. Drop them from the working frame so the calculations are correct, and keep them in a rejects frame so somebody can see what was excluded. A report that quietly loses rows is the failure mode this whole practice exists to prevent.",[10,2113,2114,2117],{},[2096,2115,2116],{},"How do I drop a column that is mostly empty?","\ndropna(axis=1, thresh=n) keeps only columns with at least n non-null values. It is useful for exploring an unfamiliar export and dangerous in a pipeline, where a column becoming empty is a signal rather than a nuisance.",[147,2119,2121],{"id":2120},"related","Related",[2123,2124,2125,2132,2139,2144,2149],"ul",{},[2126,2127,2128,2129,2131],"li",{},"Up one level: ",[19,2130,22],{"href":21}," — the wider set of strategies for gaps.",[2126,2133,2134,2138],{},[19,2135,2137],{"href":2136},"\u002Fadvanced-data-transformation-and-cleaning\u002Fhandling-missing-data-in-excel-reports\u002Ffind-and-report-missing-values-in-an-excel-file\u002F","Find and Report Missing Values in an Excel File"," — measuring the gaps before deciding what to do.",[2126,2140,2141,2143],{},[19,2142,1912],{"href":1911}," — the alternative to dropping, and when it is honest.",[2126,2145,2146,2148],{},[19,2147,530],{"href":529}," — asserting the required-column contract at the read.",[2126,2150,2151,2155],{},[19,2152,2154],{"href":2153},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fremove-blank-rows-from-excel-with-pandas\u002F","Remove Blank Rows from Excel with Pandas"," — the simpler case of rows that are entirely empty.",[2157,2158,2159],"style",{},"html pre.shiki code .sMTad, html code.shiki .sMTad{--shiki-default:#6F42C1;--shiki-dark:#FFB757}html pre.shiki code .srMev, html code.shiki .srMev{--shiki-default:#032F62;--shiki-dark:#ADDCFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .s-kum, html code.shiki .s-kum{--shiki-default:#D73A49;--shiki-dark:#FF9492}html pre.shiki code .skGVy, html code.shiki .skGVy{--shiki-default:#24292E;--shiki-dark:#F0F3F6}html pre.shiki code .sP0c6, html code.shiki .sP0c6{--shiki-default:#005CC5;--shiki-dark:#91CBFF}html pre.shiki code .s-wDw, html code.shiki .s-wDw{--shiki-default:#6A737D;--shiki-dark:#BDC4CC}html pre.shiki code .sa561, html code.shiki .sa561{--shiki-default:#E36209;--shiki-dark:#FFB757}html pre.shiki code .sSjpA, html code.shiki .sSjpA{--shiki-default:#005CC5;--shiki-dark:#FF9492}html pre.shiki code .s_Opv, html code.shiki .s_Opv{--shiki-default:#6F42C1;--shiki-dark:#DBB7FF}",{"title":157,"searchDepth":203,"depth":203,"links":2161},[2162,2163,2164,2165,2166,2167,2168,2169,2170,2171,2172,2173,2174,2175,2176],{"id":149,"depth":203,"text":150},{"id":389,"depth":203,"text":390},{"id":425,"depth":203,"text":426},{"id":534,"depth":203,"text":535},{"id":881,"depth":203,"text":882},{"id":1171,"depth":203,"text":1172},{"id":1248,"depth":203,"text":1249},{"id":1317,"depth":203,"text":1318},{"id":1681,"depth":203,"text":1682},{"id":1793,"depth":203,"text":1794},{"id":1901,"depth":203,"text":1902},{"id":1920,"depth":203,"text":1921},{"id":2081,"depth":203,"text":2082},{"id":2091,"depth":203,"text":2092},{"id":2120,"depth":203,"text":2121},"2026-09-04","dropna() with no arguments removes almost everything. Name the required columns, normalise blank-looking placeholders first, and keep the rejected rows with a reason and a source row number.","md",[2181,2183,2185,2187],{"q":2098,"a":2182},"how='any' drops a row if any column is missing, which on a wide export removes almost everything because one optional note column is blank. A subset limits the check to the columns that genuinely must be present, which is nearly always what you want.",{"q":2104,"a":2184},"No. A truly empty cell does, but a cell containing a space, an empty string, or a text placeholder like 'N\u002FA' or '-' does not — it becomes a value. Normalising those to NaN before the check is what makes dropna behave the way people expect.",{"q":2110,"a":2186},"Both. Drop them from the working frame so the calculations are correct, and keep them in a rejects frame so somebody can see what was excluded. A report that quietly loses rows is the failure mode this whole practice exists to prevent.",{"q":2116,"a":2188},"dropna(axis=1, thresh=n) keeps only columns with at least n non-null values. It is useful for exploring an unfamiliar export and dangerous in a pipeline, where a column becoming empty is a signal rather than a nuisance.",{"breadcrumb":2190},[2191,2193,2196],{"name":2192,"item":1433},"Home",{"name":2194,"item":2195},"Advanced Data Transformation and Cleaning","\u002Fadvanced-data-transformation-and-cleaning\u002F",{"name":22,"item":21},"\u002Fadvanced-data-transformation-and-cleaning\u002Fhandling-missing-data-in-excel-reports\u002Fdrop-rows-with-missing-required-fields-in-pandas",{"title":5,"description":2199},"Use dropna correctly on Excel data — subset for required columns, thresh for partial records, placeholder normalisation so blanks are really NaN, and a rejects sheet with reasons.","drop-rows-with-missing-required-fields-in-pandas","advanced-data-transformation-and-cleaning\u002Fhandling-missing-data-in-excel-reports\u002Fdrop-rows-with-missing-required-fields-in-pandas\u002Findex","how-to","UgZ4_gUlNMUm4RYGJ2H14f1al-So1C9cw4jFMBz9LdU",[2205,2209],{"title":2206,"path":2207,"stem":2208,"children":-1},"Handling Missing Data in Excel Reports with Pandas","\u002Fadvanced-data-transformation-and-cleaning\u002Fhandling-missing-data-in-excel-reports","advanced-data-transformation-and-cleaning\u002Fhandling-missing-data-in-excel-reports\u002Findex",{"title":2210,"path":2211,"stem":2212,"children":-1},"Fill Missing Values in Excel with Pandas fillna","\u002Fadvanced-data-transformation-and-cleaning\u002Fhandling-missing-data-in-excel-reports\u002Ffill-missing-values-in-excel-with-pandas-fillna","advanced-data-transformation-and-cleaning\u002Fhandling-missing-data-in-excel-reports\u002Ffill-missing-values-in-excel-with-pandas-fillna\u002Findex",1788710162364]