[{"data":1,"prerenderedAt":2531},["ShallowReactive",2],{"doc:\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fcheck-excel-data-types-with-pandas":3,"surround:\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fcheck-excel-data-types-with-pandas":2522},{"id":4,"title":5,"body":6,"dateModified":2488,"datePublished":2488,"description":2489,"extension":2490,"faq":2491,"meta":2505,"navigation":217,"path":2515,"seo":2516,"slug":2518,"stem":2519,"type":2520,"__hash__":2521},"docs\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fcheck-excel-data-types-with-pandas\u002Findex.md","Check Excel Data Types with pandas",{"type":7,"value":8,"toc":2473},"minimark",[9,39,152,157,185,189,431,434,438,444,724,742,746,749,1352,1371,1461,1465,1468,1645,1669,1673,1676,1969,1988,1992,2114,2125,2129,2132,2192,2201,2205,2332,2336,2354,2358,2361,2365,2381,2394,2408,2421,2425,2428,2437,2440,2469],[10,11,12,16,17,20,21,24,25,28,29,32,33,38],"p",{},[13,14,15],"code",{},"df.dtypes"," after a ",[13,18,19],{},"read_excel"," is the shortest description of what a spreadsheet actually contained. A column of ",[13,22,23],{},"object"," where you expected ",[13,26,27],{},"float64"," means at least one cell is not a number, and a column silently converted to ",[13,30,31],{},"int64"," may have thrown away the leading zeros of an account code. This guide, part of ",[34,35,37],"a",{"href":36},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002F","Validating Excel Data with Python",", turns that signal into a set of checks you can run on every import.",[40,41,49,50,49,54,49,58,49,65,49,72,49,79,49,84,49,88,49,93,49,97,49,101,49,104,49,108,49,116,49,121,49,125,49,128,49,130,49,132,49,134,49,138,49,142,49,144,49,146,49,148],"svg",{"viewBox":42,"role":43,"ariaLabelledBy":44,"xmlns":47,"style":48},"0 0 740 244","img",[45,46],"dt-infer-t","dt-infer-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;max-width:740px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif","\n  ",[51,52,53],"title",{"id":45},"How one bad cell changes a whole column's dtype",[55,56,57],"desc",{"id":46},"A quantity column containing 4, 2 and 7 is read as int64. The same column with one cell containing the text n\u002Fa is read as object, so every value in it becomes a Python object and arithmetic silently stops working.",[59,60],"rect",{"x":61,"y":61,"width":62,"height":63,"fill":64},"0","740","244","#ffffff",[66,67,71],"text",{"x":68,"y":69,"style":70},"185","30","font-size:12.5px;font-weight:700;fill:var(--muted,#5b6780);text-anchor:middle","all cells numeric",[59,73],{"x":74,"y":75,"width":76,"height":77,"fill":78},"60","44","250","34","#5b5cf0",[66,80,83],{"x":68,"y":81,"style":82},"66","font-size:12px;font-weight:700;fill:#ffffff;text-anchor:middle","Quantity",[59,85],{"x":74,"y":86,"width":76,"height":69,"fill":64,"stroke":87},"78","var(--line,#cdd5e6)",[66,89,92],{"x":68,"y":90,"style":91},"98","font-size:12px;fill:var(--text,#172033);text-anchor:middle","4",[59,94],{"x":74,"y":95,"width":76,"height":69,"fill":96,"stroke":87},"108","#f0f4ff",[66,98,100],{"x":68,"y":99,"style":91},"128","2",[59,102],{"x":74,"y":103,"width":76,"height":69,"fill":64,"stroke":87},"138",[66,105,107],{"x":68,"y":106,"style":91},"158","7",[59,109],{"x":74,"y":110,"width":76,"height":111,"rx":112,"fill":113,"stroke":114,"style":115},"182","38","9","#d9f4f1","var(--teal,#0f9488)","stroke-width:2px",[66,117,120],{"x":68,"y":118,"style":119},"206","font-size:12.5px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","dtype: int64 — arithmetic works",[66,122,124],{"x":123,"y":69,"style":70},"555","one cell is text",[59,126],{"x":127,"y":75,"width":76,"height":77,"fill":78},"430",[66,129,83],{"x":123,"y":81,"style":82},[59,131],{"x":127,"y":86,"width":76,"height":69,"fill":64,"stroke":87},[66,133,92],{"x":123,"y":90,"style":91},[59,135],{"x":127,"y":95,"width":76,"height":69,"fill":136,"stroke":137},"#fce9e9","var(--danger,#dc2626)",[66,139,141],{"x":123,"y":99,"style":140},"font-size:12px;font-weight:700;fill:var(--danger,#dc2626);text-anchor:middle","n\u002Fa",[59,143],{"x":127,"y":103,"width":76,"height":69,"fill":64,"stroke":87},[66,145,107],{"x":123,"y":106,"style":91},[59,147],{"x":127,"y":110,"width":76,"height":111,"rx":112,"fill":136,"stroke":137,"style":115},[66,149,151],{"x":123,"y":118,"style":150},"font-size:12.5px;font-weight:700;fill:var(--danger,#dc2626);text-anchor:middle","dtype: object — sums silently fail",[153,154,156],"h2",{"id":155},"prerequisites","Prerequisites",[158,159,164],"pre",{"className":160,"code":161,"language":162,"meta":163,"style":163},"language-bash shiki shiki-themes github-light github-dark-high-contrast","pip install pandas openpyxl\n","bash","",[13,165,166],{"__ignoreMap":163},[167,168,171,175,179,182],"span",{"class":169,"line":170},"line",1,[167,172,174],{"class":173},"sMTad","pip",[167,176,178],{"class":177},"srMev"," install",[167,180,181],{"class":177}," pandas",[167,183,184],{"class":177}," openpyxl\n",[153,186,188],{"id":187},"build-a-workbook-with-type-problems","Build a workbook with type problems",[158,190,194],{"className":191,"code":192,"language":193,"meta":163,"style":163},"language-python shiki shiki-themes github-light github-dark-high-contrast","import pandas as pd\n\npd.DataFrame([\n    {\"Code\": \"00412\", \"Quantity\": 4, \"Unit_Price\": 19.99, \"Order_Date\": \"2026-01-14\"},\n    {\"Code\": \"00733\", \"Quantity\": \"2\", \"Unit_Price\": \"49,50\", \"Order_Date\": \"15\u002F01\u002F2026\"},\n    {\"Code\": \"01120\", \"Quantity\": \"n\u002Fa\", \"Unit_Price\": 12.25, \"Order_Date\": 45678},\n    {\"Code\": \"02001\", \"Quantity\": 5, \"Unit_Price\": None, \"Order_Date\": \"2026-02-30\"},\n]).to_excel(\"types.xlsx\", index=False, sheet_name=\"Orders\")\n","python",[13,195,196,212,219,225,274,315,356,397],{"__ignoreMap":163},[167,197,198,202,206,209],{"class":169,"line":170},[167,199,201],{"class":200},"s-kum","import",[167,203,205],{"class":204},"skGVy"," pandas ",[167,207,208],{"class":200},"as",[167,210,211],{"class":204}," pd\n",[167,213,215],{"class":169,"line":214},2,[167,216,218],{"emptyLinePlaceholder":217},true,"\n",[167,220,222],{"class":169,"line":221},3,[167,223,224],{"class":204},"pd.DataFrame([\n",[167,226,228,231,234,237,240,243,246,248,251,253,256,258,261,263,266,268,271],{"class":169,"line":227},4,[167,229,230],{"class":204},"    {",[167,232,233],{"class":177},"\"Code\"",[167,235,236],{"class":204},": ",[167,238,239],{"class":177},"\"00412\"",[167,241,242],{"class":204},", ",[167,244,245],{"class":177},"\"Quantity\"",[167,247,236],{"class":204},[167,249,92],{"class":250},"sP0c6",[167,252,242],{"class":204},[167,254,255],{"class":177},"\"Unit_Price\"",[167,257,236],{"class":204},[167,259,260],{"class":250},"19.99",[167,262,242],{"class":204},[167,264,265],{"class":177},"\"Order_Date\"",[167,267,236],{"class":204},[167,269,270],{"class":177},"\"2026-01-14\"",[167,272,273],{"class":204},"},\n",[167,275,277,279,281,283,286,288,290,292,295,297,299,301,304,306,308,310,313],{"class":169,"line":276},5,[167,278,230],{"class":204},[167,280,233],{"class":177},[167,282,236],{"class":204},[167,284,285],{"class":177},"\"00733\"",[167,287,242],{"class":204},[167,289,245],{"class":177},[167,291,236],{"class":204},[167,293,294],{"class":177},"\"2\"",[167,296,242],{"class":204},[167,298,255],{"class":177},[167,300,236],{"class":204},[167,302,303],{"class":177},"\"49,50\"",[167,305,242],{"class":204},[167,307,265],{"class":177},[167,309,236],{"class":204},[167,311,312],{"class":177},"\"15\u002F01\u002F2026\"",[167,314,273],{"class":204},[167,316,318,320,322,324,327,329,331,333,336,338,340,342,345,347,349,351,354],{"class":169,"line":317},6,[167,319,230],{"class":204},[167,321,233],{"class":177},[167,323,236],{"class":204},[167,325,326],{"class":177},"\"01120\"",[167,328,242],{"class":204},[167,330,245],{"class":177},[167,332,236],{"class":204},[167,334,335],{"class":177},"\"n\u002Fa\"",[167,337,242],{"class":204},[167,339,255],{"class":177},[167,341,236],{"class":204},[167,343,344],{"class":250},"12.25",[167,346,242],{"class":204},[167,348,265],{"class":177},[167,350,236],{"class":204},[167,352,353],{"class":250},"45678",[167,355,273],{"class":204},[167,357,359,361,363,365,368,370,372,374,377,379,381,383,386,388,390,392,395],{"class":169,"line":358},7,[167,360,230],{"class":204},[167,362,233],{"class":177},[167,364,236],{"class":204},[167,366,367],{"class":177},"\"02001\"",[167,369,242],{"class":204},[167,371,245],{"class":177},[167,373,236],{"class":204},[167,375,376],{"class":250},"5",[167,378,242],{"class":204},[167,380,255],{"class":177},[167,382,236],{"class":204},[167,384,385],{"class":250},"None",[167,387,242],{"class":204},[167,389,265],{"class":177},[167,391,236],{"class":204},[167,393,394],{"class":177},"\"2026-02-30\"",[167,396,273],{"class":204},[167,398,400,403,406,408,412,415,418,420,423,425,428],{"class":169,"line":399},8,[167,401,402],{"class":204},"]).to_excel(",[167,404,405],{"class":177},"\"types.xlsx\"",[167,407,242],{"class":204},[167,409,411],{"class":410},"sa561","index",[167,413,414],{"class":200},"=",[167,416,417],{"class":250},"False",[167,419,242],{"class":204},[167,421,422],{"class":410},"sheet_name",[167,424,414],{"class":200},[167,426,427],{"class":177},"\"Orders\"",[167,429,430],{"class":204},")\n",[10,432,433],{},"Four rows carrying the four classic type problems: a code with leading zeros, a number typed as text, a European decimal comma, and dates in three incompatible forms including an Excel serial number and an impossible date.",[153,435,437],{"id":436},"look-before-you-convert","Look before you convert",[10,439,440,441,443],{},"Read once with everything as ",[13,442,23],{},", then describe what arrived. Nothing is guessed, so nothing is hidden:",[158,445,447],{"className":191,"code":446,"language":193,"meta":163,"style":163},"import pandas as pd\n\ndf = pd.read_excel(\"types.xlsx\", sheet_name=\"Orders\", dtype=object)\n\ndef profile(frame):\n    rows = []\n    for column in frame.columns:\n        series = frame[column]\n        present = series.dropna()\n        kinds = sorted({type(v).__name__ for v in present})\n        rows.append({\n            \"column\": column,\n            \"dtype\": str(series.dtype),\n            \"non_null\": int(series.notna().sum()),\n            \"python_types\": \", \".join(kinds) or \"—\",\n            \"sample\": present.iloc[0] if len(present) else None,\n        })\n    return pd.DataFrame(rows)\n\nprint(profile(df).to_string(index=False))\n",[13,448,449,459,463,494,498,510,520,534,544,555,589,595,604,618,632,655,686,692,701,706],{"__ignoreMap":163},[167,450,451,453,455,457],{"class":169,"line":170},[167,452,201],{"class":200},[167,454,205],{"class":204},[167,456,208],{"class":200},[167,458,211],{"class":204},[167,460,461],{"class":169,"line":214},[167,462,218],{"emptyLinePlaceholder":217},[167,464,465,468,470,473,475,477,479,481,483,485,488,490,492],{"class":169,"line":221},[167,466,467],{"class":204},"df ",[167,469,414],{"class":200},[167,471,472],{"class":204}," pd.read_excel(",[167,474,405],{"class":177},[167,476,242],{"class":204},[167,478,422],{"class":410},[167,480,414],{"class":200},[167,482,427],{"class":177},[167,484,242],{"class":204},[167,486,487],{"class":410},"dtype",[167,489,414],{"class":200},[167,491,23],{"class":250},[167,493,430],{"class":204},[167,495,496],{"class":169,"line":227},[167,497,218],{"emptyLinePlaceholder":217},[167,499,500,503,507],{"class":169,"line":276},[167,501,502],{"class":200},"def",[167,504,506],{"class":505},"s_Opv"," profile",[167,508,509],{"class":204},"(frame):\n",[167,511,512,515,517],{"class":169,"line":317},[167,513,514],{"class":204},"    rows ",[167,516,414],{"class":200},[167,518,519],{"class":204}," []\n",[167,521,522,525,528,531],{"class":169,"line":358},[167,523,524],{"class":200},"    for",[167,526,527],{"class":204}," column ",[167,529,530],{"class":200},"in",[167,532,533],{"class":204}," frame.columns:\n",[167,535,536,539,541],{"class":169,"line":399},[167,537,538],{"class":204},"        series ",[167,540,414],{"class":200},[167,542,543],{"class":204}," frame[column]\n",[167,545,547,550,552],{"class":169,"line":546},9,[167,548,549],{"class":204},"        present ",[167,551,414],{"class":200},[167,553,554],{"class":204}," series.dropna()\n",[167,556,558,561,563,566,569,572,575,578,581,584,586],{"class":169,"line":557},10,[167,559,560],{"class":204},"        kinds ",[167,562,414],{"class":200},[167,564,565],{"class":250}," sorted",[167,567,568],{"class":204},"({",[167,570,571],{"class":250},"type",[167,573,574],{"class":204},"(v).",[167,576,577],{"class":250},"__name__",[167,579,580],{"class":200}," for",[167,582,583],{"class":204}," v ",[167,585,530],{"class":200},[167,587,588],{"class":204}," present})\n",[167,590,592],{"class":169,"line":591},11,[167,593,594],{"class":204},"        rows.append({\n",[167,596,598,601],{"class":169,"line":597},12,[167,599,600],{"class":177},"            \"column\"",[167,602,603],{"class":204},": column,\n",[167,605,607,610,612,615],{"class":169,"line":606},13,[167,608,609],{"class":177},"            \"dtype\"",[167,611,236],{"class":204},[167,613,614],{"class":250},"str",[167,616,617],{"class":204},"(series.dtype),\n",[167,619,621,624,626,629],{"class":169,"line":620},14,[167,622,623],{"class":177},"            \"non_null\"",[167,625,236],{"class":204},[167,627,628],{"class":250},"int",[167,630,631],{"class":204},"(series.notna().sum()),\n",[167,633,635,638,640,643,646,649,652],{"class":169,"line":634},15,[167,636,637],{"class":177},"            \"python_types\"",[167,639,236],{"class":204},[167,641,642],{"class":177},"\", \"",[167,644,645],{"class":204},".join(kinds) ",[167,647,648],{"class":200},"or",[167,650,651],{"class":177}," \"—\"",[167,653,654],{"class":204},",\n",[167,656,658,661,664,666,669,672,675,678,681,684],{"class":169,"line":657},16,[167,659,660],{"class":177},"            \"sample\"",[167,662,663],{"class":204},": present.iloc[",[167,665,61],{"class":250},[167,667,668],{"class":204},"] ",[167,670,671],{"class":200},"if",[167,673,674],{"class":250}," len",[167,676,677],{"class":204},"(present) ",[167,679,680],{"class":200},"else",[167,682,683],{"class":250}," None",[167,685,654],{"class":204},[167,687,689],{"class":169,"line":688},17,[167,690,691],{"class":204},"        })\n",[167,693,695,698],{"class":169,"line":694},18,[167,696,697],{"class":200},"    return",[167,699,700],{"class":204}," pd.DataFrame(rows)\n",[167,702,704],{"class":169,"line":703},19,[167,705,218],{"emptyLinePlaceholder":217},[167,707,709,712,715,717,719,721],{"class":169,"line":708},20,[167,710,711],{"class":250},"print",[167,713,714],{"class":204},"(profile(df).to_string(",[167,716,411],{"class":410},[167,718,414],{"class":200},[167,720,417],{"class":250},[167,722,723],{"class":204},"))\n",[10,725,726,727,730,731,734,735,738,739,741],{},"The ",[13,728,729],{},"python_types"," column is the useful one. A column reporting ",[13,732,733],{},"float, str"," is a mixed column, and it is far better to learn that here than from a ",[13,736,737],{},"TypeError"," three transformations later. Columns whose values are all ",[13,740,614],{}," may still be numbers — they are simply numbers that were typed as text in Excel.",[153,743,745],{"id":744},"coerce-and-keep-the-failures","Coerce, and keep the failures",[10,747,748],{},"Conversion and validation are the same operation when you record what did not convert:",[158,750,752],{"className":191,"code":751,"language":193,"meta":163,"style":163},"import pandas as pd\n\ndef to_number(series, decimal_comma=False):\n    text = series.astype(str).str.strip()\n    text = text.str.replace(r\"[£$€\\s]\", \"\", regex=True)\n    if decimal_comma:\n        text = text.str.replace(\".\", \"\", regex=False).str.replace(\",\", \".\", regex=False)\n    else:\n        text = text.str.replace(\",\", \"\", regex=False)\n    return pd.to_numeric(text.replace({\"\": None, \"nan\": None, \"n\u002Fa\": None, \"-\": None}),\n                         errors=\"coerce\")\n\ndef coerce_with_report(frame, numeric=(), dates=()):\n    out, failures = frame.copy(), []\n    for column in numeric:\n        converted = to_number(frame[column])\n        broke = converted.isna() & frame[column].notna() & (frame[column].astype(str).str.strip() != \"\")\n        failures += [{\"row\": int(i) + 2, \"column\": column, \"value\": frame.loc[i, column],\n                      \"problem\": \"not a number\"} for i in frame.index[broke]]\n        out[column] = converted\n    for column in dates:\n        converted = pd.to_datetime(frame[column], errors=\"coerce\", format=\"mixed\", dayfirst=True)\n        broke = converted.isna() & frame[column].notna()\n        failures += [{\"row\": int(i) + 2, \"column\": column, \"value\": frame.loc[i, column],\n                      \"problem\": \"not a date\"} for i in frame.index[broke]]\n        out[column] = converted\n    return out, failures\n\ntyped, failures = coerce_with_report(df, numeric=[\"Quantity\", \"Unit_Price\"], dates=[\"Order_Date\"])\nprint(typed.dtypes)\nfor failure in failures:\n    print(failure)\n",[13,753,754,764,768,785,800,844,852,896,904,928,970,982,986,1006,1016,1027,1037,1071,1112,1136,1146,1158,1196,1210,1241,1261,1270,1278,1283,1322,1330,1343],{"__ignoreMap":163},[167,755,756,758,760,762],{"class":169,"line":170},[167,757,201],{"class":200},[167,759,205],{"class":204},[167,761,208],{"class":200},[167,763,211],{"class":204},[167,765,766],{"class":169,"line":214},[167,767,218],{"emptyLinePlaceholder":217},[167,769,770,772,775,778,780,782],{"class":169,"line":221},[167,771,502],{"class":200},[167,773,774],{"class":505}," to_number",[167,776,777],{"class":204},"(series, decimal_comma",[167,779,414],{"class":200},[167,781,417],{"class":250},[167,783,784],{"class":204},"):\n",[167,786,787,790,792,795,797],{"class":169,"line":227},[167,788,789],{"class":204},"    text ",[167,791,414],{"class":200},[167,793,794],{"class":204}," series.astype(",[167,796,614],{"class":250},[167,798,799],{"class":204},").str.strip()\n",[167,801,802,804,806,809,812,815,818,822,825,827,829,832,834,837,839,842],{"class":169,"line":276},[167,803,789],{"class":204},[167,805,414],{"class":200},[167,807,808],{"class":204}," text.str.replace(",[167,810,811],{"class":200},"r",[167,813,814],{"class":177},"\"",[167,816,817],{"class":250},"[",[167,819,821],{"class":820},"sSjpA","£$€",[167,823,824],{"class":250},"\\s]",[167,826,814],{"class":177},[167,828,242],{"class":204},[167,830,831],{"class":177},"\"\"",[167,833,242],{"class":204},[167,835,836],{"class":410},"regex",[167,838,414],{"class":200},[167,840,841],{"class":250},"True",[167,843,430],{"class":204},[167,845,846,849],{"class":169,"line":317},[167,847,848],{"class":200},"    if",[167,850,851],{"class":204}," decimal_comma:\n",[167,853,854,857,859,861,864,866,868,870,872,874,876,879,882,884,886,888,890,892,894],{"class":169,"line":358},[167,855,856],{"class":204},"        text ",[167,858,414],{"class":200},[167,860,808],{"class":204},[167,862,863],{"class":177},"\".\"",[167,865,242],{"class":204},[167,867,831],{"class":177},[167,869,242],{"class":204},[167,871,836],{"class":410},[167,873,414],{"class":200},[167,875,417],{"class":250},[167,877,878],{"class":204},").str.replace(",[167,880,881],{"class":177},"\",\"",[167,883,242],{"class":204},[167,885,863],{"class":177},[167,887,242],{"class":204},[167,889,836],{"class":410},[167,891,414],{"class":200},[167,893,417],{"class":250},[167,895,430],{"class":204},[167,897,898,901],{"class":169,"line":399},[167,899,900],{"class":200},"    else",[167,902,903],{"class":204},":\n",[167,905,906,908,910,912,914,916,918,920,922,924,926],{"class":169,"line":546},[167,907,856],{"class":204},[167,909,414],{"class":200},[167,911,808],{"class":204},[167,913,881],{"class":177},[167,915,242],{"class":204},[167,917,831],{"class":177},[167,919,242],{"class":204},[167,921,836],{"class":410},[167,923,414],{"class":200},[167,925,417],{"class":250},[167,927,430],{"class":204},[167,929,930,932,935,937,939,941,943,946,948,950,952,954,956,958,960,963,965,967],{"class":169,"line":557},[167,931,697],{"class":200},[167,933,934],{"class":204}," pd.to_numeric(text.replace({",[167,936,831],{"class":177},[167,938,236],{"class":204},[167,940,385],{"class":250},[167,942,242],{"class":204},[167,944,945],{"class":177},"\"nan\"",[167,947,236],{"class":204},[167,949,385],{"class":250},[167,951,242],{"class":204},[167,953,335],{"class":177},[167,955,236],{"class":204},[167,957,385],{"class":250},[167,959,242],{"class":204},[167,961,962],{"class":177},"\"-\"",[167,964,236],{"class":204},[167,966,385],{"class":250},[167,968,969],{"class":204},"}),\n",[167,971,972,975,977,980],{"class":169,"line":591},[167,973,974],{"class":410},"                         errors",[167,976,414],{"class":200},[167,978,979],{"class":177},"\"coerce\"",[167,981,430],{"class":204},[167,983,984],{"class":169,"line":597},[167,985,218],{"emptyLinePlaceholder":217},[167,987,988,990,993,996,998,1001,1003],{"class":169,"line":606},[167,989,502],{"class":200},[167,991,992],{"class":505}," coerce_with_report",[167,994,995],{"class":204},"(frame, numeric",[167,997,414],{"class":200},[167,999,1000],{"class":204},"(), dates",[167,1002,414],{"class":200},[167,1004,1005],{"class":204},"()):\n",[167,1007,1008,1011,1013],{"class":169,"line":620},[167,1009,1010],{"class":204},"    out, failures ",[167,1012,414],{"class":200},[167,1014,1015],{"class":204}," frame.copy(), []\n",[167,1017,1018,1020,1022,1024],{"class":169,"line":634},[167,1019,524],{"class":200},[167,1021,527],{"class":204},[167,1023,530],{"class":200},[167,1025,1026],{"class":204}," numeric:\n",[167,1028,1029,1032,1034],{"class":169,"line":657},[167,1030,1031],{"class":204},"        converted ",[167,1033,414],{"class":200},[167,1035,1036],{"class":204}," to_number(frame[column])\n",[167,1038,1039,1042,1044,1047,1050,1053,1055,1058,1060,1063,1066,1069],{"class":169,"line":688},[167,1040,1041],{"class":204},"        broke ",[167,1043,414],{"class":200},[167,1045,1046],{"class":204}," converted.isna() ",[167,1048,1049],{"class":200},"&",[167,1051,1052],{"class":204}," frame[column].notna() ",[167,1054,1049],{"class":200},[167,1056,1057],{"class":204}," (frame[column].astype(",[167,1059,614],{"class":250},[167,1061,1062],{"class":204},").str.strip() ",[167,1064,1065],{"class":200},"!=",[167,1067,1068],{"class":177}," \"\"",[167,1070,430],{"class":204},[167,1072,1073,1076,1079,1082,1085,1087,1089,1092,1095,1098,1100,1103,1106,1109],{"class":169,"line":694},[167,1074,1075],{"class":204},"        failures ",[167,1077,1078],{"class":200},"+=",[167,1080,1081],{"class":204}," [{",[167,1083,1084],{"class":177},"\"row\"",[167,1086,236],{"class":204},[167,1088,628],{"class":250},[167,1090,1091],{"class":204},"(i) ",[167,1093,1094],{"class":200},"+",[167,1096,1097],{"class":250}," 2",[167,1099,242],{"class":204},[167,1101,1102],{"class":177},"\"column\"",[167,1104,1105],{"class":204},": column, ",[167,1107,1108],{"class":177},"\"value\"",[167,1110,1111],{"class":204},": frame.loc[i, column],\n",[167,1113,1114,1117,1119,1122,1125,1128,1131,1133],{"class":169,"line":703},[167,1115,1116],{"class":177},"                      \"problem\"",[167,1118,236],{"class":204},[167,1120,1121],{"class":177},"\"not a number\"",[167,1123,1124],{"class":204},"} ",[167,1126,1127],{"class":200},"for",[167,1129,1130],{"class":204}," i ",[167,1132,530],{"class":200},[167,1134,1135],{"class":204}," frame.index[broke]]\n",[167,1137,1138,1141,1143],{"class":169,"line":708},[167,1139,1140],{"class":204},"        out[column] ",[167,1142,414],{"class":200},[167,1144,1145],{"class":204}," converted\n",[167,1147,1149,1151,1153,1155],{"class":169,"line":1148},21,[167,1150,524],{"class":200},[167,1152,527],{"class":204},[167,1154,530],{"class":200},[167,1156,1157],{"class":204}," dates:\n",[167,1159,1161,1163,1165,1168,1171,1173,1175,1177,1180,1182,1185,1187,1190,1192,1194],{"class":169,"line":1160},22,[167,1162,1031],{"class":204},[167,1164,414],{"class":200},[167,1166,1167],{"class":204}," pd.to_datetime(frame[column], ",[167,1169,1170],{"class":410},"errors",[167,1172,414],{"class":200},[167,1174,979],{"class":177},[167,1176,242],{"class":204},[167,1178,1179],{"class":410},"format",[167,1181,414],{"class":200},[167,1183,1184],{"class":177},"\"mixed\"",[167,1186,242],{"class":204},[167,1188,1189],{"class":410},"dayfirst",[167,1191,414],{"class":200},[167,1193,841],{"class":250},[167,1195,430],{"class":204},[167,1197,1199,1201,1203,1205,1207],{"class":169,"line":1198},23,[167,1200,1041],{"class":204},[167,1202,414],{"class":200},[167,1204,1046],{"class":204},[167,1206,1049],{"class":200},[167,1208,1209],{"class":204}," frame[column].notna()\n",[167,1211,1213,1215,1217,1219,1221,1223,1225,1227,1229,1231,1233,1235,1237,1239],{"class":169,"line":1212},24,[167,1214,1075],{"class":204},[167,1216,1078],{"class":200},[167,1218,1081],{"class":204},[167,1220,1084],{"class":177},[167,1222,236],{"class":204},[167,1224,628],{"class":250},[167,1226,1091],{"class":204},[167,1228,1094],{"class":200},[167,1230,1097],{"class":250},[167,1232,242],{"class":204},[167,1234,1102],{"class":177},[167,1236,1105],{"class":204},[167,1238,1108],{"class":177},[167,1240,1111],{"class":204},[167,1242,1244,1246,1248,1251,1253,1255,1257,1259],{"class":169,"line":1243},25,[167,1245,1116],{"class":177},[167,1247,236],{"class":204},[167,1249,1250],{"class":177},"\"not a date\"",[167,1252,1124],{"class":204},[167,1254,1127],{"class":200},[167,1256,1130],{"class":204},[167,1258,530],{"class":200},[167,1260,1135],{"class":204},[167,1262,1264,1266,1268],{"class":169,"line":1263},26,[167,1265,1140],{"class":204},[167,1267,414],{"class":200},[167,1269,1145],{"class":204},[167,1271,1273,1275],{"class":169,"line":1272},27,[167,1274,697],{"class":200},[167,1276,1277],{"class":204}," out, failures\n",[167,1279,1281],{"class":169,"line":1280},28,[167,1282,218],{"emptyLinePlaceholder":217},[167,1284,1286,1289,1291,1294,1297,1299,1301,1303,1305,1307,1310,1313,1315,1317,1319],{"class":169,"line":1285},29,[167,1287,1288],{"class":204},"typed, failures ",[167,1290,414],{"class":200},[167,1292,1293],{"class":204}," coerce_with_report(df, ",[167,1295,1296],{"class":410},"numeric",[167,1298,414],{"class":200},[167,1300,817],{"class":204},[167,1302,245],{"class":177},[167,1304,242],{"class":204},[167,1306,255],{"class":177},[167,1308,1309],{"class":204},"], ",[167,1311,1312],{"class":410},"dates",[167,1314,414],{"class":200},[167,1316,817],{"class":204},[167,1318,265],{"class":177},[167,1320,1321],{"class":204},"])\n",[167,1323,1325,1327],{"class":169,"line":1324},30,[167,1326,711],{"class":250},[167,1328,1329],{"class":204},"(typed.dtypes)\n",[167,1331,1333,1335,1338,1340],{"class":169,"line":1332},31,[167,1334,1127],{"class":200},[167,1336,1337],{"class":204}," failure ",[167,1339,530],{"class":200},[167,1341,1342],{"class":204}," failures:\n",[167,1344,1346,1349],{"class":169,"line":1345},32,[167,1347,1348],{"class":250},"    print",[167,1350,1351],{"class":204},"(failure)\n",[10,1353,1354,1355,1358,1359,1362,1363,1366,1367,1370],{},"Stripping currency symbols and thousands separators before ",[13,1356,1357],{},"to_numeric"," is what makes the difference between \"three quarters of the column failed\" and \"one genuinely bad value failed\". Note the ",[13,1360,1361],{},"decimal_comma"," switch: a European export where ",[13,1364,1365],{},"49,50"," means forty-nine and a half needs the opposite treatment to a US file where ",[13,1368,1369],{},"49,500"," means forty-nine thousand five hundred, and no heuristic can tell them apart reliably. Ask the source, then set the flag.",[40,1372,49,1377,49,1380,49,1383,49,1386,49,1391,49,1395,49,1399,49,1406,49,1410,49,1414,49,1418,49,1421,49,1426,49,1429,49,1433,49,1435,49,1438,49,1440,49,1443,49,1450,49,1456],{"viewBox":1373,"role":43,"ariaLabelledBy":1374,"xmlns":47,"style":48},"0 0 740 232",[1375,1376],"dt-num-t","dt-num-d",[51,1378,1379],{"id":1375},"The same string means different numbers in different locales",[55,1381,1382],{"id":1376},"The text 49,50 becomes 49.5 when read with a decimal comma and 4950 when the comma is treated as a thousands separator. The text 1,234.50 is only valid in the dot-decimal reading. The locale must come from the source, not from a guess.",[59,1384],{"x":61,"y":61,"width":62,"height":1385,"fill":64},"232",[66,1387,1390],{"x":1388,"y":77,"style":1389},"120","font-size:12px;font-weight:700;fill:var(--muted,#5b6780);text-anchor:middle","as typed",[66,1392,1394],{"x":1393,"y":77,"style":1389},"400","read as dot-decimal",[66,1396,1398],{"x":1397,"y":77,"style":1389},"630","read as comma-decimal",[59,1400],{"x":69,"y":1401,"width":1402,"height":1403,"rx":1404,"fill":1405,"stroke":87},"50","180","42","8","#f0f2f5",[66,1407,303],{"x":1388,"y":1408,"style":1409},"76","font-size:13px;fill:var(--text,#172033);text-anchor:middle;font-family:ui-monospace,Menlo,monospace",[59,1411],{"x":1412,"y":1401,"width":1413,"height":1403,"rx":1404,"fill":136,"stroke":137},"290","220",[66,1415,1417],{"x":1393,"y":1408,"style":1416},"font-size:13px;font-weight:700;fill:var(--danger,#dc2626);text-anchor:middle;font-family:ui-monospace,Menlo,monospace","4950.0",[59,1419],{"x":1420,"y":1401,"width":1402,"height":1403,"rx":1404,"fill":113,"stroke":114},"536",[66,1422,1425],{"x":1423,"y":1408,"style":1424},"626","font-size:13px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle;font-family:ui-monospace,Menlo,monospace","49.5",[59,1427],{"x":69,"y":1428,"width":1402,"height":1403,"rx":1404,"fill":1405,"stroke":87},"106",[66,1430,1432],{"x":1388,"y":1431,"style":1409},"132","\"1,234.50\"",[59,1434],{"x":1412,"y":1428,"width":1413,"height":1403,"rx":1404,"fill":113,"stroke":114},[66,1436,1437],{"x":1393,"y":1431,"style":1424},"1234.5",[59,1439],{"x":1420,"y":1428,"width":1402,"height":1403,"rx":1404,"fill":136,"stroke":137},[66,1441,1442],{"x":1423,"y":1431,"style":1416},"NaN",[59,1444],{"x":1388,"y":1445,"width":1446,"height":75,"rx":1447,"fill":1448,"stroke":1449},"170","500","10","#fdefd8","var(--gold,#b4740a)",[66,1451,1455],{"x":1452,"y":1453,"style":1454},"370","190","font-size:12px;font-weight:700;fill:var(--gold-ink,#7a4e06);text-anchor:middle","Both readings are \"correct\" — only the source can say which applies.",[66,1457,1460],{"x":1452,"y":1458,"style":1459},"208","font-size:11.5px;fill:var(--muted,#5b6780);text-anchor:middle","Record the locale per feed; never infer it per file.",[153,1462,1464],{"id":1463},"dates-three-forms-in-one-column","Dates: three forms in one column",[10,1466,1467],{},"Excel stores real dates as serial numbers counted from 1899-12-30. When a column mixes serials, ISO strings and day-first strings — which happens whenever people paste between files — handle the numbers separately:",[158,1469,1471],{"className":191,"code":1470,"language":193,"meta":163,"style":163},"import pandas as pd\n\nEXCEL_EPOCH = pd.Timestamp(\"1899-12-30\")\n\ndef to_datetime_mixed(series, dayfirst=True):\n    values = series.copy()\n    numeric = pd.to_numeric(values, errors=\"coerce\")\n    from_serial = EXCEL_EPOCH + pd.to_timedelta(numeric, unit=\"D\")\n    from_text = pd.to_datetime(values.where(numeric.isna()), errors=\"coerce\",\n                               format=\"mixed\", dayfirst=dayfirst)\n    return from_serial.fillna(from_text)\n\nparsed = to_datetime_mixed(df[\"Order_Date\"])\nprint(parsed)\n",[13,1472,1473,1483,1487,1503,1507,1523,1533,1551,1577,1595,1613,1620,1624,1638],{"__ignoreMap":163},[167,1474,1475,1477,1479,1481],{"class":169,"line":170},[167,1476,201],{"class":200},[167,1478,205],{"class":204},[167,1480,208],{"class":200},[167,1482,211],{"class":204},[167,1484,1485],{"class":169,"line":214},[167,1486,218],{"emptyLinePlaceholder":217},[167,1488,1489,1492,1495,1498,1501],{"class":169,"line":221},[167,1490,1491],{"class":250},"EXCEL_EPOCH",[167,1493,1494],{"class":200}," =",[167,1496,1497],{"class":204}," pd.Timestamp(",[167,1499,1500],{"class":177},"\"1899-12-30\"",[167,1502,430],{"class":204},[167,1504,1505],{"class":169,"line":227},[167,1506,218],{"emptyLinePlaceholder":217},[167,1508,1509,1511,1514,1517,1519,1521],{"class":169,"line":276},[167,1510,502],{"class":200},[167,1512,1513],{"class":505}," to_datetime_mixed",[167,1515,1516],{"class":204},"(series, dayfirst",[167,1518,414],{"class":200},[167,1520,841],{"class":250},[167,1522,784],{"class":204},[167,1524,1525,1528,1530],{"class":169,"line":317},[167,1526,1527],{"class":204},"    values ",[167,1529,414],{"class":200},[167,1531,1532],{"class":204}," series.copy()\n",[167,1534,1535,1538,1540,1543,1545,1547,1549],{"class":169,"line":358},[167,1536,1537],{"class":204},"    numeric ",[167,1539,414],{"class":200},[167,1541,1542],{"class":204}," pd.to_numeric(values, ",[167,1544,1170],{"class":410},[167,1546,414],{"class":200},[167,1548,979],{"class":177},[167,1550,430],{"class":204},[167,1552,1553,1556,1558,1561,1564,1567,1570,1572,1575],{"class":169,"line":399},[167,1554,1555],{"class":204},"    from_serial ",[167,1557,414],{"class":200},[167,1559,1560],{"class":250}," EXCEL_EPOCH",[167,1562,1563],{"class":200}," +",[167,1565,1566],{"class":204}," pd.to_timedelta(numeric, ",[167,1568,1569],{"class":410},"unit",[167,1571,414],{"class":200},[167,1573,1574],{"class":177},"\"D\"",[167,1576,430],{"class":204},[167,1578,1579,1582,1584,1587,1589,1591,1593],{"class":169,"line":546},[167,1580,1581],{"class":204},"    from_text ",[167,1583,414],{"class":200},[167,1585,1586],{"class":204}," pd.to_datetime(values.where(numeric.isna()), ",[167,1588,1170],{"class":410},[167,1590,414],{"class":200},[167,1592,979],{"class":177},[167,1594,654],{"class":204},[167,1596,1597,1600,1602,1604,1606,1608,1610],{"class":169,"line":557},[167,1598,1599],{"class":410},"                               format",[167,1601,414],{"class":200},[167,1603,1184],{"class":177},[167,1605,242],{"class":204},[167,1607,1189],{"class":410},[167,1609,414],{"class":200},[167,1611,1612],{"class":204},"dayfirst)\n",[167,1614,1615,1617],{"class":169,"line":591},[167,1616,697],{"class":200},[167,1618,1619],{"class":204}," from_serial.fillna(from_text)\n",[167,1621,1622],{"class":169,"line":597},[167,1623,218],{"emptyLinePlaceholder":217},[167,1625,1626,1629,1631,1634,1636],{"class":169,"line":606},[167,1627,1628],{"class":204},"parsed ",[167,1630,414],{"class":200},[167,1632,1633],{"class":204}," to_datetime_mixed(df[",[167,1635,265],{"class":177},[167,1637,1321],{"class":204},[167,1639,1640,1642],{"class":169,"line":620},[167,1641,711],{"class":250},[167,1643,1644],{"class":204},"(parsed)\n",[10,1646,1647,1649,1650,1653,1654,1657,1658,1661,1662,1664,1665,1668],{},[13,1648,353],{}," becomes a real date instead of failing, ",[13,1651,1652],{},"15\u002F01\u002F2026"," is read day-first, and ",[13,1655,1656],{},"2026-02-30"," stays ",[13,1659,1660],{},"NaT"," because no calendar has that day. The ",[13,1663,1189],{}," flag carries the same warning as the decimal comma: ",[13,1666,1667],{},"03\u002F04\u002F2026"," is either March or April depending on who typed it, and a wrong guess produces dates that are plausible, wrong, and unnoticeable.",[153,1670,1672],{"id":1671},"assert-the-schema-you-expect","Assert the schema you expect",[10,1674,1675],{},"Once coercion is done, state the contract. A single assertion catches a source system that changes a column's meaning between runs:",[158,1677,1679],{"className":191,"code":1678,"language":193,"meta":163,"style":163},"import pandas as pd\nfrom pandas.api import types as pdt\n\nEXPECTED = {\n    \"Code\": pdt.is_string_dtype,\n    \"Quantity\": pdt.is_numeric_dtype,\n    \"Unit_Price\": pdt.is_numeric_dtype,\n    \"Order_Date\": pdt.is_datetime64_any_dtype,\n}\n\ndef assert_schema(frame):\n    problems = [\n        f\"{column}: expected {check.__name__.replace('is_', '').replace('_dtype', '')}, got {frame[column].dtype}\"\n        for column, check in EXPECTED.items()\n        if column in frame.columns and not check(frame[column])\n    ]\n    if problems:\n        raise TypeError(\"schema check failed — \" + \"; \".join(problems))\n    return True\n\ntyped[\"Code\"] = typed[\"Code\"].astype(\"string\")\nassert_schema(typed)\nprint(\"schema OK\")\n",[13,1680,1681,1691,1709,1713,1723,1731,1739,1746,1754,1759,1763,1772,1782,1847,1863,1884,1889,1896,1918,1925,1929,1953,1958],{"__ignoreMap":163},[167,1682,1683,1685,1687,1689],{"class":169,"line":170},[167,1684,201],{"class":200},[167,1686,205],{"class":204},[167,1688,208],{"class":200},[167,1690,211],{"class":204},[167,1692,1693,1696,1699,1701,1704,1706],{"class":169,"line":214},[167,1694,1695],{"class":200},"from",[167,1697,1698],{"class":204}," pandas.api ",[167,1700,201],{"class":200},[167,1702,1703],{"class":204}," types ",[167,1705,208],{"class":200},[167,1707,1708],{"class":204}," pdt\n",[167,1710,1711],{"class":169,"line":221},[167,1712,218],{"emptyLinePlaceholder":217},[167,1714,1715,1718,1720],{"class":169,"line":227},[167,1716,1717],{"class":250},"EXPECTED",[167,1719,1494],{"class":200},[167,1721,1722],{"class":204}," {\n",[167,1724,1725,1728],{"class":169,"line":276},[167,1726,1727],{"class":177},"    \"Code\"",[167,1729,1730],{"class":204},": pdt.is_string_dtype,\n",[167,1732,1733,1736],{"class":169,"line":317},[167,1734,1735],{"class":177},"    \"Quantity\"",[167,1737,1738],{"class":204},": pdt.is_numeric_dtype,\n",[167,1740,1741,1744],{"class":169,"line":358},[167,1742,1743],{"class":177},"    \"Unit_Price\"",[167,1745,1738],{"class":204},[167,1747,1748,1751],{"class":169,"line":399},[167,1749,1750],{"class":177},"    \"Order_Date\"",[167,1752,1753],{"class":204},": pdt.is_datetime64_any_dtype,\n",[167,1755,1756],{"class":169,"line":546},[167,1757,1758],{"class":204},"}\n",[167,1760,1761],{"class":169,"line":557},[167,1762,218],{"emptyLinePlaceholder":217},[167,1764,1765,1767,1770],{"class":169,"line":591},[167,1766,502],{"class":200},[167,1768,1769],{"class":505}," assert_schema",[167,1771,509],{"class":204},[167,1773,1774,1777,1779],{"class":169,"line":597},[167,1775,1776],{"class":204},"    problems ",[167,1778,414],{"class":200},[167,1780,1781],{"class":204}," [\n",[167,1783,1784,1787,1789,1792,1795,1798,1801,1803,1806,1808,1811,1814,1816,1819,1822,1825,1827,1829,1832,1834,1837,1839,1842,1844],{"class":169,"line":606},[167,1785,1786],{"class":200},"        f",[167,1788,814],{"class":177},[167,1790,1791],{"class":820},"{",[167,1793,1794],{"class":204},"column",[167,1796,1797],{"class":820},"}",[167,1799,1800],{"class":177},": expected ",[167,1802,1791],{"class":820},[167,1804,1805],{"class":204},"check.",[167,1807,577],{"class":250},[167,1809,1810],{"class":204},".replace(",[167,1812,1813],{"class":177},"'is_'",[167,1815,242],{"class":204},[167,1817,1818],{"class":177},"''",[167,1820,1821],{"class":204},").replace(",[167,1823,1824],{"class":177},"'_dtype'",[167,1826,242],{"class":204},[167,1828,1818],{"class":177},[167,1830,1831],{"class":204},")",[167,1833,1797],{"class":820},[167,1835,1836],{"class":177},", got ",[167,1838,1791],{"class":820},[167,1840,1841],{"class":204},"frame[column].dtype",[167,1843,1797],{"class":820},[167,1845,1846],{"class":177},"\"\n",[167,1848,1849,1852,1855,1857,1860],{"class":169,"line":620},[167,1850,1851],{"class":200},"        for",[167,1853,1854],{"class":204}," column, check ",[167,1856,530],{"class":200},[167,1858,1859],{"class":250}," EXPECTED",[167,1861,1862],{"class":204},".items()\n",[167,1864,1865,1868,1870,1872,1875,1878,1881],{"class":169,"line":634},[167,1866,1867],{"class":200},"        if",[167,1869,527],{"class":204},[167,1871,530],{"class":200},[167,1873,1874],{"class":204}," frame.columns ",[167,1876,1877],{"class":200},"and",[167,1879,1880],{"class":200}," not",[167,1882,1883],{"class":204}," check(frame[column])\n",[167,1885,1886],{"class":169,"line":657},[167,1887,1888],{"class":204},"    ]\n",[167,1890,1891,1893],{"class":169,"line":688},[167,1892,848],{"class":200},[167,1894,1895],{"class":204}," problems:\n",[167,1897,1898,1901,1904,1907,1910,1912,1915],{"class":169,"line":694},[167,1899,1900],{"class":200},"        raise",[167,1902,1903],{"class":250}," TypeError",[167,1905,1906],{"class":204},"(",[167,1908,1909],{"class":177},"\"schema check failed — \"",[167,1911,1563],{"class":200},[167,1913,1914],{"class":177}," \"; \"",[167,1916,1917],{"class":204},".join(problems))\n",[167,1919,1920,1922],{"class":169,"line":703},[167,1921,697],{"class":200},[167,1923,1924],{"class":250}," True\n",[167,1926,1927],{"class":169,"line":708},[167,1928,218],{"emptyLinePlaceholder":217},[167,1930,1931,1934,1936,1938,1940,1943,1945,1948,1951],{"class":169,"line":1148},[167,1932,1933],{"class":204},"typed[",[167,1935,233],{"class":177},[167,1937,668],{"class":204},[167,1939,414],{"class":200},[167,1941,1942],{"class":204}," typed[",[167,1944,233],{"class":177},[167,1946,1947],{"class":204},"].astype(",[167,1949,1950],{"class":177},"\"string\"",[167,1952,430],{"class":204},[167,1954,1955],{"class":169,"line":1160},[167,1956,1957],{"class":204},"assert_schema(typed)\n",[167,1959,1960,1962,1964,1967],{"class":169,"line":1198},[167,1961,711],{"class":250},[167,1963,1906],{"class":204},[167,1965,1966],{"class":177},"\"schema OK\"",[167,1968,430],{"class":204},[10,1970,1971,1972,1975,1976,1979,1980,1983,1984,1987],{},"Keeping ",[13,1973,1974],{},"Code"," as a string is the point of that first line. Read without care, ",[13,1977,1978],{},"00412"," becomes the integer ",[13,1981,1982],{},"412",", and the join against a reference table quietly matches nothing. Any identifier — account numbers, SKUs, postcodes, phone numbers — should be a string from the moment it is read, which is one more reason to read with ",[13,1985,1986],{},"dtype=object"," and convert deliberately.",[153,1989,1991],{"id":1990},"report-the-type-failures-with-the-data","Report the type failures with the data",[158,1993,1995],{"className":191,"code":1994,"language":193,"meta":163,"style":163},"import pandas as pd\n\nwith pd.ExcelWriter(\"types_checked.xlsx\", engine=\"openpyxl\") as writer:\n    typed.to_excel(writer, sheet_name=\"Typed\", index=False)\n    pd.DataFrame(failures).to_excel(writer, sheet_name=\"Type_Issues\", index=False)\n\nprint(f\"{len(failures)} type failure(s) written to Type_Issues\")\n",[13,1996,1997,2007,2011,2040,2062,2084,2088],{"__ignoreMap":163},[167,1998,1999,2001,2003,2005],{"class":169,"line":170},[167,2000,201],{"class":200},[167,2002,205],{"class":204},[167,2004,208],{"class":200},[167,2006,211],{"class":204},[167,2008,2009],{"class":169,"line":214},[167,2010,218],{"emptyLinePlaceholder":217},[167,2012,2013,2016,2019,2022,2024,2027,2029,2032,2035,2037],{"class":169,"line":221},[167,2014,2015],{"class":200},"with",[167,2017,2018],{"class":204}," pd.ExcelWriter(",[167,2020,2021],{"class":177},"\"types_checked.xlsx\"",[167,2023,242],{"class":204},[167,2025,2026],{"class":410},"engine",[167,2028,414],{"class":200},[167,2030,2031],{"class":177},"\"openpyxl\"",[167,2033,2034],{"class":204},") ",[167,2036,208],{"class":200},[167,2038,2039],{"class":204}," writer:\n",[167,2041,2042,2045,2047,2049,2052,2054,2056,2058,2060],{"class":169,"line":227},[167,2043,2044],{"class":204},"    typed.to_excel(writer, ",[167,2046,422],{"class":410},[167,2048,414],{"class":200},[167,2050,2051],{"class":177},"\"Typed\"",[167,2053,242],{"class":204},[167,2055,411],{"class":410},[167,2057,414],{"class":200},[167,2059,417],{"class":250},[167,2061,430],{"class":204},[167,2063,2064,2067,2069,2071,2074,2076,2078,2080,2082],{"class":169,"line":276},[167,2065,2066],{"class":204},"    pd.DataFrame(failures).to_excel(writer, ",[167,2068,422],{"class":410},[167,2070,414],{"class":200},[167,2072,2073],{"class":177},"\"Type_Issues\"",[167,2075,242],{"class":204},[167,2077,411],{"class":410},[167,2079,414],{"class":200},[167,2081,417],{"class":250},[167,2083,430],{"class":204},[167,2085,2086],{"class":169,"line":317},[167,2087,218],{"emptyLinePlaceholder":217},[167,2089,2090,2092,2094,2097,2099,2101,2104,2107,2109,2112],{"class":169,"line":358},[167,2091,711],{"class":250},[167,2093,1906],{"class":204},[167,2095,2096],{"class":200},"f",[167,2098,814],{"class":177},[167,2100,1791],{"class":820},[167,2102,2103],{"class":250},"len",[167,2105,2106],{"class":204},"(failures)",[167,2108,1797],{"class":820},[167,2110,2111],{"class":177}," type failure(s) written to Type_Issues\"",[167,2113,430],{"class":204},[10,2115,726,2116,2119,2120,2124],{},[13,2117,2118],{},"row"," numbers in that sheet are spreadsheet rows, so a submitter can jump straight to the cell. Pair it with ",[34,2121,2123],{"href":2122},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fhighlight-invalid-cells-in-excel-with-python\u002F","highlighting the invalid cells"," on the original data and the fix takes seconds rather than an email exchange.",[153,2126,2128],{"id":2127},"what-each-dtype-is-telling-you","What each dtype is telling you",[10,2130,2131],{},"The dtype after a read is a summary of the whole column, and each value carries a specific message about the data behind it:",[40,2133,49,2138,49,2141,49,2144,49,2147,49,2152,49,2156,49,2161,49,2163,49,2166,49,2169,49,2172,49,2176,49,2180,49,2184,49,2189],{"viewBox":2134,"role":43,"ariaLabelledBy":2135,"xmlns":47,"style":48},"0 0 740 226",[2136,2137],"dt-map-t","dt-map-d",[51,2139,2140],{"id":2136},"What each pandas dtype implies about the spreadsheet column",[55,2142,2143],{"id":2137},"int64 means every cell was a whole number. float64 means numbers with at least one decimal or blank. object means mixed or text content and needs investigation. datetime64 means every cell parsed as a date. string means the column was read or cast as text, which is what identifiers need.",[59,2145],{"x":61,"y":61,"width":62,"height":2146,"fill":64},"226",[59,2148],{"x":2149,"y":2150,"width":2151,"height":1403,"rx":112,"fill":113,"stroke":114},"24","26","336",[66,2153,31],{"x":75,"y":2154,"style":2155},"53","font-size:12.5px;font-weight:700;fill:var(--teal,#0f9488);font-family:ui-monospace,Menlo,monospace",[66,2157,2160],{"x":2158,"y":2154,"style":2159},"380","font-size:11.5px;fill:var(--muted,#5b6780)","every cell was a whole number — safe to sum",[59,2162],{"x":2149,"y":1408,"width":2151,"height":1403,"rx":112,"fill":113,"stroke":114},[66,2164,27],{"x":75,"y":2165,"style":2155},"103",[66,2167,2168],{"x":2158,"y":2165,"style":2159},"numbers, with decimals or blanks present",[59,2170],{"x":2149,"y":2171,"width":2151,"height":1403,"rx":112,"fill":136,"stroke":137,"style":115},"126",[66,2173,23],{"x":75,"y":2174,"style":2175},"153","font-size:12.5px;font-weight:700;fill:var(--danger,#dc2626);font-family:ui-monospace,Menlo,monospace",[66,2177,2179],{"x":2158,"y":2174,"style":2178},"font-size:11.5px;fill:var(--danger,#dc2626)","mixed or text — profile it before trusting anything",[59,2181],{"x":2149,"y":2182,"width":2151,"height":1403,"rx":112,"fill":2183,"stroke":87},"176","#ebebfd",[66,2185,2188],{"x":75,"y":2186,"style":2187},"203","font-size:12.5px;font-weight:700;fill:var(--brand-strong,#4338ca);font-family:ui-monospace,Menlo,monospace","datetime64[ns] \u002F string",[66,2190,2191],{"x":2158,"y":2186,"style":2159},"parsed dates, or text you chose to keep as text",[10,2193,2194,2195,2197,2198,2200],{},"The pair to watch is ",[13,2196,31],{}," and ",[13,2199,23],{}," in the same column across two runs of the same report. It means one submission was clean and the next contained a stray note, and any code that indexed or joined on that column will behave differently between the two — usually without raising anything.",[153,2202,2204],{"id":2203},"common-pitfalls-and-gotchas","Common pitfalls and gotchas",[2206,2207,2208,2224],"table",{},[2209,2210,2211],"thead",{},[2212,2213,2214,2218,2221],"tr",{},[2215,2216,2217],"th",{},"Symptom",[2215,2219,2220],{},"Cause",[2215,2222,2223],{},"Fix",[2225,2226,2227,2241,2258,2277,2288,2299,2317],"tbody",{},[2212,2228,2229,2235,2238],{},[2230,2231,2232,2233],"td",{},"Numeric column has dtype ",[13,2234,23],{},[2230,2236,2237],{},"One or more non-numeric cells",[2230,2239,2240],{},"Profile the column, then coerce with a failure report",[2212,2242,2243,2246,2249],{},[2230,2244,2245],{},"Leading zeros disappeared",[2230,2247,2248],{},"Column was inferred as integer",[2230,2250,2251,2252,2254,2255,2257],{},"Read as ",[13,2253,23],{},"\u002F",[13,2256,614],{}," and keep it a string",[2212,2259,2260,2265,2274],{},[2230,2261,2262,2264],{},[13,2263,737],{}," when summing",[2230,2266,2267,2268,2197,2270,2273],{},"Mixed ",[13,2269,614],{},[13,2271,2272],{},"float"," in one column",[2230,2275,2276],{},"Coerce before any arithmetic",[2212,2278,2279,2282,2285],{},[2230,2280,2281],{},"Dates off by four years",[2230,2283,2284],{},"Workbook uses the 1904 date system",[2230,2286,2287],{},"Use the 1904 epoch for that file",[2212,2289,2290,2293,2296],{},[2230,2291,2292],{},"European decimals ten times too large",[2230,2294,2295],{},"Comma treated as a thousands separator",[2230,2297,2298],{},"Set the decimal convention per source feed",[2212,2300,2301,2307,2310],{},[2230,2302,2303,2306],{},[13,2304,2305],{},"to_datetime"," warns about inconsistent parsing",[2230,2308,2309],{},"Genuinely mixed formats",[2230,2311,2312,2313,2316],{},"Pass ",[13,2314,2315],{},"format=\"mixed\"",", and split serials out first",[2212,2318,2319,2326,2329],{},[2230,2320,2321,2197,2323,2325],{},[13,2322,1442],{},[13,2324,1660],{}," counted as valid",[2230,2327,2328],{},"Coerced values never checked",[2230,2330,2331],{},"Compare against the original to find what broke",[153,2333,2335],{"id":2334},"performance-and-scale-notes","Performance and scale notes",[10,2337,2338,2339,2342,2343,2346,2347,2349,2350,2353],{},"Profiling with a ",[13,2340,2341],{},"type()"," set per column is a Python-level loop, so on very large frames sample it — ",[13,2344,2345],{},"frame.head(5000)"," is enough to reveal a mixed column. Coercion itself is vectorised and fast; the slow part is usually the regular expression stripping currency symbols, which you can skip for columns that arrive clean. Reading with ",[13,2348,1986],{}," costs more memory than letting pandas infer, so for a genuinely large workbook validate a sample with ",[13,2351,2352],{},"nrows",", then read the full file with the dtypes you have just proven correct.",[153,2355,2357],{"id":2356},"conclusion","Conclusion",[10,2359,2360],{},"Type checking a spreadsheet is three deliberate moves rather than one hopeful read: read raw so nothing is guessed, profile what actually arrived, then coerce each column deliberately while recording every value that refused to convert. Keep identifiers as strings, get the decimal and date conventions from the source rather than from a guess, and finish with a schema assertion so a changed source system fails loudly instead of quietly reporting the wrong numbers.",[153,2362,2364],{"id":2363},"frequently-asked-questions","Frequently asked questions",[10,2366,2367,2374,2375,2377,2378,2380],{},[2368,2369,2370,2371,2373],"strong",{},"Why is my numeric column dtype ",[13,2372,23],{},"?","\nAt least one cell is not a number — usually a stray ",[13,2376,335],{},", a dash, or a number typed with a thousands separator. pandas falls back to ",[13,2379,23],{}," for the whole column.",[10,2382,2383,2386,2387,2390,2391,2393],{},[2368,2384,2385],{},"How do I see which values broke the conversion?","\nCoerce with ",[13,2388,2389],{},"errors=\"coerce\"",", then compare the result against the original: rows that are ",[13,2392,1442],{}," afterwards but were non-blank before are exactly the failures.",[10,2395,2396,2404,2405,2407],{},[2368,2397,2398,2399,2401,2402,2373],{},"Should I pass ",[13,2400,487],{}," to ",[13,2403,19],{},"\nPass ",[13,2406,1986],{}," when validating, so nothing is guessed, then coerce explicitly. Passing a strict dtype up front raises on the first bad value and tells you nothing about the rest.",[10,2409,2410,2413,2414,2417,2418,2420],{},[2368,2411,2412],{},"Why did my ID column lose its leading zeros?","\nIt was inferred as numeric. Read it as a string with ",[13,2415,2416],{},"dtype={\"Code\": str}"," — or read everything as ",[13,2419,23],{}," and never let the inference run.",[153,2422,2424],{"id":2423},"related","Related",[10,2426,2427],{},"Up to the parent guide:",[2429,2430,2431],"ul",{},[2432,2433,2434,2436],"li",{},[34,2435,37],{"href":36}," — the validation layer these checks belong to.",[10,2438,2439],{},"Related guides:",[2429,2441,2442,2449,2455,2462],{},[2432,2443,2444,2448],{},[34,2445,2447],{"href":2446},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fvalidate-excel-columns-before-import-with-pandas\u002F","Validate Excel Columns Before Import with pandas"," — the structural gate that runs first.",[2432,2450,2451,2454],{},[34,2452,2453],{"href":2122},"Highlight Invalid Cells in Excel with Python"," — showing the failures in the workbook.",[2432,2456,2457,2461],{},[34,2458,2460],{"href":2459},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002F","Cleaning Excel Data with pandas"," — normalising values once their types are known.",[2432,2463,2464,2468],{},[34,2465,2467],{"href":2466},"\u002Fformatting-and-charting-excel-reports-with-python\u002Fapplying-number-and-date-formats-in-excel\u002Fformat-dates-in-excel-cells-with-python\u002F","Format Dates in Excel Cells with Python"," — writing dates back so Excel shows them properly.",[2470,2471,2472],"style",{},"html pre.shiki code .sMTad, html code.shiki .sMTad{--shiki-default:#6F42C1;--shiki-dark:#FFB757}html pre.shiki code .srMev, html code.shiki .srMev{--shiki-default:#032F62;--shiki-dark:#ADDCFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .s-kum, html code.shiki .s-kum{--shiki-default:#D73A49;--shiki-dark:#FF9492}html pre.shiki code .skGVy, html code.shiki .skGVy{--shiki-default:#24292E;--shiki-dark:#F0F3F6}html pre.shiki code .sP0c6, html code.shiki .sP0c6{--shiki-default:#005CC5;--shiki-dark:#91CBFF}html pre.shiki code .sa561, html code.shiki .sa561{--shiki-default:#E36209;--shiki-dark:#FFB757}html pre.shiki code .s_Opv, html code.shiki .s_Opv{--shiki-default:#6F42C1;--shiki-dark:#DBB7FF}html pre.shiki code .sSjpA, html code.shiki .sSjpA{--shiki-default:#005CC5;--shiki-dark:#FF9492}",{"title":163,"searchDepth":214,"depth":214,"links":2474},[2475,2476,2477,2478,2479,2480,2481,2482,2483,2484,2485,2486,2487],{"id":155,"depth":214,"text":156},{"id":187,"depth":214,"text":188},{"id":436,"depth":214,"text":437},{"id":744,"depth":214,"text":745},{"id":1463,"depth":214,"text":1464},{"id":1671,"depth":214,"text":1672},{"id":1990,"depth":214,"text":1991},{"id":2127,"depth":214,"text":2128},{"id":2203,"depth":214,"text":2204},{"id":2334,"depth":214,"text":2335},{"id":2356,"depth":214,"text":2357},{"id":2363,"depth":214,"text":2364},{"id":2423,"depth":214,"text":2424},"2026-08-01","Find out what pandas actually read from a spreadsheet — object columns hiding mixed types, numbers stored as text, Excel date serials — and coerce them with a record of every failure.","md",[2492,2495,2498,2501],{"q":2493,"a":2494},"Why is my numeric column dtype object?","At least one cell is not a number — usually a stray \"n\u002Fa\", a dash, or a number typed with a thousands separator. pandas falls back to object for the whole column.",{"q":2385,"a":2496},{"Coerce with errors=\"coerce\", then compare the result against the original":2497},"rows that are NaN afterwards but were non-blank before are exactly the failures.",{"q":2499,"a":2500},"Should I pass dtype to read_excel?","Pass dtype=object when validating, so nothing is guessed, then coerce explicitly. Passing a strict dtype up front raises on the first bad value and tells you nothing about the rest.",{"q":2412,"a":2502},{"It was inferred as numeric":2503},{" Read it as a string with dtype={\"Code\"":2504},"str} — or read everything as object and never let the inference run.",{"breadcrumb":2506},[2507,2509,2512,2513],{"name":2508,"item":2254},"Home",{"name":2510,"item":2511},"Advanced Data Transformation and Cleaning","\u002Fadvanced-data-transformation-and-cleaning\u002F",{"name":37,"item":36},{"name":5,"item":2514},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fcheck-excel-data-types-with-pandas\u002F","\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fcheck-excel-data-types-with-pandas",{"title":5,"description":2517},"Inspect dtypes after read_excel, detect mixed-type object columns, convert numbers and dates with errors=coerce, and assert an expected schema before a report runs.","check-excel-data-types-with-pandas","advanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fcheck-excel-data-types-with-pandas\u002Findex","how-to","4KscRNVjD3I_4iYyX6FKpeSns9YaIMWE4dBNR-TxBTA",[2523,2527],{"title":2524,"path":2525,"stem":2526,"children":-1},"Add Dropdown Data Validation to Excel with openpyxl","\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fadd-dropdown-data-validation-to-excel-with-openpyxl","advanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fadd-dropdown-data-validation-to-excel-with-openpyxl\u002Findex",{"title":2528,"path":2529,"stem":2530,"children":-1},"Compare Two Excel Files for Differences with Python","\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fcompare-two-excel-files-for-differences-with-python","advanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fcompare-two-excel-files-for-differences-with-python\u002Findex",1785584462686]