[{"data":1,"prerenderedAt":2992},["ShallowReactive",2],{"doc:\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Fconcatenate-excel-sheets-with-different-columns":3,"surround:\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Fconcatenate-excel-sheets-with-different-columns":2984},{"id":4,"title":5,"body":6,"dateModified":2960,"datePublished":2960,"description":2961,"extension":2962,"faq":2963,"meta":2975,"navigation":258,"path":2976,"seo":2977,"slug":2980,"stem":2981,"type":2982,"__hash__":2983},"docs\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Fconcatenate-excel-sheets-with-different-columns\u002Findex.md","Concatenate Excel Sheets with Different Columns",{"type":7,"value":8,"toc":2947},"minimark",[9,46,180,185,213,216,498,505,509,512,726,742,746,749,894,904,1062,1065,1069,1072,1613,1620,1634,1638,1648,1730,2010,2013,2028,2034,2119,2126,2130,2229,2232,2367,2374,2378,2507,2511,2516,2604,2620,2764,2767,2780,2809,2817,2821,2824,2828,2840,2856,2875,2892,2903,2907,2943],[10,11,12,13,17,18,21,22,25,26,29,30,33,34,36,37,39,40,45],"p",{},"A monthly export is never quite the same file twice. One month a column is called ",[14,15,16],"code",{},"revenue",", the next it is ",[14,19,20],{},"Revenue",", then somebody adds ",[14,23,24],{},"currency"," and drops ",[14,27,28],{},"notes",". Stack them naively with ",[14,31,32],{},"pd.concat"," and you get a frame with ",[14,35,16],{}," and ",[14,38,20],{}," as separate columns, each two-thirds empty, and totals that are silently wrong. This guide covers aligning the headers first, tracking where each row came from, and reporting the drift instead of absorbing it. It extends ",[41,42,44],"a",{"href":43},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002F","Merging and Joining Excel DataFrames",".",[47,48,57,58,57,62,57,66,57,73,57,80,57,90,57,95,57,98,57,102,57,105,57,109,57,128,57,138,57,147,57,152,57,156,57,161,57,166,57,171,57,175],"svg",{"viewBox":49,"role":50,"ariaLabel":51,"ariaLabelledBy":52,"xmlns":55,"style":56},"0 0 800 264","img","Three monthly files with drifting headers concatenated naively produce duplicate near-identical columns, while normalising names first produces one clean aligned frame.",[53,54],"cat-t","cat-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;max-width:800px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif","\n  ",[59,60,61],"title",{"id":53},"Header drift across three monthly files, and what concat does with it",[63,64,65],"desc",{"id":54},"Three monthly exports have slightly different headers: June uses revenue, July uses Revenue with a capital, August uses total_revenue and adds a currency column. Concatenating them directly produces three separate revenue columns each about a third populated, because concat matches column names as exact strings. Applying a rename map first collapses all three into one canonical revenue column, and currency simply appears with nulls for the months that lacked it.",[67,68],"rect",{"x":69,"y":69,"width":70,"height":71,"fill":72},"0","800","264","#ffffff",[74,75,79],"text",{"x":76,"y":77,"style":78},"160","28","font-size:12px;font-weight:700;fill:var(--muted,#5b6780);text-anchor:middle","three monthly files",[67,81],{"x":82,"y":83,"width":84,"height":85,"rx":86,"fill":87,"stroke":88,"style":89},"20","42","280","34","6","#ebebfd","var(--brand,#5b5cf0)","stroke-width:2px",[74,91,94],{"x":76,"y":92,"style":93},"64","font-size:10.5px;fill:var(--text,#172033);text-anchor:middle","June · region, revenue",[67,96],{"x":82,"y":97,"width":84,"height":85,"rx":86,"fill":87,"stroke":88,"style":89},"82",[74,99,101],{"x":76,"y":100,"style":93},"104","July · region, Revenue",[67,103],{"x":82,"y":104,"width":84,"height":85,"rx":86,"fill":87,"stroke":88,"style":89},"122",[74,106,108],{"x":76,"y":107,"style":93},"144","August · region, total_revenue, currency",[110,111,114,115,114,119,114,122,114,125,57],"g",{"stroke":112,"style":89,"fill":113},"var(--line,#cdd5e6)","none","\n    ",[116,117],"path",{"d":118},"M300 59 H 330 V 78 H 358",[116,120],{"d":121},"M300 99 H 330 V 78",[116,123],{"d":124},"M300 139 H 330 V 78",[116,126],{"d":127},"M300 139 H 330 V 196 H 358",[110,129,114,131,114,135,57],{"fill":130},"#5b5cf0",[132,133],"polygon",{"points":134},"366,78 354,72 354,84",[132,136],{"points":137},"366,196 354,190 354,202",[67,139],{"x":140,"y":141,"width":142,"height":143,"rx":144,"fill":145,"stroke":146,"style":89},"374","40","410","78","13","#fee8f2","var(--accent,#f43f8f)",[74,148,151],{"x":149,"y":92,"style":150},"579","font-size:12px;font-weight:700;fill:var(--accent-ink,#be185d);text-anchor:middle","concat as-is",[74,153,155],{"x":149,"y":154,"style":93},"86","revenue · Revenue · total_revenue · currency",[74,157,160],{"x":149,"y":158,"style":159},"106","font-size:10.5px;font-weight:700;fill:var(--accent-ink,#be185d);text-anchor:middle","three columns, each two-thirds empty",[67,162],{"x":140,"y":163,"width":142,"height":143,"rx":144,"fill":164,"stroke":165,"style":89},"158","#d9f4f1","var(--teal,#0f9488)",[74,167,170],{"x":149,"y":168,"style":169},"182","font-size:12px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","rename map, then concat",[74,172,174],{"x":149,"y":173,"style":93},"204","region · revenue · currency · source_file",[74,176,179],{"x":149,"y":177,"style":178},"224","font-size:10.5px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","one revenue column; currency null where absent",[181,182,184],"h2",{"id":183},"prerequisites","Prerequisites",[186,187,192],"pre",{"className":188,"code":189,"language":190,"meta":191,"style":191},"language-bash shiki shiki-themes github-light github-dark-high-contrast","pip install pandas openpyxl\n","bash","",[14,193,194],{"__ignoreMap":191},[195,196,199,203,207,210],"span",{"class":197,"line":198},"line",1,[195,200,202],{"class":201},"sMTad","pip",[195,204,206],{"class":205},"srMev"," install",[195,208,209],{"class":205}," pandas",[195,211,212],{"class":205}," openpyxl\n",[10,214,215],{},"Three months with realistic drift:",[186,217,221],{"className":218,"code":219,"language":220,"meta":191,"style":191},"language-python shiki shiki-themes github-light github-dark-high-contrast","from pathlib import Path\nimport pandas as pd\n\nPath(\"monthly\").mkdir(exist_ok=True)\n\npd.DataFrame({\"region\": [\"North\", \"South\"],\n              \"revenue\": [5150.00, 4268.50]}).to_excel(\n    \"monthly\u002F2026-06.xlsx\", index=False)\n\npd.DataFrame({\"Region\": [\"North\", \"West\"],\n              \"Revenue\": [4980.25, 3511.25]}).to_excel(\n    \"monthly\u002F2026-07.xlsx\", index=False)\n\npd.DataFrame({\"region \": [\"North\", \"South\", \"East\"],\n              \"total_revenue\": [\"5,402.75\", \"3140.75\", \"2980.10\"],\n              \"currency\": [\"EUR\", \"EUR\", \"EUR\"]}).to_excel(\n    \"monthly\u002F2026-08.xlsx\", index=False)\n","python",[14,222,223,239,253,260,286,291,315,334,352,357,376,394,410,415,438,461,482],{"__ignoreMap":191},[195,224,225,229,233,236],{"class":197,"line":198},[195,226,228],{"class":227},"s-kum","from",[195,230,232],{"class":231},"skGVy"," pathlib ",[195,234,235],{"class":227},"import",[195,237,238],{"class":231}," Path\n",[195,240,242,244,247,250],{"class":197,"line":241},2,[195,243,235],{"class":227},[195,245,246],{"class":231}," pandas ",[195,248,249],{"class":227},"as",[195,251,252],{"class":231}," pd\n",[195,254,256],{"class":197,"line":255},3,[195,257,259],{"emptyLinePlaceholder":258},true,"\n",[195,261,263,266,269,272,276,279,283],{"class":197,"line":262},4,[195,264,265],{"class":231},"Path(",[195,267,268],{"class":205},"\"monthly\"",[195,270,271],{"class":231},").mkdir(",[195,273,275],{"class":274},"sa561","exist_ok",[195,277,278],{"class":227},"=",[195,280,282],{"class":281},"sP0c6","True",[195,284,285],{"class":231},")\n",[195,287,289],{"class":197,"line":288},5,[195,290,259],{"emptyLinePlaceholder":258},[195,292,294,297,300,303,306,309,312],{"class":197,"line":293},6,[195,295,296],{"class":231},"pd.DataFrame({",[195,298,299],{"class":205},"\"region\"",[195,301,302],{"class":231},": [",[195,304,305],{"class":205},"\"North\"",[195,307,308],{"class":231},", ",[195,310,311],{"class":205},"\"South\"",[195,313,314],{"class":231},"],\n",[195,316,318,321,323,326,328,331],{"class":197,"line":317},7,[195,319,320],{"class":205},"              \"revenue\"",[195,322,302],{"class":231},[195,324,325],{"class":281},"5150.00",[195,327,308],{"class":231},[195,329,330],{"class":281},"4268.50",[195,332,333],{"class":231},"]}).to_excel(\n",[195,335,337,340,342,345,347,350],{"class":197,"line":336},8,[195,338,339],{"class":205},"    \"monthly\u002F2026-06.xlsx\"",[195,341,308],{"class":231},[195,343,344],{"class":274},"index",[195,346,278],{"class":227},[195,348,349],{"class":281},"False",[195,351,285],{"class":231},[195,353,355],{"class":197,"line":354},9,[195,356,259],{"emptyLinePlaceholder":258},[195,358,360,362,365,367,369,371,374],{"class":197,"line":359},10,[195,361,296],{"class":231},[195,363,364],{"class":205},"\"Region\"",[195,366,302],{"class":231},[195,368,305],{"class":205},[195,370,308],{"class":231},[195,372,373],{"class":205},"\"West\"",[195,375,314],{"class":231},[195,377,379,382,384,387,389,392],{"class":197,"line":378},11,[195,380,381],{"class":205},"              \"Revenue\"",[195,383,302],{"class":231},[195,385,386],{"class":281},"4980.25",[195,388,308],{"class":231},[195,390,391],{"class":281},"3511.25",[195,393,333],{"class":231},[195,395,397,400,402,404,406,408],{"class":197,"line":396},12,[195,398,399],{"class":205},"    \"monthly\u002F2026-07.xlsx\"",[195,401,308],{"class":231},[195,403,344],{"class":274},[195,405,278],{"class":227},[195,407,349],{"class":281},[195,409,285],{"class":231},[195,411,413],{"class":197,"line":412},13,[195,414,259],{"emptyLinePlaceholder":258},[195,416,418,420,423,425,427,429,431,433,436],{"class":197,"line":417},14,[195,419,296],{"class":231},[195,421,422],{"class":205},"\"region \"",[195,424,302],{"class":231},[195,426,305],{"class":205},[195,428,308],{"class":231},[195,430,311],{"class":205},[195,432,308],{"class":231},[195,434,435],{"class":205},"\"East\"",[195,437,314],{"class":231},[195,439,441,444,446,449,451,454,456,459],{"class":197,"line":440},15,[195,442,443],{"class":205},"              \"total_revenue\"",[195,445,302],{"class":231},[195,447,448],{"class":205},"\"5,402.75\"",[195,450,308],{"class":231},[195,452,453],{"class":205},"\"3140.75\"",[195,455,308],{"class":231},[195,457,458],{"class":205},"\"2980.10\"",[195,460,314],{"class":231},[195,462,464,467,469,472,474,476,478,480],{"class":197,"line":463},16,[195,465,466],{"class":205},"              \"currency\"",[195,468,302],{"class":231},[195,470,471],{"class":205},"\"EUR\"",[195,473,308],{"class":231},[195,475,471],{"class":205},[195,477,308],{"class":231},[195,479,471],{"class":205},[195,481,333],{"class":231},[195,483,485,488,490,492,494,496],{"class":197,"line":484},17,[195,486,487],{"class":205},"    \"monthly\u002F2026-08.xlsx\"",[195,489,308],{"class":231},[195,491,344],{"class":274},[195,493,278],{"class":227},[195,495,349],{"class":281},[195,497,285],{"class":231},[10,499,500,501,504],{},"Note the trailing space in August's ",[14,502,503],{},"region "," — that alone is enough to create a duplicate column.",[181,506,508],{"id":507},"step-1-see-the-drift-before-combining","Step 1 — See the drift before combining",[10,510,511],{},"Compare the headers across files first, so you know what you are dealing with:",[186,513,515],{"className":218,"code":514,"language":220,"meta":191,"style":191},"from pathlib import Path\nimport pandas as pd\n\ndef header_matrix(pattern=\"monthly\u002F*.xlsx\"):\n    \"\"\"Which columns appear in which files?\"\"\"\n    headers = {}\n    for path in sorted(Path().glob(pattern)):\n        headers[path.stem] = list(pd.read_excel(path, nrows=0).columns)\n\n    every = sorted({c for cols in headers.values() for c in cols})\n    return pd.DataFrame(\n        {name: [c in cols for c in every] for name, cols in headers.items()},\n        index=every,\n    )\n\nmatrix = header_matrix()\nprint(matrix.to_string())\n",[14,516,517,527,537,541,561,566,576,593,616,620,653,661,689,699,704,708,718],{"__ignoreMap":191},[195,518,519,521,523,525],{"class":197,"line":198},[195,520,228],{"class":227},[195,522,232],{"class":231},[195,524,235],{"class":227},[195,526,238],{"class":231},[195,528,529,531,533,535],{"class":197,"line":241},[195,530,235],{"class":227},[195,532,246],{"class":231},[195,534,249],{"class":227},[195,536,252],{"class":231},[195,538,539],{"class":197,"line":255},[195,540,259],{"emptyLinePlaceholder":258},[195,542,543,546,550,553,555,558],{"class":197,"line":262},[195,544,545],{"class":227},"def",[195,547,549],{"class":548},"s_Opv"," header_matrix",[195,551,552],{"class":231},"(pattern",[195,554,278],{"class":227},[195,556,557],{"class":205},"\"monthly\u002F*.xlsx\"",[195,559,560],{"class":231},"):\n",[195,562,563],{"class":197,"line":288},[195,564,565],{"class":205},"    \"\"\"Which columns appear in which files?\"\"\"\n",[195,567,568,571,573],{"class":197,"line":293},[195,569,570],{"class":231},"    headers ",[195,572,278],{"class":227},[195,574,575],{"class":231}," {}\n",[195,577,578,581,584,587,590],{"class":197,"line":317},[195,579,580],{"class":227},"    for",[195,582,583],{"class":231}," path ",[195,585,586],{"class":227},"in",[195,588,589],{"class":281}," sorted",[195,591,592],{"class":231},"(Path().glob(pattern)):\n",[195,594,595,598,600,603,606,609,611,613],{"class":197,"line":336},[195,596,597],{"class":231},"        headers[path.stem] ",[195,599,278],{"class":227},[195,601,602],{"class":281}," list",[195,604,605],{"class":231},"(pd.read_excel(path, ",[195,607,608],{"class":274},"nrows",[195,610,278],{"class":227},[195,612,69],{"class":281},[195,614,615],{"class":231},").columns)\n",[195,617,618],{"class":197,"line":354},[195,619,259],{"emptyLinePlaceholder":258},[195,621,622,625,627,629,632,635,638,640,643,645,648,650],{"class":197,"line":359},[195,623,624],{"class":231},"    every ",[195,626,278],{"class":227},[195,628,589],{"class":281},[195,630,631],{"class":231},"({c ",[195,633,634],{"class":227},"for",[195,636,637],{"class":231}," cols ",[195,639,586],{"class":227},[195,641,642],{"class":231}," headers.values() ",[195,644,634],{"class":227},[195,646,647],{"class":231}," c ",[195,649,586],{"class":227},[195,651,652],{"class":231}," cols})\n",[195,654,655,658],{"class":197,"line":378},[195,656,657],{"class":227},"    return",[195,659,660],{"class":231}," pd.DataFrame(\n",[195,662,663,666,668,670,672,674,676,679,681,684,686],{"class":197,"line":396},[195,664,665],{"class":231},"        {name: [c ",[195,667,586],{"class":227},[195,669,637],{"class":231},[195,671,634],{"class":227},[195,673,647],{"class":231},[195,675,586],{"class":227},[195,677,678],{"class":231}," every] ",[195,680,634],{"class":227},[195,682,683],{"class":231}," name, cols ",[195,685,586],{"class":227},[195,687,688],{"class":231}," headers.items()},\n",[195,690,691,694,696],{"class":197,"line":412},[195,692,693],{"class":274},"        index",[195,695,278],{"class":227},[195,697,698],{"class":231},"every,\n",[195,700,701],{"class":197,"line":417},[195,702,703],{"class":231},"    )\n",[195,705,706],{"class":197,"line":440},[195,707,259],{"emptyLinePlaceholder":258},[195,709,710,713,715],{"class":197,"line":463},[195,711,712],{"class":231},"matrix ",[195,714,278],{"class":227},[195,716,717],{"class":231}," header_matrix()\n",[195,719,720,723],{"class":197,"line":484},[195,721,722],{"class":281},"print",[195,724,725],{"class":231},"(matrix.to_string())\n",[10,727,728,729,732,733,308,736,36,739,741],{},"Reading with ",[14,730,731],{},"nrows=0"," fetches only the header, which makes this cheap even over dozens of large files. The output makes the problem obvious: ",[14,734,735],{},"Region",[14,737,738],{},"region",[14,740,503],{}," each appear in exactly one file.",[181,743,745],{"id":744},"step-2-normalise-the-header-names","Step 2 — Normalise the header names",[10,747,748],{},"Most drift is cosmetic — case, spacing, punctuation. Handle it mechanically before anything else:",[186,750,752],{"className":218,"code":751,"language":220,"meta":191,"style":191},"import re\n\ndef normalise_header(name):\n    \"\"\"Canonical form for a column name: lowercase, underscore-separated.\"\"\"\n    text = str(name).strip().lower()\n    text = re.sub(r\"[^\\w]+\", \"_\", text)     # spaces and punctuation to _\n    return re.sub(r\"_+\", \"_\", text).strip(\"_\")\n\nprint(normalise_header(\"region \"))        # region\nprint(normalise_header(\"Total Revenue\"))  # total_revenue\n",[14,753,754,761,765,775,780,793,834,860,864,879],{"__ignoreMap":191},[195,755,756,758],{"class":197,"line":198},[195,757,235],{"class":227},[195,759,760],{"class":231}," re\n",[195,762,763],{"class":197,"line":241},[195,764,259],{"emptyLinePlaceholder":258},[195,766,767,769,772],{"class":197,"line":255},[195,768,545],{"class":227},[195,770,771],{"class":548}," normalise_header",[195,773,774],{"class":231},"(name):\n",[195,776,777],{"class":197,"line":262},[195,778,779],{"class":205},"    \"\"\"Canonical form for a column name: lowercase, underscore-separated.\"\"\"\n",[195,781,782,785,787,790],{"class":197,"line":288},[195,783,784],{"class":231},"    text ",[195,786,278],{"class":227},[195,788,789],{"class":281}," str",[195,791,792],{"class":231},"(name).strip().lower()\n",[195,794,795,797,799,802,805,808,811,814,817,820,822,824,827,830],{"class":197,"line":293},[195,796,784],{"class":231},[195,798,278],{"class":227},[195,800,801],{"class":231}," re.sub(",[195,803,804],{"class":227},"r",[195,806,807],{"class":205},"\"",[195,809,810],{"class":281},"[",[195,812,813],{"class":227},"^",[195,815,816],{"class":281},"\\w]",[195,818,819],{"class":227},"+",[195,821,807],{"class":205},[195,823,308],{"class":231},[195,825,826],{"class":205},"\"_\"",[195,828,829],{"class":231},", text)     ",[195,831,833],{"class":832},"s-wDw","# spaces and punctuation to _\n",[195,835,836,838,840,842,845,847,849,851,853,856,858],{"class":197,"line":317},[195,837,657],{"class":227},[195,839,801],{"class":231},[195,841,804],{"class":227},[195,843,844],{"class":205},"\"_",[195,846,819],{"class":227},[195,848,807],{"class":205},[195,850,308],{"class":231},[195,852,826],{"class":205},[195,854,855],{"class":231},", text).strip(",[195,857,826],{"class":205},[195,859,285],{"class":231},[195,861,862],{"class":197,"line":336},[195,863,259],{"emptyLinePlaceholder":258},[195,865,866,868,871,873,876],{"class":197,"line":354},[195,867,722],{"class":281},[195,869,870],{"class":231},"(normalise_header(",[195,872,422],{"class":205},[195,874,875],{"class":231},"))        ",[195,877,878],{"class":832},"# region\n",[195,880,881,883,885,888,891],{"class":197,"line":359},[195,882,722],{"class":281},[195,884,870],{"class":231},[195,886,887],{"class":205},"\"Total Revenue\"",[195,889,890],{"class":231},"))  ",[195,892,893],{"class":832},"# total_revenue\n",[10,895,896,897,900,901,903],{},"That collapses the accidental variants. The genuine renames — ",[14,898,899],{},"total_revenue"," really meaning the same thing as ",[14,902,16],{}," — need an explicit map, because no rule can infer them:",[186,905,907],{"className":218,"code":906,"language":220,"meta":191,"style":191},"RENAMES = {\n    \"total_revenue\": \"revenue\",\n    \"sales_value\": \"revenue\",\n    \"rev\": \"revenue\",\n    \"area\": \"region\",\n    \"region_name\": \"region\",\n}\n\ndef canonical_columns(df, renames=RENAMES):\n    \"\"\"Normalise header spelling, then apply the known rename map.\"\"\"\n    df = df.rename(columns=normalise_header)\n    return df.rename(columns={k: v for k, v in renames.items()\n                              if k in df.columns})\n",[14,908,909,920,934,945,956,967,978,983,987,1003,1008,1026,1049],{"__ignoreMap":191},[195,910,911,914,917],{"class":197,"line":198},[195,912,913],{"class":281},"RENAMES",[195,915,916],{"class":227}," =",[195,918,919],{"class":231}," {\n",[195,921,922,925,928,931],{"class":197,"line":241},[195,923,924],{"class":205},"    \"total_revenue\"",[195,926,927],{"class":231},": ",[195,929,930],{"class":205},"\"revenue\"",[195,932,933],{"class":231},",\n",[195,935,936,939,941,943],{"class":197,"line":255},[195,937,938],{"class":205},"    \"sales_value\"",[195,940,927],{"class":231},[195,942,930],{"class":205},[195,944,933],{"class":231},[195,946,947,950,952,954],{"class":197,"line":262},[195,948,949],{"class":205},"    \"rev\"",[195,951,927],{"class":231},[195,953,930],{"class":205},[195,955,933],{"class":231},[195,957,958,961,963,965],{"class":197,"line":288},[195,959,960],{"class":205},"    \"area\"",[195,962,927],{"class":231},[195,964,299],{"class":205},[195,966,933],{"class":231},[195,968,969,972,974,976],{"class":197,"line":293},[195,970,971],{"class":205},"    \"region_name\"",[195,973,927],{"class":231},[195,975,299],{"class":205},[195,977,933],{"class":231},[195,979,980],{"class":197,"line":317},[195,981,982],{"class":231},"}\n",[195,984,985],{"class":197,"line":336},[195,986,259],{"emptyLinePlaceholder":258},[195,988,989,991,994,997,999,1001],{"class":197,"line":354},[195,990,545],{"class":227},[195,992,993],{"class":548}," canonical_columns",[195,995,996],{"class":231},"(df, renames",[195,998,278],{"class":227},[195,1000,913],{"class":281},[195,1002,560],{"class":231},[195,1004,1005],{"class":197,"line":359},[195,1006,1007],{"class":205},"    \"\"\"Normalise header spelling, then apply the known rename map.\"\"\"\n",[195,1009,1010,1013,1015,1018,1021,1023],{"class":197,"line":378},[195,1011,1012],{"class":231},"    df ",[195,1014,278],{"class":227},[195,1016,1017],{"class":231}," df.rename(",[195,1019,1020],{"class":274},"columns",[195,1022,278],{"class":227},[195,1024,1025],{"class":231},"normalise_header)\n",[195,1027,1028,1030,1032,1034,1036,1039,1041,1044,1046],{"class":197,"line":396},[195,1029,657],{"class":227},[195,1031,1017],{"class":231},[195,1033,1020],{"class":274},[195,1035,278],{"class":227},[195,1037,1038],{"class":231},"{k: v ",[195,1040,634],{"class":227},[195,1042,1043],{"class":231}," k, v ",[195,1045,586],{"class":227},[195,1047,1048],{"class":231}," renames.items()\n",[195,1050,1051,1054,1057,1059],{"class":197,"line":412},[195,1052,1053],{"class":227},"                              if",[195,1055,1056],{"class":231}," k ",[195,1058,586],{"class":227},[195,1060,1061],{"class":231}," df.columns})\n",[10,1063,1064],{},"Keep the map in one place and treat adding to it as a deliberate act. That is what turns \"the numbers went wrong in August\" into \"August introduced a new spelling, add one line\".",[181,1066,1068],{"id":1067},"step-3-combine-tracking-the-source","Step 3 — Combine, tracking the source",[10,1070,1071],{},"Read, canonicalise, tag, concatenate:",[186,1073,1075],{"className":218,"code":1074,"language":220,"meta":191,"style":191},"from pathlib import Path\nimport pandas as pd\n\ndef combine(pattern=\"monthly\u002F*.xlsx\", renames=RENAMES, required=(\"region\",)):\n    \"\"\"Stack every matching file into one frame, aligned and tagged.\"\"\"\n    frames, seen = [], {}\n\n    for path in sorted(Path().glob(pattern)):\n        df = canonical_columns(pd.read_excel(path), renames)\n\n        missing = set(required) - set(df.columns)\n        if missing:\n            raise ValueError(\n                f\"{path.name} is missing required column(s): \"\n                f\"{', '.join(sorted(missing))}\"\n            )\n\n        df[\"source_file\"] = path.name\n        seen[path.name] = set(df.columns)\n        frames.append(df)\n\n    if not frames:\n        raise FileNotFoundError(f\"no files matched {pattern}\")\n\n    combined = pd.concat(frames, ignore_index=True, sort=False)\n\n    # Report columns that appear in only some files — usually a signal.\n    everywhere = set.intersection(*seen.values())\n    partial = sorted(set(combined.columns) - everywhere)\n    if partial:\n        print(\"columns present in only some files:\", partial)\n        for name in partial:\n            files = [f for f, cols in seen.items() if name in cols]\n            print(f\"  {name:\u003C16} {len(files)}\u002F{len(seen)} file(s): \"\n                  f\"{', '.join(files)}\")\n\n    return combined\n\ncombined = combine()\nprint(combined.head())\n",[14,1076,1077,1087,1097,1101,1134,1139,1149,1153,1165,1175,1179,1200,1208,1219,1239,1264,1269,1273,1290,1302,1308,1313,1325,1353,1358,1387,1392,1398,1417,1440,1448,1462,1475,1506,1555,1576,1581,1589,1594,1605],{"__ignoreMap":191},[195,1078,1079,1081,1083,1085],{"class":197,"line":198},[195,1080,228],{"class":227},[195,1082,232],{"class":231},[195,1084,235],{"class":227},[195,1086,238],{"class":231},[195,1088,1089,1091,1093,1095],{"class":197,"line":241},[195,1090,235],{"class":227},[195,1092,246],{"class":231},[195,1094,249],{"class":227},[195,1096,252],{"class":231},[195,1098,1099],{"class":197,"line":255},[195,1100,259],{"emptyLinePlaceholder":258},[195,1102,1103,1105,1108,1110,1112,1114,1117,1119,1121,1124,1126,1129,1131],{"class":197,"line":262},[195,1104,545],{"class":227},[195,1106,1107],{"class":548}," combine",[195,1109,552],{"class":231},[195,1111,278],{"class":227},[195,1113,557],{"class":205},[195,1115,1116],{"class":231},", renames",[195,1118,278],{"class":227},[195,1120,913],{"class":281},[195,1122,1123],{"class":231},", required",[195,1125,278],{"class":227},[195,1127,1128],{"class":231},"(",[195,1130,299],{"class":205},[195,1132,1133],{"class":231},",)):\n",[195,1135,1136],{"class":197,"line":288},[195,1137,1138],{"class":205},"    \"\"\"Stack every matching file into one frame, aligned and tagged.\"\"\"\n",[195,1140,1141,1144,1146],{"class":197,"line":293},[195,1142,1143],{"class":231},"    frames, seen ",[195,1145,278],{"class":227},[195,1147,1148],{"class":231}," [], {}\n",[195,1150,1151],{"class":197,"line":317},[195,1152,259],{"emptyLinePlaceholder":258},[195,1154,1155,1157,1159,1161,1163],{"class":197,"line":336},[195,1156,580],{"class":227},[195,1158,583],{"class":231},[195,1160,586],{"class":227},[195,1162,589],{"class":281},[195,1164,592],{"class":231},[195,1166,1167,1170,1172],{"class":197,"line":354},[195,1168,1169],{"class":231},"        df ",[195,1171,278],{"class":227},[195,1173,1174],{"class":231}," canonical_columns(pd.read_excel(path), renames)\n",[195,1176,1177],{"class":197,"line":359},[195,1178,259],{"emptyLinePlaceholder":258},[195,1180,1181,1184,1186,1189,1192,1195,1197],{"class":197,"line":378},[195,1182,1183],{"class":231},"        missing ",[195,1185,278],{"class":227},[195,1187,1188],{"class":281}," set",[195,1190,1191],{"class":231},"(required) ",[195,1193,1194],{"class":227},"-",[195,1196,1188],{"class":281},[195,1198,1199],{"class":231},"(df.columns)\n",[195,1201,1202,1205],{"class":197,"line":396},[195,1203,1204],{"class":227},"        if",[195,1206,1207],{"class":231}," missing:\n",[195,1209,1210,1213,1216],{"class":197,"line":412},[195,1211,1212],{"class":227},"            raise",[195,1214,1215],{"class":281}," ValueError",[195,1217,1218],{"class":231},"(\n",[195,1220,1221,1224,1226,1230,1233,1236],{"class":197,"line":417},[195,1222,1223],{"class":227},"                f",[195,1225,807],{"class":205},[195,1227,1229],{"class":1228},"sSjpA","{",[195,1231,1232],{"class":231},"path.name",[195,1234,1235],{"class":1228},"}",[195,1237,1238],{"class":205}," is missing required column(s): \"\n",[195,1240,1241,1243,1245,1247,1250,1253,1256,1259,1261],{"class":197,"line":440},[195,1242,1223],{"class":227},[195,1244,807],{"class":205},[195,1246,1229],{"class":1228},[195,1248,1249],{"class":205},"', '",[195,1251,1252],{"class":231},".join(",[195,1254,1255],{"class":281},"sorted",[195,1257,1258],{"class":231},"(missing))",[195,1260,1235],{"class":1228},[195,1262,1263],{"class":205},"\"\n",[195,1265,1266],{"class":197,"line":463},[195,1267,1268],{"class":231},"            )\n",[195,1270,1271],{"class":197,"line":484},[195,1272,259],{"emptyLinePlaceholder":258},[195,1274,1276,1279,1282,1285,1287],{"class":197,"line":1275},18,[195,1277,1278],{"class":231},"        df[",[195,1280,1281],{"class":205},"\"source_file\"",[195,1283,1284],{"class":231},"] ",[195,1286,278],{"class":227},[195,1288,1289],{"class":231}," path.name\n",[195,1291,1293,1296,1298,1300],{"class":197,"line":1292},19,[195,1294,1295],{"class":231},"        seen[path.name] ",[195,1297,278],{"class":227},[195,1299,1188],{"class":281},[195,1301,1199],{"class":231},[195,1303,1305],{"class":197,"line":1304},20,[195,1306,1307],{"class":231},"        frames.append(df)\n",[195,1309,1311],{"class":197,"line":1310},21,[195,1312,259],{"emptyLinePlaceholder":258},[195,1314,1316,1319,1322],{"class":197,"line":1315},22,[195,1317,1318],{"class":227},"    if",[195,1320,1321],{"class":227}," not",[195,1323,1324],{"class":231}," frames:\n",[195,1326,1328,1331,1334,1336,1339,1342,1344,1347,1349,1351],{"class":197,"line":1327},23,[195,1329,1330],{"class":227},"        raise",[195,1332,1333],{"class":281}," FileNotFoundError",[195,1335,1128],{"class":231},[195,1337,1338],{"class":227},"f",[195,1340,1341],{"class":205},"\"no files matched ",[195,1343,1229],{"class":1228},[195,1345,1346],{"class":231},"pattern",[195,1348,1235],{"class":1228},[195,1350,807],{"class":205},[195,1352,285],{"class":231},[195,1354,1356],{"class":197,"line":1355},24,[195,1357,259],{"emptyLinePlaceholder":258},[195,1359,1361,1364,1366,1369,1372,1374,1376,1378,1381,1383,1385],{"class":197,"line":1360},25,[195,1362,1363],{"class":231},"    combined ",[195,1365,278],{"class":227},[195,1367,1368],{"class":231}," pd.concat(frames, ",[195,1370,1371],{"class":274},"ignore_index",[195,1373,278],{"class":227},[195,1375,282],{"class":281},[195,1377,308],{"class":231},[195,1379,1380],{"class":274},"sort",[195,1382,278],{"class":227},[195,1384,349],{"class":281},[195,1386,285],{"class":231},[195,1388,1390],{"class":197,"line":1389},26,[195,1391,259],{"emptyLinePlaceholder":258},[195,1393,1395],{"class":197,"line":1394},27,[195,1396,1397],{"class":832},"    # Report columns that appear in only some files — usually a signal.\n",[195,1399,1401,1404,1406,1408,1411,1414],{"class":197,"line":1400},28,[195,1402,1403],{"class":231},"    everywhere ",[195,1405,278],{"class":227},[195,1407,1188],{"class":281},[195,1409,1410],{"class":231},".intersection(",[195,1412,1413],{"class":227},"*",[195,1415,1416],{"class":231},"seen.values())\n",[195,1418,1420,1423,1425,1427,1429,1432,1435,1437],{"class":197,"line":1419},29,[195,1421,1422],{"class":231},"    partial ",[195,1424,278],{"class":227},[195,1426,589],{"class":281},[195,1428,1128],{"class":231},[195,1430,1431],{"class":281},"set",[195,1433,1434],{"class":231},"(combined.columns) ",[195,1436,1194],{"class":227},[195,1438,1439],{"class":231}," everywhere)\n",[195,1441,1443,1445],{"class":197,"line":1442},30,[195,1444,1318],{"class":227},[195,1446,1447],{"class":231}," partial:\n",[195,1449,1451,1454,1456,1459],{"class":197,"line":1450},31,[195,1452,1453],{"class":281},"        print",[195,1455,1128],{"class":231},[195,1457,1458],{"class":205},"\"columns present in only some files:\"",[195,1460,1461],{"class":231},", partial)\n",[195,1463,1465,1468,1471,1473],{"class":197,"line":1464},32,[195,1466,1467],{"class":227},"        for",[195,1469,1470],{"class":231}," name ",[195,1472,586],{"class":227},[195,1474,1447],{"class":231},[195,1476,1478,1481,1483,1486,1488,1491,1493,1496,1499,1501,1503],{"class":197,"line":1477},33,[195,1479,1480],{"class":231},"            files ",[195,1482,278],{"class":227},[195,1484,1485],{"class":231}," [f ",[195,1487,634],{"class":227},[195,1489,1490],{"class":231}," f, cols ",[195,1492,586],{"class":227},[195,1494,1495],{"class":231}," seen.items() ",[195,1497,1498],{"class":227},"if",[195,1500,1470],{"class":231},[195,1502,586],{"class":227},[195,1504,1505],{"class":231}," cols]\n",[195,1507,1509,1512,1514,1516,1519,1521,1524,1527,1529,1532,1535,1538,1540,1543,1545,1547,1550,1552],{"class":197,"line":1508},34,[195,1510,1511],{"class":281},"            print",[195,1513,1128],{"class":231},[195,1515,1338],{"class":227},[195,1517,1518],{"class":205},"\"  ",[195,1520,1229],{"class":1228},[195,1522,1523],{"class":231},"name",[195,1525,1526],{"class":227},":\u003C16",[195,1528,1235],{"class":1228},[195,1530,1531],{"class":1228}," {",[195,1533,1534],{"class":281},"len",[195,1536,1537],{"class":231},"(files)",[195,1539,1235],{"class":1228},[195,1541,1542],{"class":205},"\u002F",[195,1544,1229],{"class":1228},[195,1546,1534],{"class":281},[195,1548,1549],{"class":231},"(seen)",[195,1551,1235],{"class":1228},[195,1553,1554],{"class":205}," file(s): \"\n",[195,1556,1558,1561,1563,1565,1567,1570,1572,1574],{"class":197,"line":1557},35,[195,1559,1560],{"class":227},"                  f",[195,1562,807],{"class":205},[195,1564,1229],{"class":1228},[195,1566,1249],{"class":205},[195,1568,1569],{"class":231},".join(files)",[195,1571,1235],{"class":1228},[195,1573,807],{"class":205},[195,1575,285],{"class":231},[195,1577,1579],{"class":197,"line":1578},36,[195,1580,259],{"emptyLinePlaceholder":258},[195,1582,1584,1586],{"class":197,"line":1583},37,[195,1585,657],{"class":227},[195,1587,1588],{"class":231}," combined\n",[195,1590,1592],{"class":197,"line":1591},38,[195,1593,259],{"emptyLinePlaceholder":258},[195,1595,1597,1600,1602],{"class":197,"line":1596},39,[195,1598,1599],{"class":231},"combined ",[195,1601,278],{"class":227},[195,1603,1604],{"class":231}," combine()\n",[195,1606,1608,1610],{"class":197,"line":1607},40,[195,1609,722],{"class":281},[195,1611,1612],{"class":231},"(combined.head())\n",[10,1614,1615,1616,1619],{},"The ",[14,1617,1618],{},"source_file"," column is worth more than it looks. When a total is wrong, the first question is which file contributed the bad rows, and without a source column that question needs the whole pipeline re-run to answer.",[10,1621,1622,1625,1626,1629,1630,1633],{},[14,1623,1624],{},"ignore_index=True"," renumbers the result so the index is unique — without it, three files each starting at zero produce a frame with repeated index values, which then breaks ",[14,1627,1628],{},"loc"," lookups in confusing ways. And ",[14,1631,1632],{},"sort=False"," keeps the column order from the first frame rather than sorting alphabetically, which reads better in the output.",[181,1635,1637],{"id":1636},"step-4-reconcile-the-dtypes","Step 4 — Reconcile the dtypes",[10,1639,1640,1643,1644,1647],{},[14,1641,1642],{},"concat"," aligns names, not types. A column read as float in two files and text in the third becomes ",[14,1645,1646],{},"object",", and every subsequent sum concatenates strings:",[47,1649,57,1655,57,1658,57,1661,57,1664,57,1667,57,1672,57,1674,57,1677,57,1679,57,1683,57,1694,57,1697,57,1703,57,1708,57,1713,57,1715,57,1720,57,1724],{"viewBox":1650,"role":50,"ariaLabel":1651,"ariaLabelledBy":1652,"xmlns":55,"style":56},"0 0 800 226","Dtype collision: two files supply a float revenue column and one supplies text, so the concatenated column becomes object dtype and arithmetic silently breaks.",[1653,1654],"dt-t","dt-d",[59,1656,1657],{"id":1653},"One text column turns the whole combined column into object dtype",[63,1659,1660],{"id":1654},"June and July supply revenue as float64. August supplies it as text because the export wrote formatted strings. Concatenating produces an object column, so summing it concatenates strings rather than adding numbers and no error is raised. Coercing each file's column to a numeric type before combining keeps the result float64.",[67,1662],{"x":69,"y":69,"width":70,"height":1663,"fill":72},"226",[67,1665],{"x":82,"y":83,"width":1666,"height":85,"rx":86,"fill":164,"stroke":165,"style":89},"230",[74,1668,1671],{"x":1669,"y":92,"style":1670},"135","font-size:11px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","June · revenue float64",[67,1673],{"x":82,"y":97,"width":1666,"height":85,"rx":86,"fill":164,"stroke":165,"style":89},[74,1675,1676],{"x":1669,"y":100,"style":1670},"July · revenue float64",[67,1678],{"x":82,"y":104,"width":1666,"height":85,"rx":86,"fill":145,"stroke":146,"style":89},[74,1680,1682],{"x":1669,"y":107,"style":1681},"font-size:11px;font-weight:700;fill:var(--accent-ink,#be185d);text-anchor:middle","August · revenue object",[110,1684,114,1685,114,1688,114,1691,57],{"stroke":112,"style":89,"fill":113},[116,1686],{"d":1687},"M250 59 H 284 V 99 H 314",[116,1689],{"d":1690},"M250 99 H 318",[116,1692],{"d":1693},"M250 139 H 284 V 99",[132,1695],{"points":1696,"fill":130},"326,99 314,93 314,105",[67,1698],{"x":1699,"y":83,"width":1700,"height":1701,"rx":1702,"fill":145,"stroke":146,"style":89},"334","450","52","11",[74,1704,1707],{"x":1705,"y":92,"style":1706},"559","font-size:11.5px;font-weight:700;fill:var(--accent-ink,#be185d);text-anchor:middle","combined revenue → object",[74,1709,1712],{"x":1705,"y":1710,"style":1711},"83","font-size:10.5px;fill:var(--muted,#5b6780);text-anchor:middle","sum() concatenates strings · no error raised",[67,1714],{"x":1699,"y":158,"width":1700,"height":1701,"rx":1702,"fill":164,"stroke":165,"style":89},[74,1716,1719],{"x":1705,"y":1717,"style":1718},"128","font-size:11.5px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","coerce before concat → float64",[74,1721,1723],{"x":1705,"y":1722,"style":1711},"147","to_numeric on each file, then combine",[74,1725,1729],{"x":1726,"y":1727,"style":1728},"400","196","font-size:11px;fill:var(--muted,#5b6780);text-anchor:middle","check combined.dtypes afterwards — an object column where you expect a number is the signal",[186,1731,1733],{"className":218,"code":1732,"language":220,"meta":191,"style":191},"import pandas as pd\n\nSCHEMA = {\n    \"region\": \"string\",\n    \"revenue\": \"float64\",\n    \"currency\": \"string\",\n}\n\ndef coerce(df, schema=SCHEMA):\n    \"\"\"Force each known column to its intended dtype before combining.\"\"\"\n    out = df.copy()\n    for name, dtype in schema.items():\n        if name not in out.columns:\n            continue\n        if dtype.startswith(\"float\") or dtype.startswith(\"Int\"):\n            cleaned = out[name].astype(\"string\").str.replace(\n                r\"[^\\d.\\-]\", \"\", regex=True\n            )\n            out[name] = pd.to_numeric(cleaned, errors=\"coerce\")\n            if dtype.startswith(\"Int\"):\n                out[name] = out[name].astype(dtype)\n        else:\n            out[name] = out[name].astype(dtype)\n    return out\n",[14,1734,1735,1745,1749,1758,1770,1782,1793,1797,1801,1817,1822,1832,1844,1859,1864,1887,1902,1942,1946,1966,1977,1987,1995,2003],{"__ignoreMap":191},[195,1736,1737,1739,1741,1743],{"class":197,"line":198},[195,1738,235],{"class":227},[195,1740,246],{"class":231},[195,1742,249],{"class":227},[195,1744,252],{"class":231},[195,1746,1747],{"class":197,"line":241},[195,1748,259],{"emptyLinePlaceholder":258},[195,1750,1751,1754,1756],{"class":197,"line":255},[195,1752,1753],{"class":281},"SCHEMA",[195,1755,916],{"class":227},[195,1757,919],{"class":231},[195,1759,1760,1763,1765,1768],{"class":197,"line":262},[195,1761,1762],{"class":205},"    \"region\"",[195,1764,927],{"class":231},[195,1766,1767],{"class":205},"\"string\"",[195,1769,933],{"class":231},[195,1771,1772,1775,1777,1780],{"class":197,"line":288},[195,1773,1774],{"class":205},"    \"revenue\"",[195,1776,927],{"class":231},[195,1778,1779],{"class":205},"\"float64\"",[195,1781,933],{"class":231},[195,1783,1784,1787,1789,1791],{"class":197,"line":293},[195,1785,1786],{"class":205},"    \"currency\"",[195,1788,927],{"class":231},[195,1790,1767],{"class":205},[195,1792,933],{"class":231},[195,1794,1795],{"class":197,"line":317},[195,1796,982],{"class":231},[195,1798,1799],{"class":197,"line":336},[195,1800,259],{"emptyLinePlaceholder":258},[195,1802,1803,1805,1808,1811,1813,1815],{"class":197,"line":354},[195,1804,545],{"class":227},[195,1806,1807],{"class":548}," coerce",[195,1809,1810],{"class":231},"(df, schema",[195,1812,278],{"class":227},[195,1814,1753],{"class":281},[195,1816,560],{"class":231},[195,1818,1819],{"class":197,"line":359},[195,1820,1821],{"class":205},"    \"\"\"Force each known column to its intended dtype before combining.\"\"\"\n",[195,1823,1824,1827,1829],{"class":197,"line":378},[195,1825,1826],{"class":231},"    out ",[195,1828,278],{"class":227},[195,1830,1831],{"class":231}," df.copy()\n",[195,1833,1834,1836,1839,1841],{"class":197,"line":396},[195,1835,580],{"class":227},[195,1837,1838],{"class":231}," name, dtype ",[195,1840,586],{"class":227},[195,1842,1843],{"class":231}," schema.items():\n",[195,1845,1846,1848,1850,1853,1856],{"class":197,"line":412},[195,1847,1204],{"class":227},[195,1849,1470],{"class":231},[195,1851,1852],{"class":227},"not",[195,1854,1855],{"class":227}," in",[195,1857,1858],{"class":231}," out.columns:\n",[195,1860,1861],{"class":197,"line":417},[195,1862,1863],{"class":227},"            continue\n",[195,1865,1866,1868,1871,1874,1877,1880,1882,1885],{"class":197,"line":440},[195,1867,1204],{"class":227},[195,1869,1870],{"class":231}," dtype.startswith(",[195,1872,1873],{"class":205},"\"float\"",[195,1875,1876],{"class":231},") ",[195,1878,1879],{"class":227},"or",[195,1881,1870],{"class":231},[195,1883,1884],{"class":205},"\"Int\"",[195,1886,560],{"class":231},[195,1888,1889,1892,1894,1897,1899],{"class":197,"line":463},[195,1890,1891],{"class":231},"            cleaned ",[195,1893,278],{"class":227},[195,1895,1896],{"class":231}," out[name].astype(",[195,1898,1767],{"class":205},[195,1900,1901],{"class":231},").str.replace(\n",[195,1903,1904,1907,1909,1911,1913,1916,1918,1922,1925,1927,1929,1932,1934,1937,1939],{"class":197,"line":484},[195,1905,1906],{"class":227},"                r",[195,1908,807],{"class":205},[195,1910,810],{"class":281},[195,1912,813],{"class":227},[195,1914,1915],{"class":281},"\\d",[195,1917,45],{"class":1228},[195,1919,1921],{"class":1920},"s_b0D","\\-",[195,1923,1924],{"class":281},"]",[195,1926,807],{"class":205},[195,1928,308],{"class":231},[195,1930,1931],{"class":205},"\"\"",[195,1933,308],{"class":231},[195,1935,1936],{"class":274},"regex",[195,1938,278],{"class":227},[195,1940,1941],{"class":281},"True\n",[195,1943,1944],{"class":197,"line":1275},[195,1945,1268],{"class":231},[195,1947,1948,1951,1953,1956,1959,1961,1964],{"class":197,"line":1292},[195,1949,1950],{"class":231},"            out[name] ",[195,1952,278],{"class":227},[195,1954,1955],{"class":231}," pd.to_numeric(cleaned, ",[195,1957,1958],{"class":274},"errors",[195,1960,278],{"class":227},[195,1962,1963],{"class":205},"\"coerce\"",[195,1965,285],{"class":231},[195,1967,1968,1971,1973,1975],{"class":197,"line":1304},[195,1969,1970],{"class":227},"            if",[195,1972,1870],{"class":231},[195,1974,1884],{"class":205},[195,1976,560],{"class":231},[195,1978,1979,1982,1984],{"class":197,"line":1310},[195,1980,1981],{"class":231},"                out[name] ",[195,1983,278],{"class":227},[195,1985,1986],{"class":231}," out[name].astype(dtype)\n",[195,1988,1989,1992],{"class":197,"line":1315},[195,1990,1991],{"class":227},"        else",[195,1993,1994],{"class":231},":\n",[195,1996,1997,1999,2001],{"class":197,"line":1327},[195,1998,1950],{"class":231},[195,2000,278],{"class":227},[195,2002,1986],{"class":231},[195,2004,2005,2007],{"class":197,"line":1355},[195,2006,657],{"class":227},[195,2008,2009],{"class":231}," out\n",[10,2011,2012],{},"Slot it into the reader and the combined frame comes out typed:",[186,2014,2016],{"className":218,"code":2015,"language":220,"meta":191,"style":191},"df = coerce(canonical_columns(pd.read_excel(path)))\n",[14,2017,2018],{"__ignoreMap":191},[195,2019,2020,2023,2025],{"class":197,"line":198},[195,2021,2022],{"class":231},"df ",[195,2024,278],{"class":227},[195,2026,2027],{"class":231}," coerce(canonical_columns(pd.read_excel(path)))\n",[10,2029,2030,2031,2033],{},"Then verify, because a silent ",[14,2032,1646],{}," column is exactly the failure this is meant to prevent:",[186,2035,2037],{"className":218,"code":2036,"language":220,"meta":191,"style":191},"for name, expected in SCHEMA.items():\n    if name in combined.columns and str(combined[name].dtype) != expected:\n        print(f\"WARNING {name}: {combined[name].dtype}, expected {expected}\")\n",[14,2038,2039,2054,2079],{"__ignoreMap":191},[195,2040,2041,2043,2046,2048,2051],{"class":197,"line":198},[195,2042,634],{"class":227},[195,2044,2045],{"class":231}," name, expected ",[195,2047,586],{"class":227},[195,2049,2050],{"class":281}," SCHEMA",[195,2052,2053],{"class":231},".items():\n",[195,2055,2056,2058,2060,2062,2065,2068,2070,2073,2076],{"class":197,"line":241},[195,2057,1318],{"class":227},[195,2059,1470],{"class":231},[195,2061,586],{"class":227},[195,2063,2064],{"class":231}," combined.columns ",[195,2066,2067],{"class":227},"and",[195,2069,789],{"class":281},[195,2071,2072],{"class":231},"(combined[name].dtype) ",[195,2074,2075],{"class":227},"!=",[195,2077,2078],{"class":231}," expected:\n",[195,2080,2081,2083,2085,2087,2090,2092,2094,2096,2098,2100,2103,2105,2108,2110,2113,2115,2117],{"class":197,"line":255},[195,2082,1453],{"class":281},[195,2084,1128],{"class":231},[195,2086,1338],{"class":227},[195,2088,2089],{"class":205},"\"WARNING ",[195,2091,1229],{"class":1228},[195,2093,1523],{"class":231},[195,2095,1235],{"class":1228},[195,2097,927],{"class":205},[195,2099,1229],{"class":1228},[195,2101,2102],{"class":231},"combined[name].dtype",[195,2104,1235],{"class":1228},[195,2106,2107],{"class":205},", expected ",[195,2109,1229],{"class":1228},[195,2111,2112],{"class":231},"expected",[195,2114,1235],{"class":1228},[195,2116,807],{"class":205},[195,2118,285],{"class":231},[10,2120,2121,2122,45],{},"The fuller text-to-number treatment is in ",[41,2123,2125],{"href":2124},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fconvert-excel-text-columns-to-numbers-with-pandas\u002F","converting Excel text columns to numbers",[181,2127,2129],{"id":2128},"step-5-reconcile-the-row-counts","Step 5 — Reconcile the row counts",[47,2131,57,2136,57,2139,57,2142,57,2144,57,2147,57,2153,57,2156,57,2160,57,2163,57,2167,57,2170,57,2175,57,2179,57,2190,57,2201,57,2207,57,2212,57,2215,57,2219,57,2224],{"viewBox":1650,"role":50,"ariaLabel":2132,"ariaLabelledBy":2133,"xmlns":55,"style":56},"Row reconciliation: the sum of each source file's row count must equal the combined frame's row count, catching both a file that failed to read and one read twice.",[2134,2135],"recon-t","recon-d",[59,2137,2138],{"id":2134},"The row count is the cheapest correctness check there is",[63,2140,2141],{"id":2135},"Three source files contributing forty, thirty-eight and forty-two rows should produce a combined frame of one hundred and twenty. If the combined count is lower, a file failed to read or a filter dropped rows. If it is higher, a file was included twice, which is what happens when a glob pattern matches both the original and a backup copy. Grouping the combined frame by its source column shows immediately which file is responsible.",[67,2143],{"x":69,"y":69,"width":70,"height":1663,"fill":72},[67,2145],{"x":82,"y":1701,"width":2146,"height":85,"rx":86,"fill":87,"stroke":88,"style":89},"150",[74,2148,2152],{"x":2149,"y":2150,"style":2151},"95","74","font-size:11px;font-weight:700;fill:var(--brand-strong,#4338ca);text-anchor:middle","June · 40 rows",[67,2154],{"x":82,"y":2155,"width":2146,"height":85,"rx":86,"fill":87,"stroke":88,"style":89},"92",[74,2157,2159],{"x":2149,"y":2158,"style":2151},"114","July · 38 rows",[67,2161],{"x":82,"y":2162,"width":2146,"height":85,"rx":86,"fill":87,"stroke":88,"style":89},"132",[74,2164,2166],{"x":2149,"y":2165,"style":2151},"154","August · 42 rows",[74,2168,278],{"x":1727,"y":2158,"style":2169},"font-size:20px;font-weight:700;fill:var(--brand,#5b5cf0);text-anchor:middle",[67,2171],{"x":177,"y":154,"width":2172,"height":2173,"rx":2174,"fill":164,"stroke":165,"style":89},"164","46","10",[74,2176,2178],{"x":2177,"y":2158,"style":169},"306","expect 120",[110,2180,114,2181,114,2184,114,2187,57],{"stroke":112,"style":89,"fill":113},[116,2182],{"d":2183},"M388 109 H 420 V 54 H 452",[116,2185],{"d":2186},"M388 109 H 424",[116,2188],{"d":2189},"M388 109 H 420 V 164 H 452",[110,2191,114,2192,114,2195,114,2198,57],{"fill":130},[132,2193],{"points":2194},"460,54 448,48 448,60",[132,2196],{"points":2197},"460,109 448,103 448,115",[132,2199],{"points":2200},"460,164 448,158 448,170",[67,2202],{"x":2203,"y":2204,"width":2205,"height":2206,"rx":2174,"fill":145,"stroke":146,"style":89},"468","32","316","44",[74,2208,2211],{"x":2209,"y":2210,"style":1681},"626","60","fewer → a file failed to read",[67,2213],{"x":2203,"y":2214,"width":2205,"height":2206,"rx":2174,"fill":164,"stroke":165,"style":89},"87",[74,2216,2218],{"x":2209,"y":2217,"style":1670},"115","exactly 120 → nothing lost",[67,2220],{"x":2203,"y":2221,"width":2205,"height":2206,"rx":2174,"fill":2222,"stroke":2223,"style":89},"142","#fdefd8","var(--gold,#b4740a)",[74,2225,2228],{"x":2209,"y":2226,"style":2227},"170","font-size:11px;font-weight:700;fill:var(--gold-ink,#7a4e06);text-anchor:middle","more → a file was included twice",[10,2230,2231],{},"A combine should conserve rows exactly. Asserting it catches a file that failed to read and one that got read twice:",[186,2233,2235],{"className":218,"code":2234,"language":220,"meta":191,"style":191},"from pathlib import Path\nimport pandas as pd\n\nexpected = sum(\n    len(pd.read_excel(p)) for p in sorted(Path().glob(\"monthly\u002F*.xlsx\"))\n)\nassert len(combined) == expected, (\n    f\"combined has {len(combined)} rows, source files have {expected}\"\n)\n\nprint(combined.groupby(\"source_file\").size().to_string())\n",[14,2236,2237,2247,2257,2261,2273,2298,2302,2319,2347,2351,2355],{"__ignoreMap":191},[195,2238,2239,2241,2243,2245],{"class":197,"line":198},[195,2240,228],{"class":227},[195,2242,232],{"class":231},[195,2244,235],{"class":227},[195,2246,238],{"class":231},[195,2248,2249,2251,2253,2255],{"class":197,"line":241},[195,2250,235],{"class":227},[195,2252,246],{"class":231},[195,2254,249],{"class":227},[195,2256,252],{"class":231},[195,2258,2259],{"class":197,"line":255},[195,2260,259],{"emptyLinePlaceholder":258},[195,2262,2263,2266,2268,2271],{"class":197,"line":262},[195,2264,2265],{"class":231},"expected ",[195,2267,278],{"class":227},[195,2269,2270],{"class":281}," sum",[195,2272,1218],{"class":231},[195,2274,2275,2278,2281,2283,2286,2288,2290,2293,2295],{"class":197,"line":288},[195,2276,2277],{"class":281},"    len",[195,2279,2280],{"class":231},"(pd.read_excel(p)) ",[195,2282,634],{"class":227},[195,2284,2285],{"class":231}," p ",[195,2287,586],{"class":227},[195,2289,589],{"class":281},[195,2291,2292],{"class":231},"(Path().glob(",[195,2294,557],{"class":205},[195,2296,2297],{"class":231},"))\n",[195,2299,2300],{"class":197,"line":293},[195,2301,285],{"class":231},[195,2303,2304,2307,2310,2313,2316],{"class":197,"line":317},[195,2305,2306],{"class":227},"assert",[195,2308,2309],{"class":281}," len",[195,2311,2312],{"class":231},"(combined) ",[195,2314,2315],{"class":227},"==",[195,2317,2318],{"class":231}," expected, (\n",[195,2320,2321,2324,2327,2329,2331,2334,2336,2339,2341,2343,2345],{"class":197,"line":336},[195,2322,2323],{"class":227},"    f",[195,2325,2326],{"class":205},"\"combined has ",[195,2328,1229],{"class":1228},[195,2330,1534],{"class":281},[195,2332,2333],{"class":231},"(combined)",[195,2335,1235],{"class":1228},[195,2337,2338],{"class":205}," rows, source files have ",[195,2340,1229],{"class":1228},[195,2342,2112],{"class":231},[195,2344,1235],{"class":1228},[195,2346,1263],{"class":205},[195,2348,2349],{"class":197,"line":354},[195,2350,285],{"class":231},[195,2352,2353],{"class":197,"line":359},[195,2354,259],{"emptyLinePlaceholder":258},[195,2356,2357,2359,2362,2364],{"class":197,"line":378},[195,2358,722],{"class":281},[195,2360,2361],{"class":231},"(combined.groupby(",[195,2363,1281],{"class":205},[195,2365,2366],{"class":231},").size().to_string())\n",[10,2368,2369,2370,45],{},"Where each file also carries a total you can check against, comparing sums per source is stronger still — the reconciliation idea developed in ",[41,2371,2373],{"href":2372},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fcompare-two-excel-files-for-differences-with-python\u002F","comparing two Excel files for differences",[181,2375,2377],{"id":2376},"common-pitfalls-and-fixes","Common pitfalls and fixes",[2379,2380,2381,2397],"table",{},[2382,2383,2384],"thead",{},[2385,2386,2387,2391,2394],"tr",{},[2388,2389,2390],"th",{},"Symptom",[2388,2392,2393],{},"Cause",[2388,2395,2396],{},"Fix",[2398,2399,2400,2412,2423,2437,2452,2468,2482,2496],"tbody",{},[2385,2401,2402,2406,2409],{},[2403,2404,2405],"td",{},"Two near-identical columns",[2403,2407,2408],{},"Case or whitespace differs",[2403,2410,2411],{},"Normalise header names first.",[2385,2413,2414,2417,2420],{},[2403,2415,2416],{},"A column is mostly empty",[2403,2418,2419],{},"It only exists in some files",[2403,2421,2422],{},"Report partial columns; add a rename if it is the same field.",[2385,2424,2425,2431,2434],{},[2403,2426,2427,2430],{},[14,2428,2429],{},"sum()"," concatenates strings",[2403,2432,2433],{},"Dtype collision fell back to object",[2403,2435,2436],{},"Coerce each file before combining.",[2385,2438,2439,2442,2447],{},[2403,2440,2441],{},"Repeated index values",[2403,2443,2444,2446],{},[14,2445,1371],{}," not set",[2403,2448,2449,2450,45],{},"Pass ",[14,2451,1624],{},[2385,2453,2454,2457,2464],{},[2403,2455,2456],{},"Columns reordered alphabetically",[2403,2458,2459,2460,2463],{},"Default ",[14,2461,2462],{},"sort=True"," behaviour",[2403,2465,2449,2466,45],{},[14,2467,1632],{},[2385,2469,2470,2473,2476],{},[2403,2471,2472],{},"Cannot tell which file a bad row came from",[2403,2474,2475],{},"No provenance",[2403,2477,2478,2479,2481],{},"Add a ",[14,2480,1618],{}," column.",[2385,2483,2484,2487,2490],{},[2403,2485,2486],{},"Combine is very slow",[2403,2488,2489],{},"Appending in a loop",[2403,2491,2492,2493,2495],{},"Build a list and ",[14,2494,1642],{}," once.",[2385,2497,2498,2501,2504],{},[2403,2499,2500],{},"A file silently contributed nothing",[2403,2502,2503],{},"Read failed or matched no rows",[2403,2505,2506],{},"Assert the row counts reconcile.",[181,2508,2510],{"id":2509},"performance-and-scale-notes","Performance and scale notes",[10,2512,2513,2515],{},[14,2514,32],{}," allocates one new frame and copies each input into it — a single linear pass. Accumulating with repeated concatenation in a loop is quadratic, because every iteration copies everything gathered so far:",[186,2517,2519],{"className":218,"code":2518,"language":220,"meta":191,"style":191},"import pandas as pd\n\n# Wrong: copies the accumulated frame on every iteration.\ncombined = pd.DataFrame()\nfor path in paths:\n    combined = pd.concat([combined, pd.read_excel(path)])\n\n# Right: one allocation, one copy per input.\ncombined = pd.concat([pd.read_excel(p) for p in paths], ignore_index=True)\n",[14,2520,2521,2531,2535,2540,2549,2560,2569,2573,2578],{"__ignoreMap":191},[195,2522,2523,2525,2527,2529],{"class":197,"line":198},[195,2524,235],{"class":227},[195,2526,246],{"class":231},[195,2528,249],{"class":227},[195,2530,252],{"class":231},[195,2532,2533],{"class":197,"line":241},[195,2534,259],{"emptyLinePlaceholder":258},[195,2536,2537],{"class":197,"line":255},[195,2538,2539],{"class":832},"# Wrong: copies the accumulated frame on every iteration.\n",[195,2541,2542,2544,2546],{"class":197,"line":262},[195,2543,1599],{"class":231},[195,2545,278],{"class":227},[195,2547,2548],{"class":231}," pd.DataFrame()\n",[195,2550,2551,2553,2555,2557],{"class":197,"line":288},[195,2552,634],{"class":227},[195,2554,583],{"class":231},[195,2556,586],{"class":227},[195,2558,2559],{"class":231}," paths:\n",[195,2561,2562,2564,2566],{"class":197,"line":293},[195,2563,1363],{"class":231},[195,2565,278],{"class":227},[195,2567,2568],{"class":231}," pd.concat([combined, pd.read_excel(path)])\n",[195,2570,2571],{"class":197,"line":317},[195,2572,259],{"emptyLinePlaceholder":258},[195,2574,2575],{"class":197,"line":336},[195,2576,2577],{"class":832},"# Right: one allocation, one copy per input.\n",[195,2579,2580,2582,2584,2587,2589,2591,2593,2596,2598,2600,2602],{"class":197,"line":354},[195,2581,1599],{"class":231},[195,2583,278],{"class":227},[195,2585,2586],{"class":231}," pd.concat([pd.read_excel(p) ",[195,2588,634],{"class":227},[195,2590,2285],{"class":231},[195,2592,586],{"class":227},[195,2594,2595],{"class":231}," paths], ",[195,2597,1371],{"class":274},[195,2599,278],{"class":227},[195,2601,282],{"class":281},[195,2603,285],{"class":231},[10,2605,2606,2607,2611,2612,2615,2616,2619],{},"Three further habits. ",[2608,2609,2610],"strong",{},"Read only the columns you need"," with ",[14,2613,2614],{},"usecols"," — after canonicalising you know the target names, so a wide export contributes only its relevant columns. ",[2608,2617,2618],{},"Read the files in parallel",", since parsing dominates and each file is independent:",[186,2621,2623],{"className":218,"code":2622,"language":220,"meta":191,"style":191},"from concurrent.futures import ThreadPoolExecutor\nfrom pathlib import Path\nimport pandas as pd\n\ndef read_one(path):\n    df = coerce(canonical_columns(pd.read_excel(path)))\n    df[\"source_file\"] = path.name\n    return df\n\npaths = sorted(Path().glob(\"monthly\u002F*.xlsx\"))\nwith ThreadPoolExecutor(max_workers=4) as pool:\n    combined = pd.concat(list(pool.map(read_one, paths)), ignore_index=True)\n",[14,2624,2625,2637,2647,2657,2661,2671,2679,2692,2699,2703,2718,2741],{"__ignoreMap":191},[195,2626,2627,2629,2632,2634],{"class":197,"line":198},[195,2628,228],{"class":227},[195,2630,2631],{"class":231}," concurrent.futures ",[195,2633,235],{"class":227},[195,2635,2636],{"class":231}," ThreadPoolExecutor\n",[195,2638,2639,2641,2643,2645],{"class":197,"line":241},[195,2640,228],{"class":227},[195,2642,232],{"class":231},[195,2644,235],{"class":227},[195,2646,238],{"class":231},[195,2648,2649,2651,2653,2655],{"class":197,"line":255},[195,2650,235],{"class":227},[195,2652,246],{"class":231},[195,2654,249],{"class":227},[195,2656,252],{"class":231},[195,2658,2659],{"class":197,"line":262},[195,2660,259],{"emptyLinePlaceholder":258},[195,2662,2663,2665,2668],{"class":197,"line":288},[195,2664,545],{"class":227},[195,2666,2667],{"class":548}," read_one",[195,2669,2670],{"class":231},"(path):\n",[195,2672,2673,2675,2677],{"class":197,"line":293},[195,2674,1012],{"class":231},[195,2676,278],{"class":227},[195,2678,2027],{"class":231},[195,2680,2681,2684,2686,2688,2690],{"class":197,"line":317},[195,2682,2683],{"class":231},"    df[",[195,2685,1281],{"class":205},[195,2687,1284],{"class":231},[195,2689,278],{"class":227},[195,2691,1289],{"class":231},[195,2693,2694,2696],{"class":197,"line":336},[195,2695,657],{"class":227},[195,2697,2698],{"class":231}," df\n",[195,2700,2701],{"class":197,"line":354},[195,2702,259],{"emptyLinePlaceholder":258},[195,2704,2705,2708,2710,2712,2714,2716],{"class":197,"line":359},[195,2706,2707],{"class":231},"paths ",[195,2709,278],{"class":227},[195,2711,589],{"class":281},[195,2713,2292],{"class":231},[195,2715,557],{"class":205},[195,2717,2297],{"class":231},[195,2719,2720,2723,2726,2729,2731,2734,2736,2738],{"class":197,"line":378},[195,2721,2722],{"class":227},"with",[195,2724,2725],{"class":231}," ThreadPoolExecutor(",[195,2727,2728],{"class":274},"max_workers",[195,2730,278],{"class":227},[195,2732,2733],{"class":281},"4",[195,2735,1876],{"class":231},[195,2737,249],{"class":227},[195,2739,2740],{"class":231}," pool:\n",[195,2742,2743,2745,2747,2750,2753,2756,2758,2760,2762],{"class":197,"line":396},[195,2744,1363],{"class":231},[195,2746,278],{"class":227},[195,2748,2749],{"class":231}," pd.concat(",[195,2751,2752],{"class":281},"list",[195,2754,2755],{"class":231},"(pool.map(read_one, paths)), ",[195,2757,1371],{"class":274},[195,2759,278],{"class":227},[195,2761,282],{"class":281},[195,2763,285],{"class":231},[10,2765,2766],{},"Threads work here because the Excel parsers release the GIL during I\u002FO and much of the parsing; for very large files, processes are better still.",[10,2768,2769,2772,2773,2775,2776,2779],{},[2608,2770,2771],{},"Convert the category-like columns after combining, not before."," Making ",[14,2774,738],{}," a ",[14,2777,2778],{},"category"," in each frame separately gives each its own categories, and concatenating those falls back to object. Convert once on the result:",[186,2781,2783],{"className":218,"code":2782,"language":220,"meta":191,"style":191},"combined[\"region\"] = combined[\"region\"].astype(\"category\")\n",[14,2784,2785],{"__ignoreMap":191},[195,2786,2787,2790,2792,2794,2796,2799,2801,2804,2807],{"class":197,"line":198},[195,2788,2789],{"class":231},"combined[",[195,2791,299],{"class":205},[195,2793,1284],{"class":231},[195,2795,278],{"class":227},[195,2797,2798],{"class":231}," combined[",[195,2800,299],{"class":205},[195,2802,2803],{"class":231},"].astype(",[195,2805,2806],{"class":205},"\"category\"",[195,2808,285],{"class":231},[10,2810,2811,2812,2816],{},"For a corpus too large to hold at once, write each canonicalised frame to Parquet and let a columnar reader handle the union — or convert the sources once, as described in ",[41,2813,2815],{"href":2814},"\u002Fgetting-started-with-python-excel-automation\u002Fworking-with-multiple-excel-sheets-in-python\u002Fcombine-multiple-excel-files-into-one-python\u002F","combining multiple Excel files into one",", so the expensive parse happens a single time.",[181,2818,2820],{"id":2819},"conclusion","Conclusion",[10,2822,2823],{},"Concatenating files with drifting headers is an alignment problem, not a stacking one. Compare the headers first so you know what changed, normalise the cosmetic variation mechanically, and keep an explicit rename map for the genuine renames that no rule can infer. Tag every row with its source file, coerce the dtypes before combining so a text column cannot poison the result, and report the columns that appear in only some files rather than absorbing them silently. Then assert the row counts reconcile — a combine that quietly dropped a file looks exactly like one that worked.",[181,2825,2827],{"id":2826},"frequently-asked-questions","Frequently asked questions",[10,2829,2830,2833,2835,2836,2839],{},[2608,2831,2832],{},"What happens to columns that only appear in some files?",[14,2834,32],{}," keeps every column it sees and fills the missing ones with ",[14,2837,2838],{},"NaN",". That is usually what you want, but it means a typo in one file's header silently becomes a new mostly-empty column instead of an error.",[10,2841,2842,2845,2846,308,2848,36,2850,2852,2853,2855],{},[2608,2843,2844],{},"How do I stop a renamed column becoming two columns?","\nApply an explicit rename map before concatenating, so ",[14,2847,16],{},[14,2849,20],{},[14,2851,899],{}," all become one canonical name. Relying on ",[14,2854,1642],{}," to align them will not work — it matches on exact strings.",[10,2857,2858,2868,2870,2871,2874],{},[2608,2859,2860,2861,2863,2864,2867],{},"Should I use ",[14,2862,1642],{}," or ",[14,2865,2866],{},"append","?",[14,2869,1642],{},". ",[14,2872,2873],{},"DataFrame.append"," was removed, and concatenating a list of frames in one call is far faster than repeatedly appending, which copies the whole accumulated frame each time.",[10,2876,2877,2880,2881,2884,2885,2887,2888,2891],{},[2608,2878,2879],{},"How do I know which file a row came from?","\nAdd a source column before concatenating, or pass ",[14,2882,2883],{},"keys"," to ",[14,2886,32],{}," to build a hierarchical index. The source column is usually more convenient because it survives a ",[14,2889,2890],{},"reset_index"," and writes to Excel cleanly.",[10,2893,2894,2897,2899,2900,2902],{},[2608,2895,2896],{},"What if the same column has different dtypes across files?",[14,2898,1642],{}," falls back to ",[14,2901,1646],{}," dtype, which silently disables arithmetic. Coerce each column to its intended type before combining, and check dtypes on the result.",[181,2904,2906],{"id":2905},"related","Related",[2908,2909,2910,2917,2923,2930,2936],"ul",{},[2911,2912,2913,2914,2916],"li",{},"Up to the parent: ",[41,2915,44],{"href":43}," — the join operations alongside this stack.",[2911,2918,2919,2922],{},[41,2920,2921],{"href":2814},"Combine Multiple Excel Files into One with Python"," — the simpler case where headers match.",[2911,2924,2925,2929],{},[41,2926,2928],{"href":2927},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Ffind-rows-in-one-excel-file-missing-from-another\u002F","Find Rows in One Excel File Missing from Another"," — reconciling the combined result.",[2911,2931,2932,2935],{},[41,2933,2934],{"href":2124},"Convert Excel Text Columns to Numbers with pandas"," — fixing the dtype collisions.",[2911,2937,2938,2942],{},[41,2939,2941],{"href":2940},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fvalidate-excel-columns-before-import-with-pandas\u002F","Validate Excel Columns Before Import with pandas"," — failing loudly when a file's shape changes.",[2944,2945,2946],"style",{},"html pre.shiki code .sMTad, html code.shiki .sMTad{--shiki-default:#6F42C1;--shiki-dark:#FFB757}html pre.shiki code .srMev, html code.shiki .srMev{--shiki-default:#032F62;--shiki-dark:#ADDCFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .s-kum, html code.shiki .s-kum{--shiki-default:#D73A49;--shiki-dark:#FF9492}html pre.shiki code .skGVy, html code.shiki .skGVy{--shiki-default:#24292E;--shiki-dark:#F0F3F6}html pre.shiki code .sa561, html code.shiki .sa561{--shiki-default:#E36209;--shiki-dark:#FFB757}html pre.shiki code .sP0c6, html code.shiki .sP0c6{--shiki-default:#005CC5;--shiki-dark:#91CBFF}html pre.shiki code .s_Opv, html code.shiki .s_Opv{--shiki-default:#6F42C1;--shiki-dark:#DBB7FF}html pre.shiki code .s-wDw, html code.shiki .s-wDw{--shiki-default:#6A737D;--shiki-dark:#BDC4CC}html pre.shiki code .sSjpA, html code.shiki .sSjpA{--shiki-default:#005CC5;--shiki-dark:#FF9492}html pre.shiki code .s_b0D, html code.shiki .s_b0D{--shiki-default:#22863A;--shiki-default-font-weight:bold;--shiki-dark:#72F088;--shiki-dark-font-weight:bold}",{"title":191,"searchDepth":241,"depth":241,"links":2948},[2949,2950,2951,2952,2953,2954,2955,2956,2957,2958,2959],{"id":183,"depth":241,"text":184},{"id":507,"depth":241,"text":508},{"id":744,"depth":241,"text":745},{"id":1067,"depth":241,"text":1068},{"id":1636,"depth":241,"text":1637},{"id":2128,"depth":241,"text":2129},{"id":2376,"depth":241,"text":2377},{"id":2509,"depth":241,"text":2510},{"id":2819,"depth":241,"text":2820},{"id":2826,"depth":241,"text":2827},{"id":2905,"depth":241,"text":2906},"2026-08-15","Stack monthly exports whose columns drift — align headers, map renamed fields, keep a source column, and report what each file was missing before you combine them.","md",[2964,2966,2968,2971,2973],{"q":2832,"a":2965},"pd.concat keeps every column it sees and fills the missing ones with NaN. That is usually what you want, but it means a typo in one file's header silently becomes a new mostly-empty column instead of an error.",{"q":2844,"a":2967},"Apply an explicit rename map before concatenating, so revenue, Revenue and total_revenue all become one canonical name. Relying on concat to align them will not work — it matches on exact strings.",{"q":2969,"a":2970},"Should I use concat or append?","concat. DataFrame.append was removed, and concatenating a list of frames in one call is far faster than repeatedly appending, which copies the whole accumulated frame each time.",{"q":2879,"a":2972},"Add a source column before concatenating, or pass keys to pd.concat to build a hierarchical index. The source column is usually more convenient because it survives a reset_index and writes to Excel cleanly.",{"q":2896,"a":2974},"concat falls back to object dtype, which silently disables arithmetic. Coerce each column to its intended type before combining, and check dtypes on the result.",{},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Fconcatenate-excel-sheets-with-different-columns",{"title":2978,"description":2979},"Concatenate Excel Sheets with Different Columns (pandas)","Combine Excel files whose headers changed between months: normalise names, apply a rename map, use pd.concat safely, track the source, and report unexpected columns.","concatenate-excel-sheets-with-different-columns","advanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Fconcatenate-excel-sheets-with-different-columns\u002Findex","how-to","aTUznR2GVhh6bW8H2E6-8fP-7XxRxJQ_-5tNy1z-C_g",[2985,2989],{"title":2986,"path":2987,"stem":2988,"children":-1},"Merging and Joining Excel DataFrames with Pandas","\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes","advanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Findex",{"title":2928,"path":2990,"stem":2991,"children":-1},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Ffind-rows-in-one-excel-file-missing-from-another","advanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Ffind-rows-in-one-excel-file-missing-from-another\u002Findex",1786800028621]