[{"data":1,"prerenderedAt":2128},["ShallowReactive",2],{"doc:\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Fmerge-on-keys-that-differ-in-case-and-spacing-with-pandas":3,"surround:\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Fmerge-on-keys-that-differ-in-case-and-spacing-with-pandas":2094,"ancestors:\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Fmerge-on-keys-that-differ-in-case-and-spacing-with-pandas":2103,"children:\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Fmerge-on-keys-that-differ-in-case-and-spacing-with-pandas":2108,"related:\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Fmerge-on-keys-that-differ-in-case-and-spacing-with-pandas":2109},{"id":4,"title":5,"body":6,"dateModified":2059,"datePublished":2059,"description":2060,"extension":2061,"faq":2062,"keyword":2071,"kind":2072,"meta":2073,"navigation":127,"path":2083,"seo":2084,"slug":2087,"stem":2088,"topics":2089,"type":2092,"__hash__":2093},"docs\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Fmerge-on-keys-that-differ-in-case-and-spacing-with-pandas\u002Findex.md","Merge on Keys That Differ in Case and Spacing with pandas",{"type":7,"value":8,"toc":2045},"minimark",[9,36,41,69,95,99,311,345,349,556,586,740,744,761,831,841,973,990,1119,1123,1141,1293,1320,1324,1327,1427,1438,1503,1507,1510,1877,1880,1884,1912,1916,1960,1964,1980,1984,1990,1996,2002,2008,2012,2041],[10,11,12,13,17,18,21,22,25,26,29,30,35],"p",{},"The orders file and the product master both have a product code. ",[14,15,16],"code",{},"orders.merge(products, on=\"SKU\")"," should match every order, and matches about half. Look at the unmatched rows and the codes look identical — until you notice that one side has ",[14,19,20],{},"\"SKU-001 \""," with a trailing space, the other ",[14,23,24],{},"\"sku-001\""," in lower case; one export dropped the leading zeros from numeric IDs; and one cell copied from a web page contains a non-breaking space that looks exactly like a normal one. ",[14,27,28],{},"merge"," compares keys exactly, character for character, so every one of these is a non-match. The fix is a normalised join key on both sides, used for matching only, with the original values kept for display. This page belongs to ",[31,32,34],"a",{"href":33},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002F","Merging and Joining Excel DataFrames",".",[37,38,40],"h2",{"id":39},"prerequisites","Prerequisites",[42,43,48],"pre",{"className":44,"code":45,"language":46,"meta":47,"style":47},"language-bash shiki shiki-themes github-light github-dark-high-contrast","pip install pandas openpyxl\n","bash","",[14,49,50],{"__ignoreMap":47},[51,52,55,59,63,66],"span",{"class":53,"line":54},"line",1,[51,56,58],{"class":57},"sMTad","pip",[51,60,62],{"class":61},"srMev"," install",[51,64,65],{"class":61}," pandas",[51,67,68],{"class":61}," openpyxl\n",[10,70,71,74,75,78,79,82,83,86,87,90,91,94],{},[14,72,73],{},"merge_inputs.xlsx"," has an ",[14,76,77],{},"Orders"," sheet, a ",[14,80,81],{},"Products"," master and a ",[14,84,85],{},"Customers"," master. The order SKUs carry every common defect — trailing space, lower case, a non-breaking space, a zero-width space at the end, a space instead of a hyphen — plus one genuinely unknown code. The customer numbers sometimes lost their leading zeros. And the product master has a trap of its own: ",[14,88,89],{},"Sku-006"," and ",[14,92,93],{},"SKU-006"," are two rows that differ only in case.",[37,96,98],{"id":97},"step-1-see-how-much-an-exact-merge-misses","Step 1 — See how much an exact merge misses",[42,100,104],{"className":101,"code":102,"language":103,"meta":47,"style":47},"language-python shiki shiki-themes github-light github-dark-high-contrast","import pandas as pd\n\nbook = pd.read_excel(\"merge_inputs.xlsx\", sheet_name=None, dtype=str)\norders, products, customers = book[\"Orders\"], book[\"Products\"], book[\"Customers\"]\nexact = orders.merge(products, on=\"SKU\", how=\"left\", indicator=True)\nprint(exact[[\"OrderID\", \"SKU\", \"Product\", \"_merge\"]].assign(SKU=exact[\"SKU\"].map(repr)).to_string(index=False))\nprint(exact[\"_merge\"].value_counts().to_dict())\n","python",[14,105,106,122,129,170,198,239,298],{"__ignoreMap":47},[51,107,108,112,116,119],{"class":53,"line":54},[51,109,111],{"class":110},"s-kum","import",[51,113,115],{"class":114},"skGVy"," pandas ",[51,117,118],{"class":110},"as",[51,120,121],{"class":114}," pd\n",[51,123,125],{"class":53,"line":124},2,[51,126,128],{"emptyLinePlaceholder":127},true,"\n",[51,130,132,135,138,141,144,147,151,153,157,159,162,164,167],{"class":53,"line":131},3,[51,133,134],{"class":114},"book ",[51,136,137],{"class":110},"=",[51,139,140],{"class":114}," pd.read_excel(",[51,142,143],{"class":61},"\"merge_inputs.xlsx\"",[51,145,146],{"class":114},", ",[51,148,150],{"class":149},"sa561","sheet_name",[51,152,137],{"class":110},[51,154,156],{"class":155},"sP0c6","None",[51,158,146],{"class":114},[51,160,161],{"class":149},"dtype",[51,163,137],{"class":110},[51,165,166],{"class":155},"str",[51,168,169],{"class":114},")\n",[51,171,173,176,178,181,184,187,190,192,195],{"class":53,"line":172},4,[51,174,175],{"class":114},"orders, products, customers ",[51,177,137],{"class":110},[51,179,180],{"class":114}," book[",[51,182,183],{"class":61},"\"Orders\"",[51,185,186],{"class":114},"], book[",[51,188,189],{"class":61},"\"Products\"",[51,191,186],{"class":114},[51,193,194],{"class":61},"\"Customers\"",[51,196,197],{"class":114},"]\n",[51,199,201,204,206,209,212,214,217,219,222,224,227,229,232,234,237],{"class":53,"line":200},5,[51,202,203],{"class":114},"exact ",[51,205,137],{"class":110},[51,207,208],{"class":114}," orders.merge(products, ",[51,210,211],{"class":149},"on",[51,213,137],{"class":110},[51,215,216],{"class":61},"\"SKU\"",[51,218,146],{"class":114},[51,220,221],{"class":149},"how",[51,223,137],{"class":110},[51,225,226],{"class":61},"\"left\"",[51,228,146],{"class":114},[51,230,231],{"class":149},"indicator",[51,233,137],{"class":110},[51,235,236],{"class":155},"True",[51,238,169],{"class":114},[51,240,242,245,248,251,253,255,257,260,262,265,268,271,273,276,278,281,284,287,290,292,295],{"class":53,"line":241},6,[51,243,244],{"class":155},"print",[51,246,247],{"class":114},"(exact[[",[51,249,250],{"class":61},"\"OrderID\"",[51,252,146],{"class":114},[51,254,216],{"class":61},[51,256,146],{"class":114},[51,258,259],{"class":61},"\"Product\"",[51,261,146],{"class":114},[51,263,264],{"class":61},"\"_merge\"",[51,266,267],{"class":114},"]].assign(",[51,269,270],{"class":149},"SKU",[51,272,137],{"class":110},[51,274,275],{"class":114},"exact[",[51,277,216],{"class":61},[51,279,280],{"class":114},"].map(",[51,282,283],{"class":155},"repr",[51,285,286],{"class":114},")).to_string(",[51,288,289],{"class":149},"index",[51,291,137],{"class":110},[51,293,294],{"class":155},"False",[51,296,297],{"class":114},"))\n",[51,299,301,303,306,308],{"class":53,"line":300},7,[51,302,244],{"class":155},[51,304,305],{"class":114},"(exact[",[51,307,264],{"class":61},[51,309,310],{"class":114},"].value_counts().to_dict())\n",[10,312,313,314,317,318,321,322,325,326,329,330,332,333,336,337,340,341,344],{},"Reading everything as text (",[14,315,316],{},"dtype=str",") prevents pandas from turning ",[14,319,320],{},"00412"," into the number 412 before you get a chance to decide. ",[14,323,324],{},"indicator=True"," adds a ",[14,327,328],{},"_merge"," column saying whether each row matched. Printing the keys with ",[14,331,283],{}," reveals what the eye cannot: ",[14,334,335],{},"'SKU\\xa0003'"," contains a non-breaking space and ",[14,338,339],{},"'SKU-004​'"," a zero-width space. None of the seven orders matched — not even ",[14,342,343],{},"SKU-001",", whose trailing space is enough to make it different.",[37,346,348],{"id":347},"step-2-build-a-normalised-key","Step 2 — Build a normalised key",[42,350,352],{"className":101,"code":351,"language":103,"meta":47,"style":47},"import re\nimport unicodedata\n\ndef norm_key(value):\n    s = unicodedata.normalize(\"NFKC\", str(value))\n    s = s.replace(\"​\", \"\").replace(\"\", \"\")       # zero-width space, BOM\n    s = re.sub(r\"[\\s\\-_]+\", \"\", s)                           # spaces (incl. NBSP after NFKC), hyphens, underscores\n    return s.casefold()\n\nfor raw in [\"SKU-001 \", \"sku-002\", \"SKU 003\", \"SKU-004​\", \"SKU 005\"]:\n    print(repr(raw), \"->\", norm_key(raw))\n",[14,353,354,361,368,372,384,404,438,482,491,496,536],{"__ignoreMap":47},[51,355,356,358],{"class":53,"line":54},[51,357,111],{"class":110},[51,359,360],{"class":114}," re\n",[51,362,363,365],{"class":53,"line":124},[51,364,111],{"class":110},[51,366,367],{"class":114}," unicodedata\n",[51,369,370],{"class":53,"line":131},[51,371,128],{"emptyLinePlaceholder":127},[51,373,374,377,381],{"class":53,"line":172},[51,375,376],{"class":110},"def",[51,378,380],{"class":379},"s_Opv"," norm_key",[51,382,383],{"class":114},"(value):\n",[51,385,386,389,391,394,397,399,401],{"class":53,"line":200},[51,387,388],{"class":114},"    s ",[51,390,137],{"class":110},[51,392,393],{"class":114}," unicodedata.normalize(",[51,395,396],{"class":61},"\"NFKC\"",[51,398,146],{"class":114},[51,400,166],{"class":155},[51,402,403],{"class":114},"(value))\n",[51,405,406,408,410,413,416,418,421,424,427,429,431,434],{"class":53,"line":241},[51,407,388],{"class":114},[51,409,137],{"class":110},[51,411,412],{"class":114}," s.replace(",[51,414,415],{"class":61},"\"​\"",[51,417,146],{"class":114},[51,419,420],{"class":61},"\"\"",[51,422,423],{"class":114},").replace(",[51,425,426],{"class":61},"\"\"",[51,428,146],{"class":114},[51,430,420],{"class":61},[51,432,433],{"class":114},")       ",[51,435,437],{"class":436},"s-wDw","# zero-width space, BOM\n",[51,439,440,442,444,447,450,453,456,460,464,467,470,472,474,476,479],{"class":53,"line":300},[51,441,388],{"class":114},[51,443,137],{"class":110},[51,445,446],{"class":114}," re.sub(",[51,448,449],{"class":110},"r",[51,451,452],{"class":61},"\"",[51,454,455],{"class":155},"[\\s",[51,457,459],{"class":458},"s_b0D","\\-",[51,461,463],{"class":462},"sSjpA","_",[51,465,466],{"class":155},"]",[51,468,469],{"class":110},"+",[51,471,452],{"class":61},[51,473,146],{"class":114},[51,475,420],{"class":61},[51,477,478],{"class":114},", s)                           ",[51,480,481],{"class":436},"# spaces (incl. NBSP after NFKC), hyphens, underscores\n",[51,483,485,488],{"class":53,"line":484},8,[51,486,487],{"class":110},"    return",[51,489,490],{"class":114}," s.casefold()\n",[51,492,494],{"class":53,"line":493},9,[51,495,128],{"emptyLinePlaceholder":127},[51,497,499,502,505,508,511,513,515,518,520,523,525,528,530,533],{"class":53,"line":498},10,[51,500,501],{"class":110},"for",[51,503,504],{"class":114}," raw ",[51,506,507],{"class":110},"in",[51,509,510],{"class":114}," [",[51,512,20],{"class":61},[51,514,146],{"class":114},[51,516,517],{"class":61},"\"sku-002\"",[51,519,146],{"class":114},[51,521,522],{"class":61},"\"SKU 003\"",[51,524,146],{"class":114},[51,526,527],{"class":61},"\"SKU-004​\"",[51,529,146],{"class":114},[51,531,532],{"class":61},"\"SKU 005\"",[51,534,535],{"class":114},"]:\n",[51,537,539,542,545,547,550,553],{"class":53,"line":538},11,[51,540,541],{"class":155},"    print",[51,543,544],{"class":114},"(",[51,546,283],{"class":155},[51,548,549],{"class":114},"(raw), ",[51,551,552],{"class":61},"\"->\"",[51,554,555],{"class":114},", norm_key(raw))\n",[10,557,558,559,562,563,146,566,90,569,572,573,576,577,580,581,585],{},"Each step removes one class of difference. ",[14,560,561],{},"NFKC"," normalisation turns lookalike characters — the non-breaking space, full-width letters pasted from some systems — into their plain forms. The zero-width characters are removed explicitly because no normalisation touches them. Spaces, hyphens and underscores are dropped entirely, so ",[14,564,565],{},"SKU 005",[14,567,568],{},"SKU-005",[14,570,571],{},"SKU_005"," agree. ",[14,574,575],{},"casefold"," is a stronger ",[14,578,579],{},"lower"," that also handles characters such as the German ß. The rule must be the ",[582,583,584],"em",{},"same function"," on both sides — that is what makes the keys comparable.",[587,588,596,597,596,601,596,605,596,613,596,621,596,628,596,634,596,639,596,642,596,647,596,652,596,656,596,659,596,663,596,666,596,670,596,675,596,679,596,683,596,687,596,689,596,692,596,694,596,697,596,699,596,702,596,704,596,707,596,712,596,715,596,721,596,726,596,734],"svg",{"viewBox":589,"role":590,"ariaLabelledBy":591,"xmlns":594,"style":595},"192 0 376 224","img",[592,593],"mk-norm-t","mk-norm-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;max-width:376px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif","\n  ",[598,599,600],"title",{"id":592},"Five spellings normalised to one key format",[602,603,604],"desc",{"id":593},"On the left five order SKUs with a trailing space, lower case, a non-breaking space, a zero-width space and a space instead of a hyphen. On the right their normalised keys sku001 to sku005, all in the same format.",[606,607],"rect",{"x":608,"y":609,"width":610,"height":611,"fill":612},"192","0","376","224","#ffffff",[606,614],{"x":615,"y":616,"width":617,"height":618,"rx":609,"fill":612,"stroke":619,"style":620},"208.2","40.0","122.1","28.0","var(--line,#cdd5e6)","stroke-width:1px",[622,623,627],"text",{"x":624,"y":625,"style":626},"269.2","59.0","font-size:10.5px;font-weight:700;fill:var(--text,#172033);text-anchor:middle","raw",[606,629],{"x":615,"y":630,"width":617,"height":618,"rx":609,"fill":631,"stroke":632,"style":633},"68.0","#fdefd8","var(--gold,#b4740a)","stroke-width:1.5px",[622,635,638],{"x":624,"y":636,"style":637},"87.0","font-size:10.5px;font-weight:700;fill:var(--gold-ink,#7a4e06);text-anchor:middle","'SKU-001 '",[606,640],{"x":615,"y":641,"width":617,"height":618,"rx":609,"fill":612,"stroke":619,"style":620},"96.0",[622,643,646],{"x":624,"y":644,"style":645},"115.0","font-size:10.5px;font-weight:400;fill:var(--text,#172033);text-anchor:middle","'sku-002'",[606,648],{"x":615,"y":649,"width":617,"height":618,"rx":609,"fill":650,"stroke":651,"style":633},"124.0","#fce9e9","var(--danger,#dc2626)",[622,653,335],{"x":624,"y":654,"style":655},"143.0","font-size:10.5px;font-weight:700;fill:var(--accent-ink,#be185d);text-anchor:middle",[606,657],{"x":615,"y":658,"width":617,"height":618,"rx":609,"fill":650,"stroke":651,"style":633},"152.0",[622,660,662],{"x":624,"y":661,"style":655},"171.0","'SKU-004\\u200b'",[606,664],{"x":615,"y":665,"width":617,"height":618,"rx":609,"fill":612,"stroke":619,"style":620},"180.0",[622,667,669],{"x":624,"y":668,"style":645},"199.0","'SKU 005'",[606,671],{"x":672,"y":616,"width":673,"height":618,"rx":609,"fill":674,"stroke":619,"style":633},"486.1","65.8","#e7ebef",[622,676,678],{"x":677,"y":625,"style":626},"519.0","key",[606,680],{"x":672,"y":630,"width":673,"height":618,"rx":609,"fill":681,"stroke":682,"style":633},"#d9f4f1","var(--teal,#0f9488)",[622,684,686],{"x":677,"y":636,"style":685},"font-size:10.5px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","sku001",[606,688],{"x":672,"y":641,"width":673,"height":618,"rx":609,"fill":681,"stroke":682,"style":633},[622,690,691],{"x":677,"y":644,"style":685},"sku002",[606,693],{"x":672,"y":649,"width":673,"height":618,"rx":609,"fill":681,"stroke":682,"style":633},[622,695,696],{"x":677,"y":654,"style":685},"sku003",[606,698],{"x":672,"y":658,"width":673,"height":618,"rx":609,"fill":681,"stroke":682,"style":633},[622,700,701],{"x":677,"y":661,"style":685},"sku004",[606,703],{"x":672,"y":665,"width":673,"height":618,"rx":609,"fill":681,"stroke":682,"style":633},[622,705,706],{"x":677,"y":668,"style":685},"sku005",[622,708,711],{"x":624,"y":709,"style":710},"29.0","font-size:12.5px;font-weight:700;fill:var(--muted,#5b6780);text-anchor:middle","raw SKU (repr)",[622,713,714],{"x":677,"y":709,"style":710},"join key",[53,716],{"x1":717,"y1":649,"x2":718,"y2":649,"stroke":719,"style":720},"340.2","472.1","var(--brand,#5b5cf0)","stroke-width:2px",[722,723],"polygon",{"points":724,"fill":725},"476.1,124.0 467.1,129.0 467.1,119.0","#5b5cf0",[606,727],{"x":728,"y":729,"width":730,"height":731,"rx":732,"fill":612,"stroke":619,"style":733},"361.8","91.0","92.7","22.0","11","stroke-width:1.2px",[622,735,739],{"x":736,"y":737,"style":738},"408.2","106.0","font-size:11.5px;font-weight:400;fill:var(--brand-strong,#4338ca);text-anchor:middle;font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace","norm_key()",[37,741,743],{"id":742},"step-3-check-for-collisions-before-merging","Step 3 — Check for collisions before merging",[10,745,746,747,750,751,753,754,756,757,760],{},"Normalising can make two ",[582,748,749],{},"different"," master records identical. In the product master, ",[14,752,89],{}," (an old record) and ",[14,755,93],{}," both become ",[14,758,759],{},"sku006",". Merging on that key would duplicate the matching order. Find such collisions first:",[42,762,764],{"className":101,"code":763,"language":103,"meta":47,"style":47},"products[\"Key\"] = products[\"SKU\"].map(norm_key)\ncollisions = products[products[\"Key\"].duplicated(keep=False)].sort_values(\"Key\")\nprint(collisions.to_string(index=False))\n",[14,765,766,787,816],{"__ignoreMap":47},[51,767,768,771,774,777,779,782,784],{"class":53,"line":54},[51,769,770],{"class":114},"products[",[51,772,773],{"class":61},"\"Key\"",[51,775,776],{"class":114},"] ",[51,778,137],{"class":110},[51,780,781],{"class":114}," products[",[51,783,216],{"class":61},[51,785,786],{"class":114},"].map(norm_key)\n",[51,788,789,792,794,797,799,802,805,807,809,812,814],{"class":53,"line":124},[51,790,791],{"class":114},"collisions ",[51,793,137],{"class":110},[51,795,796],{"class":114}," products[products[",[51,798,773],{"class":61},[51,800,801],{"class":114},"].duplicated(",[51,803,804],{"class":149},"keep",[51,806,137],{"class":110},[51,808,294],{"class":155},[51,810,811],{"class":114},")].sort_values(",[51,813,773],{"class":61},[51,815,169],{"class":114},[51,817,818,820,823,825,827,829],{"class":53,"line":131},[51,819,244],{"class":155},[51,821,822],{"class":114},"(collisions.to_string(",[51,824,289],{"class":149},[51,826,137],{"class":110},[51,828,294],{"class":155},[51,830,297],{"class":114},[10,832,833,834,836,837,840],{},"The master has two rows for ",[14,835,759],{},", and they disagree on the product name. That is a data-quality problem for the owner of the master, not something a script should guess at. Resolve it — here by keeping the current record — and enforce uniqueness on every later run with ",[14,838,839],{},"validate",":",[42,842,844],{"className":101,"code":843,"language":103,"meta":47,"style":47},"products_clean = products[products[\"SKU\"] != \"Sku-006\"]\norders[\"Key\"] = orders[\"SKU\"].map(norm_key)\njoined = orders.merge(products_clean[[\"Key\", \"Product\"]], on=\"Key\", how=\"left\",\n                      validate=\"many_to_one\", indicator=True)\nprint(joined[[\"OrderID\", \"SKU\", \"Product\", \"_merge\"]].to_string(index=False))\n",[14,845,846,867,885,921,941],{"__ignoreMap":47},[51,847,848,851,853,855,857,859,862,865],{"class":53,"line":54},[51,849,850],{"class":114},"products_clean ",[51,852,137],{"class":110},[51,854,796],{"class":114},[51,856,216],{"class":61},[51,858,776],{"class":114},[51,860,861],{"class":110},"!=",[51,863,864],{"class":61}," \"Sku-006\"",[51,866,197],{"class":114},[51,868,869,872,874,876,878,881,883],{"class":53,"line":124},[51,870,871],{"class":114},"orders[",[51,873,773],{"class":61},[51,875,776],{"class":114},[51,877,137],{"class":110},[51,879,880],{"class":114}," orders[",[51,882,216],{"class":61},[51,884,786],{"class":114},[51,886,887,890,892,895,897,899,901,904,906,908,910,912,914,916,918],{"class":53,"line":131},[51,888,889],{"class":114},"joined ",[51,891,137],{"class":110},[51,893,894],{"class":114}," orders.merge(products_clean[[",[51,896,773],{"class":61},[51,898,146],{"class":114},[51,900,259],{"class":61},[51,902,903],{"class":114},"]], ",[51,905,211],{"class":149},[51,907,137],{"class":110},[51,909,773],{"class":61},[51,911,146],{"class":114},[51,913,221],{"class":149},[51,915,137],{"class":110},[51,917,226],{"class":61},[51,919,920],{"class":114},",\n",[51,922,923,926,928,931,933,935,937,939],{"class":53,"line":172},[51,924,925],{"class":149},"                      validate",[51,927,137],{"class":110},[51,929,930],{"class":61},"\"many_to_one\"",[51,932,146],{"class":114},[51,934,231],{"class":149},[51,936,137],{"class":110},[51,938,236],{"class":155},[51,940,169],{"class":114},[51,942,943,945,948,950,952,954,956,958,960,962,965,967,969,971],{"class":53,"line":200},[51,944,244],{"class":155},[51,946,947],{"class":114},"(joined[[",[51,949,250],{"class":61},[51,951,146],{"class":114},[51,953,216],{"class":61},[51,955,146],{"class":114},[51,957,259],{"class":61},[51,959,146],{"class":114},[51,961,264],{"class":61},[51,963,964],{"class":114},"]].to_string(",[51,966,289],{"class":149},[51,968,137],{"class":110},[51,970,294],{"class":155},[51,972,297],{"class":114},[10,974,975,978,979,982,983,985,986,35],{},[14,976,977],{},"validate=\"many_to_one\""," raises a ",[14,980,981],{},"MergeError"," if the right-hand keys are not unique, so a new collision in next month's master stops the job instead of silently doubling orders. Now six of the seven orders match; the original ",[14,984,270],{}," column is untouched for display. More on merge validation in ",[31,987,989],{"href":988},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Fvalidate-a-merge-to-avoid-duplicate-rows\u002F","Validate a Merge to Avoid Duplicate Rows",[587,991,596,997,596,1000,596,1003,596,1008,596,1015,596,1026,596,1032,596,1036,596,1042,596,1048,596,1052,596,1057,596,1061,596,1066,596,1070,596,1075,596,1078,596,1085,596,1090,596,1094,596,1098,596,1103,596,1106,596,1111,596,1114],{"viewBox":992,"role":590,"ariaLabelledBy":993,"xmlns":594,"style":996},"4 0 752 233",[994,995],"mk-collide-t","mk-collide-d","width:100%;max-width:752px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif",[598,998,999],{"id":994},"What normalising can break",[602,1001,1002],{"id":995},"Normalising matches order keys that differed in case and spacing. But it can also make two different master rows identical — Sku-006 and SKU-006 both become sku006 — which would duplicate matching orders unless the master is de-duplicated and the merge validated.",[606,1004],{"x":1005,"y":609,"width":1006,"height":1007,"fill":612},"4","752","233",[606,1009],{"x":1010,"y":1011,"width":1012,"height":1013,"rx":1014,"fill":681,"stroke":682,"style":720},"20.0","16.0","351.0","201.0","14",[1016,1017,1019],"g",{"transform":1018},"translate(182.5,29.0) scale(1.083)",[1020,1021],"path",{"d":1022,"fill":1023,"stroke":1024,"style":1025},"M5 12.5l4.5 4.5L19 7","none","var(--teal-ink,#0b6157)","stroke-width:1.66px;stroke-linecap:round;stroke-linejoin:round",[622,1027,1031],{"x":1028,"y":1029,"style":1030},"195.5","78.0","font-size:14px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","Fixed by normalising",[53,1033],{"x1":1034,"y1":729,"x2":1035,"y2":729,"stroke":682,"style":620},"36.0","355.0",[622,1037,1041],{"x":1038,"y":1039,"style":1040},"60.0","111.0","font-size:12px;font-weight:400;fill:var(--text,#172033);text-anchor:start","trailing spaces",[1016,1043,1045],{"transform":1044},"translate(38.5,99.5) scale(0.625)",[1020,1046],{"d":1022,"fill":1023,"stroke":682,"style":1047},"stroke-width:2.88px;stroke-linecap:round;stroke-linejoin:round",[622,1049,1051],{"x":1038,"y":1050,"style":1040},"137.0","case differences",[1016,1053,1055],{"transform":1054},"translate(38.5,125.5) scale(0.625)",[1020,1056],{"d":1022,"fill":1023,"stroke":682,"style":1047},[622,1058,1060],{"x":1038,"y":1059,"style":1040},"163.0","NBSP and zero-width spaces",[1016,1062,1064],{"transform":1063},"translate(38.5,151.5) scale(0.625)",[1020,1065],{"d":1022,"fill":1023,"stroke":682,"style":1047},[622,1067,1069],{"x":1038,"y":1068,"style":1040},"189.0","space vs hyphen",[1016,1071,1073],{"transform":1072},"translate(38.5,177.5) scale(0.625)",[1020,1074],{"d":1022,"fill":1023,"stroke":682,"style":1047},[606,1076],{"x":1077,"y":1011,"width":1012,"height":1013,"rx":1014,"fill":631,"stroke":632,"style":720},"389.0",[1016,1079,1081],{"transform":1080},"translate(551.5,29.0) scale(1.083)",[1020,1082],{"d":1083,"fill":1023,"stroke":1084,"style":1025},"M12 3.5l9.5 17h-19z M12 10v5 M12 17.8v.4","var(--gold-ink,#7a4e06)",[622,1086,1089],{"x":1087,"y":1029,"style":1088},"564.5","font-size:14px;font-weight:700;fill:var(--gold-ink,#7a4e06);text-anchor:middle","Created by normalising",[53,1091],{"x1":1092,"y1":729,"x2":1093,"y2":729,"stroke":632,"style":620},"405.0","724.0",[622,1095,1097],{"x":1096,"y":1039,"style":1040},"429.0","Sku-006 and SKU-006 collide",[1016,1099,1101],{"transform":1100},"translate(407.5,99.5) scale(0.625)",[1020,1102],{"d":1083,"fill":1023,"stroke":632,"style":1047},[622,1104,1105],{"x":1096,"y":1050,"style":1040},"one order would match twice",[1016,1107,1109],{"transform":1108},"translate(407.5,125.5) scale(0.625)",[1020,1110],{"d":1083,"fill":1023,"stroke":632,"style":1047},[622,1112,1113],{"x":1096,"y":1059,"style":1040},"caught by validate=\"many_to_one\"",[1016,1115,1117],{"transform":1116},"translate(407.5,151.5) scale(0.625)",[1020,1118],{"d":1022,"fill":1023,"stroke":682,"style":1047},[37,1120,1122],{"id":1121},"step-4-numeric-ids-that-lost-their-leading-zeros","Step 4 — Numeric IDs that lost their leading zeros",[10,1124,1125,1126,146,1129,90,1131,1134,1135,90,1137,1140],{},"Customer numbers are a different kind of key: ",[14,1127,1128],{},"412",[14,1130,320],{},[14,1132,1133],{},"0412"," should match, but ",[14,1136,1128],{},[14,1138,1139],{},"4120"," must not. Pad every ID to the master's width:",[42,1142,1144],{"className":101,"code":1143,"language":103,"meta":47,"style":47},"width = customers[\"CustomerNo\"].str.len().max()\ncustomers[\"Key\"] = customers[\"CustomerNo\"].str.strip().str.zfill(width)\norders[\"CustKey\"] = orders[\"CustomerNo\"].str.strip().str.zfill(width)\nwith_customers = joined.merge(customers[[\"Key\", \"Customer\"]].rename(columns={\"Key\": \"CustKey\"}),\n                              left_on=orders[\"CustKey\"], right_on=\"CustKey\", how=\"left\")\nprint(with_customers[[\"OrderID\", \"CustomerNo\", \"Customer\"]].to_string(index=False))\n",[14,1145,1146,1162,1180,1197,1235,1266],{"__ignoreMap":47},[51,1147,1148,1151,1153,1156,1159],{"class":53,"line":54},[51,1149,1150],{"class":114},"width ",[51,1152,137],{"class":110},[51,1154,1155],{"class":114}," customers[",[51,1157,1158],{"class":61},"\"CustomerNo\"",[51,1160,1161],{"class":114},"].str.len().max()\n",[51,1163,1164,1167,1169,1171,1173,1175,1177],{"class":53,"line":124},[51,1165,1166],{"class":114},"customers[",[51,1168,773],{"class":61},[51,1170,776],{"class":114},[51,1172,137],{"class":110},[51,1174,1155],{"class":114},[51,1176,1158],{"class":61},[51,1178,1179],{"class":114},"].str.strip().str.zfill(width)\n",[51,1181,1182,1184,1187,1189,1191,1193,1195],{"class":53,"line":131},[51,1183,871],{"class":114},[51,1185,1186],{"class":61},"\"CustKey\"",[51,1188,776],{"class":114},[51,1190,137],{"class":110},[51,1192,880],{"class":114},[51,1194,1158],{"class":61},[51,1196,1179],{"class":114},[51,1198,1199,1202,1204,1207,1209,1211,1214,1217,1220,1222,1225,1227,1230,1232],{"class":53,"line":172},[51,1200,1201],{"class":114},"with_customers ",[51,1203,137],{"class":110},[51,1205,1206],{"class":114}," joined.merge(customers[[",[51,1208,773],{"class":61},[51,1210,146],{"class":114},[51,1212,1213],{"class":61},"\"Customer\"",[51,1215,1216],{"class":114},"]].rename(",[51,1218,1219],{"class":149},"columns",[51,1221,137],{"class":110},[51,1223,1224],{"class":114},"{",[51,1226,773],{"class":61},[51,1228,1229],{"class":114},": ",[51,1231,1186],{"class":61},[51,1233,1234],{"class":114},"}),\n",[51,1236,1237,1240,1242,1244,1246,1249,1252,1254,1256,1258,1260,1262,1264],{"class":53,"line":200},[51,1238,1239],{"class":149},"                              left_on",[51,1241,137],{"class":110},[51,1243,871],{"class":114},[51,1245,1186],{"class":61},[51,1247,1248],{"class":114},"], ",[51,1250,1251],{"class":149},"right_on",[51,1253,137],{"class":110},[51,1255,1186],{"class":61},[51,1257,146],{"class":114},[51,1259,221],{"class":149},[51,1261,137],{"class":110},[51,1263,226],{"class":61},[51,1265,169],{"class":114},[51,1267,1268,1270,1273,1275,1277,1279,1281,1283,1285,1287,1289,1291],{"class":53,"line":241},[51,1269,244],{"class":155},[51,1271,1272],{"class":114},"(with_customers[[",[51,1274,250],{"class":61},[51,1276,146],{"class":114},[51,1278,1158],{"class":61},[51,1280,146],{"class":114},[51,1282,1213],{"class":61},[51,1284,964],{"class":114},[51,1286,289],{"class":149},[51,1288,137],{"class":110},[51,1290,294],{"class":155},[51,1292,297],{"class":114},[10,1294,1295,1298,1299,1301,1302,1304,1305,1308,1309,1312,1313,1315,1316,35],{},[14,1296,1297],{},"zfill"," pads with zeros on the left to the master's five digits, so ",[14,1300,1128],{}," becomes ",[14,1303,320],{}," and matches Müller GmbH. Order 7's customer ",[14,1306,1307],{},"2001"," pads to ",[14,1310,1311],{},"02001"," and matches Oakridge Farms. Use this only for keys that are genuinely numeric with lost padding; applying it to alphanumeric codes would invent matches. Leading zeros on the way ",[582,1314,507],{}," are covered in ",[31,1317,1319],{"href":1318},"\u002Fgetting-started-with-python-excel-automation\u002Freading-excel-files-with-pandas\u002Fkeep-leading-zeros-when-reading-excel-with-pandas\u002F","Keep Leading Zeros When Reading Excel with pandas",[37,1321,1323],{"id":1322},"step-5-report-what-still-does-not-match","Step 5 — Report what still does not match",[10,1325,1326],{},"Whatever remains unmatched after normalisation is a real difference, and the people who own the data need to see it:",[42,1328,1330],{"className":101,"code":1329,"language":103,"meta":47,"style":47},"unmatched = joined[joined[\"_merge\"] == \"left_only\"][[\"OrderID\", \"SKU\", \"Qty\"]]\nprint(unmatched.to_string(index=False))\nsummary = joined[\"_merge\"].map({\"both\": \"matched\", \"left_only\": \"unmatched\"}).value_counts()\nprint(summary.to_dict())\n",[14,1331,1332,1369,1384,1420],{"__ignoreMap":47},[51,1333,1334,1337,1339,1342,1344,1346,1349,1352,1355,1357,1359,1361,1363,1366],{"class":53,"line":54},[51,1335,1336],{"class":114},"unmatched ",[51,1338,137],{"class":110},[51,1340,1341],{"class":114}," joined[joined[",[51,1343,264],{"class":61},[51,1345,776],{"class":114},[51,1347,1348],{"class":110},"==",[51,1350,1351],{"class":61}," \"left_only\"",[51,1353,1354],{"class":114},"][[",[51,1356,250],{"class":61},[51,1358,146],{"class":114},[51,1360,216],{"class":61},[51,1362,146],{"class":114},[51,1364,1365],{"class":61},"\"Qty\"",[51,1367,1368],{"class":114},"]]\n",[51,1370,1371,1373,1376,1378,1380,1382],{"class":53,"line":124},[51,1372,244],{"class":155},[51,1374,1375],{"class":114},"(unmatched.to_string(",[51,1377,289],{"class":149},[51,1379,137],{"class":110},[51,1381,294],{"class":155},[51,1383,297],{"class":114},[51,1385,1386,1389,1391,1394,1396,1399,1402,1404,1407,1409,1412,1414,1417],{"class":53,"line":131},[51,1387,1388],{"class":114},"summary ",[51,1390,137],{"class":110},[51,1392,1393],{"class":114}," joined[",[51,1395,264],{"class":61},[51,1397,1398],{"class":114},"].map({",[51,1400,1401],{"class":61},"\"both\"",[51,1403,1229],{"class":114},[51,1405,1406],{"class":61},"\"matched\"",[51,1408,146],{"class":114},[51,1410,1411],{"class":61},"\"left_only\"",[51,1413,1229],{"class":114},[51,1415,1416],{"class":61},"\"unmatched\"",[51,1418,1419],{"class":114},"}).value_counts()\n",[51,1421,1422,1424],{"class":53,"line":172},[51,1423,244],{"class":155},[51,1425,1426],{"class":114},"(summary.to_dict())\n",[10,1428,1429,1430,1433,1434,35],{},"Order 7 references ",[14,1431,1432],{},"SKU-099",", which is not in the master in any spelling — a new product that has not been set up, or a typo. Listing it, rather than dropping it, keeps the totals honest; the full pattern for finding rows missing from another file is in ",[31,1435,1437],{"href":1436},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Ffind-rows-in-one-excel-file-missing-from-another\u002F","Find Rows in One Excel File Missing from Another",[587,1439,596,1445,596,1448,596,1451,596,1455,596,1459,596,1465,596,1469,596,1474,596,1477,596,1481,596,1484,596,1489,596,1491,596,1495,596,1498],{"viewBox":1440,"role":590,"ariaLabelledBy":1441,"xmlns":594,"style":1444},"4 0 703 140",[1442,1443],"mk-result-t","mk-result-d","width:100%;max-width:703px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif",[598,1446,1447],{"id":1442},"Orders matched to the product master",[602,1449,1450],{"id":1443},"An exact merge matched none of the seven orders. After normalising keys and resolving the duplicate master record, six of seven matched; the one remaining order references a product code that does not exist.",[606,1452],{"x":1005,"y":609,"width":1453,"height":1454,"fill":612},"703","140",[606,1456],{"x":1010,"y":1011,"width":1457,"height":618,"rx":1458,"fill":674,"stroke":619,"style":620},"133.5","7",[622,1460,1464],{"x":1461,"y":1462,"style":1463},"86.8","34.5","font-size:12.5px;font-weight:700;fill:var(--text,#172033);text-anchor:middle","exact merge",[606,1466],{"x":1467,"y":1011,"width":1468,"height":618,"rx":1458,"fill":650,"stroke":651,"style":633},"163.5","6.0",[622,1470,1473],{"x":1471,"y":1462,"style":1472},"177.5","font-size:12px;font-weight:700;fill:var(--accent-ink,#be185d);text-anchor:start","0 of 7",[606,1475],{"x":1010,"y":1476,"width":1457,"height":618,"rx":1458,"fill":674,"stroke":619,"style":620},"56.0",[622,1478,1480],{"x":1461,"y":1479,"style":1463},"74.5","normalised keys",[606,1482],{"x":1467,"y":1476,"width":1483,"height":618,"rx":1458,"fill":681,"stroke":682,"style":633},"485.3",[622,1485,1488],{"x":1486,"y":1479,"style":1487},"656.8","font-size:12px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:start","6 of 7",[606,1490],{"x":1010,"y":641,"width":1457,"height":618,"rx":1458,"fill":674,"stroke":619,"style":620},[622,1492,1494],{"x":1461,"y":1493,"style":1463},"114.5","still unmatched",[606,1496],{"x":1467,"y":641,"width":1497,"height":618,"rx":1458,"fill":631,"stroke":632,"style":633},"80.9",[622,1499,1502],{"x":1500,"y":1493,"style":1501},"252.4","font-size:12px;font-weight:700;fill:var(--gold-ink,#7a4e06);text-anchor:start","1 (SKU-099)",[37,1504,1506],{"id":1505},"step-6-keep-the-rule-in-one-reusable-function","Step 6 — Keep the rule in one reusable function",[10,1508,1509],{},"Every report that joins on these codes needs the same rule, and a rule copied into five scripts drifts. Wrap normalisation, the collision check, the merge and the report into one function:",[42,1511,1513],{"className":101,"code":1512,"language":103,"meta":47,"style":47},"def normalised_merge(left, right, on, key_func=norm_key, how=\"left\"):\n    left = left.assign(_key=left[on].map(key_func))\n    right = right.assign(_key=right[on].map(key_func))\n    dupes = right[right[\"_key\"].duplicated(keep=False)]\n    if not dupes.empty:\n        raise ValueError(f\"lookup keys collide after normalising: {sorted(dupes[on])}\")\n    merged = left.merge(right.drop(columns=on), on=\"_key\", how=how,\n                        validate=\"many_to_one\", indicator=True)\n    unmatched = merged.loc[merged[\"_merge\"] == \"left_only\", on].tolist()\n    return merged.drop(columns=[\"_key\", \"_merge\"]), unmatched\n\ntry:\n    normalised_merge(orders[[\"OrderID\", \"SKU\", \"Qty\"]], products[[\"SKU\", \"Product\"]], on=\"SKU\")\nexcept ValueError as exc:\n    print(\"ValueError:\", exc)\nresult, missing = normalised_merge(orders[[\"OrderID\", \"SKU\", \"Qty\"]], products_clean[[\"SKU\", \"Product\"]], on=\"SKU\")\nprint(result.to_string(index=False))\nprint(\"unmatched:\", missing)\n",[14,1514,1515,1537,1555,1572,1596,1607,1638,1670,1689,1710,1733,1737,1746,1781,1795,1808,1848,1864],{"__ignoreMap":47},[51,1516,1517,1519,1522,1525,1527,1530,1532,1534],{"class":53,"line":54},[51,1518,376],{"class":110},[51,1520,1521],{"class":379}," normalised_merge",[51,1523,1524],{"class":114},"(left, right, on, key_func",[51,1526,137],{"class":110},[51,1528,1529],{"class":114},"norm_key, how",[51,1531,137],{"class":110},[51,1533,226],{"class":61},[51,1535,1536],{"class":114},"):\n",[51,1538,1539,1542,1544,1547,1550,1552],{"class":53,"line":124},[51,1540,1541],{"class":114},"    left ",[51,1543,137],{"class":110},[51,1545,1546],{"class":114}," left.assign(",[51,1548,1549],{"class":149},"_key",[51,1551,137],{"class":110},[51,1553,1554],{"class":114},"left[on].map(key_func))\n",[51,1556,1557,1560,1562,1565,1567,1569],{"class":53,"line":131},[51,1558,1559],{"class":114},"    right ",[51,1561,137],{"class":110},[51,1563,1564],{"class":114}," right.assign(",[51,1566,1549],{"class":149},[51,1568,137],{"class":110},[51,1570,1571],{"class":114},"right[on].map(key_func))\n",[51,1573,1574,1577,1579,1582,1585,1587,1589,1591,1593],{"class":53,"line":172},[51,1575,1576],{"class":114},"    dupes ",[51,1578,137],{"class":110},[51,1580,1581],{"class":114}," right[right[",[51,1583,1584],{"class":61},"\"_key\"",[51,1586,801],{"class":114},[51,1588,804],{"class":149},[51,1590,137],{"class":110},[51,1592,294],{"class":155},[51,1594,1595],{"class":114},")]\n",[51,1597,1598,1601,1604],{"class":53,"line":200},[51,1599,1600],{"class":110},"    if",[51,1602,1603],{"class":110}," not",[51,1605,1606],{"class":114}," dupes.empty:\n",[51,1608,1609,1612,1615,1617,1620,1623,1625,1628,1631,1634,1636],{"class":53,"line":241},[51,1610,1611],{"class":110},"        raise",[51,1613,1614],{"class":155}," ValueError",[51,1616,544],{"class":114},[51,1618,1619],{"class":110},"f",[51,1621,1622],{"class":61},"\"lookup keys collide after normalising: ",[51,1624,1224],{"class":462},[51,1626,1627],{"class":155},"sorted",[51,1629,1630],{"class":114},"(dupes[on])",[51,1632,1633],{"class":462},"}",[51,1635,452],{"class":61},[51,1637,169],{"class":114},[51,1639,1640,1643,1645,1648,1650,1652,1655,1657,1659,1661,1663,1665,1667],{"class":53,"line":300},[51,1641,1642],{"class":114},"    merged ",[51,1644,137],{"class":110},[51,1646,1647],{"class":114}," left.merge(right.drop(",[51,1649,1219],{"class":149},[51,1651,137],{"class":110},[51,1653,1654],{"class":114},"on), ",[51,1656,211],{"class":149},[51,1658,137],{"class":110},[51,1660,1584],{"class":61},[51,1662,146],{"class":114},[51,1664,221],{"class":149},[51,1666,137],{"class":110},[51,1668,1669],{"class":114},"how,\n",[51,1671,1672,1675,1677,1679,1681,1683,1685,1687],{"class":53,"line":484},[51,1673,1674],{"class":149},"                        validate",[51,1676,137],{"class":110},[51,1678,930],{"class":61},[51,1680,146],{"class":114},[51,1682,231],{"class":149},[51,1684,137],{"class":110},[51,1686,236],{"class":155},[51,1688,169],{"class":114},[51,1690,1691,1694,1696,1699,1701,1703,1705,1707],{"class":53,"line":493},[51,1692,1693],{"class":114},"    unmatched ",[51,1695,137],{"class":110},[51,1697,1698],{"class":114}," merged.loc[merged[",[51,1700,264],{"class":61},[51,1702,776],{"class":114},[51,1704,1348],{"class":110},[51,1706,1351],{"class":61},[51,1708,1709],{"class":114},", on].tolist()\n",[51,1711,1712,1714,1717,1719,1721,1724,1726,1728,1730],{"class":53,"line":498},[51,1713,487],{"class":110},[51,1715,1716],{"class":114}," merged.drop(",[51,1718,1219],{"class":149},[51,1720,137],{"class":110},[51,1722,1723],{"class":114},"[",[51,1725,1584],{"class":61},[51,1727,146],{"class":114},[51,1729,264],{"class":61},[51,1731,1732],{"class":114},"]), unmatched\n",[51,1734,1735],{"class":53,"line":538},[51,1736,128],{"emptyLinePlaceholder":127},[51,1738,1740,1743],{"class":53,"line":1739},12,[51,1741,1742],{"class":110},"try",[51,1744,1745],{"class":114},":\n",[51,1747,1749,1752,1754,1756,1758,1760,1762,1765,1767,1769,1771,1773,1775,1777,1779],{"class":53,"line":1748},13,[51,1750,1751],{"class":114},"    normalised_merge(orders[[",[51,1753,250],{"class":61},[51,1755,146],{"class":114},[51,1757,216],{"class":61},[51,1759,146],{"class":114},[51,1761,1365],{"class":61},[51,1763,1764],{"class":114},"]], products[[",[51,1766,216],{"class":61},[51,1768,146],{"class":114},[51,1770,259],{"class":61},[51,1772,903],{"class":114},[51,1774,211],{"class":149},[51,1776,137],{"class":110},[51,1778,216],{"class":61},[51,1780,169],{"class":114},[51,1782,1784,1787,1789,1792],{"class":53,"line":1783},14,[51,1785,1786],{"class":110},"except",[51,1788,1614],{"class":155},[51,1790,1791],{"class":110}," as",[51,1793,1794],{"class":114}," exc:\n",[51,1796,1798,1800,1802,1805],{"class":53,"line":1797},15,[51,1799,541],{"class":155},[51,1801,544],{"class":114},[51,1803,1804],{"class":61},"\"ValueError:\"",[51,1806,1807],{"class":114},", exc)\n",[51,1809,1811,1814,1816,1819,1821,1823,1825,1827,1829,1832,1834,1836,1838,1840,1842,1844,1846],{"class":53,"line":1810},16,[51,1812,1813],{"class":114},"result, missing ",[51,1815,137],{"class":110},[51,1817,1818],{"class":114}," normalised_merge(orders[[",[51,1820,250],{"class":61},[51,1822,146],{"class":114},[51,1824,216],{"class":61},[51,1826,146],{"class":114},[51,1828,1365],{"class":61},[51,1830,1831],{"class":114},"]], products_clean[[",[51,1833,216],{"class":61},[51,1835,146],{"class":114},[51,1837,259],{"class":61},[51,1839,903],{"class":114},[51,1841,211],{"class":149},[51,1843,137],{"class":110},[51,1845,216],{"class":61},[51,1847,169],{"class":114},[51,1849,1851,1853,1856,1858,1860,1862],{"class":53,"line":1850},17,[51,1852,244],{"class":155},[51,1854,1855],{"class":114},"(result.to_string(",[51,1857,289],{"class":149},[51,1859,137],{"class":110},[51,1861,294],{"class":155},[51,1863,297],{"class":114},[51,1865,1867,1869,1871,1874],{"class":53,"line":1866},18,[51,1868,244],{"class":155},[51,1870,544],{"class":114},[51,1872,1873],{"class":61},"\"unmatched:\"",[51,1875,1876],{"class":114},", missing)\n",[10,1878,1879],{},"Against the raw master the function refuses to run and names the colliding codes; against the cleaned master it returns the joined orders and the list of unmatched codes in one call. Importing it from a shared module means a change to the rule — say, a new separator character that turns up in one supplier's files — applies to every report at once. The helper raises on collisions instead of picking one silently, because the right answer depends on business knowledge the script does not have.",[37,1881,1883],{"id":1882},"performance-notes","Performance notes",[10,1885,1886,1889,1890,1893,1894,1897,1898,146,1901,146,1904,1907,1908,1911],{},[14,1887,1888],{},"Series.map"," with a Python function runs once per value, which is fine for tens of thousands of keys and slow for millions. For large tables, normalise the ",[582,1891,1892],{},"unique"," keys only and map back — ",[14,1895,1896],{},"uniq = s.unique(); lookup = dict(zip(uniq, map(norm_key, uniq))); s.map(lookup)"," — because codes usually repeat heavily, or use vectorised string methods (",[14,1899,1900],{},"str.normalize(\"NFKC\")",[14,1902,1903],{},"str.replace(r\"[\\s\\-_]+\", \"\", regex=True)",[14,1905,1906],{},"str.casefold()",", plus a ",[14,1909,1910],{},"str.replace"," for the zero-width characters), which perform the same steps column-wise. The merge itself is a hash join and costs the same as any other merge.",[37,1913,1915],{"id":1914},"common-pitfalls","Common pitfalls",[1917,1918,1919,1927,1933,1939,1947],"ul",{},[1920,1921,1922,1926],"li",{},[1923,1924,1925],"strong",{},"Normalising only one side."," Both sides must go through the same function.",[1920,1928,1929,1932],{},[1923,1930,1931],{},"Overwriting the original key."," Keep the raw value for display and audit; join on a separate column.",[1920,1934,1935,1938],{},[1923,1936,1937],{},"Ignoring collisions."," Normalisation can merge distinct master records; check and validate.",[1920,1940,1941,1944,1945,35],{},[1923,1942,1943],{},"Reading IDs as numbers."," Leading zeros vanish before you can normalise; read keys with ",[14,1946,316],{},[1920,1948,1949,1952,1953,90,1956,1959],{},[1923,1950,1951],{},"Over-normalising."," Removing every non-alphanumeric character can make ",[14,1954,1955],{},"A-12",[14,1957,1958],{},"A1-2"," equal; remove only the separators that are noise in your data.",[37,1961,1963],{"id":1962},"conclusion","Conclusion",[10,1965,1966,1967,1969,1970,146,1972,1974,1975,90,1977,1979],{},"Exact merges fail on keys that differ in case, spacing, invisible characters, separators or lost leading zeros. Read keys as text, build one normalisation function — ",[14,1968,561],{},", remove zero-width characters, drop separators, ",[14,1971,575],{},[14,1973,1297],{}," for numeric IDs — and apply it to both sides into a separate key column. Check the master for keys that collide after normalising, merge with ",[14,1976,977],{},[14,1978,324],{},", and report the rows that still do not match.",[37,1981,1983],{"id":1982},"frequently-asked-questions","Frequently asked questions",[10,1985,1986,1989],{},[1923,1987,1988],{},"Why does my pandas merge miss rows that look identical?","\nmerge compares keys character by character. Trailing spaces, different case, non-breaking or zero-width spaces, or different separators make keys unequal even when they look the same. Print them with repr() to see the hidden characters.",[10,1991,1992,1995],{},[1923,1993,1994],{},"How do I merge case-insensitively in pandas?","\nCreate a key column on both DataFrames with the same normalisation — strip the text and casefold it — and merge on that column instead of the original.",[10,1997,1998,2001],{},[1923,1999,2000],{},"How do I match IDs when one file lost the leading zeros?","\nRead both columns as text and pad them to the same width with str.zfill(width), using the width of the master's IDs, then merge on the padded keys.",[10,2003,2004,2007],{},[1923,2005,2006],{},"How do I stop a normalised merge from duplicating rows?","\nCheck the lookup table for keys that become duplicates after normalisation, resolve them, and merge with validate=\"many_to_one\" so any future duplicate raises an error.",[37,2009,2011],{"id":2010},"related","Related",[1917,2013,2014,2020,2027,2034],{},[1920,2015,2016,2017,2019],{},"Up one level: ",[31,2018,34],{"href":33}," — combining sheets and files.",[1920,2021,2022,2026],{},[31,2023,2025],{"href":2024},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fstrip-whitespace-and-normalise-text-columns-with-pandas\u002F","Strip Whitespace and Normalise Text Columns with pandas"," — the same clean-up on whole columns.",[1920,2028,2029,2033],{},[31,2030,2032],{"href":2031},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Ffuzzy-match-names-between-two-excel-files-with-python\u002F","Fuzzy Match Names Between Two Excel Files with Python"," — when keys differ by more than formatting.",[1920,2035,2036,2040],{},[31,2037,2039],{"href":2038},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fremove-non-printable-characters-from-excel-data\u002F","Remove Non-Printable Characters from Excel Data"," — finding hidden characters.",[2042,2043,2044],"style",{},"html pre.shiki code .sMTad, html code.shiki .sMTad{--shiki-default:#6F42C1;--shiki-dark:#FFB757}html pre.shiki code .srMev, html code.shiki .srMev{--shiki-default:#032F62;--shiki-dark:#ADDCFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .s-kum, html code.shiki .s-kum{--shiki-default:#D73A49;--shiki-dark:#FF9492}html pre.shiki code .skGVy, html code.shiki .skGVy{--shiki-default:#24292E;--shiki-dark:#F0F3F6}html pre.shiki code .sa561, html code.shiki .sa561{--shiki-default:#E36209;--shiki-dark:#FFB757}html pre.shiki code .sP0c6, html code.shiki .sP0c6{--shiki-default:#005CC5;--shiki-dark:#91CBFF}html pre.shiki code .s_Opv, html code.shiki .s_Opv{--shiki-default:#6F42C1;--shiki-dark:#DBB7FF}html pre.shiki code .s-wDw, html code.shiki .s-wDw{--shiki-default:#6A737D;--shiki-dark:#BDC4CC}html pre.shiki code .s_b0D, html code.shiki .s_b0D{--shiki-default:#22863A;--shiki-default-font-weight:bold;--shiki-dark:#72F088;--shiki-dark-font-weight:bold}html pre.shiki code .sSjpA, html code.shiki .sSjpA{--shiki-default:#005CC5;--shiki-dark:#FF9492}",{"title":47,"searchDepth":124,"depth":124,"links":2046},[2047,2048,2049,2050,2051,2052,2053,2054,2055,2056,2057,2058],{"id":39,"depth":124,"text":40},{"id":97,"depth":124,"text":98},{"id":347,"depth":124,"text":348},{"id":742,"depth":124,"text":743},{"id":1121,"depth":124,"text":1122},{"id":1322,"depth":124,"text":1323},{"id":1505,"depth":124,"text":1506},{"id":1882,"depth":124,"text":1883},{"id":1914,"depth":124,"text":1915},{"id":1962,"depth":124,"text":1963},{"id":1982,"depth":124,"text":1983},{"id":2010,"depth":124,"text":2011},"2026-10-02","A merge that should match every row matches half of them because one file says \"SKU-001 \" and the other \"sku-001\", or an ID has lost its leading zeros, or an invisible non-breaking space hides in a cell. Build a normalised join key on both sides, merge on it while keeping the original values, detect keys that collapse into collisions, and report what still does not match.","md",[2063,2065,2067,2069],{"q":1988,"a":2064},"merge compares keys character by character. Trailing spaces, different case, non-breaking or zero-width spaces, or different separators make keys unequal even when they look the same. Print them with repr() to see the hidden characters.",{"q":1994,"a":2066},"Create a key column on both DataFrames with the same normalisation — strip the text and casefold it — and merge on that column instead of the original.",{"q":2000,"a":2068},"Read both columns as text and pad them to the same width with str.zfill(width), using the width of the master's IDs, then merge on the padded keys.",{"q":2006,"a":2070},"Check the lookup table for keys that become duplicates after normalisation, resolve them, and merge with validate=\"many_to_one\" so any future duplicate raises an error.","merge on keys that differ in case and spacing with pandas",null,{"breadcrumb":2074},[2075,2078,2081],{"name":2076,"item":2077},"Home","\u002F",{"name":2079,"item":2080},"Advanced Data Transformation and Cleaning for Excel Automation","\u002Fadvanced-data-transformation-and-cleaning\u002F",{"name":2082,"item":33},"Merging and Joining Excel DataFrames with Pandas","\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Fmerge-on-keys-that-differ-in-case-and-spacing-with-pandas",{"title":2085,"description":2086},"Merge Excel Data on Keys with Different Case and Spacing","Fix pandas merges that miss rows because of case, spaces, non-breaking spaces, zero-width characters, hyphens or lost leading zeros: a normalised key column on both sides, collision checks with validate, indicator=True for unmatched rows.","merge-on-keys-that-differ-in-case-and-spacing-with-pandas","advanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Fmerge-on-keys-that-differ-in-case-and-spacing-with-pandas\u002Findex",[2090,2091],"joins-and-lookups","data-cleaning","how-to","zDX6vmyH5jyablBVF9X1XPmhV6TpzMuEMjqV06FGIAc",[2095,2099],{"title":2096,"path":2097,"stem":2098,"children":-1},"Merge More Than Two Excel Files on a Key Column","\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Fmerge-more-than-two-excel-files-on-a-key-column","advanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Fmerge-more-than-two-excel-files-on-a-key-column\u002Findex",{"title":2100,"path":2101,"stem":2102,"children":-1},"Merge Two Excel Files on a Common Column in Python","\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Fmerge-two-excel-files-on-common-column-python","advanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Fmerge-two-excel-files-on-common-column-python\u002Findex",[2104,2106],{"path":2105,"title":2079},"\u002Fadvanced-data-transformation-and-cleaning",{"path":2107,"title":2082},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes",[],[2110,2113,2116,2119,2122,2125],{"path":2111,"title":2112,"kind":2092},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fcompare-two-excel-files-for-differences-with-python","Compare Two Excel Files for Differences with Python",{"path":2114,"title":2115,"kind":2092},"\u002Fautomating-reporting-workflows\u002Fbuilding-finance-and-operations-reports\u002Fconvert-multi-currency-transactions-with-an-fx-rate-table","Convert Multi-Currency Transactions with an FX Rate Table",{"path":2117,"title":2118,"kind":2092},"\u002Fadvanced-data-transformation-and-cleaning\u002Fapplying-conditional-formatting-with-openpyxl\u002Fhighlight-differences-between-two-columns-with-openpyxl","Highlight Differences Between Two Columns with openpyxl",{"path":2120,"title":2121,"kind":2092},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fmap-free-text-job-titles-to-standard-categories","Map Free-Text Job Titles to Standard Categories",{"path":2123,"title":2124,"kind":2092},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fstandardise-category-values-with-a-mapping-table","Standardise Category Values with a Mapping Table in pandas",{"path":2126,"title":2127,"kind":2092},"\u002Fadvanced-data-transformation-and-cleaning\u002Fexcel-formula-equivalents-in-pandas\u002Fapproximate-match-vlookup-in-pandas","Approximate-Match VLOOKUP for Tiered Rates in pandas",1790972096430]