[{"data":1,"prerenderedAt":1978},["ShallowReactive",2],{"doc:\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fremove-accents-and-normalise-unicode-in-excel-text":3,"surround:\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fremove-accents-and-normalise-unicode-in-excel-text":1944,"ancestors:\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fremove-accents-and-normalise-unicode-in-excel-text":1953,"children:\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fremove-accents-and-normalise-unicode-in-excel-text":1958,"related:\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fremove-accents-and-normalise-unicode-in-excel-text":1959},{"id":4,"title":5,"body":6,"dateModified":1910,"datePublished":1910,"description":1911,"extension":1912,"faq":1913,"keyword":1922,"kind":1923,"meta":1924,"navigation":118,"path":1934,"seo":1935,"slug":1938,"stem":1939,"topics":1940,"type":1942,"__hash__":1943},"docs\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fremove-accents-and-normalise-unicode-in-excel-text\u002Findex.md","Remove Accents and Normalise Unicode in Excel Text",{"type":7,"value":8,"toc":1897},"minimark",[9,33,38,66,75,79,82,374,401,512,516,530,659,689,693,700,987,1021,1187,1191,1194,1342,1349,1353,1356,1619,1633,1755,1759,1773,1777,1818,1822,1832,1836,1842,1848,1854,1860,1864,1893],[10,11,12,13,17,18,22,23,26,27,32],"p",{},"Two customer lists both contain \"Café Señor\", yet a merge in pandas finds no match. ",[14,15,16],"code",{},"drop_duplicates"," leaves \"Müller GmbH\" in twice. A filter for \"Zürich\" misses half the rows. Nothing is visibly different — the text ",[19,20,21],"em",{},"looks"," identical — but the underlying characters are not. Unicode allows the same visible letter to be stored in more than one way: an \"é\" can be one character or an \"e\" followed by a combining accent; a space can be a normal space, a non-breaking space or a zero-width space that is not visible at all; a digit can be a normal \"1\" or a full-width \"１\" pasted from an East Asian system. Excel displays them all the same. This page shows how to detect these differences, normalise text with Python's ",[14,24,25],{},"unicodedata",", build accent-free keys for matching, and keep the original spelling for display. It belongs to ",[28,29,31],"a",{"href":30},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002F","Cleaning Excel Data with pandas",".",[34,35,37],"h2",{"id":36},"prerequisites","Prerequisites",[39,40,45],"pre",{"className":41,"code":42,"language":43,"meta":44,"style":44},"language-bash shiki shiki-themes github-light github-dark-high-contrast","pip install pandas openpyxl\n","bash","",[14,46,47],{"__ignoreMap":44},[48,49,52,56,60,63],"span",{"class":50,"line":51},"line",1,[48,53,55],{"class":54},"sMTad","pip",[48,57,59],{"class":58},"srMev"," install",[48,61,62],{"class":58}," pandas",[48,64,65],{"class":58}," openpyxl\n",[10,67,68,71,72,74],{},[14,69,70],{},"customers.xlsx"," has two sheets: a CRM list of six customers and a sheet of web orders for the same customers, typed or pasted differently. To the eye, several names are identical; underneath, one web name uses a decomposed accent, one a non-breaking space and one a hidden zero-width space, two were typed in capitals without accents, and one CRM name contains the typographic ligature \"fi\" from a word processor. Everything needed is in Python's standard library ",[14,73,25],{}," module.",[34,76,78],{"id":77},"step-1-see-that-identical-text-does-not-match","Step 1 — See that \"identical\" text does not match",[10,80,81],{},"Try the obvious merge and then look at the actual characters:",[39,83,87],{"className":84,"code":85,"language":86,"meta":44,"style":44},"language-python shiki shiki-themes github-light github-dark-high-contrast","import unicodedata\nimport pandas as pd\n\ncrm = pd.read_excel(\"customers.xlsx\", sheet_name=\"CRM\")\nweb = pd.read_excel(\"customers.xlsx\", sheet_name=\"Web orders\")\nplain = web.merge(crm, on=\"Customer\", how=\"left\")\nprint(plain[[\"Customer\", \"Account\"]].to_string(index=False))\n\ndef reveal(text):\n    return \" \".join(f\"U+{ord(ch):04X}\" if not ch.isascii() or not ch.isprintable() else ch for ch in text)\n\nfor name in web[\"Customer\"][:3]:\n    print(repr(name), \"→\", reveal(name))\n","python",[14,88,89,99,113,120,150,173,204,235,240,253,324,329,353],{"__ignoreMap":44},[48,90,91,95],{"class":50,"line":51},[48,92,94],{"class":93},"s-kum","import",[48,96,98],{"class":97},"skGVy"," unicodedata\n",[48,100,102,104,107,110],{"class":50,"line":101},2,[48,103,94],{"class":93},[48,105,106],{"class":97}," pandas ",[48,108,109],{"class":93},"as",[48,111,112],{"class":97}," pd\n",[48,114,116],{"class":50,"line":115},3,[48,117,119],{"emptyLinePlaceholder":118},true,"\n",[48,121,123,126,129,132,135,138,142,144,147],{"class":50,"line":122},4,[48,124,125],{"class":97},"crm ",[48,127,128],{"class":93},"=",[48,130,131],{"class":97}," pd.read_excel(",[48,133,134],{"class":58},"\"customers.xlsx\"",[48,136,137],{"class":97},", ",[48,139,141],{"class":140},"sa561","sheet_name",[48,143,128],{"class":93},[48,145,146],{"class":58},"\"CRM\"",[48,148,149],{"class":97},")\n",[48,151,153,156,158,160,162,164,166,168,171],{"class":50,"line":152},5,[48,154,155],{"class":97},"web ",[48,157,128],{"class":93},[48,159,131],{"class":97},[48,161,134],{"class":58},[48,163,137],{"class":97},[48,165,141],{"class":140},[48,167,128],{"class":93},[48,169,170],{"class":58},"\"Web orders\"",[48,172,149],{"class":97},[48,174,176,179,181,184,187,189,192,194,197,199,202],{"class":50,"line":175},6,[48,177,178],{"class":97},"plain ",[48,180,128],{"class":93},[48,182,183],{"class":97}," web.merge(crm, ",[48,185,186],{"class":140},"on",[48,188,128],{"class":93},[48,190,191],{"class":58},"\"Customer\"",[48,193,137],{"class":97},[48,195,196],{"class":140},"how",[48,198,128],{"class":93},[48,200,201],{"class":58},"\"left\"",[48,203,149],{"class":97},[48,205,207,211,214,216,218,221,224,227,229,232],{"class":50,"line":206},7,[48,208,210],{"class":209},"sP0c6","print",[48,212,213],{"class":97},"(plain[[",[48,215,191],{"class":58},[48,217,137],{"class":97},[48,219,220],{"class":58},"\"Account\"",[48,222,223],{"class":97},"]].to_string(",[48,225,226],{"class":140},"index",[48,228,128],{"class":93},[48,230,231],{"class":209},"False",[48,233,234],{"class":97},"))\n",[48,236,238],{"class":50,"line":237},8,[48,239,119],{"emptyLinePlaceholder":118},[48,241,243,246,250],{"class":50,"line":242},9,[48,244,245],{"class":93},"def",[48,247,249],{"class":248},"s_Opv"," reveal",[48,251,252],{"class":97},"(text):\n",[48,254,256,259,262,265,268,271,275,278,281,284,287,290,293,296,299,302,304,307,310,313,316,318,321],{"class":50,"line":255},10,[48,257,258],{"class":93},"    return",[48,260,261],{"class":58}," \" \"",[48,263,264],{"class":97},".join(",[48,266,267],{"class":93},"f",[48,269,270],{"class":58},"\"U+",[48,272,274],{"class":273},"sSjpA","{",[48,276,277],{"class":209},"ord",[48,279,280],{"class":97},"(ch)",[48,282,283],{"class":93},":04X",[48,285,286],{"class":273},"}",[48,288,289],{"class":58},"\"",[48,291,292],{"class":93}," if",[48,294,295],{"class":93}," not",[48,297,298],{"class":97}," ch.isascii() ",[48,300,301],{"class":93},"or",[48,303,295],{"class":93},[48,305,306],{"class":97}," ch.isprintable() ",[48,308,309],{"class":93},"else",[48,311,312],{"class":97}," ch ",[48,314,315],{"class":93},"for",[48,317,312],{"class":97},[48,319,320],{"class":93},"in",[48,322,323],{"class":97}," text)\n",[48,325,327],{"class":50,"line":326},11,[48,328,119],{"emptyLinePlaceholder":118},[48,330,332,334,337,339,342,344,347,350],{"class":50,"line":331},12,[48,333,315],{"class":93},[48,335,336],{"class":97}," name ",[48,338,320],{"class":93},[48,340,341],{"class":97}," web[",[48,343,191],{"class":58},[48,345,346],{"class":97},"][:",[48,348,349],{"class":209},"3",[48,351,352],{"class":97},"]:\n",[48,354,356,359,362,365,368,371],{"class":50,"line":355},13,[48,357,358],{"class":209},"    print",[48,360,361],{"class":97},"(",[48,363,364],{"class":209},"repr",[48,366,367],{"class":97},"(name), ",[48,369,370],{"class":58},"\"→\"",[48,372,373],{"class":97},", reveal(name))\n",[10,375,376,377,380,381,384,385,388,389,392,393,396,397,400],{},"The merge finds no account for any of the web orders. Revealing the code points shows why: the web \"Café\" is ",[14,378,379],{},"e"," followed by ",[14,382,383],{},"U+0301"," (a combining acute accent) where the CRM has the single character ",[14,386,387],{},"U+00E9","; \"Müller GmbH\" has ",[14,390,391],{},"U+00A0",", a non-breaking space, between the words; and \"Zürich Optik\" hides a ",[14,394,395],{},"U+200B"," zero-width space before the normal space. A helper like ",[14,398,399],{},"reveal"," is worth keeping around: when two strings look equal and are not, printing their code points answers the question in seconds.",[402,403,411,412,411,416,411,420,411,428,411,438,411,449,411,456,411,462,411,467,411,471,411,475,411,479,411,484,411,491,411,496,411,500,411,503,411,506,411,509],"svg",{"viewBox":404,"role":405,"ariaLabelledBy":406,"xmlns":409,"style":410},"4 0 752 233","img",[407,408],"uni-same-t","uni-same-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;max-width:752px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif","\n  ",[413,414,415],"title",{"id":407},"Same on screen, different underneath",[417,418,419],"desc",{"id":408},"Café can be stored as five characters with é as U+00E9, or as six with e followed by the combining accent U+0301. A space can be the normal U+0020, a non-breaking U+00A0, or be preceded by an invisible zero-width space U+200B. Excel shows them identically but they do not compare equal.",[421,422],"rect",{"x":423,"y":424,"width":425,"height":426,"fill":427},"4","0","752","233","#ffffff",[421,429],{"x":430,"y":431,"width":432,"height":433,"rx":434,"fill":435,"stroke":436,"style":437},"20.0","16.0","351.0","201.0","14","#e7ebef","var(--line,#cdd5e6)","stroke-width:2px",[439,440,442],"g",{"transform":441},"translate(182.5,29.0) scale(1.083)",[443,444],"path",{"d":445,"fill":446,"stroke":447,"style":448},"M2 12s4-7 10-7 10 7 10 7-4 7-10 7S2 12 2 12z M12 9a3 3 0 1 0 .01 0z","none","var(--text,#172033)","stroke-width:1.66px;stroke-linecap:round;stroke-linejoin:round",[450,451,455],"text",{"x":452,"y":453,"style":454},"195.5","78.0","font-size:14px;font-weight:700;fill:var(--text,#172033);text-anchor:middle","what Excel shows",[50,457],{"x1":458,"y1":459,"x2":460,"y2":459,"stroke":436,"style":461},"36.0","91.0","355.0","stroke-width:1px",[450,463,466],{"x":452,"y":464,"style":465},"111.0","font-size:12px;font-weight:400;fill:var(--text,#172033);text-anchor:middle","Café Señor",[450,468,470],{"x":452,"y":469,"style":465},"137.0","Müller GmbH",[450,472,474],{"x":452,"y":473,"style":465},"163.0","Zürich Optik",[450,476,478],{"x":452,"y":477,"style":465},"189.0","Office fitters",[421,480],{"x":481,"y":431,"width":432,"height":433,"rx":434,"fill":482,"stroke":483,"style":437},"389.0","#fce9e9","var(--danger,#dc2626)",[439,485,487],{"transform":486},"translate(551.5,29.0) scale(1.083)",[443,488],{"d":489,"fill":446,"stroke":490,"style":448},"M11 4a7 7 0 1 0 .01 0z M16 16l5 5","var(--accent-ink,#be185d)",[450,492,495],{"x":493,"y":453,"style":494},"564.5","font-size:14px;font-weight:700;fill:var(--accent-ink,#be185d);text-anchor:middle","what is stored",[50,497],{"x1":498,"y1":459,"x2":499,"y2":459,"stroke":483,"style":461},"405.0","724.0",[450,501,502],{"x":493,"y":464,"style":465},"e + U+0301 (combining)",[450,504,505],{"x":493,"y":469,"style":465},"U+00A0 non-breaking space",[450,507,508],{"x":493,"y":473,"style":465},"U+200B zero-width + space",[450,510,511],{"x":493,"y":477,"style":465},"U+FB01 fi ligature",[34,513,515],{"id":514},"step-2-normalise-to-one-form-with-nfc-or-nfkc","Step 2 — Normalise to one form with NFC or NFKC",[10,517,518,521,522,525,526,529],{},[14,519,520],{},"unicodedata.normalize"," converts text to a standard ",[19,523,524],{},"normal form",". NFC composes letters and accents into single characters wherever possible; NFKC additionally replaces ",[19,527,528],{},"compatibility"," characters — ligatures, full-width forms, superscripts, non-breaking spaces — with their plain equivalents:",[39,531,533],{"className":84,"code":532,"language":86,"meta":44,"style":44},"samples = {\"decomposed é\": \"Café\", \"nbsp\": \"Müller GmbH\", \"ligature\": \"Office fitters\", \"full-width\": \"ＳＫＵ１２\"}\nfor label, s in samples.items():\n    print(f\"{label:13} NFC={unicodedata.normalize('NFC', s)!r:18} NFKC={unicodedata.normalize('NFKC', s)!r}\")\n",[14,534,535,587,599],{"__ignoreMap":44},[48,536,537,540,542,545,548,551,554,556,559,561,564,566,569,571,574,576,579,581,584],{"class":50,"line":51},[48,538,539],{"class":97},"samples ",[48,541,128],{"class":93},[48,543,544],{"class":97}," {",[48,546,547],{"class":58},"\"decomposed é\"",[48,549,550],{"class":97},": ",[48,552,553],{"class":58},"\"Café\"",[48,555,137],{"class":97},[48,557,558],{"class":58},"\"nbsp\"",[48,560,550],{"class":97},[48,562,563],{"class":58},"\"Müller GmbH\"",[48,565,137],{"class":97},[48,567,568],{"class":58},"\"ligature\"",[48,570,550],{"class":97},[48,572,573],{"class":58},"\"Office fitters\"",[48,575,137],{"class":97},[48,577,578],{"class":58},"\"full-width\"",[48,580,550],{"class":97},[48,582,583],{"class":58},"\"ＳＫＵ１２\"",[48,585,586],{"class":97},"}\n",[48,588,589,591,594,596],{"class":50,"line":101},[48,590,315],{"class":93},[48,592,593],{"class":97}," label, s ",[48,595,320],{"class":93},[48,597,598],{"class":97}," samples.items():\n",[48,600,601,603,605,607,609,611,614,617,619,622,624,627,630,633,636,638,641,643,645,648,650,653,655,657],{"class":50,"line":115},[48,602,358],{"class":209},[48,604,361],{"class":97},[48,606,267],{"class":93},[48,608,289],{"class":58},[48,610,274],{"class":273},[48,612,613],{"class":97},"label",[48,615,616],{"class":93},":13",[48,618,286],{"class":273},[48,620,621],{"class":58}," NFC=",[48,623,274],{"class":273},[48,625,626],{"class":97},"unicodedata.normalize(",[48,628,629],{"class":58},"'NFC'",[48,631,632],{"class":97},", s)",[48,634,635],{"class":93},"!r:18",[48,637,286],{"class":273},[48,639,640],{"class":58}," NFKC=",[48,642,274],{"class":273},[48,644,626],{"class":97},[48,646,647],{"class":58},"'NFKC'",[48,649,632],{"class":97},[48,651,652],{"class":93},"!r",[48,654,286],{"class":273},[48,656,289],{"class":58},[48,658,149],{"class":97},[10,660,661,662,665,666,669,670,673,674,677,678,681,682,677,685,688],{},"NFC fixes the decomposed accent — ",[14,663,664],{},"Cafe"," + combining accent becomes ",[14,667,668],{},"Café"," with a single ",[14,671,672],{},"é"," — but leaves the other three alone, because a ligature or a non-breaking space is a different character, not just a different encoding. NFKC fixes all four: the ligature ",[14,675,676],{},"fi"," becomes ",[14,679,680],{},"fi",", the non-breaking space a normal space and the full-width ",[14,683,684],{},"ＳＫＵ１２",[14,686,687],{},"SKU12",". For data cleaning, NFKC is usually the right choice; NFC is the safe choice when text will be displayed or printed and typographic characters should be kept. Neither removes zero-width characters, which is the next step.",[34,690,692],{"id":691},"step-3-a-cleaning-function-for-names-and-keys","Step 3 — A cleaning function for names and keys",[10,694,695,696,699],{},"Combine NFKC with removal of invisible format characters and collapsing of whitespace. That gives a cleaned display value; for ",[19,697,698],{},"matching",", go further — case-fold and strip accents:",[39,701,703],{"className":84,"code":702,"language":86,"meta":44,"style":44},"import re\n\nINVISIBLE = re.compile(r\"[​-‏⁠]\")      # zero-width spaces, joiners, marks, BOM\n\ndef clean_text(text):\n    text = unicodedata.normalize(\"NFKC\", str(text))\n    text = INVISIBLE.sub(\"\", text)\n    return re.sub(r\"\\s+\", \" \", text).strip()\n\ndef match_key(text):\n    decomposed = unicodedata.normalize(\"NFKD\", clean_text(text))\n    no_accents = \"\".join(ch for ch in decomposed if not unicodedata.combining(ch))\n    return no_accents.casefold()\n\nfor raw in [\"Zürich​ Optik\", \"STRASSE & SÖHNE\", \"Straße & Söhne\", \"Ångström AB\", \"ANGSTROM AB\"]:\n    print(f\"{raw!r:24} clean={clean_text(raw)!r:20} key={match_key(raw)!r}\")\n",[14,704,705,712,716,750,754,763,784,802,829,833,842,857,887,894,899,937],{"__ignoreMap":44},[48,706,707,709],{"class":50,"line":51},[48,708,94],{"class":93},[48,710,711],{"class":97}," re\n",[48,713,714],{"class":50,"line":101},[48,715,119],{"emptyLinePlaceholder":118},[48,717,718,721,724,727,730,732,735,738,741,743,746],{"class":50,"line":115},[48,719,720],{"class":209},"INVISIBLE",[48,722,723],{"class":93}," =",[48,725,726],{"class":97}," re.compile(",[48,728,729],{"class":93},"r",[48,731,289],{"class":58},[48,733,734],{"class":209},"[",[48,736,737],{"class":273},"​-‏⁠",[48,739,740],{"class":209},"]",[48,742,289],{"class":58},[48,744,745],{"class":97},")      ",[48,747,749],{"class":748},"s-wDw","# zero-width spaces, joiners, marks, BOM\n",[48,751,752],{"class":50,"line":122},[48,753,119],{"emptyLinePlaceholder":118},[48,755,756,758,761],{"class":50,"line":152},[48,757,245],{"class":93},[48,759,760],{"class":248}," clean_text",[48,762,252],{"class":97},[48,764,765,768,770,773,776,778,781],{"class":50,"line":175},[48,766,767],{"class":97},"    text ",[48,769,128],{"class":93},[48,771,772],{"class":97}," unicodedata.normalize(",[48,774,775],{"class":58},"\"NFKC\"",[48,777,137],{"class":97},[48,779,780],{"class":209},"str",[48,782,783],{"class":97},"(text))\n",[48,785,786,788,790,793,796,799],{"class":50,"line":206},[48,787,767],{"class":97},[48,789,128],{"class":93},[48,791,792],{"class":209}," INVISIBLE",[48,794,795],{"class":97},".sub(",[48,797,798],{"class":58},"\"\"",[48,800,801],{"class":97},", text)\n",[48,803,804,806,809,811,813,816,819,821,823,826],{"class":50,"line":237},[48,805,258],{"class":93},[48,807,808],{"class":97}," re.sub(",[48,810,729],{"class":93},[48,812,289],{"class":58},[48,814,815],{"class":209},"\\s",[48,817,818],{"class":93},"+",[48,820,289],{"class":58},[48,822,137],{"class":97},[48,824,825],{"class":58},"\" \"",[48,827,828],{"class":97},", text).strip()\n",[48,830,831],{"class":50,"line":242},[48,832,119],{"emptyLinePlaceholder":118},[48,834,835,837,840],{"class":50,"line":255},[48,836,245],{"class":93},[48,838,839],{"class":248}," match_key",[48,841,252],{"class":97},[48,843,844,847,849,851,854],{"class":50,"line":326},[48,845,846],{"class":97},"    decomposed ",[48,848,128],{"class":93},[48,850,772],{"class":97},[48,852,853],{"class":58},"\"NFKD\"",[48,855,856],{"class":97},", clean_text(text))\n",[48,858,859,862,864,867,870,872,874,876,879,882,884],{"class":50,"line":331},[48,860,861],{"class":97},"    no_accents ",[48,863,128],{"class":93},[48,865,866],{"class":58}," \"\"",[48,868,869],{"class":97},".join(ch ",[48,871,315],{"class":93},[48,873,312],{"class":97},[48,875,320],{"class":93},[48,877,878],{"class":97}," decomposed ",[48,880,881],{"class":93},"if",[48,883,295],{"class":93},[48,885,886],{"class":97}," unicodedata.combining(ch))\n",[48,888,889,891],{"class":50,"line":355},[48,890,258],{"class":93},[48,892,893],{"class":97}," no_accents.casefold()\n",[48,895,897],{"class":50,"line":896},14,[48,898,119],{"emptyLinePlaceholder":118},[48,900,902,904,907,909,912,915,917,920,922,925,927,930,932,935],{"class":50,"line":901},15,[48,903,315],{"class":93},[48,905,906],{"class":97}," raw ",[48,908,320],{"class":93},[48,910,911],{"class":97}," [",[48,913,914],{"class":58},"\"Zürich​ Optik\"",[48,916,137],{"class":97},[48,918,919],{"class":58},"\"STRASSE & SÖHNE\"",[48,921,137],{"class":97},[48,923,924],{"class":58},"\"Straße & Söhne\"",[48,926,137],{"class":97},[48,928,929],{"class":58},"\"Ångström AB\"",[48,931,137],{"class":97},[48,933,934],{"class":58},"\"ANGSTROM AB\"",[48,936,352],{"class":97},[48,938,940,942,944,946,948,950,953,956,958,961,963,966,969,971,974,976,979,981,983,985],{"class":50,"line":939},16,[48,941,358],{"class":209},[48,943,361],{"class":97},[48,945,267],{"class":93},[48,947,289],{"class":58},[48,949,274],{"class":273},[48,951,952],{"class":97},"raw",[48,954,955],{"class":93},"!r:24",[48,957,286],{"class":273},[48,959,960],{"class":58}," clean=",[48,962,274],{"class":273},[48,964,965],{"class":97},"clean_text(raw)",[48,967,968],{"class":93},"!r:20",[48,970,286],{"class":273},[48,972,973],{"class":58}," key=",[48,975,274],{"class":273},[48,977,978],{"class":97},"match_key(raw)",[48,980,652],{"class":93},[48,982,286],{"class":273},[48,984,289],{"class":58},[48,986,149],{"class":97},[10,988,989,992,993,996,997,1000,1001,1004,1005,1008,1009,1012,1013,1016,1017,1020],{},[14,990,991],{},"clean_text"," produces a tidy, still-accented value suitable for display: the zero-width space is gone and spacing is normal. ",[14,994,995],{},"match_key"," decomposes characters with NFKD so each accent becomes a separate combining mark, drops the marks with ",[14,998,999],{},"unicodedata.combining",", and case-folds. ",[14,1002,1003],{},"casefold"," rather than ",[14,1006,1007],{},"lower"," matters for German: ",[14,1010,1011],{},"\"Straße\".casefold()"," is ",[14,1014,1015],{},"\"strasse\"",", so \"Straße & Söhne\" and \"STRASSE & SÖHNE\" produce the same key, which ",[14,1018,1019],{},"lower()"," would not achieve. \"Ångström AB\" and \"ANGSTROM AB\" match for the same reason. The key is for comparing, never for showing — \"Muller\" is not how Mr Müller spells his name.",[402,1022,411,1028,411,1031,411,1034,411,1039,411,1045,411,1049,411,1055,411,1059,411,1064,411,1066,411,1070,411,1073,411,1076,411,1078,411,1082,411,1085,411,1088,411,1090,411,1093,411,1096,411,1099,411,1101,411,1104,411,1107,411,1111,411,1113,411,1116,411,1121,411,1126,411,1131,411,1136,411,1139,411,1142,411,1145,411,1149,411,1152,411,1155,411,1160,411,1163,411,1169,411,1174,411,1182],{"viewBox":1023,"role":405,"ariaLabelledBy":1024,"xmlns":409,"style":1027},"79 0 601 220",[1025,1026],"uni-key-t","uni-key-d","width:100%;max-width:601px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif",[413,1029,1030],{"id":1025},"From raw text to a display value and a match key",[417,1032,1033],{"id":1026},"Straße & Söhne and STRASSE & SÖHNE clean to different display values but the same match key, strasse & sohne, after NFKC normalisation, removal of invisible characters, NFKD decomposition, removal of combining accents and casefolding.",[421,1035],{"x":1036,"y":424,"width":1037,"height":1038,"fill":427},"79","601","220",[421,1040],{"x":1041,"y":1042,"width":1043,"height":1044,"rx":424,"fill":435,"stroke":436,"style":461},"95.9","40.0","32.0","24.0",[421,1046],{"x":1047,"y":1042,"width":1048,"height":1044,"rx":424,"fill":435,"stroke":436,"style":461},"127.9","125.0",[450,1050,1054],{"x":1051,"y":1052,"style":1053},"190.4","57.0","font-size:11px;font-weight:600;fill:var(--muted,#5b6780);text-anchor:middle","A",[421,1056],{"x":1041,"y":1057,"width":1043,"height":1058,"rx":424,"fill":435,"stroke":436,"style":461},"64.0","28.0",[450,1060,1063],{"x":1061,"y":1062,"style":1053},"111.9","83.0","1",[421,1065],{"x":1047,"y":1057,"width":1048,"height":1058,"rx":424,"fill":427,"stroke":436,"style":461},[450,1067,1069],{"x":1051,"y":1062,"style":1068},"font-size:10.5px;font-weight:700;fill:var(--text,#172033);text-anchor:middle","raw value",[421,1071],{"x":1041,"y":1072,"width":1043,"height":1058,"rx":424,"fill":435,"stroke":436,"style":461},"92.0",[450,1074,1075],{"x":1061,"y":464,"style":1053},"2",[421,1077],{"x":1047,"y":1072,"width":1048,"height":1058,"rx":424,"fill":427,"stroke":436,"style":461},[450,1079,1081],{"x":1051,"y":464,"style":1080},"font-size:10.5px;font-weight:400;fill:var(--text,#172033);text-anchor:middle","Straße & Söhne",[421,1083],{"x":1041,"y":1084,"width":1043,"height":1058,"rx":424,"fill":435,"stroke":436,"style":461},"120.0",[450,1086,349],{"x":1061,"y":1087,"style":1053},"139.0",[421,1089],{"x":1047,"y":1084,"width":1048,"height":1058,"rx":424,"fill":427,"stroke":436,"style":461},[450,1091,1092],{"x":1051,"y":1087,"style":1080},"STRASSE & SÖHNE",[421,1094],{"x":1041,"y":1095,"width":1043,"height":1058,"rx":424,"fill":435,"stroke":436,"style":461},"148.0",[450,1097,423],{"x":1061,"y":1098,"style":1053},"167.0",[421,1100],{"x":1047,"y":1095,"width":1048,"height":1058,"rx":424,"fill":427,"stroke":436,"style":461},[450,1102,1103],{"x":1051,"y":1098,"style":1080},"Ångström AB",[421,1105],{"x":1041,"y":1106,"width":1043,"height":1058,"rx":424,"fill":435,"stroke":436,"style":461},"176.0",[450,1108,1110],{"x":1061,"y":1109,"style":1053},"195.0","5",[421,1112],{"x":1047,"y":1106,"width":1048,"height":1058,"rx":424,"fill":427,"stroke":436,"style":461},[450,1114,1115],{"x":1051,"y":1109,"style":1080},"ANGSTROM AB",[421,1117],{"x":1118,"y":1042,"width":1119,"height":1058,"rx":424,"fill":435,"stroke":436,"style":1120},"554.2","109.9","stroke-width:1.5px",[450,1122,1125],{"x":1123,"y":1124,"style":1068},"609.1","59.0","match key",[421,1127],{"x":1118,"y":1128,"width":1119,"height":1058,"rx":424,"fill":1129,"stroke":1130,"style":1120},"68.0","#d9f4f1","var(--teal,#0f9488)",[450,1132,1135],{"x":1123,"y":1133,"style":1134},"87.0","font-size:10.5px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","strasse & sohne",[421,1137],{"x":1118,"y":1138,"width":1119,"height":1058,"rx":424,"fill":1129,"stroke":1130,"style":1120},"96.0",[450,1140,1135],{"x":1123,"y":1141,"style":1134},"115.0",[421,1143],{"x":1118,"y":1144,"width":1119,"height":1058,"rx":424,"fill":1129,"stroke":1130,"style":1120},"124.0",[450,1146,1148],{"x":1123,"y":1147,"style":1134},"143.0","angstrom ab",[421,1150],{"x":1118,"y":1151,"width":1119,"height":1058,"rx":424,"fill":1129,"stroke":1130,"style":1120},"152.0",[450,1153,1148],{"x":1123,"y":1154,"style":1134},"171.0",[450,1156,952],{"x":1157,"y":1158,"style":1159},"174.4","29.0","font-size:12.5px;font-weight:700;fill:var(--muted,#5b6780);text-anchor:middle",[450,1161,1162],{"x":1123,"y":1158,"style":1159},"key",[50,1164],{"x1":1165,"y1":1166,"x2":1167,"y2":1166,"stroke":1168,"style":437},"262.9","110.0","540.2","var(--brand,#5b5cf0)",[1170,1171],"polygon",{"points":1172,"fill":1173},"544.2,110.0 535.2,115.0 535.2,105.0","#5b5cf0",[421,1175],{"x":1176,"y":1177,"width":1178,"height":1179,"rx":1180,"fill":427,"stroke":436,"style":1181},"299.0","77.0","209.0","22.0","11","stroke-width:1.2px",[450,1183,1186],{"x":1184,"y":1072,"style":1185},"403.5","font-size:11.5px;font-weight:400;fill:var(--brand-strong,#4338ca);text-anchor:middle;font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace","clean_text() → match_key()",[34,1188,1190],{"id":1189},"step-4-match-on-keys-keep-the-originals","Step 4 — Match on keys, keep the originals",[10,1192,1193],{},"Merge on the keys, while keeping each side's original text for display:",[39,1195,1197],{"className":84,"code":1196,"language":86,"meta":44,"style":44},"crm[\"key\"] = crm[\"Customer\"].map(match_key)\nweb[\"key\"] = web[\"Customer\"].map(match_key)\nmatched = web.merge(crm[[\"key\", \"Account\", \"Customer\"]], on=\"key\", how=\"left\", suffixes=(\"\", \" (CRM)\"))\nprint(matched[[\"Customer\", \"Customer (CRM)\", \"Account\", \"Order total\"]].to_string(index=False))\nprint(\"unmatched:\", matched[\"Account\"].isna().sum())\n",[14,1198,1199,1220,1237,1292,1325],{"__ignoreMap":44},[48,1200,1201,1204,1207,1210,1212,1215,1217],{"class":50,"line":51},[48,1202,1203],{"class":97},"crm[",[48,1205,1206],{"class":58},"\"key\"",[48,1208,1209],{"class":97},"] ",[48,1211,128],{"class":93},[48,1213,1214],{"class":97}," crm[",[48,1216,191],{"class":58},[48,1218,1219],{"class":97},"].map(match_key)\n",[48,1221,1222,1225,1227,1229,1231,1233,1235],{"class":50,"line":101},[48,1223,1224],{"class":97},"web[",[48,1226,1206],{"class":58},[48,1228,1209],{"class":97},[48,1230,128],{"class":93},[48,1232,341],{"class":97},[48,1234,191],{"class":58},[48,1236,1219],{"class":97},[48,1238,1239,1242,1244,1247,1249,1251,1253,1255,1257,1260,1262,1264,1266,1268,1270,1272,1274,1276,1279,1281,1283,1285,1287,1290],{"class":50,"line":115},[48,1240,1241],{"class":97},"matched ",[48,1243,128],{"class":93},[48,1245,1246],{"class":97}," web.merge(crm[[",[48,1248,1206],{"class":58},[48,1250,137],{"class":97},[48,1252,220],{"class":58},[48,1254,137],{"class":97},[48,1256,191],{"class":58},[48,1258,1259],{"class":97},"]], ",[48,1261,186],{"class":140},[48,1263,128],{"class":93},[48,1265,1206],{"class":58},[48,1267,137],{"class":97},[48,1269,196],{"class":140},[48,1271,128],{"class":93},[48,1273,201],{"class":58},[48,1275,137],{"class":97},[48,1277,1278],{"class":140},"suffixes",[48,1280,128],{"class":93},[48,1282,361],{"class":97},[48,1284,798],{"class":58},[48,1286,137],{"class":97},[48,1288,1289],{"class":58},"\" (CRM)\"",[48,1291,234],{"class":97},[48,1293,1294,1296,1299,1301,1303,1306,1308,1310,1312,1315,1317,1319,1321,1323],{"class":50,"line":122},[48,1295,210],{"class":209},[48,1297,1298],{"class":97},"(matched[[",[48,1300,191],{"class":58},[48,1302,137],{"class":97},[48,1304,1305],{"class":58},"\"Customer (CRM)\"",[48,1307,137],{"class":97},[48,1309,220],{"class":58},[48,1311,137],{"class":97},[48,1313,1314],{"class":58},"\"Order total\"",[48,1316,223],{"class":97},[48,1318,226],{"class":140},[48,1320,128],{"class":93},[48,1322,231],{"class":209},[48,1324,234],{"class":97},[48,1326,1327,1329,1331,1334,1337,1339],{"class":50,"line":152},[48,1328,210],{"class":209},[48,1330,361],{"class":97},[48,1332,1333],{"class":58},"\"unmatched:\"",[48,1335,1336],{"class":97},", matched[",[48,1338,220],{"class":58},[48,1340,1341],{"class":97},"].isna().sum())\n",[10,1343,1344,1345,32],{},"All six web orders now find their account. The output shows both spellings side by side — the web's shouted \"STRASSE & SÖHNE\" next to the CRM's \"Straße & Söhne\" — and the report can use the CRM's version for display. Matching on accent-free keys can occasionally join names that are genuinely different only by an accent; check the keys for duplicates on the reference side before merging, as in ",[28,1346,1348],{"href":1347},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Fvalidate-a-merge-to-avoid-duplicate-rows\u002F","Validate a Merge to Avoid Duplicate Rows",[34,1350,1352],{"id":1351},"step-5-smart-quotes-dashes-and-other-look-alikes","Step 5 — Smart quotes, dashes and other look-alikes",[10,1354,1355],{},"Text pasted from word processors also brings typographic punctuation that NFKC leaves alone: curly quotes, en and em dashes, ellipses. If a downstream system expects plain ASCII, or you match on these characters, translate them explicitly:",[39,1357,1359],{"className":84,"code":1358,"language":86,"meta":44,"style":44},"PUNCT = str.maketrans({\"‘\": \"'\", \"’\": \"'\", \"“\": '\"', \"”\": '\"',\n                       \"–\": \"-\", \"—\": \"-\", \"…\": \"...\", \"´\": \"'\"})\npasted = \"O’Brien – “Premium” range…\"\nprint(pasted, \"→\", pasted.translate(PUNCT))\n\ndef find_odd(series):\n    odd = series.map(lambda s: sorted({f\"U+{ord(c):04X} {unicodedata.name(c, '?')}\" for c in str(s)\n                                      if not c.isascii() and unicodedata.category(c)[0] in \"ZPC\"}))\n    return odd[odd.str.len() > 0]\n\nprint(find_odd(web[\"Customer\"]).to_dict())\n",[14,1360,1361,1413,1454,1464,1480,1484,1494,1559,1587,1603,1607],{"__ignoreMap":44},[48,1362,1363,1366,1368,1371,1374,1377,1379,1382,1384,1387,1389,1391,1393,1396,1398,1401,1403,1406,1408,1410],{"class":50,"line":51},[48,1364,1365],{"class":209},"PUNCT",[48,1367,723],{"class":93},[48,1369,1370],{"class":209}," str",[48,1372,1373],{"class":97},".maketrans({",[48,1375,1376],{"class":58},"\"‘\"",[48,1378,550],{"class":97},[48,1380,1381],{"class":58},"\"'\"",[48,1383,137],{"class":97},[48,1385,1386],{"class":58},"\"’\"",[48,1388,550],{"class":97},[48,1390,1381],{"class":58},[48,1392,137],{"class":97},[48,1394,1395],{"class":58},"\"“\"",[48,1397,550],{"class":97},[48,1399,1400],{"class":58},"'\"'",[48,1402,137],{"class":97},[48,1404,1405],{"class":58},"\"”\"",[48,1407,550],{"class":97},[48,1409,1400],{"class":58},[48,1411,1412],{"class":97},",\n",[48,1414,1415,1418,1420,1423,1425,1428,1430,1432,1434,1437,1439,1442,1444,1447,1449,1451],{"class":50,"line":101},[48,1416,1417],{"class":58},"                       \"–\"",[48,1419,550],{"class":97},[48,1421,1422],{"class":58},"\"-\"",[48,1424,137],{"class":97},[48,1426,1427],{"class":58},"\"—\"",[48,1429,550],{"class":97},[48,1431,1422],{"class":58},[48,1433,137],{"class":97},[48,1435,1436],{"class":58},"\"…\"",[48,1438,550],{"class":97},[48,1440,1441],{"class":58},"\"...\"",[48,1443,137],{"class":97},[48,1445,1446],{"class":58},"\"´\"",[48,1448,550],{"class":97},[48,1450,1381],{"class":58},[48,1452,1453],{"class":97},"})\n",[48,1455,1456,1459,1461],{"class":50,"line":115},[48,1457,1458],{"class":97},"pasted ",[48,1460,128],{"class":93},[48,1462,1463],{"class":58}," \"O’Brien – “Premium” range…\"\n",[48,1465,1466,1468,1471,1473,1476,1478],{"class":50,"line":122},[48,1467,210],{"class":209},[48,1469,1470],{"class":97},"(pasted, ",[48,1472,370],{"class":58},[48,1474,1475],{"class":97},", pasted.translate(",[48,1477,1365],{"class":209},[48,1479,234],{"class":97},[48,1481,1482],{"class":50,"line":152},[48,1483,119],{"emptyLinePlaceholder":118},[48,1485,1486,1488,1491],{"class":50,"line":175},[48,1487,245],{"class":93},[48,1489,1490],{"class":248}," find_odd",[48,1492,1493],{"class":97},"(series):\n",[48,1495,1496,1499,1501,1504,1507,1510,1513,1516,1518,1520,1522,1524,1527,1529,1531,1533,1536,1539,1542,1544,1546,1549,1552,1554,1556],{"class":50,"line":206},[48,1497,1498],{"class":97},"    odd ",[48,1500,128],{"class":93},[48,1502,1503],{"class":97}," series.map(",[48,1505,1506],{"class":93},"lambda",[48,1508,1509],{"class":97}," s: ",[48,1511,1512],{"class":209},"sorted",[48,1514,1515],{"class":97},"({",[48,1517,267],{"class":93},[48,1519,270],{"class":58},[48,1521,274],{"class":273},[48,1523,277],{"class":209},[48,1525,1526],{"class":97},"(c)",[48,1528,283],{"class":93},[48,1530,286],{"class":273},[48,1532,544],{"class":273},[48,1534,1535],{"class":97},"unicodedata.name(c, ",[48,1537,1538],{"class":58},"'?'",[48,1540,1541],{"class":97},")",[48,1543,286],{"class":273},[48,1545,289],{"class":58},[48,1547,1548],{"class":93}," for",[48,1550,1551],{"class":97}," c ",[48,1553,320],{"class":93},[48,1555,1370],{"class":209},[48,1557,1558],{"class":97},"(s)\n",[48,1560,1561,1564,1566,1569,1572,1575,1577,1579,1581,1584],{"class":50,"line":237},[48,1562,1563],{"class":93},"                                      if",[48,1565,295],{"class":93},[48,1567,1568],{"class":97}," c.isascii() ",[48,1570,1571],{"class":93},"and",[48,1573,1574],{"class":97}," unicodedata.category(c)[",[48,1576,424],{"class":209},[48,1578,1209],{"class":97},[48,1580,320],{"class":93},[48,1582,1583],{"class":58}," \"ZPC\"",[48,1585,1586],{"class":97},"}))\n",[48,1588,1589,1591,1594,1597,1600],{"class":50,"line":242},[48,1590,258],{"class":93},[48,1592,1593],{"class":97}," odd[odd.str.len() ",[48,1595,1596],{"class":93},">",[48,1598,1599],{"class":209}," 0",[48,1601,1602],{"class":97},"]\n",[48,1604,1605],{"class":50,"line":255},[48,1606,119],{"emptyLinePlaceholder":118},[48,1608,1609,1611,1614,1616],{"class":50,"line":326},[48,1610,210],{"class":209},[48,1612,1613],{"class":97},"(find_odd(web[",[48,1615,191],{"class":58},[48,1617,1618],{"class":97},"]).to_dict())\n",[10,1620,1621,1624,1625,1628,1629,32],{},[14,1622,1623],{},"str.translate"," with a mapping table replaces each typographic character with its plain equivalent in one pass: O'Brien keeps its apostrophe, but now a straight one, so it matches the same name typed on a keyboard. ",[14,1626,1627],{},"find_odd"," lists every non-ASCII space, punctuation or control character in a column, with its official Unicode name — a quick audit of what a column actually contains, run before deciding what to normalise. Mojibake — text that was decoded with the wrong encoding, such as \"Café\" — is a different problem with a different fix, covered in ",[28,1630,1632],{"href":1631},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Ffix-garbled-characters-in-excel-data\u002F","Fix Garbled Characters in Excel Data",[402,1634,411,1639,411,1642,411,1645,411,1648,411,1653,411,1659,411,1665,411,1670,411,1675,411,1678,411,1681,411,1688,411,1694,411,1698,411,1702,411,1705,411,1708,411,1713,411,1720,411,1725,411,1728,411,1732,411,1735,411,1738,411,1745,411,1750,411,1753],{"viewBox":1635,"role":405,"ariaLabelledBy":1636,"xmlns":409,"style":410},"4 0 752 142",[1637,1638],"uni-flow-t","uni-flow-d",[413,1640,1641],{"id":1637},"A text-normalisation routine",[417,1643,1644],{"id":1638},"Audit the column for unusual characters, normalise with NFKC, remove zero-width characters and collapse spaces for a clean display value, translate typographic punctuation if needed, and build a casefolded, accent-free key used only for matching.",[421,1646],{"x":423,"y":424,"width":425,"height":1647,"fill":427},"142",[421,1649],{"x":430,"y":431,"width":1650,"height":1651,"rx":1652,"fill":435,"stroke":436,"style":437},"153.0","110.5","12",[439,1654,1656],{"transform":1655},"translate(84.5,37.8) scale(1.000)",[443,1657],{"d":489,"fill":446,"stroke":447,"style":1658},"stroke-width:1.80px;stroke-linecap:round;stroke-linejoin:round",[450,1660,1664],{"x":1661,"y":1662,"style":1663},"96.5","84.2","font-size:13.5px;font-weight:700;fill:var(--text,#172033);text-anchor:middle","Audit",[450,1666,1669],{"x":1661,"y":1667,"style":1668},"103.8","font-size:11.5px;font-weight:400;fill:var(--text,#172033);text-anchor:middle","find_odd(), reveal()",[50,1671],{"x1":1672,"y1":1673,"x2":1674,"y2":1673,"stroke":1168,"style":437},"178.0","71.2","200.0",[1170,1676],{"points":1677,"fill":1173},"204.0,71.2 195.0,76.2 195.0,66.2",[421,1679],{"x":1178,"y":431,"width":1650,"height":1651,"rx":1652,"fill":1680,"stroke":1168,"style":437},"#f0f4ff",[439,1682,1684],{"transform":1683},"translate(273.5,29.0) scale(1.000)",[443,1685],{"d":1686,"fill":446,"stroke":1687,"style":1658},"M20 11a8 8 0 1 0-2.3 5.7 M20 4v7h-7","var(--brand-strong,#4338ca)",[450,1689,1693],{"x":1690,"y":1691,"style":1692},"285.5","75.5","font-size:13.5px;font-weight:700;fill:var(--brand-strong,#4338ca);text-anchor:middle","NFKC + invisible",[450,1695,1697],{"x":1690,"y":1696,"style":1668},"95.0","one form, no",[450,1699,1701],{"x":1690,"y":1700,"style":1668},"112.5","zero-width",[50,1703],{"x1":1704,"y1":1673,"x2":481,"y2":1673,"stroke":1168,"style":437},"367.0",[1170,1706],{"points":1707,"fill":1173},"393.0,71.2 384.0,76.2 384.0,66.2",[421,1709],{"x":1710,"y":431,"width":1650,"height":1651,"rx":1652,"fill":1711,"stroke":1712,"style":437},"398.0","#fdefd8","var(--gold,#b4740a)",[439,1714,1716],{"transform":1715},"translate(462.5,37.8) scale(1.000)",[443,1717],{"d":1718,"fill":446,"stroke":1719,"style":1658},"M6 3h9l4 4v14H6z M15 3v4h4","var(--gold-ink,#7a4e06)",[450,1721,1724],{"x":1722,"y":1662,"style":1723},"474.5","font-size:13.5px;font-weight:700;fill:var(--gold-ink,#7a4e06);text-anchor:middle","Punctuation",[450,1726,1727],{"x":1722,"y":1667,"style":1668},"smart quotes → ASCII",[50,1729],{"x1":1730,"y1":1673,"x2":1731,"y2":1673,"stroke":1168,"style":437},"556.0","578.0",[1170,1733],{"points":1734,"fill":1173},"582.0,71.2 573.0,76.2 573.0,66.2",[421,1736],{"x":1737,"y":431,"width":1650,"height":1651,"rx":1652,"fill":1129,"stroke":1130,"style":437},"587.0",[439,1739,1741],{"transform":1740},"translate(651.5,29.0) scale(1.000)",[443,1742],{"d":1743,"fill":446,"stroke":1744,"style":1658},"M10 14a4 4 0 0 0 5.7 0l3-3a4 4 0 0 0-5.7-5.7l-1 1 M14 10a4 4 0 0 0-5.7 0l-3 3a4 4 0 0 0 5.7 5.7l1-1","var(--teal-ink,#0b6157)",[450,1746,1749],{"x":1747,"y":1691,"style":1748},"663.5","font-size:13.5px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","Match key",[450,1751,1752],{"x":1747,"y":1696,"style":1668},"NFKD, no accents,",[450,1754,1003],{"x":1747,"y":1700,"style":1668},[34,1756,1758],{"id":1757},"where-to-normalise","Where to normalise",[10,1760,1761,1762,1765,1766,1769,1770,1772],{},"Normalise as early as possible — right after reading each source — so every later step, from filters to joins to pivot tables, sees one form. Store the cleaned display value (NFKC, no invisible characters) in place of the raw text if the raw form has no value of its own, or alongside it if the original must be preserved for audit; compute match keys on the fly or in a separate column, never in place of the name. When writing results back to Excel, write the cleaned values: a workbook full of non-breaking spaces and zero-width characters will cause the same lookup failures for whoever uses it next, this time in ",[14,1763,1764],{},"VLOOKUP"," and ",[14,1767,1768],{},"COUNTIF"," formulas that are even harder to debug than pandas merges. And when a new source joins the pipeline, run the ",[14,1771,1627],{}," audit on its text columns first — it takes a second and tells you exactly which of these problems that source brings.",[34,1774,1776],{"id":1775},"common-pitfalls","Common pitfalls",[1778,1779,1780,1788,1794,1800,1812],"ul",{},[1781,1782,1783,1787],"li",{},[1784,1785,1786],"strong",{},"Assuming text that looks equal is equal."," Print code points when comparisons fail.",[1781,1789,1790,1793],{},[1784,1791,1792],{},"NFC when the problem is a non-breaking space or ligature."," Use NFKC for cleaning.",[1781,1795,1796,1799],{},[1784,1797,1798],{},"Forgetting zero-width characters."," No normal form removes them; strip them explicitly.",[1781,1801,1802,1807,1808,1811],{},[1784,1803,1804,1806],{},[14,1805,1019],{}," for matching."," Use ",[14,1809,1810],{},"casefold()",", which handles ß and other special cases.",[1781,1813,1814,1817],{},[1784,1815,1816],{},"Displaying the accent-free key."," Keep the original spelling for anything people read.",[34,1819,1821],{"id":1820},"conclusion","Conclusion",[10,1823,1824,1825,1828,1829,1831],{},"Text from different sources can look identical and still not match, because Unicode stores accents, spaces, ligatures and full-width characters in more than one way and allows invisible characters. Normalise with ",[14,1826,1827],{},"unicodedata.normalize(\"NFKC\", ...)",", remove zero-width characters and collapse whitespace for a clean display value, translate typographic punctuation when needed, and build a separate match key with NFKD, accent removal and ",[14,1830,1810],{},". Merge and de-duplicate on the keys, and keep the original spellings for display.",[34,1833,1835],{"id":1834},"frequently-asked-questions","Frequently asked questions",[10,1837,1838,1841],{},[1784,1839,1840],{},"Why don't two identical-looking strings match in pandas?","\nThey contain different Unicode characters, such as a decomposed accent, a non-breaking space or an invisible zero-width space. Normalise both with unicodedata and remove invisible characters before comparing.",[10,1843,1844,1847],{},[1784,1845,1846],{},"What is the difference between NFC and NFKC?","\nNFC combines letters and accents into single characters. NFKC also replaces compatibility characters, such as ligatures, full-width letters and non-breaking spaces, with their plain equivalents.",[10,1849,1850,1853],{},[1784,1851,1852],{},"How do I remove accents from text in Python?","\nNormalise with NFKD so accents become separate combining marks, then keep only characters for which unicodedata.combining returns 0.",[10,1855,1856,1859],{},[1784,1857,1858],{},"Should I store text without accents?","\nUse accent-free text only as a key for matching and de-duplication. Keep and display the original spelling.",[34,1861,1863],{"id":1862},"related","Related",[1778,1865,1866,1872,1879,1886],{},[1781,1867,1868,1869,1871],{},"Up one level: ",[28,1870,31],{"href":30}," — tidy data before analysis.",[1781,1873,1874,1878],{},[28,1875,1877],{"href":1876},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fremove-non-printable-characters-from-excel-data\u002F","Remove Non-Printable Characters from Excel Data"," — control characters and other invisibles.",[1781,1880,1881,1885],{},[28,1882,1884],{"href":1883},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Fmerge-on-keys-that-differ-in-case-and-spacing-with-pandas\u002F","Merge on Keys That Differ in Case and Spacing with pandas"," — simpler key normalisation.",[1781,1887,1888,1892],{},[28,1889,1891],{"href":1890},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Ffix-the-capitalisation-of-names-with-exceptions-in-pandas\u002F","Fix the Capitalisation of Names with Exceptions in pandas"," — the display side of name cleaning.",[1894,1895,1896],"style",{},"html pre.shiki code .sMTad, html code.shiki .sMTad{--shiki-default:#6F42C1;--shiki-dark:#FFB757}html pre.shiki code .srMev, html code.shiki .srMev{--shiki-default:#032F62;--shiki-dark:#ADDCFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .s-kum, html code.shiki .s-kum{--shiki-default:#D73A49;--shiki-dark:#FF9492}html pre.shiki code .skGVy, html code.shiki .skGVy{--shiki-default:#24292E;--shiki-dark:#F0F3F6}html pre.shiki code .sa561, html code.shiki .sa561{--shiki-default:#E36209;--shiki-dark:#FFB757}html pre.shiki code .sP0c6, html code.shiki .sP0c6{--shiki-default:#005CC5;--shiki-dark:#91CBFF}html pre.shiki code .s_Opv, html code.shiki .s_Opv{--shiki-default:#6F42C1;--shiki-dark:#DBB7FF}html pre.shiki code .sSjpA, html code.shiki .sSjpA{--shiki-default:#005CC5;--shiki-dark:#FF9492}html pre.shiki code .s-wDw, html code.shiki .s-wDw{--shiki-default:#6A737D;--shiki-dark:#BDC4CC}",{"title":44,"searchDepth":101,"depth":101,"links":1898},[1899,1900,1901,1902,1903,1904,1905,1906,1907,1908,1909],{"id":36,"depth":101,"text":37},{"id":77,"depth":101,"text":78},{"id":514,"depth":101,"text":515},{"id":691,"depth":101,"text":692},{"id":1189,"depth":101,"text":1190},{"id":1351,"depth":101,"text":1352},{"id":1757,"depth":101,"text":1758},{"id":1775,"depth":101,"text":1776},{"id":1820,"depth":101,"text":1821},{"id":1834,"depth":101,"text":1835},{"id":1862,"depth":101,"text":1863},"2026-10-02","Make text from different sources compare equal when it looks equal: Unicode normal forms NFC and NFKC for composed and decomposed accents, full-width characters, ligatures and non-breaking spaces, removing zero-width characters, casefold instead of lower, stripping accents for matching keys while keeping the original for display, and converting smart quotes and dashes.","md",[1914,1916,1918,1920],{"q":1840,"a":1915},"They contain different Unicode characters, such as a decomposed accent, a non-breaking space or an invisible zero-width space. Normalise both with unicodedata and remove invisible characters before comparing.",{"q":1846,"a":1917},"NFC combines letters and accents into single characters. NFKC also replaces compatibility characters, such as ligatures, full-width letters and non-breaking spaces, with their plain equivalents.",{"q":1852,"a":1919},"Normalise with NFKD so accents become separate combining marks, then keep only characters for which unicodedata.combining returns 0.",{"q":1858,"a":1921},"Use accent-free text only as a key for matching and de-duplication. Keep and display the original spelling.","remove accents and normalise unicode in excel text",null,{"breadcrumb":1925},[1926,1929,1932],{"name":1927,"item":1928},"Home","\u002F",{"name":1930,"item":1931},"Advanced Data Transformation and Cleaning for Excel Automation","\u002Fadvanced-data-transformation-and-cleaning\u002F",{"name":1933,"item":30},"Cleaning Excel Data With Pandas: A Production Workflow","\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fremove-accents-and-normalise-unicode-in-excel-text",{"title":1936,"description":1937},"Remove Accents and Normalise Unicode in Excel Data with Python","Fix text that looks identical but does not match: unicodedata NFC and NFKC, decomposed accents, non-breaking and zero-width spaces, casefold, accent-free match keys, and smart quotes converted to plain ASCII.","remove-accents-and-normalise-unicode-in-excel-text","advanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fremove-accents-and-normalise-unicode-in-excel-text\u002Findex",[1941],"data-cleaning","how-to","wX2dqUNIQesQTo_MZzIvj50IWvpa5McNBacrpT6-xYA",[1945,1949],{"title":1946,"path":1947,"stem":1948,"children":-1},"Pandas: Drop Duplicates From an Excel Column","\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fpandas-drop-duplicates-from-excel-column","advanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fpandas-drop-duplicates-from-excel-column\u002Findex",{"title":1950,"path":1951,"stem":1952,"children":-1},"Remove Blank Rows From Excel With Pandas","\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fremove-blank-rows-from-excel-with-pandas","advanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fremove-blank-rows-from-excel-with-pandas\u002Findex",[1954,1956],{"path":1955,"title":1933},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas",{"path":1957,"title":1930},"\u002Fadvanced-data-transformation-and-cleaning",[],[1960,1963,1966,1969,1972,1975],{"path":1961,"title":1962,"kind":1942},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Ffind-duplicate-rows-in-excel-with-python","Find Duplicate Rows in Excel with Python",{"path":1964,"title":1965,"kind":1942},"\u002Fadvanced-data-transformation-and-cleaning\u002Fapplying-conditional-formatting-with-openpyxl\u002Fhighlight-duplicate-values-in-excel-with-openpyxl","Highlight Duplicate Values in Excel with openpyxl",{"path":1967,"title":1968,"kind":1942},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fclean-mixed-currency-amounts-into-amount-and-currency-columns","Clean Mixed-Currency Amounts into Amount and Currency Columns",{"path":1970,"title":1971,"kind":1942},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fcoalesce-several-excel-columns-into-one-with-pandas","Coalesce Several Excel Columns into One with pandas",{"path":1973,"title":1974,"kind":1942},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fconvert-currency-strings-to-numbers-with-pandas","Convert Currency Strings Like \"$1,234.56\" and \"(500)\" to Numbers with pandas",{"path":1976,"title":1977,"kind":1942},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fconvert-excel-text-columns-to-numbers-with-pandas","Convert Excel Text Columns to Numbers with pandas",1790972082896]