[{"data":1,"prerenderedAt":2481},["ShallowReactive",2],{"doc:\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fstrip-whitespace-and-normalise-text-columns-with-pandas":3,"surround:\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fstrip-whitespace-and-normalise-text-columns-with-pandas":2472},{"id":4,"title":5,"body":6,"dateModified":2447,"datePublished":2447,"description":2448,"extension":2449,"faq":2450,"meta":2463,"navigation":192,"path":2464,"seo":2465,"slug":2468,"stem":2469,"type":2470,"__hash__":2471},"docs\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fstrip-whitespace-and-normalise-text-columns-with-pandas\u002Findex.md","Strip Whitespace and Normalise Text Columns with pandas",{"type":7,"value":8,"toc":2435},"minimark",[9,24,128,133,161,164,320,324,331,388,391,625,628,631,683,687,697,988,1009,1013,1021,1166,1209,1218,1296,1300,1303,1392,1606,1609,1770,1781,1785,1928,1932,1994,2001,2011,2056,2066,2238,2245,2313,2321,2325,2331,2335,2344,2354,2374,2384,2390,2394,2431],[10,11,12,13,17,18,23],"p",{},"Two spreadsheets both say ",[14,15,16],"code",{},"North",". The join matches nothing. This is the single most common data-cleaning problem coming out of Excel, and it is invisible by construction: a trailing space, a non-breaking space pasted from a web page, or an accented character stored two different ways all render identically on screen and compare as different strings. This guide covers finding them, fixing them, and building a normalised key column that makes joins and group-bys reliable. It is part of ",[19,20,22],"a",{"href":21},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002F","Cleaning Excel Data with pandas",".",[25,26,35,36,35,40,35,44,35,51,35,58,35,62,35,70,35,75,35,81,35,86,35,89,35,92,35,96,35,100,35,103,35,106,35,108,35,111,35,114,35,117,35,119,35,122],"svg",{"viewBox":27,"role":28,"ariaLabel":29,"ariaLabelledBy":30,"xmlns":33,"style":34},"0 0 800 252","img","Four values that all display as North but compare as different strings: plain, trailing space, non-breaking space, and a double inner space.",[31,32],"ws-t","ws-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;max-width:800px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif","\n  ",[37,38,39],"title",{"id":31},"Four distinct strings that all look like \"North\"",[41,42,43],"desc",{"id":32},"Four cell values rendered identically on screen. The first is the plain word North. The second has a trailing ordinary space. The third uses a non-breaking space instead of a normal one in a two-word name. The fourth has two spaces between the words. All four display the same way, all four compare as different strings, and a group-by therefore produces four groups where the reader expects one.",[45,46],"rect",{"x":47,"y":47,"width":48,"height":49,"fill":50},"0","800","252","#ffffff",[52,53,57],"text",{"x":54,"y":55,"style":56},"180","30","font-size:12px;font-weight:700;fill:var(--muted,#5b6780);text-anchor:middle","what the reader sees",[52,59,61],{"x":60,"y":55,"style":56},"580","what Python sees",[45,63],{"x":55,"y":64,"width":65,"height":66,"rx":67,"fill":50,"stroke":68,"style":69},"44","300","34","6","var(--line,#cdd5e6)","stroke-width:2px",[52,71,74],{"x":54,"y":72,"style":73},"66","font-size:11.5px;fill:var(--text,#172033);text-anchor:middle","North Region",[45,76],{"x":77,"y":64,"width":78,"height":66,"rx":67,"fill":79,"stroke":80,"style":69},"376","394","#d9f4f1","var(--teal,#0f9488)",[52,82,85],{"x":83,"y":72,"style":84},"573","font-size:11px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","'North Region'",[45,87],{"x":55,"y":88,"width":65,"height":66,"rx":67,"fill":50,"stroke":68,"style":69},"86",[52,90,74],{"x":54,"y":91,"style":73},"108",[45,93],{"x":77,"y":88,"width":78,"height":66,"rx":67,"fill":94,"stroke":95,"style":69},"#fee8f2","var(--accent,#f43f8f)",[52,97,99],{"x":83,"y":91,"style":98},"font-size:11px;font-weight:700;fill:var(--accent-ink,#be185d);text-anchor:middle","'North Region '  ← trailing space",[45,101],{"x":55,"y":102,"width":65,"height":66,"rx":67,"fill":50,"stroke":68,"style":69},"128",[52,104,74],{"x":54,"y":105,"style":73},"150",[45,107],{"x":77,"y":102,"width":78,"height":66,"rx":67,"fill":94,"stroke":95,"style":69},[52,109,110],{"x":83,"y":105,"style":98},"'North Region'  ← non-breaking",[45,112],{"x":55,"y":113,"width":65,"height":66,"rx":67,"fill":50,"stroke":68,"style":69},"170",[52,115,74],{"x":54,"y":116,"style":73},"192",[45,118],{"x":77,"y":113,"width":78,"height":66,"rx":67,"fill":94,"stroke":95,"style":69},[52,120,121],{"x":83,"y":116,"style":98},"'North  Region'  ← double space",[52,123,127],{"x":124,"y":125,"style":126},"400","232","font-size:11.5px;font-weight:700;fill:var(--accent-ink,#be185d);text-anchor:middle","a group-by produces four groups where the reader expects one",[129,130,132],"h2",{"id":131},"prerequisites","Prerequisites",[134,135,140],"pre",{"className":136,"code":137,"language":138,"meta":139,"style":139},"language-bash shiki shiki-themes github-light github-dark-high-contrast","pip install pandas openpyxl\n","bash","",[14,141,142],{"__ignoreMap":139},[143,144,147,151,155,158],"span",{"class":145,"line":146},"line",1,[143,148,150],{"class":149},"sMTad","pip",[143,152,154],{"class":153},"srMev"," install",[143,156,157],{"class":153}," pandas",[143,159,160],{"class":153}," openpyxl\n",[10,162,163],{},"A frame with every problem in it:",[134,165,169],{"className":166,"code":167,"language":168,"meta":139,"style":139},"language-python shiki shiki-themes github-light github-dark-high-contrast","import pandas as pd\n\ndf = pd.DataFrame({\n    \"region\": [\"North Region\", \"North Region \", \"North Region\",\n               \"North  Region\", \"north region\", \"Nörth Region\"],\n    \"revenue\": [5150.00, 4268.50, 3511.25, 2980.10, 3140.75, 1820.00],\n})\n\nprint(df[\"region\"].nunique())     # 6 — every one is distinct\n","python",[14,170,171,187,194,206,232,251,290,296,301],{"__ignoreMap":139},[143,172,173,177,181,184],{"class":145,"line":146},[143,174,176],{"class":175},"s-kum","import",[143,178,180],{"class":179},"skGVy"," pandas ",[143,182,183],{"class":175},"as",[143,185,186],{"class":179}," pd\n",[143,188,190],{"class":145,"line":189},2,[143,191,193],{"emptyLinePlaceholder":192},true,"\n",[143,195,197,200,203],{"class":145,"line":196},3,[143,198,199],{"class":179},"df ",[143,201,202],{"class":175},"=",[143,204,205],{"class":179}," pd.DataFrame({\n",[143,207,209,212,215,218,221,224,226,229],{"class":145,"line":208},4,[143,210,211],{"class":153},"    \"region\"",[143,213,214],{"class":179},": [",[143,216,217],{"class":153},"\"North Region\"",[143,219,220],{"class":179},", ",[143,222,223],{"class":153},"\"North Region \"",[143,225,220],{"class":179},[143,227,228],{"class":153},"\"North Region\"",[143,230,231],{"class":179},",\n",[143,233,235,238,240,243,245,248],{"class":145,"line":234},5,[143,236,237],{"class":153},"               \"North  Region\"",[143,239,220],{"class":179},[143,241,242],{"class":153},"\"north region\"",[143,244,220],{"class":179},[143,246,247],{"class":153},"\"Nörth Region\"",[143,249,250],{"class":179},"],\n",[143,252,254,257,259,263,265,268,270,273,275,278,280,283,285,288],{"class":145,"line":253},6,[143,255,256],{"class":153},"    \"revenue\"",[143,258,214],{"class":179},[143,260,262],{"class":261},"sP0c6","5150.00",[143,264,220],{"class":179},[143,266,267],{"class":261},"4268.50",[143,269,220],{"class":179},[143,271,272],{"class":261},"3511.25",[143,274,220],{"class":179},[143,276,277],{"class":261},"2980.10",[143,279,220],{"class":179},[143,281,282],{"class":261},"3140.75",[143,284,220],{"class":179},[143,286,287],{"class":261},"1820.00",[143,289,250],{"class":179},[143,291,293],{"class":145,"line":292},7,[143,294,295],{"class":179},"})\n",[143,297,299],{"class":145,"line":298},8,[143,300,193],{"emptyLinePlaceholder":192},[143,302,304,307,310,313,316],{"class":145,"line":303},9,[143,305,306],{"class":261},"print",[143,308,309],{"class":179},"(df[",[143,311,312],{"class":153},"\"region\"",[143,314,315],{"class":179},"].nunique())     ",[143,317,319],{"class":318},"s-wDw","# 6 — every one is distinct\n",[129,321,323],{"id":322},"step-1-see-what-is-actually-there","Step 1 — See what is actually there",[10,325,326,327,330],{},"Never diagnose whitespace by eye. ",[14,328,329],{},"repr"," shows the escapes:",[134,332,334],{"className":166,"code":333,"language":168,"meta":139,"style":139},"for value in df[\"region\"]:\n    print(repr(value))\n# 'North Region'\n# 'North Region '\n# 'North\\xa0Region'\n# 'North  Region'\n",[14,335,336,355,368,373,378,383],{"__ignoreMap":139},[143,337,338,341,344,347,350,352],{"class":145,"line":146},[143,339,340],{"class":175},"for",[143,342,343],{"class":179}," value ",[143,345,346],{"class":175},"in",[143,348,349],{"class":179}," df[",[143,351,312],{"class":153},[143,353,354],{"class":179},"]:\n",[143,356,357,360,363,365],{"class":145,"line":189},[143,358,359],{"class":261},"    print",[143,361,362],{"class":179},"(",[143,364,329],{"class":261},[143,366,367],{"class":179},"(value))\n",[143,369,370],{"class":145,"line":196},[143,371,372],{"class":318},"# 'North Region'\n",[143,374,375],{"class":145,"line":208},[143,376,377],{"class":318},"# 'North Region '\n",[143,379,380],{"class":145,"line":234},[143,381,382],{"class":318},"# 'North\\xa0Region'\n",[143,384,385],{"class":145,"line":253},[143,386,387],{"class":318},"# 'North  Region'\n",[10,389,390],{},"For a systematic view, count the values that would change under cleaning:",[134,392,394],{"className":166,"code":393,"language":168,"meta":139,"style":139},"import pandas as pd\n\ndef whitespace_report(series):\n    \"\"\"Summarise the invisible problems in a text column.\"\"\"\n    text = series.astype(\"string\")\n    return {\n        \"values\": len(text),\n        \"distinct\": int(text.nunique()),\n        \"leading or trailing space\": int((text != text.str.strip()).sum()),\n        \"double inner space\": int(text.str.contains(r\"\\s{2,}\", na=False).sum()),\n        \"non-breaking space\": int(text.str.contains(\" \", na=False).sum()),\n        \"zero-width or BOM\": int(\n            text.str.contains(\"[​‌‍]\", na=False).sum()\n        ),\n    }\n\nprint(whitespace_report(df[\"region\"]))\n",[14,395,396,406,410,422,427,443,451,465,478,496,537,562,575,595,601,607,612],{"__ignoreMap":139},[143,397,398,400,402,404],{"class":145,"line":146},[143,399,176],{"class":175},[143,401,180],{"class":179},[143,403,183],{"class":175},[143,405,186],{"class":179},[143,407,408],{"class":145,"line":189},[143,409,193],{"emptyLinePlaceholder":192},[143,411,412,415,419],{"class":145,"line":196},[143,413,414],{"class":175},"def",[143,416,418],{"class":417},"s_Opv"," whitespace_report",[143,420,421],{"class":179},"(series):\n",[143,423,424],{"class":145,"line":208},[143,425,426],{"class":153},"    \"\"\"Summarise the invisible problems in a text column.\"\"\"\n",[143,428,429,432,434,437,440],{"class":145,"line":234},[143,430,431],{"class":179},"    text ",[143,433,202],{"class":175},[143,435,436],{"class":179}," series.astype(",[143,438,439],{"class":153},"\"string\"",[143,441,442],{"class":179},")\n",[143,444,445,448],{"class":145,"line":253},[143,446,447],{"class":175},"    return",[143,449,450],{"class":179}," {\n",[143,452,453,456,459,462],{"class":145,"line":292},[143,454,455],{"class":153},"        \"values\"",[143,457,458],{"class":179},": ",[143,460,461],{"class":261},"len",[143,463,464],{"class":179},"(text),\n",[143,466,467,470,472,475],{"class":145,"line":298},[143,468,469],{"class":153},"        \"distinct\"",[143,471,458],{"class":179},[143,473,474],{"class":261},"int",[143,476,477],{"class":179},"(text.nunique()),\n",[143,479,480,483,485,487,490,493],{"class":145,"line":303},[143,481,482],{"class":153},"        \"leading or trailing space\"",[143,484,458],{"class":179},[143,486,474],{"class":261},[143,488,489],{"class":179},"((text ",[143,491,492],{"class":175},"!=",[143,494,495],{"class":179}," text.str.strip()).sum()),\n",[143,497,499,502,504,506,509,512,515,518,521,523,525,529,531,534],{"class":145,"line":498},10,[143,500,501],{"class":153},"        \"double inner space\"",[143,503,458],{"class":179},[143,505,474],{"class":261},[143,507,508],{"class":179},"(text.str.contains(",[143,510,511],{"class":175},"r",[143,513,514],{"class":153},"\"",[143,516,517],{"class":261},"\\s",[143,519,520],{"class":175},"{2,}",[143,522,514],{"class":153},[143,524,220],{"class":179},[143,526,528],{"class":527},"sa561","na",[143,530,202],{"class":175},[143,532,533],{"class":261},"False",[143,535,536],{"class":179},").sum()),\n",[143,538,540,543,545,547,549,552,554,556,558,560],{"class":145,"line":539},11,[143,541,542],{"class":153},"        \"non-breaking space\"",[143,544,458],{"class":179},[143,546,474],{"class":261},[143,548,508],{"class":179},[143,550,551],{"class":153},"\" \"",[143,553,220],{"class":179},[143,555,528],{"class":527},[143,557,202],{"class":175},[143,559,533],{"class":261},[143,561,536],{"class":179},[143,563,565,568,570,572],{"class":145,"line":564},12,[143,566,567],{"class":153},"        \"zero-width or BOM\"",[143,569,458],{"class":179},[143,571,474],{"class":261},[143,573,574],{"class":179},"(\n",[143,576,578,581,584,586,588,590,592],{"class":145,"line":577},13,[143,579,580],{"class":179},"            text.str.contains(",[143,582,583],{"class":153},"\"[​‌‍]\"",[143,585,220],{"class":179},[143,587,528],{"class":527},[143,589,202],{"class":175},[143,591,533],{"class":261},[143,593,594],{"class":179},").sum()\n",[143,596,598],{"class":145,"line":597},14,[143,599,600],{"class":179},"        ),\n",[143,602,604],{"class":145,"line":603},15,[143,605,606],{"class":179},"    }\n",[143,608,610],{"class":145,"line":609},16,[143,611,193],{"emptyLinePlaceholder":192},[143,613,615,617,620,622],{"class":145,"line":614},17,[143,616,306],{"class":261},[143,618,619],{"class":179},"(whitespace_report(df[",[143,621,312],{"class":153},[143,623,624],{"class":179},"]))\n",[10,626,627],{},"Each line of that report maps to a specific fix, which is why it is worth producing before touching anything. Leading and trailing spaces come from manual entry and from exports that pad to a fixed width. Double inner spaces usually come from concatenation in the source system — a first and last name joined with a space where the first name already ended in one. Non-breaking spaces almost always arrive by copy-and-paste from a web page or a PDF, and they cluster in exactly the columns somebody assembled by hand. Zero-width characters and the byte-order mark come from encoding round-trips, and are the hardest to spot because they occupy no visual width at all.",[10,629,630],{},"Running that on every text column of an import turns a mystery into a checklist:",[134,632,634],{"className":166,"code":633,"language":168,"meta":139,"style":139},"text_columns = df.select_dtypes(include=[\"object\", \"string\"]).columns\nfor name in text_columns:\n    print(name, whitespace_report(df[name]))\n",[14,635,636,664,676],{"__ignoreMap":139},[143,637,638,641,643,646,649,651,654,657,659,661],{"class":145,"line":146},[143,639,640],{"class":179},"text_columns ",[143,642,202],{"class":175},[143,644,645],{"class":179}," df.select_dtypes(",[143,647,648],{"class":527},"include",[143,650,202],{"class":175},[143,652,653],{"class":179},"[",[143,655,656],{"class":153},"\"object\"",[143,658,220],{"class":179},[143,660,439],{"class":153},[143,662,663],{"class":179},"]).columns\n",[143,665,666,668,671,673],{"class":145,"line":189},[143,667,340],{"class":175},[143,669,670],{"class":179}," name ",[143,672,346],{"class":175},[143,674,675],{"class":179}," text_columns:\n",[143,677,678,680],{"class":145,"line":196},[143,679,359],{"class":261},[143,681,682],{"class":179},"(name, whitespace_report(df[name]))\n",[129,684,686],{"id":685},"step-2-clean-in-the-right-order","Step 2 — Clean in the right order",[10,688,689,690,692,693,696],{},"Order matters. Strip alone leaves the non-breaking space in the middle, and collapsing before replacing does not touch it because ",[14,691,517],{}," in the regex engine does not always match ",[14,694,695],{},"\\xa0"," in a byte-oriented context. Replace the specific characters first, then collapse, then strip:",[134,698,700],{"className":166,"code":699,"language":168,"meta":139,"style":139},"import pandas as pd\n\nINVISIBLE = {\n    \" \": \" \",     # non-breaking space\n    \" \": \" \",     # figure space\n    \" \": \" \",     # narrow no-break space\n    \"​\": \"\",      # zero-width space\n    \"‌\": \"\",      # zero-width non-joiner\n    \"‍\": \"\",      # zero-width joiner\n    \"\": \"\",      # byte-order mark\n}\n\ndef clean_text(series):\n    \"\"\"Normalise whitespace in a text column, preserving None.\"\"\"\n    text = series.astype(\"string\")\n    for bad, good in INVISIBLE.items():\n        text = text.str.replace(bad, good, regex=False)\n    text = text.str.replace(r\"\\s+\", \" \", regex=True)     # collapse runs\n    return text.str.strip()\n\ndf[\"region_clean\"] = clean_text(df[\"region\"])\nprint(df[\"region_clean\"].nunique())      # 3, down from 6\n",[14,701,702,712,716,726,742,756,770,786,800,814,828,833,837,846,851,863,879,898,938,946,951,973],{"__ignoreMap":139},[143,703,704,706,708,710],{"class":145,"line":146},[143,705,176],{"class":175},[143,707,180],{"class":179},[143,709,183],{"class":175},[143,711,186],{"class":179},[143,713,714],{"class":145,"line":189},[143,715,193],{"emptyLinePlaceholder":192},[143,717,718,721,724],{"class":145,"line":196},[143,719,720],{"class":261},"INVISIBLE",[143,722,723],{"class":175}," =",[143,725,450],{"class":179},[143,727,728,731,733,736,739],{"class":145,"line":208},[143,729,730],{"class":153},"    \" \"",[143,732,458],{"class":179},[143,734,735],{"class":153},"\" \"",[143,737,738],{"class":179},",     ",[143,740,741],{"class":318},"# non-breaking space\n",[143,743,744,747,749,751,753],{"class":145,"line":234},[143,745,746],{"class":153},"    \" \"",[143,748,458],{"class":179},[143,750,735],{"class":153},[143,752,738],{"class":179},[143,754,755],{"class":318},"# figure space\n",[143,757,758,761,763,765,767],{"class":145,"line":253},[143,759,760],{"class":153},"    \" \"",[143,762,458],{"class":179},[143,764,735],{"class":153},[143,766,738],{"class":179},[143,768,769],{"class":318},"# narrow no-break space\n",[143,771,772,775,777,780,783],{"class":145,"line":292},[143,773,774],{"class":153},"    \"​\"",[143,776,458],{"class":179},[143,778,779],{"class":153},"\"\"",[143,781,782],{"class":179},",      ",[143,784,785],{"class":318},"# zero-width space\n",[143,787,788,791,793,795,797],{"class":145,"line":298},[143,789,790],{"class":153},"    \"‌\"",[143,792,458],{"class":179},[143,794,779],{"class":153},[143,796,782],{"class":179},[143,798,799],{"class":318},"# zero-width non-joiner\n",[143,801,802,805,807,809,811],{"class":145,"line":303},[143,803,804],{"class":153},"    \"‍\"",[143,806,458],{"class":179},[143,808,779],{"class":153},[143,810,782],{"class":179},[143,812,813],{"class":318},"# zero-width joiner\n",[143,815,816,819,821,823,825],{"class":145,"line":498},[143,817,818],{"class":153},"    \"\"",[143,820,458],{"class":179},[143,822,779],{"class":153},[143,824,782],{"class":179},[143,826,827],{"class":318},"# byte-order mark\n",[143,829,830],{"class":145,"line":539},[143,831,832],{"class":179},"}\n",[143,834,835],{"class":145,"line":564},[143,836,193],{"emptyLinePlaceholder":192},[143,838,839,841,844],{"class":145,"line":577},[143,840,414],{"class":175},[143,842,843],{"class":417}," clean_text",[143,845,421],{"class":179},[143,847,848],{"class":145,"line":597},[143,849,850],{"class":153},"    \"\"\"Normalise whitespace in a text column, preserving None.\"\"\"\n",[143,852,853,855,857,859,861],{"class":145,"line":603},[143,854,431],{"class":179},[143,856,202],{"class":175},[143,858,436],{"class":179},[143,860,439],{"class":153},[143,862,442],{"class":179},[143,864,865,868,871,873,876],{"class":145,"line":609},[143,866,867],{"class":175},"    for",[143,869,870],{"class":179}," bad, good ",[143,872,346],{"class":175},[143,874,875],{"class":261}," INVISIBLE",[143,877,878],{"class":179},".items():\n",[143,880,881,884,886,889,892,894,896],{"class":145,"line":614},[143,882,883],{"class":179},"        text ",[143,885,202],{"class":175},[143,887,888],{"class":179}," text.str.replace(bad, good, ",[143,890,891],{"class":527},"regex",[143,893,202],{"class":175},[143,895,533],{"class":261},[143,897,442],{"class":179},[143,899,901,903,905,908,910,912,914,917,919,921,923,925,927,929,932,935],{"class":145,"line":900},18,[143,902,431],{"class":179},[143,904,202],{"class":175},[143,906,907],{"class":179}," text.str.replace(",[143,909,511],{"class":175},[143,911,514],{"class":153},[143,913,517],{"class":261},[143,915,916],{"class":175},"+",[143,918,514],{"class":153},[143,920,220],{"class":179},[143,922,735],{"class":153},[143,924,220],{"class":179},[143,926,891],{"class":527},[143,928,202],{"class":175},[143,930,931],{"class":261},"True",[143,933,934],{"class":179},")     ",[143,936,937],{"class":318},"# collapse runs\n",[143,939,941,943],{"class":145,"line":940},19,[143,942,447],{"class":175},[143,944,945],{"class":179}," text.str.strip()\n",[143,947,949],{"class":145,"line":948},20,[143,950,193],{"emptyLinePlaceholder":192},[143,952,954,957,960,963,965,968,970],{"class":145,"line":953},21,[143,955,956],{"class":179},"df[",[143,958,959],{"class":153},"\"region_clean\"",[143,961,962],{"class":179},"] ",[143,964,202],{"class":175},[143,966,967],{"class":179}," clean_text(df[",[143,969,312],{"class":153},[143,971,972],{"class":179},"])\n",[143,974,976,978,980,982,985],{"class":145,"line":975},22,[143,977,306],{"class":261},[143,979,309],{"class":179},[143,981,959],{"class":153},[143,983,984],{"class":179},"].nunique())      ",[143,986,987],{"class":318},"# 3, down from 6\n",[10,989,990,993,994,997,998,1001,1002,1004,1005,1008],{},[14,991,992],{},"astype(\"string\")"," rather than ",[14,995,996],{},"astype(str)"," is deliberate: the nullable string dtype keeps missing values as ",[14,999,1000],{},"\u003CNA>",", whereas ",[14,1003,996],{}," turns them into the literal text ",[14,1006,1007],{},"\"nan\"",", which then survives every subsequent clean and quietly becomes a category.",[129,1010,1012],{"id":1011},"step-3-build-a-comparison-key","Step 3 — Build a comparison key",[10,1014,1015,1016,1020],{},"Cleaning whitespace leaves case and accents. For a ",[1017,1018,1019],"em",{},"key"," used to join or group, fold both — but keep the original for display:",[134,1022,1024],{"className":166,"code":1023,"language":168,"meta":139,"style":139},"import unicodedata\nimport pandas as pd\n\ndef comparison_key(series):\n    \"\"\"A normalised key for joining and grouping. Not for display.\"\"\"\n    text = clean_text(series)\n\n    # NFKC folds compatibility forms and composes accents consistently.\n    text = text.map(\n        lambda v: unicodedata.normalize(\"NFKC\", v) if pd.notna(v) else v\n    )\n    return text.str.casefold()\n\ndf[\"region_key\"] = comparison_key(df[\"region\"])\nprint(df.groupby(\"region_key\")[\"revenue\"].sum())\n",[14,1025,1026,1033,1043,1047,1056,1061,1070,1074,1079,1088,1114,1119,1126,1130,1148],{"__ignoreMap":139},[143,1027,1028,1030],{"class":145,"line":146},[143,1029,176],{"class":175},[143,1031,1032],{"class":179}," unicodedata\n",[143,1034,1035,1037,1039,1041],{"class":145,"line":189},[143,1036,176],{"class":175},[143,1038,180],{"class":179},[143,1040,183],{"class":175},[143,1042,186],{"class":179},[143,1044,1045],{"class":145,"line":196},[143,1046,193],{"emptyLinePlaceholder":192},[143,1048,1049,1051,1054],{"class":145,"line":208},[143,1050,414],{"class":175},[143,1052,1053],{"class":417}," comparison_key",[143,1055,421],{"class":179},[143,1057,1058],{"class":145,"line":234},[143,1059,1060],{"class":153},"    \"\"\"A normalised key for joining and grouping. Not for display.\"\"\"\n",[143,1062,1063,1065,1067],{"class":145,"line":253},[143,1064,431],{"class":179},[143,1066,202],{"class":175},[143,1068,1069],{"class":179}," clean_text(series)\n",[143,1071,1072],{"class":145,"line":292},[143,1073,193],{"emptyLinePlaceholder":192},[143,1075,1076],{"class":145,"line":298},[143,1077,1078],{"class":318},"    # NFKC folds compatibility forms and composes accents consistently.\n",[143,1080,1081,1083,1085],{"class":145,"line":303},[143,1082,431],{"class":179},[143,1084,202],{"class":175},[143,1086,1087],{"class":179}," text.map(\n",[143,1089,1090,1093,1096,1099,1102,1105,1108,1111],{"class":145,"line":498},[143,1091,1092],{"class":175},"        lambda",[143,1094,1095],{"class":179}," v: unicodedata.normalize(",[143,1097,1098],{"class":153},"\"NFKC\"",[143,1100,1101],{"class":179},", v) ",[143,1103,1104],{"class":175},"if",[143,1106,1107],{"class":179}," pd.notna(v) ",[143,1109,1110],{"class":175},"else",[143,1112,1113],{"class":179}," v\n",[143,1115,1116],{"class":145,"line":539},[143,1117,1118],{"class":179},"    )\n",[143,1120,1121,1123],{"class":145,"line":564},[143,1122,447],{"class":175},[143,1124,1125],{"class":179}," text.str.casefold()\n",[143,1127,1128],{"class":145,"line":577},[143,1129,193],{"emptyLinePlaceholder":192},[143,1131,1132,1134,1137,1139,1141,1144,1146],{"class":145,"line":597},[143,1133,956],{"class":179},[143,1135,1136],{"class":153},"\"region_key\"",[143,1138,962],{"class":179},[143,1140,202],{"class":175},[143,1142,1143],{"class":179}," comparison_key(df[",[143,1145,312],{"class":153},[143,1147,972],{"class":179},[143,1149,1150,1152,1155,1157,1160,1163],{"class":145,"line":603},[143,1151,306],{"class":261},[143,1153,1154],{"class":179},"(df.groupby(",[143,1156,1136],{"class":153},[143,1158,1159],{"class":179},")[",[143,1161,1162],{"class":153},"\"revenue\"",[143,1164,1165],{"class":179},"].sum())\n",[10,1167,1168,1169,1178,1179,1181,1182,1185,1186,1189,1190,1193,1194,1197,1198,1201,1202,1205,1206,23],{},"Two choices worth understanding. ",[1170,1171,1172,993,1175],"strong",{},[14,1173,1174],{},"casefold",[14,1176,1177],{},"lower"," handles cases ",[14,1180,1177],{}," misses — the German ",[14,1183,1184],{},"ß"," folds to ",[14,1187,1188],{},"ss",", so ",[14,1191,1192],{},"STRASSE"," and ",[14,1195,1196],{},"Straße"," match. ",[1170,1199,1200],{},"NFKC rather than NFC"," additionally folds compatibility characters, so a full-width ",[14,1203,1204],{},"Ｎ"," pasted from a Japanese-locale system matches an ordinary ",[14,1207,1208],{},"N",[10,1210,1211,1212,1193,1215,1217],{},"Stripping accents entirely is a further step, and one to take deliberately rather than by default — it makes ",[14,1213,1214],{},"Nörth",[14,1216,16],{}," match, which is right for a fuzzy lookup and wrong if the two are genuinely different places:",[134,1219,1221],{"className":166,"code":1220,"language":168,"meta":139,"style":139},"import unicodedata\n\ndef strip_accents(value):\n    \"\"\"Remove combining marks: 'Nörth' -> 'North'. Use with care.\"\"\"\n    decomposed = unicodedata.normalize(\"NFKD\", str(value))\n    return \"\".join(c for c in decomposed if not unicodedata.combining(c))\n",[14,1222,1223,1229,1233,1243,1248,1268],{"__ignoreMap":139},[143,1224,1225,1227],{"class":145,"line":146},[143,1226,176],{"class":175},[143,1228,1032],{"class":179},[143,1230,1231],{"class":145,"line":189},[143,1232,193],{"emptyLinePlaceholder":192},[143,1234,1235,1237,1240],{"class":145,"line":196},[143,1236,414],{"class":175},[143,1238,1239],{"class":417}," strip_accents",[143,1241,1242],{"class":179},"(value):\n",[143,1244,1245],{"class":145,"line":208},[143,1246,1247],{"class":153},"    \"\"\"Remove combining marks: 'Nörth' -> 'North'. Use with care.\"\"\"\n",[143,1249,1250,1253,1255,1258,1261,1263,1266],{"class":145,"line":234},[143,1251,1252],{"class":179},"    decomposed ",[143,1254,202],{"class":175},[143,1256,1257],{"class":179}," unicodedata.normalize(",[143,1259,1260],{"class":153},"\"NFKD\"",[143,1262,220],{"class":179},[143,1264,1265],{"class":261},"str",[143,1267,367],{"class":179},[143,1269,1270,1272,1275,1278,1280,1283,1285,1288,1290,1293],{"class":145,"line":253},[143,1271,447],{"class":175},[143,1273,1274],{"class":153}," \"\"",[143,1276,1277],{"class":179},".join(c ",[143,1279,340],{"class":175},[143,1281,1282],{"class":179}," c ",[143,1284,346],{"class":175},[143,1286,1287],{"class":179}," decomposed ",[143,1289,1104],{"class":175},[143,1291,1292],{"class":175}," not",[143,1294,1295],{"class":179}," unicodedata.combining(c))\n",[129,1297,1299],{"id":1298},"step-4-keep-the-original-alongside-the-key","Step 4 — Keep the original alongside the key",[10,1301,1302],{},"The pattern that works in a real pipeline is three columns, not one: the value as supplied, a cleaned display version, and a key.",[25,1304,35,1310,35,1313,35,1316,35,1319,35,1325,35,1331,35,1336,35,1341,35,1353,35,1363,35,1368,35,1373,35,1378,35,1383,35,1388],{"viewBox":1305,"role":28,"ariaLabel":1306,"ariaLabelledBy":1307,"xmlns":33,"style":34},"0 0 800 236","Three columns from one: the original as supplied for audit, a cleaned version for display, and a folded key for joining and grouping.",[1308,1309],"cols-t","cols-d",[37,1311,1312],{"id":1308},"Keep three versions of a text column, not one",[41,1314,1315],{"id":1309},"One supplied value produces three columns. The original is retained untouched so a report can show exactly what was provided and an audit can trace it. The cleaned version has its whitespace normalised and is what appears in output. The key is additionally case-folded and Unicode-normalised, and is used only for joining and grouping. Overwriting the original in place loses the ability to answer what was actually supplied.",[45,1317],{"x":47,"y":47,"width":48,"height":1318,"fill":50},"236",[45,1320],{"x":1321,"y":88,"width":1322,"height":72,"rx":1323,"fill":1324,"stroke":68,"style":69},"14","176","12","#f0f2f5",[52,1326,1330],{"x":1327,"y":1328,"style":1329},"102","70","font-size:11px;font-weight:700;fill:var(--muted,#5b6780);text-anchor:middle","as supplied",[52,1332,1335],{"x":1327,"y":1333,"style":1334},"116","font-size:11px;font-weight:700;fill:var(--text,#172033);text-anchor:middle","'North Region '",[52,1337,1340],{"x":1327,"y":1338,"style":1339},"138","font-size:10px;fill:var(--muted,#5b6780);text-anchor:middle","never overwrite this",[1342,1343,1345,1346,1345,1350,35],"g",{"stroke":68,"style":69,"fill":1344},"none","\n    ",[1347,1348],"path",{"d":1349},"M190 106 H 224 V 60 H 258",[1347,1351],{"d":1352},"M190 132 H 224 V 178 H 258",[1342,1354,1345,1356,1345,1360,35],{"fill":1355},"#5b5cf0",[1357,1358],"polygon",{"points":1359},"266,60 254,54 254,66",[1357,1361],{"points":1362},"266,178 254,172 254,184",[45,1364],{"x":1365,"y":55,"width":1366,"height":1367,"rx":1323,"fill":79,"stroke":80,"style":69},"274","510","62",[52,1369,1372],{"x":65,"y":1370,"style":1371},"54","font-size:12px;font-weight:700;fill:var(--teal-ink,#0b6157)","region_clean — for display",[52,1374,1377],{"x":65,"y":1375,"style":1376},"76","font-size:10.5px;fill:var(--muted,#5b6780)","'North Region' · whitespace normalised, case preserved",[45,1379],{"x":1365,"y":1380,"width":1366,"height":1367,"rx":1323,"fill":1381,"stroke":1382,"style":69},"148","#ebebfd","var(--brand,#5b5cf0)",[52,1384,1387],{"x":65,"y":1385,"style":1386},"172","font-size:12px;font-weight:700;fill:var(--brand-strong,#4338ca)","region_key — for joining and grouping",[52,1389,1391],{"x":65,"y":1390,"style":1376},"194","'north region' · case-folded and Unicode-normalised",[134,1393,1395],{"className":166,"code":1394,"language":168,"meta":139,"style":139},"import pandas as pd\n\ndef add_text_key(df, column):\n    \"\"\"Add cleaned and key variants of a text column, keeping the original.\"\"\"\n    out = df.copy()\n    out[f\"{column}_clean\"] = clean_text(out[column])\n    out[f\"{column}_key\"] = comparison_key(out[column])\n    return out\n\ndf = add_text_key(df, \"region\")\n\n# Join on the key; report on the clean value.\nsummary = (\n    df.groupby(\"region_key\")\n      .agg(display=(\"region_clean\", \"first\"), revenue=(\"revenue\", \"sum\"))\n      .reset_index(drop=True)\n)\nprint(summary)\n",[14,1396,1397,1407,1411,1421,1426,1436,1466,1490,1497,1501,1514,1518,1523,1533,1542,1581,1595,1599],{"__ignoreMap":139},[143,1398,1399,1401,1403,1405],{"class":145,"line":146},[143,1400,176],{"class":175},[143,1402,180],{"class":179},[143,1404,183],{"class":175},[143,1406,186],{"class":179},[143,1408,1409],{"class":145,"line":189},[143,1410,193],{"emptyLinePlaceholder":192},[143,1412,1413,1415,1418],{"class":145,"line":196},[143,1414,414],{"class":175},[143,1416,1417],{"class":417}," add_text_key",[143,1419,1420],{"class":179},"(df, column):\n",[143,1422,1423],{"class":145,"line":208},[143,1424,1425],{"class":153},"    \"\"\"Add cleaned and key variants of a text column, keeping the original.\"\"\"\n",[143,1427,1428,1431,1433],{"class":145,"line":234},[143,1429,1430],{"class":179},"    out ",[143,1432,202],{"class":175},[143,1434,1435],{"class":179}," df.copy()\n",[143,1437,1438,1441,1444,1446,1450,1453,1456,1459,1461,1463],{"class":145,"line":253},[143,1439,1440],{"class":179},"    out[",[143,1442,1443],{"class":175},"f",[143,1445,514],{"class":153},[143,1447,1449],{"class":1448},"sSjpA","{",[143,1451,1452],{"class":179},"column",[143,1454,1455],{"class":1448},"}",[143,1457,1458],{"class":153},"_clean\"",[143,1460,962],{"class":179},[143,1462,202],{"class":175},[143,1464,1465],{"class":179}," clean_text(out[column])\n",[143,1467,1468,1470,1472,1474,1476,1478,1480,1483,1485,1487],{"class":145,"line":292},[143,1469,1440],{"class":179},[143,1471,1443],{"class":175},[143,1473,514],{"class":153},[143,1475,1449],{"class":1448},[143,1477,1452],{"class":179},[143,1479,1455],{"class":1448},[143,1481,1482],{"class":153},"_key\"",[143,1484,962],{"class":179},[143,1486,202],{"class":175},[143,1488,1489],{"class":179}," comparison_key(out[column])\n",[143,1491,1492,1494],{"class":145,"line":298},[143,1493,447],{"class":175},[143,1495,1496],{"class":179}," out\n",[143,1498,1499],{"class":145,"line":303},[143,1500,193],{"emptyLinePlaceholder":192},[143,1502,1503,1505,1507,1510,1512],{"class":145,"line":498},[143,1504,199],{"class":179},[143,1506,202],{"class":175},[143,1508,1509],{"class":179}," add_text_key(df, ",[143,1511,312],{"class":153},[143,1513,442],{"class":179},[143,1515,1516],{"class":145,"line":539},[143,1517,193],{"emptyLinePlaceholder":192},[143,1519,1520],{"class":145,"line":564},[143,1521,1522],{"class":318},"# Join on the key; report on the clean value.\n",[143,1524,1525,1528,1530],{"class":145,"line":577},[143,1526,1527],{"class":179},"summary ",[143,1529,202],{"class":175},[143,1531,1532],{"class":179}," (\n",[143,1534,1535,1538,1540],{"class":145,"line":597},[143,1536,1537],{"class":179},"    df.groupby(",[143,1539,1136],{"class":153},[143,1541,442],{"class":179},[143,1543,1544,1547,1550,1552,1554,1556,1558,1561,1564,1567,1569,1571,1573,1575,1578],{"class":145,"line":603},[143,1545,1546],{"class":179},"      .agg(",[143,1548,1549],{"class":527},"display",[143,1551,202],{"class":175},[143,1553,362],{"class":179},[143,1555,959],{"class":153},[143,1557,220],{"class":179},[143,1559,1560],{"class":153},"\"first\"",[143,1562,1563],{"class":179},"), ",[143,1565,1566],{"class":527},"revenue",[143,1568,202],{"class":175},[143,1570,362],{"class":179},[143,1572,1162],{"class":153},[143,1574,220],{"class":179},[143,1576,1577],{"class":153},"\"sum\"",[143,1579,1580],{"class":179},"))\n",[143,1582,1583,1586,1589,1591,1593],{"class":145,"line":609},[143,1584,1585],{"class":179},"      .reset_index(",[143,1587,1588],{"class":527},"drop",[143,1590,202],{"class":175},[143,1592,931],{"class":261},[143,1594,442],{"class":179},[143,1596,1597],{"class":145,"line":614},[143,1598,442],{"class":179},[143,1600,1601,1603],{"class":145,"line":900},[143,1602,306],{"class":261},[143,1604,1605],{"class":179},"(summary)\n",[10,1607,1608],{},"Joining two files then becomes reliable, because both sides are folded the same way:",[134,1610,1612],{"className":166,"code":1611,"language":168,"meta":139,"style":139},"left = add_text_key(pd.read_excel(\"sales.xlsx\"), \"region\")\nright = add_text_key(pd.read_excel(\"targets.xlsx\"), \"region\")\n\nmerged = left.merge(right, on=\"region_key\", how=\"left\",\n                    suffixes=(\"\", \"_target\"), indicator=True)\n\nunmatched = merged.loc[merged[\"_merge\"] == \"left_only\", \"region_clean\"].unique()\nif len(unmatched):\n    print(\"still unmatched after normalising:\", list(unmatched))\n",[14,1613,1614,1633,1651,1655,1684,1711,1715,1743,1753],{"__ignoreMap":139},[143,1615,1616,1619,1621,1624,1627,1629,1631],{"class":145,"line":146},[143,1617,1618],{"class":179},"left ",[143,1620,202],{"class":175},[143,1622,1623],{"class":179}," add_text_key(pd.read_excel(",[143,1625,1626],{"class":153},"\"sales.xlsx\"",[143,1628,1563],{"class":179},[143,1630,312],{"class":153},[143,1632,442],{"class":179},[143,1634,1635,1638,1640,1642,1645,1647,1649],{"class":145,"line":189},[143,1636,1637],{"class":179},"right ",[143,1639,202],{"class":175},[143,1641,1623],{"class":179},[143,1643,1644],{"class":153},"\"targets.xlsx\"",[143,1646,1563],{"class":179},[143,1648,312],{"class":153},[143,1650,442],{"class":179},[143,1652,1653],{"class":145,"line":196},[143,1654,193],{"emptyLinePlaceholder":192},[143,1656,1657,1660,1662,1665,1668,1670,1672,1674,1677,1679,1682],{"class":145,"line":208},[143,1658,1659],{"class":179},"merged ",[143,1661,202],{"class":175},[143,1663,1664],{"class":179}," left.merge(right, ",[143,1666,1667],{"class":527},"on",[143,1669,202],{"class":175},[143,1671,1136],{"class":153},[143,1673,220],{"class":179},[143,1675,1676],{"class":527},"how",[143,1678,202],{"class":175},[143,1680,1681],{"class":153},"\"left\"",[143,1683,231],{"class":179},[143,1685,1686,1689,1691,1693,1695,1697,1700,1702,1705,1707,1709],{"class":145,"line":234},[143,1687,1688],{"class":527},"                    suffixes",[143,1690,202],{"class":175},[143,1692,362],{"class":179},[143,1694,779],{"class":153},[143,1696,220],{"class":179},[143,1698,1699],{"class":153},"\"_target\"",[143,1701,1563],{"class":179},[143,1703,1704],{"class":527},"indicator",[143,1706,202],{"class":175},[143,1708,931],{"class":261},[143,1710,442],{"class":179},[143,1712,1713],{"class":145,"line":253},[143,1714,193],{"emptyLinePlaceholder":192},[143,1716,1717,1720,1722,1725,1728,1730,1733,1736,1738,1740],{"class":145,"line":292},[143,1718,1719],{"class":179},"unmatched ",[143,1721,202],{"class":175},[143,1723,1724],{"class":179}," merged.loc[merged[",[143,1726,1727],{"class":153},"\"_merge\"",[143,1729,962],{"class":179},[143,1731,1732],{"class":175},"==",[143,1734,1735],{"class":153}," \"left_only\"",[143,1737,220],{"class":179},[143,1739,959],{"class":153},[143,1741,1742],{"class":179},"].unique()\n",[143,1744,1745,1747,1750],{"class":145,"line":298},[143,1746,1104],{"class":175},[143,1748,1749],{"class":261}," len",[143,1751,1752],{"class":179},"(unmatched):\n",[143,1754,1755,1757,1759,1762,1764,1767],{"class":145,"line":303},[143,1756,359],{"class":261},[143,1758,362],{"class":179},[143,1760,1761],{"class":153},"\"still unmatched after normalising:\"",[143,1763,220],{"class":179},[143,1765,1766],{"class":261},"list",[143,1768,1769],{"class":179},"(unmatched))\n",[10,1771,1772,1773,1776,1777,23],{},"The ",[14,1774,1775],{},"indicator=True"," and the unmatched report matter — normalising fixes the invisible mismatches and leaves the genuine ones, which are exactly the rows worth a human look. The join mechanics are covered in ",[19,1778,1780],{"href":1779},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Fmerge-two-excel-files-on-common-column-python\u002F","merging two Excel files on a common column",[129,1782,1784],{"id":1783},"common-pitfalls-and-fixes","Common pitfalls and fixes",[1786,1787,1788,1804],"table",{},[1789,1790,1791],"thead",{},[1792,1793,1794,1798,1801],"tr",{},[1795,1796,1797],"th",{},"Symptom",[1795,1799,1800],{},"Cause",[1795,1802,1803],{},"Fix",[1805,1806,1807,1819,1830,1852,1870,1883,1903,1914],"tbody",{},[1792,1808,1809,1813,1816],{},[1810,1811,1812],"td",{},"Join matches nothing",[1810,1814,1815],{},"Trailing or non-breaking space",[1810,1817,1818],{},"Clean both sides into a key column.",[1792,1820,1821,1824,1827],{},[1810,1822,1823],{},"Group-by shows near-duplicate groups",[1810,1825,1826],{},"Case or whitespace variants",[1810,1828,1829],{},"Group on a case-folded key.",[1792,1831,1832,1839,1847],{},[1810,1833,1834,1835,1838],{},"Literal ",[14,1836,1837],{},"nan"," values appear",[1810,1840,1841,1843,1844],{},[14,1842,996],{}," on a column with ",[14,1845,1846],{},"NaN",[1810,1848,1849,1850,23],{},"Use ",[14,1851,992],{},[1792,1853,1854,1860,1863],{},[1810,1855,1856,1859],{},[14,1857,1858],{},"str.strip()"," left the value unchanged",[1810,1861,1862],{},"Non-breaking space, not a normal one",[1810,1864,1865,1866,1869],{},"Replace ",[14,1867,1868],{}," "," explicitly first.",[1792,1871,1872,1875,1878],{},[1810,1873,1874],{},"Accented names still differ",[1810,1876,1877],{},"Two Unicode representations",[1810,1879,1880,23],{},[14,1881,1882],{},"unicodedata.normalize(\"NFKC\", ...)",[1792,1884,1885,1892,1899],{},[1810,1886,1887,1889,1890],{},[14,1888,1192],{}," does not match ",[14,1891,1196],{},[1810,1893,1894,1896,1897],{},[14,1895,1177],{}," does not fold ",[14,1898,1184],{},[1810,1900,1849,1901,23],{},[14,1902,1174],{},[1792,1904,1905,1908,1911],{},[1810,1906,1907],{},"Report shows lower-cased names",[1810,1909,1910],{},"Key column used for display",[1810,1912,1913],{},"Keep a separate cleaned display column.",[1792,1915,1916,1919,1922],{},[1810,1917,1918],{},"An invisible character survives cleaning",[1810,1920,1921],{},"Not in the replacement map",[1810,1923,1924,1925,1927],{},"Print ",[14,1926,329],{}," and add it.",[129,1929,1931],{"id":1930},"performance-and-scale-notes","Performance and scale notes",[25,1933,35,1939,35,1942,35,1945,35,1948,35,1952,35,1957,35,1963,35,1967,35,1971,35,1976,35,1981,35,1985,35,1990],{"viewBox":1934,"role":28,"ariaLabel":1935,"ariaLabelledBy":1936,"xmlns":33,"style":34},"0 0 800 224","Cleaning every row versus cleaning the distinct values and mapping: with low cardinality the lookup approach does a fraction of the work.",[1937,1938],"lookup-t","lookup-d",[37,1940,1941],{"id":1937},"Clean the distinct values, then map",[41,1943,1944],{"id":1938},"A million-row region column holding twenty distinct values. Cleaning row by row runs the regular expressions a million times. Extracting the distinct values first means cleaning twenty strings and then applying a fast hash-based map across the column. The saving is proportional to the ratio between row count and cardinality, so it is enormous for category-like columns and nil for free text.",[45,1946],{"x":47,"y":47,"width":48,"height":1947,"fill":50},"224",[52,1949,1951],{"x":124,"y":1950,"style":56},"28","1,000,000 rows · 20 distinct region names",[52,1953,1956],{"x":1954,"y":1375,"style":1955},"152","font-size:11.5px;font-weight:700;fill:var(--text,#172033);text-anchor:end","clean every row",[45,1958],{"x":1959,"y":1960,"width":1961,"height":55,"rx":1962,"fill":94,"stroke":95,"style":69},"168","56","596","5",[52,1964,1966],{"x":1965,"y":1375,"style":98},"466","1,000,000 regex passes",[52,1968,1970],{"x":1954,"y":1969,"style":1955},"132","clean, then map",[45,1972],{"x":1959,"y":1973,"width":1974,"height":55,"rx":1962,"fill":1975},"112","18","#0f9488",[52,1977,1980],{"x":1978,"y":1969,"style":1979},"196","font-size:11px;font-weight:700;fill:var(--teal-ink,#0b6157)","20 regex passes",[45,1982],{"x":1983,"y":1973,"width":1984,"height":55,"rx":1962,"fill":79,"stroke":80,"style":69},"322","120",[52,1986,1989],{"x":1987,"y":1969,"style":1988},"382","font-size:10.5px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","+ one hash map",[52,1991,1993],{"x":124,"y":54,"style":1992},"font-size:11px;fill:var(--muted,#5b6780);text-anchor:middle","the saving scales with rows divided by cardinality — enormous for categories, nil for free text",[10,1995,1996,1997,2000],{},"pandas string operations are vectorised but run in Python for ",[14,1998,1999],{},"object"," dtype. Two changes make a large clean substantially faster.",[10,2002,2003,2010],{},[1170,2004,2005,2006,2009],{},"Use the nullable ",[14,2007,2008],{},"string"," dtype",", or the Arrow-backed variant where available. It stores data more compactly and dispatches to faster kernels:",[134,2012,2014],{"className":166,"code":2013,"language":168,"meta":139,"style":139},"import pandas as pd\n\ndf[\"region\"] = df[\"region\"].astype(\"string[pyarrow]\")   # if pyarrow installed\n",[14,2015,2016,2026,2030],{"__ignoreMap":139},[143,2017,2018,2020,2022,2024],{"class":145,"line":146},[143,2019,176],{"class":175},[143,2021,180],{"class":179},[143,2023,183],{"class":175},[143,2025,186],{"class":179},[143,2027,2028],{"class":145,"line":189},[143,2029,193],{"emptyLinePlaceholder":192},[143,2031,2032,2034,2036,2038,2040,2042,2044,2047,2050,2053],{"class":145,"line":196},[143,2033,956],{"class":179},[143,2035,312],{"class":153},[143,2037,962],{"class":179},[143,2039,202],{"class":175},[143,2041,349],{"class":179},[143,2043,312],{"class":153},[143,2045,2046],{"class":179},"].astype(",[143,2048,2049],{"class":153},"\"string[pyarrow]\"",[143,2051,2052],{"class":179},")   ",[143,2054,2055],{"class":318},"# if pyarrow installed\n",[10,2057,2058,2061,2062,2065],{},[1170,2059,2060],{},"Combine the replacements into one pass."," Six sequential ",[14,2063,2064],{},"str.replace"," calls each walk the column; a single translation table walks it once:",[134,2067,2069],{"className":166,"code":2068,"language":168,"meta":139,"style":139},"TRANSLATION = str.maketrans({\n    \" \": \" \", \" \": \" \", \" \": \" \",\n    \"​\": \"\", \"‌\": \"\", \"‍\": \"\", \"\": \"\",\n})\n\ndef clean_text_fast(series):\n    text = series.astype(\"string\")\n    text = text.map(lambda v: v.translate(TRANSLATION) if pd.notna(v) else v)\n    return text.str.replace(r\"\\s+\", \" \", regex=True).str.strip()\n",[14,2070,2071,2084,2112,2149,2153,2157,2166,2178,2207],{"__ignoreMap":139},[143,2072,2073,2076,2078,2081],{"class":145,"line":146},[143,2074,2075],{"class":261},"TRANSLATION",[143,2077,723],{"class":175},[143,2079,2080],{"class":261}," str",[143,2082,2083],{"class":179},".maketrans({\n",[143,2085,2086,2088,2090,2092,2094,2097,2099,2101,2103,2106,2108,2110],{"class":145,"line":189},[143,2087,730],{"class":153},[143,2089,458],{"class":179},[143,2091,735],{"class":153},[143,2093,220],{"class":179},[143,2095,2096],{"class":153},"\" \"",[143,2098,458],{"class":179},[143,2100,735],{"class":153},[143,2102,220],{"class":179},[143,2104,2105],{"class":153},"\" \"",[143,2107,458],{"class":179},[143,2109,735],{"class":153},[143,2111,231],{"class":179},[143,2113,2114,2116,2118,2120,2122,2125,2127,2129,2131,2134,2136,2138,2140,2143,2145,2147],{"class":145,"line":196},[143,2115,774],{"class":153},[143,2117,458],{"class":179},[143,2119,779],{"class":153},[143,2121,220],{"class":179},[143,2123,2124],{"class":153},"\"‌\"",[143,2126,458],{"class":179},[143,2128,779],{"class":153},[143,2130,220],{"class":179},[143,2132,2133],{"class":153},"\"‍\"",[143,2135,458],{"class":179},[143,2137,779],{"class":153},[143,2139,220],{"class":179},[143,2141,2142],{"class":153},"\"\"",[143,2144,458],{"class":179},[143,2146,779],{"class":153},[143,2148,231],{"class":179},[143,2150,2151],{"class":145,"line":208},[143,2152,295],{"class":179},[143,2154,2155],{"class":145,"line":234},[143,2156,193],{"emptyLinePlaceholder":192},[143,2158,2159,2161,2164],{"class":145,"line":253},[143,2160,414],{"class":175},[143,2162,2163],{"class":417}," clean_text_fast",[143,2165,421],{"class":179},[143,2167,2168,2170,2172,2174,2176],{"class":145,"line":292},[143,2169,431],{"class":179},[143,2171,202],{"class":175},[143,2173,436],{"class":179},[143,2175,439],{"class":153},[143,2177,442],{"class":179},[143,2179,2180,2182,2184,2187,2190,2193,2195,2198,2200,2202,2204],{"class":145,"line":298},[143,2181,431],{"class":179},[143,2183,202],{"class":175},[143,2185,2186],{"class":179}," text.map(",[143,2188,2189],{"class":175},"lambda",[143,2191,2192],{"class":179}," v: v.translate(",[143,2194,2075],{"class":261},[143,2196,2197],{"class":179},") ",[143,2199,1104],{"class":175},[143,2201,1107],{"class":179},[143,2203,1110],{"class":175},[143,2205,2206],{"class":179}," v)\n",[143,2208,2209,2211,2213,2215,2217,2219,2221,2223,2225,2227,2229,2231,2233,2235],{"class":145,"line":303},[143,2210,447],{"class":175},[143,2212,907],{"class":179},[143,2214,511],{"class":175},[143,2216,514],{"class":153},[143,2218,517],{"class":261},[143,2220,916],{"class":175},[143,2222,514],{"class":153},[143,2224,220],{"class":179},[143,2226,735],{"class":153},[143,2228,220],{"class":179},[143,2230,891],{"class":527},[143,2232,202],{"class":175},[143,2234,931],{"class":261},[143,2236,2237],{"class":179},").str.strip()\n",[10,2239,2240,2241,2244],{},"A third habit matters more than either: ",[1170,2242,2243],{},"clean on the distinct values, not on every row."," A million-row column of region names holds perhaps twenty distinct values, and cleaning twenty strings then mapping is orders of magnitude cheaper:",[134,2246,2248],{"className":166,"code":2247,"language":168,"meta":139,"style":139},"import pandas as pd\n\ndef clean_via_lookup(series):\n    \"\"\"Clean each distinct value once, then map.\"\"\"\n    distinct = pd.Series(series.dropna().unique())\n    lookup = dict(zip(distinct, clean_text(distinct)))\n    return series.map(lookup)\n",[14,2249,2250,2260,2264,2273,2278,2288,2306],{"__ignoreMap":139},[143,2251,2252,2254,2256,2258],{"class":145,"line":146},[143,2253,176],{"class":175},[143,2255,180],{"class":179},[143,2257,183],{"class":175},[143,2259,186],{"class":179},[143,2261,2262],{"class":145,"line":189},[143,2263,193],{"emptyLinePlaceholder":192},[143,2265,2266,2268,2271],{"class":145,"line":196},[143,2267,414],{"class":175},[143,2269,2270],{"class":417}," clean_via_lookup",[143,2272,421],{"class":179},[143,2274,2275],{"class":145,"line":208},[143,2276,2277],{"class":153},"    \"\"\"Clean each distinct value once, then map.\"\"\"\n",[143,2279,2280,2283,2285],{"class":145,"line":234},[143,2281,2282],{"class":179},"    distinct ",[143,2284,202],{"class":175},[143,2286,2287],{"class":179}," pd.Series(series.dropna().unique())\n",[143,2289,2290,2293,2295,2298,2300,2303],{"class":145,"line":253},[143,2291,2292],{"class":179},"    lookup ",[143,2294,202],{"class":175},[143,2296,2297],{"class":261}," dict",[143,2299,362],{"class":179},[143,2301,2302],{"class":261},"zip",[143,2304,2305],{"class":179},"(distinct, clean_text(distinct)))\n",[143,2307,2308,2310],{"class":145,"line":292},[143,2309,447],{"class":175},[143,2311,2312],{"class":179}," series.map(lookup)\n",[10,2314,2315,2316,2320],{},"That trick applies to any per-value transformation on a low-cardinality column, and it is the same reasoning behind deduplicating before parsing dates in ",[19,2317,2319],{"href":2318},"\u002Fadvanced-data-transformation-and-cleaning\u002Fworking-with-dates-and-times-in-excel-data\u002Fparse-excel-dates-into-python-datetimes-with-pandas\u002F","parsing Excel dates with pandas",". Where the column genuinely has high cardinality — free-text notes, for instance — the lookup gains nothing, and the vectorised path is the right one.",[129,2322,2324],{"id":2323},"conclusion","Conclusion",[10,2326,2327,2328,2330],{},"Whitespace problems from Excel are invisible by definition, so diagnose with ",[14,2329,329],{}," and a report rather than by eye. Clean in order: replace the specific invisible characters, collapse runs of whitespace, then strip. Build a separate key column that is additionally Unicode-normalised and case-folded, use it for every join and group-by, and keep the original untouched so reports show what was actually supplied. Then clean the distinct values rather than every row, and a million-row column costs no more than a twenty-value one.",[129,2332,2334],{"id":2333},"frequently-asked-questions","Frequently asked questions",[10,2336,2337,2340,2341,2343],{},[1170,2338,2339],{},"Why does my join fail when the values look identical?","\nOne side almost certainly has trailing whitespace or a non-breaking space. Both render as a normal gap, so the values look the same on screen while comparing as different strings. Print the ",[14,2342,329],{}," of a failing value to see what is really there.",[10,2345,2346,2353],{},[1170,2347,2348,2349,2352],{},"Does ",[14,2350,2351],{},"str.strip"," remove non-breaking spaces?","\nNot by default in older pandas versions, and it is safest not to rely on it. Replace the specific characters first — non-breaking space, zero-width space and the byte-order mark — then strip.",[10,2355,2356,2365,2367,2368,2370,2371,2373],{},[1170,2357,2358,2359,2361,2362,2364],{},"Should I use ",[14,2360,1177],{}," or ",[14,2363,1174],{},"?",[14,2366,1174],{}," for comparison keys, because it handles cases ",[14,2369,1177],{}," misses, such as the German sharp s folding to a double s. Use ",[14,2372,1177],{}," only when you are producing text for display.",[10,2375,2376,2379,2380,2383],{},[1170,2377,2378],{},"Why do accented characters compare as different?","\nThe same character can be stored as one code point or as a base letter plus a combining accent. Normalise with ",[14,2381,2382],{},"unicodedata.normalize"," to NFC or NFKC so both forms become identical before comparing.",[10,2385,2386,2389],{},[1170,2387,2388],{},"Should I clean the values or keep the originals?","\nKeep both. Clean into a new key column used for joining and grouping, and leave the original for display, so a report still shows the value exactly as it was supplied.",[129,2391,2393],{"id":2392},"related","Related",[2395,2396,2397,2404,2411,2418,2424],"ul",{},[2398,2399,2400,2401,2403],"li",{},"Up to the parent: ",[19,2402,22],{"href":21}," — the wider cleaning toolkit.",[2398,2405,2406,2410],{},[19,2407,2409],{"href":2408},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fconvert-excel-text-columns-to-numbers-with-pandas\u002F","Convert Excel Text Columns to Numbers with pandas"," — the numeric equivalent of this problem.",[2398,2412,2413,2417],{},[19,2414,2416],{"href":2415},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fpandas-drop-duplicates-from-excel-column\u002F","pandas: Drop Duplicates from an Excel Column"," — deduplicating once the values are normalised.",[2398,2419,2420,2423],{},[19,2421,2422],{"href":1779},"Merge Two Excel Files on a Common Column with Python"," — the join that whitespace was breaking.",[2398,2425,2426,2430],{},[19,2427,2429],{"href":2428},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fvalidate-excel-columns-before-import-with-pandas\u002F","Validate Excel Columns Before Import with pandas"," — catching the problem at the boundary.",[2432,2433,2434],"style",{},"html pre.shiki code .sMTad, html code.shiki .sMTad{--shiki-default:#6F42C1;--shiki-dark:#FFB757}html pre.shiki code .srMev, html code.shiki .srMev{--shiki-default:#032F62;--shiki-dark:#ADDCFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .s-kum, html code.shiki .s-kum{--shiki-default:#D73A49;--shiki-dark:#FF9492}html pre.shiki code .skGVy, html code.shiki .skGVy{--shiki-default:#24292E;--shiki-dark:#F0F3F6}html pre.shiki code .sP0c6, html code.shiki .sP0c6{--shiki-default:#005CC5;--shiki-dark:#91CBFF}html pre.shiki code .s-wDw, html code.shiki .s-wDw{--shiki-default:#6A737D;--shiki-dark:#BDC4CC}html pre.shiki code .s_Opv, html code.shiki .s_Opv{--shiki-default:#6F42C1;--shiki-dark:#DBB7FF}html pre.shiki code .sa561, html code.shiki .sa561{--shiki-default:#E36209;--shiki-dark:#FFB757}html pre.shiki code .sSjpA, html code.shiki .sSjpA{--shiki-default:#005CC5;--shiki-dark:#FF9492}",{"title":139,"searchDepth":189,"depth":189,"links":2436},[2437,2438,2439,2440,2441,2442,2443,2444,2445,2446],{"id":131,"depth":189,"text":132},{"id":322,"depth":189,"text":323},{"id":685,"depth":189,"text":686},{"id":1011,"depth":189,"text":1012},{"id":1298,"depth":189,"text":1299},{"id":1783,"depth":189,"text":1784},{"id":1930,"depth":189,"text":1931},{"id":2323,"depth":189,"text":2324},{"id":2333,"depth":189,"text":2334},{"id":2392,"depth":189,"text":2393},"2026-08-15","Trailing spaces break joins and group-bys silently. Clean Excel text columns in pandas — strip, collapse inner whitespace, non-breaking spaces, case folding and accents.","md",[2451,2453,2456,2459,2461],{"q":2339,"a":2452},"One side almost certainly has trailing whitespace or a non-breaking space. Both render as a normal gap, so the values look the same on screen while comparing as different strings. Print the repr of a failing value to see what is really there.",{"q":2454,"a":2455},"Does str.strip remove non-breaking spaces?","Not by default in older pandas versions, and it is safest not to rely on it. Replace the specific characters first — non-breaking space, zero-width space and the byte-order mark — then strip.",{"q":2457,"a":2458},"Should I use lower or casefold?","casefold for comparison keys, because it handles cases lower misses, such as the German sharp s folding to a double s. Use lower only when you are producing text for display.",{"q":2378,"a":2460},"The same character can be stored as one code point or as a base letter plus a combining accent. Normalise with unicodedata.normalize to NFC so both forms become identical before comparing.",{"q":2388,"a":2462},"Keep both. Clean into a new key column used for joining and grouping, and leave the original for display, so a report still shows the value exactly as it was supplied.",{},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fstrip-whitespace-and-normalise-text-columns-with-pandas",{"title":2466,"description":2467},"Strip Whitespace from Excel Text Columns in pandas","Fix invisible text problems from Excel exports: str.strip, collapsing inner spaces, non-breaking and zero-width characters, Unicode normalisation and safe case folding.","strip-whitespace-and-normalise-text-columns-with-pandas","advanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fstrip-whitespace-and-normalise-text-columns-with-pandas\u002Findex","how-to","aJB5wGYUs7g3IQcqdJTLBBWAU7L0g3k6uDDopQjqEks",[2473,2477],{"title":2474,"path":2475,"stem":2476,"children":-1},"Remove Blank Rows From Excel With Pandas","\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fremove-blank-rows-from-excel-with-pandas","advanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fremove-blank-rows-from-excel-with-pandas\u002Findex",{"title":2478,"path":2479,"stem":2480,"children":-1},"Creating Pivot Tables from Excel Data with Pandas","\u002Fadvanced-data-transformation-and-cleaning\u002Fcreating-pivot-tables-from-excel-data","advanced-data-transformation-and-cleaning\u002Fcreating-pivot-tables-from-excel-data\u002Findex",1786800028572]