[{"data":1,"prerenderedAt":1938},["ShallowReactive",2],{"doc:\u002Fadvanced-data-transformation-and-cleaning\u002Freading-excel-with-polars-and-arrow\u002Fconvert-excel-files-to-parquet-with-python":3,"surround:\u002Fadvanced-data-transformation-and-cleaning\u002Freading-excel-with-polars-and-arrow\u002Fconvert-excel-files-to-parquet-with-python":1930},{"id":4,"title":5,"body":6,"dateModified":1907,"datePublished":1907,"description":1908,"extension":1909,"faq":1910,"meta":1921,"navigation":238,"path":1922,"seo":1923,"slug":1926,"stem":1927,"type":1928,"__hash__":1929},"docs\u002Fadvanced-data-transformation-and-cleaning\u002Freading-excel-with-polars-and-arrow\u002Fconvert-excel-files-to-parquet-with-python\u002Findex.md","Convert Excel Files to Parquet with Python",{"type":7,"value":8,"toc":1892},"minimark",[9,19,142,147,150,198,204,208,211,302,305,348,362,366,369,574,581,585,588,832,839,972,976,979,1100,1103,1156,1159,1186,1190,1193,1257,1264,1268,1271,1395,1402,1406,1409,1652,1659,1725,1729,1774,1778,1789,1793,1799,1803,1809,1815,1833,1842,1848,1852,1888],[10,11,12,13,18],"p",{},"If the same workbook is read more than once, parsing it more than once is wasted work. Converting to Parquet — a typed, columnar, compressed format — turns a slow Excel read into a fast columnar one and carries the schema with the data, so no later job has to guess whether an order number is text. This guide converts single files and whole folders, keeps types intact, refreshes incrementally, and queries the result lazily. It completes the toolkit in ",[14,15,17],"a",{"href":16},"\u002Fadvanced-data-transformation-and-cleaning\u002Freading-excel-with-polars-and-arrow\u002F","Reading Excel with Polars and Arrow",".",[20,21,29,30,29,34,29,38,29,45,29,52,29,57,29,65,29,72,29,78,29,81,29,85,29,88,29,92,29,97,29,102,29,108,29,113,29,116,29,120,29,124,29,129,29,132,29,136,29,138],"svg",{"viewBox":22,"role":23,"ariaLabelledBy":24,"xmlns":27,"style":28},"0 0 760 240","img",[25,26],"pq-t","pq-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;max-width:760px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif","\n  ",[31,32,33],"title",{"id":25},"Row storage versus columnar storage",[35,36,37],"desc",{"id":26},"An xlsx stores cells row by row inside XML, so any read parses everything; Parquet stores each column contiguously with its type, so a query reads only the columns it needs.",[39,40],"rect",{"x":41,"y":41,"width":42,"height":43,"fill":44},"0","760","240","#ffffff",[46,47,51],"text",{"x":48,"y":49,"style":50},"196","28","font-size:12.5px;font-weight:700;fill:var(--accent-ink,#be185d);text-anchor:middle","xlsx — rows of XML cells",[46,53,56],{"x":54,"y":49,"style":55},"566","font-size:12.5px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","Parquet — typed columns",[58,59],"line",{"x1":60,"y1":61,"x2":60,"y2":62,"stroke":63,"style":64},"380","40","214","var(--line,#cdd5e6)","stroke-width:1px",[39,66],{"x":67,"y":68,"width":69,"height":67,"rx":70,"fill":71,"stroke":63},"34","46","322","7","#fee8f2",[46,73,77],{"x":74,"y":75,"style":76},"195","68","font-size:11.5px;fill:var(--accent-ink,#be185d);text-anchor:middle","row 1: date, region, product, revenue",[39,79],{"x":67,"y":80,"width":69,"height":67,"rx":70,"fill":71,"stroke":63},"88",[46,82,84],{"x":74,"y":83,"style":76},"110","row 2: date, region, product, revenue",[39,86],{"x":67,"y":87,"width":69,"height":67,"rx":70,"fill":71,"stroke":63},"130",[46,89,91],{"x":74,"y":90,"style":76},"152","row 3: date, region, product, revenue",[39,93],{"x":67,"y":94,"width":69,"height":95,"rx":70,"fill":96,"stroke":63},"172","42","#f0f2f5",[46,98,101],{"x":74,"y":99,"style":100},"198","font-size:11.5px;fill:var(--text,#172033);text-anchor:middle","every read parses every cell",[39,103],{"x":104,"y":68,"width":105,"height":106,"rx":70,"fill":107,"stroke":63},"404","76","118","#d9f4f1",[46,109,112],{"x":110,"y":83,"style":111},"442","font-size:11.5px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","date",[39,114],{"x":115,"y":68,"width":105,"height":106,"rx":70,"fill":107,"stroke":63},"486",[46,117,119],{"x":118,"y":83,"style":111},"524","region",[39,121],{"x":122,"y":68,"width":105,"height":106,"rx":70,"fill":123,"stroke":63},"568","#ebebfd",[46,125,128],{"x":126,"y":83,"style":127},"606","font-size:11.5px;font-weight:700;fill:var(--brand-strong,#4338ca);text-anchor:middle","product",[39,130],{"x":131,"y":68,"width":105,"height":106,"rx":70,"fill":107,"stroke":63},"650",[46,133,135],{"x":134,"y":83,"style":111},"688","revenue",[39,137],{"x":104,"y":94,"width":69,"height":95,"rx":70,"fill":96,"stroke":63},[46,139,141],{"x":140,"y":99,"style":100},"565","a query touches only the columns it names",[143,144,146],"h2",{"id":145},"prerequisites","Prerequisites",[10,148,149],{},"Either library will do the conversion; pick whichever your project already uses:",[151,152,157],"pre",{"className":153,"code":154,"language":155,"meta":156,"style":156},"language-bash shiki shiki-themes github-light github-dark-high-contrast","pip install pandas openpyxl pyarrow        # pandas route\npip install \"polars[excel]\"                # polars route (pyarrow optional)\n","bash","",[158,159,160,185],"code",{"__ignoreMap":156},[161,162,164,168,172,175,178,181],"span",{"class":58,"line":163},1,[161,165,167],{"class":166},"sMTad","pip",[161,169,171],{"class":170},"srMev"," install",[161,173,174],{"class":170}," pandas",[161,176,177],{"class":170}," openpyxl",[161,179,180],{"class":170}," pyarrow",[161,182,184],{"class":183},"s-wDw","        # pandas route\n",[161,186,188,190,192,195],{"class":58,"line":187},2,[161,189,167],{"class":166},[161,191,171],{"class":170},[161,193,194],{"class":170}," \"polars[excel]\"",[161,196,197],{"class":183},"                # polars route (pyarrow optional)\n",[10,199,200,203],{},[158,201,202],{},"pyarrow"," is what actually reads and writes the Parquet format in the pandas route.",[143,205,207],{"id":206},"convert-one-workbook","Convert one workbook",[10,209,210],{},"The pandas version is two lines:",[151,212,216],{"className":213,"code":214,"language":215,"meta":156,"style":156},"language-python shiki shiki-themes github-light github-dark-high-contrast","import pandas as pd\n\ndf = pd.read_excel(\"sales.xlsx\", engine=\"calamine\")\ndf.to_parquet(\"sales.parquet\", compression=\"zstd\", index=False)\n","python",[158,217,218,234,240,270],{"__ignoreMap":156},[161,219,220,224,228,231],{"class":58,"line":163},[161,221,223],{"class":222},"s-kum","import",[161,225,227],{"class":226},"skGVy"," pandas ",[161,229,230],{"class":222},"as",[161,232,233],{"class":226}," pd\n",[161,235,236],{"class":58,"line":187},[161,237,239],{"emptyLinePlaceholder":238},true,"\n",[161,241,243,246,249,252,255,258,262,264,267],{"class":58,"line":242},3,[161,244,245],{"class":226},"df ",[161,247,248],{"class":222},"=",[161,250,251],{"class":226}," pd.read_excel(",[161,253,254],{"class":170},"\"sales.xlsx\"",[161,256,257],{"class":226},", ",[161,259,261],{"class":260},"sa561","engine",[161,263,248],{"class":222},[161,265,266],{"class":170},"\"calamine\"",[161,268,269],{"class":226},")\n",[161,271,273,276,279,281,284,286,289,291,294,296,300],{"class":58,"line":272},4,[161,274,275],{"class":226},"df.to_parquet(",[161,277,278],{"class":170},"\"sales.parquet\"",[161,280,257],{"class":226},[161,282,283],{"class":260},"compression",[161,285,248],{"class":222},[161,287,288],{"class":170},"\"zstd\"",[161,290,257],{"class":226},[161,292,293],{"class":260},"index",[161,295,248],{"class":222},[161,297,299],{"class":298},"sP0c6","False",[161,301,269],{"class":226},[10,303,304],{},"The Polars version is one:",[151,306,308],{"className":213,"code":307,"language":215,"meta":156,"style":156},"import polars as pl\n\npl.read_excel(\"sales.xlsx\").write_parquet(\"sales.parquet\", compression=\"zstd\")\n",[158,309,310,322,326],{"__ignoreMap":156},[161,311,312,314,317,319],{"class":58,"line":163},[161,313,223],{"class":222},[161,315,316],{"class":226}," polars ",[161,318,230],{"class":222},[161,320,321],{"class":226}," pl\n",[161,323,324],{"class":58,"line":187},[161,325,239],{"emptyLinePlaceholder":238},[161,327,328,331,333,336,338,340,342,344,346],{"class":58,"line":242},[161,329,330],{"class":226},"pl.read_excel(",[161,332,254],{"class":170},[161,334,335],{"class":226},").write_parquet(",[161,337,278],{"class":170},[161,339,257],{"class":226},[161,341,283],{"class":260},[161,343,248],{"class":222},[161,345,288],{"class":170},[161,347,269],{"class":226},[10,349,350,353,354,357,358,361],{},[158,351,352],{},"index=False"," matters in pandas: writing the index adds a column that every later read has to ignore. ",[158,355,356],{},"zstd"," compresses better than the default ",[158,359,360],{},"snappy"," at a small cost in write time, which is the right trade for a file written once and read many times.",[143,363,365],{"id":364},"fix-the-types-before-you-freeze-them","Fix the types before you freeze them",[10,367,368],{},"Parquet stores the schema, so whatever types you convert with are the types every downstream job inherits. That is a feature — provided the types are right at the moment of conversion:",[151,370,372],{"className":213,"code":371,"language":215,"meta":156,"style":156},"import pandas as pd\n\ndf = pd.read_excel(\n    \"sales.xlsx\",\n    engine=\"calamine\",\n    dtype={\"order_id\": \"string\", \"postcode\": \"string\"},   # keep leading zeros\n    parse_dates=[\"order_date\"],\n)\ndf[\"region\"] = df[\"region\"].astype(\"category\")            # small and fast to group\ndf[\"revenue\"] = pd.to_numeric(df[\"revenue\"], errors=\"coerce\")\n\nprint(df.dtypes)\ndf.to_parquet(\"sales.parquet\", compression=\"zstd\", index=False)\n",[158,373,374,384,388,397,405,417,452,469,474,505,535,540,549],{"__ignoreMap":156},[161,375,376,378,380,382],{"class":58,"line":163},[161,377,223],{"class":222},[161,379,227],{"class":226},[161,381,230],{"class":222},[161,383,233],{"class":226},[161,385,386],{"class":58,"line":187},[161,387,239],{"emptyLinePlaceholder":238},[161,389,390,392,394],{"class":58,"line":242},[161,391,245],{"class":226},[161,393,248],{"class":222},[161,395,396],{"class":226}," pd.read_excel(\n",[161,398,399,402],{"class":58,"line":272},[161,400,401],{"class":170},"    \"sales.xlsx\"",[161,403,404],{"class":226},",\n",[161,406,408,411,413,415],{"class":58,"line":407},5,[161,409,410],{"class":260},"    engine",[161,412,248],{"class":222},[161,414,266],{"class":170},[161,416,404],{"class":226},[161,418,420,423,425,428,431,434,437,439,442,444,446,449],{"class":58,"line":419},6,[161,421,422],{"class":260},"    dtype",[161,424,248],{"class":222},[161,426,427],{"class":226},"{",[161,429,430],{"class":170},"\"order_id\"",[161,432,433],{"class":226},": ",[161,435,436],{"class":170},"\"string\"",[161,438,257],{"class":226},[161,440,441],{"class":170},"\"postcode\"",[161,443,433],{"class":226},[161,445,436],{"class":170},[161,447,448],{"class":226},"},   ",[161,450,451],{"class":183},"# keep leading zeros\n",[161,453,455,458,460,463,466],{"class":58,"line":454},7,[161,456,457],{"class":260},"    parse_dates",[161,459,248],{"class":222},[161,461,462],{"class":226},"[",[161,464,465],{"class":170},"\"order_date\"",[161,467,468],{"class":226},"],\n",[161,470,472],{"class":58,"line":471},8,[161,473,269],{"class":226},[161,475,477,480,483,486,488,491,493,496,499,502],{"class":58,"line":476},9,[161,478,479],{"class":226},"df[",[161,481,482],{"class":170},"\"region\"",[161,484,485],{"class":226},"] ",[161,487,248],{"class":222},[161,489,490],{"class":226}," df[",[161,492,482],{"class":170},[161,494,495],{"class":226},"].astype(",[161,497,498],{"class":170},"\"category\"",[161,500,501],{"class":226},")            ",[161,503,504],{"class":183},"# small and fast to group\n",[161,506,508,510,513,515,517,520,522,525,528,530,533],{"class":58,"line":507},10,[161,509,479],{"class":226},[161,511,512],{"class":170},"\"revenue\"",[161,514,485],{"class":226},[161,516,248],{"class":222},[161,518,519],{"class":226}," pd.to_numeric(df[",[161,521,512],{"class":170},[161,523,524],{"class":226},"], ",[161,526,527],{"class":260},"errors",[161,529,248],{"class":222},[161,531,532],{"class":170},"\"coerce\"",[161,534,269],{"class":226},[161,536,538],{"class":58,"line":537},11,[161,539,239],{"emptyLinePlaceholder":238},[161,541,543,546],{"class":58,"line":542},12,[161,544,545],{"class":298},"print",[161,547,548],{"class":226},"(df.dtypes)\n",[161,550,552,554,556,558,560,562,564,566,568,570,572],{"class":58,"line":551},13,[161,553,275],{"class":226},[161,555,278],{"class":170},[161,557,257],{"class":226},[161,559,283],{"class":260},[161,561,248],{"class":222},[161,563,288],{"class":170},[161,565,257],{"class":226},[161,567,293],{"class":260},[161,569,248],{"class":222},[161,571,299],{"class":298},[161,573,269],{"class":226},[10,575,576,577,18],{},"Converting with sloppy types bakes the sloppiness in permanently, and the mistake is invisible until a join misses. The identifier problem in particular is worth being deliberate about — see ",[14,578,580],{"href":579},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fconvert-excel-text-columns-to-numbers-with-pandas\u002F","Convert Excel text columns to numbers with pandas",[143,582,584],{"id":583},"convert-a-folder-and-record-where-each-row-came-from","Convert a folder, and record where each row came from",[10,586,587],{},"Monthly exports are the common case. Convert each file once, tag its origin, and keep the outputs beside each other so they can be scanned as a set:",[151,589,591],{"className":213,"code":590,"language":215,"meta":156,"style":156},"\"\"\"Convert every workbook in a folder, skipping ones already up to date.\"\"\"\nfrom pathlib import Path\n\nimport polars as pl\n\nSRC = Path(\"monthly\")\nDST = Path(\"cache\")\nDST.mkdir(exist_ok=True)\n\nfor src in sorted(SRC.glob(\"*.xlsx\")):\n    dst = DST \u002F f\"{src.stem}.parquet\"\n    if dst.exists() and dst.stat().st_mtime >= src.stat().st_mtime:\n        continue                                    # already converted, unchanged\n    (pl.read_excel(src)\n       .with_columns(source_file=pl.lit(src.name), period=pl.lit(src.stem[-7:]))\n       .write_parquet(dst, compression=\"zstd\"))\n    print(\"converted\", src.name)\n",[158,592,593,598,611,615,625,629,645,659,676,680,708,739,759,767,773,803,818],{"__ignoreMap":156},[161,594,595],{"class":58,"line":163},[161,596,597],{"class":170},"\"\"\"Convert every workbook in a folder, skipping ones already up to date.\"\"\"\n",[161,599,600,603,606,608],{"class":58,"line":187},[161,601,602],{"class":222},"from",[161,604,605],{"class":226}," pathlib ",[161,607,223],{"class":222},[161,609,610],{"class":226}," Path\n",[161,612,613],{"class":58,"line":242},[161,614,239],{"emptyLinePlaceholder":238},[161,616,617,619,621,623],{"class":58,"line":272},[161,618,223],{"class":222},[161,620,316],{"class":226},[161,622,230],{"class":222},[161,624,321],{"class":226},[161,626,627],{"class":58,"line":407},[161,628,239],{"emptyLinePlaceholder":238},[161,630,631,634,637,640,643],{"class":58,"line":419},[161,632,633],{"class":298},"SRC",[161,635,636],{"class":222}," =",[161,638,639],{"class":226}," Path(",[161,641,642],{"class":170},"\"monthly\"",[161,644,269],{"class":226},[161,646,647,650,652,654,657],{"class":58,"line":454},[161,648,649],{"class":298},"DST",[161,651,636],{"class":222},[161,653,639],{"class":226},[161,655,656],{"class":170},"\"cache\"",[161,658,269],{"class":226},[161,660,661,663,666,669,671,674],{"class":58,"line":471},[161,662,649],{"class":298},[161,664,665],{"class":226},".mkdir(",[161,667,668],{"class":260},"exist_ok",[161,670,248],{"class":222},[161,672,673],{"class":298},"True",[161,675,269],{"class":226},[161,677,678],{"class":58,"line":476},[161,679,239],{"emptyLinePlaceholder":238},[161,681,682,685,688,691,694,697,699,702,705],{"class":58,"line":507},[161,683,684],{"class":222},"for",[161,686,687],{"class":226}," src ",[161,689,690],{"class":222},"in",[161,692,693],{"class":298}," sorted",[161,695,696],{"class":226},"(",[161,698,633],{"class":298},[161,700,701],{"class":226},".glob(",[161,703,704],{"class":170},"\"*.xlsx\"",[161,706,707],{"class":226},")):\n",[161,709,710,713,715,718,721,724,727,730,733,736],{"class":58,"line":537},[161,711,712],{"class":226},"    dst ",[161,714,248],{"class":222},[161,716,717],{"class":298}," DST",[161,719,720],{"class":222}," \u002F",[161,722,723],{"class":222}," f",[161,725,726],{"class":170},"\"",[161,728,427],{"class":729},"sSjpA",[161,731,732],{"class":226},"src.stem",[161,734,735],{"class":729},"}",[161,737,738],{"class":170},".parquet\"\n",[161,740,741,744,747,750,753,756],{"class":58,"line":542},[161,742,743],{"class":222},"    if",[161,745,746],{"class":226}," dst.exists() ",[161,748,749],{"class":222},"and",[161,751,752],{"class":226}," dst.stat().st_mtime ",[161,754,755],{"class":222},">=",[161,757,758],{"class":226}," src.stat().st_mtime:\n",[161,760,761,764],{"class":58,"line":551},[161,762,763],{"class":222},"        continue",[161,765,766],{"class":183},"                                    # already converted, unchanged\n",[161,768,770],{"class":58,"line":769},14,[161,771,772],{"class":226},"    (pl.read_excel(src)\n",[161,774,776,779,782,784,787,790,792,795,798,800],{"class":58,"line":775},15,[161,777,778],{"class":226},"       .with_columns(",[161,780,781],{"class":260},"source_file",[161,783,248],{"class":222},[161,785,786],{"class":226},"pl.lit(src.name), ",[161,788,789],{"class":260},"period",[161,791,248],{"class":222},[161,793,794],{"class":226},"pl.lit(src.stem[",[161,796,797],{"class":222},"-",[161,799,70],{"class":298},[161,801,802],{"class":226},":]))\n",[161,804,806,809,811,813,815],{"class":58,"line":805},16,[161,807,808],{"class":226},"       .write_parquet(dst, ",[161,810,283],{"class":260},[161,812,248],{"class":222},[161,814,288],{"class":170},[161,816,817],{"class":226},"))\n",[161,819,821,824,826,829],{"class":58,"line":820},17,[161,822,823],{"class":298},"    print",[161,825,696],{"class":226},[161,827,828],{"class":170},"\"converted\"",[161,830,831],{"class":226},", src.name)\n",[10,833,834,835,838],{},"The ",[158,836,837],{},"mtime"," comparison makes the script idempotent: run it as often as you like and it only does work for files that actually changed. That property is what lets it sit at the top of a nightly job without doubling the run time.",[20,840,29,845,29,848,29,851,29,854,29,859,29,865,29,871,29,876,29,882,29,887,29,891,29,896,29,899,29,905,29,909,29,915,29,919,29,925,29,929,29,933,29,937,29,940,29,945,29,949,29,952,29,955,29,958,29,962,29,968],{"viewBox":841,"role":23,"ariaLabelledBy":842,"xmlns":27,"style":28},"0 0 760 222",[843,844],"pq2-t","pq2-d",[31,846,847],{"id":843},"Convert once, then query the cache every run",[35,849,850],{"id":844},"Incoming workbooks are converted to Parquet only when they change, and every downstream report scans the Parquet cache instead of re-parsing spreadsheets.",[39,852],{"x":41,"y":41,"width":42,"height":853,"fill":44},"222",[46,855,858],{"x":60,"y":856,"style":857},"26","font-size:13px;font-weight:600;fill:var(--muted,#5b6780);text-anchor:middle","The Excel parse happens once per file, ever",[39,860],{"x":49,"y":861,"width":862,"height":863,"rx":864,"fill":96,"stroke":63},"48","150","60","11",[46,866,870],{"x":867,"y":868,"style":869},"103","74","font-size:12px;font-weight:700;fill:var(--text,#172033);text-anchor:middle","incoming",[46,872,875],{"x":867,"y":873,"style":874},"94","font-size:11.5px;fill:var(--muted,#5b6780);text-anchor:middle","*.xlsx",[58,877],{"x1":878,"y1":879,"x2":62,"y2":879,"stroke":880,"style":881},"178","78","var(--brand,#5b5cf0)","stroke-width:2px",[883,884],"polygon",{"points":885,"fill":886},"214,78 204,73 204,83","#5b5cf0",[39,888],{"x":889,"y":861,"width":890,"height":863,"rx":864,"fill":123,"stroke":63},"218","164",[46,892,895],{"x":893,"y":868,"style":894},"300","font-size:12px;font-weight:700;fill:var(--brand-strong,#4338ca);text-anchor:middle","changed since",[46,897,898],{"x":893,"y":873,"style":894},"last convert?",[58,900],{"x1":901,"y1":902,"x2":903,"y2":902,"stroke":904,"style":881},"382","66","418","var(--teal,#0f9488)",[883,906],{"points":907,"fill":908},"418,66 408,61 408,71","#0f766e",[46,910,914],{"x":911,"y":912,"style":913},"388","56","font-size:11px;fill:var(--muted,#5b6780);text-anchor:start","yes",[39,916],{"x":917,"y":61,"width":862,"height":918,"rx":864,"fill":107,"stroke":63},"422","52",[46,920,924],{"x":921,"y":922,"style":923},"497","72","font-size:12px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","parse and write",[58,926],{"x1":901,"y1":873,"x2":903,"y2":927,"stroke":928,"style":881},"132","var(--muted,#5b6780)",[883,930],{"points":931,"fill":932},"418,132 406,129 411,121","#5b6780",[46,934,936],{"x":911,"y":935,"style":913},"124","no",[39,938],{"x":917,"y":939,"width":862,"height":918,"rx":864,"fill":96,"stroke":63},"112",[46,941,944],{"x":921,"y":942,"style":943},"144","font-size:12px;fill:var(--text,#172033);text-anchor:middle","skip",[58,946],{"x1":947,"y1":902,"x2":948,"y2":873,"stroke":904,"style":881},"572","608",[883,950],{"points":951,"fill":908},"608,94 596,91 601,83",[58,953],{"x1":947,"y1":954,"x2":948,"y2":83,"stroke":928,"style":881},"138",[883,956],{"points":957,"fill":932},"608,110 601,121 596,113",[39,959],{"x":960,"y":105,"width":935,"height":918,"rx":864,"fill":961,"stroke":63},"612","#fdefd8",[46,963,967],{"x":964,"y":965,"style":966},"674","108","font-size:12px;font-weight:700;fill:var(--gold-ink,#7a4e06);text-anchor:middle","cache\u002F*.parquet",[46,969,971],{"x":60,"y":970,"style":874},"200","Reports scan the cache; a rerun after a failure costs almost nothing",[143,973,975],{"id":974},"query-the-cache-instead-of-the-spreadsheets","Query the cache instead of the spreadsheets",[10,977,978],{},"Once converted, reads get dramatically cheaper — especially through Polars' lazy scan, which reads only the columns and row groups a query needs:",[151,980,982],{"className":213,"code":981,"language":215,"meta":156,"style":156},"import polars as pl\n\ntop = (\n    pl.scan_parquet(\"cache\u002F*.parquet\")\n      .filter(pl.col(\"region\") == \"EMEA\")\n      .group_by(\"product\")\n      .agg(revenue=pl.col(\"revenue\").sum())\n      .sort(\"revenue\", descending=True)\n      .head(20)\n      .collect()\n)\n",[158,983,984,994,998,1008,1018,1036,1046,1063,1081,1091,1096],{"__ignoreMap":156},[161,985,986,988,990,992],{"class":58,"line":163},[161,987,223],{"class":222},[161,989,316],{"class":226},[161,991,230],{"class":222},[161,993,321],{"class":226},[161,995,996],{"class":58,"line":187},[161,997,239],{"emptyLinePlaceholder":238},[161,999,1000,1003,1005],{"class":58,"line":242},[161,1001,1002],{"class":226},"top ",[161,1004,248],{"class":222},[161,1006,1007],{"class":226}," (\n",[161,1009,1010,1013,1016],{"class":58,"line":272},[161,1011,1012],{"class":226},"    pl.scan_parquet(",[161,1014,1015],{"class":170},"\"cache\u002F*.parquet\"",[161,1017,269],{"class":226},[161,1019,1020,1023,1025,1028,1031,1034],{"class":58,"line":407},[161,1021,1022],{"class":226},"      .filter(pl.col(",[161,1024,482],{"class":170},[161,1026,1027],{"class":226},") ",[161,1029,1030],{"class":222},"==",[161,1032,1033],{"class":170}," \"EMEA\"",[161,1035,269],{"class":226},[161,1037,1038,1041,1044],{"class":58,"line":419},[161,1039,1040],{"class":226},"      .group_by(",[161,1042,1043],{"class":170},"\"product\"",[161,1045,269],{"class":226},[161,1047,1048,1051,1053,1055,1058,1060],{"class":58,"line":454},[161,1049,1050],{"class":226},"      .agg(",[161,1052,135],{"class":260},[161,1054,248],{"class":222},[161,1056,1057],{"class":226},"pl.col(",[161,1059,512],{"class":170},[161,1061,1062],{"class":226},").sum())\n",[161,1064,1065,1068,1070,1072,1075,1077,1079],{"class":58,"line":471},[161,1066,1067],{"class":226},"      .sort(",[161,1069,512],{"class":170},[161,1071,257],{"class":226},[161,1073,1074],{"class":260},"descending",[161,1076,248],{"class":222},[161,1078,673],{"class":298},[161,1080,269],{"class":226},[161,1082,1083,1086,1089],{"class":58,"line":476},[161,1084,1085],{"class":226},"      .head(",[161,1087,1088],{"class":298},"20",[161,1090,269],{"class":226},[161,1092,1093],{"class":58,"line":507},[161,1094,1095],{"class":226},"      .collect()\n",[161,1097,1098],{"class":58,"line":537},[161,1099,269],{"class":226},[10,1101,1102],{},"pandas reads the same files, and can push a column selection down too:",[151,1104,1106],{"className":213,"code":1105,"language":215,"meta":156,"style":156},"import pandas as pd\n\ndf = pd.read_parquet(\"cache\u002F\", columns=[\"region\", \"product\", \"revenue\"])\n",[158,1107,1108,1118,1122],{"__ignoreMap":156},[161,1109,1110,1112,1114,1116],{"class":58,"line":163},[161,1111,223],{"class":222},[161,1113,227],{"class":226},[161,1115,230],{"class":222},[161,1117,233],{"class":226},[161,1119,1120],{"class":58,"line":187},[161,1121,239],{"emptyLinePlaceholder":238},[161,1123,1124,1126,1128,1131,1134,1136,1139,1141,1143,1145,1147,1149,1151,1153],{"class":58,"line":242},[161,1125,245],{"class":226},[161,1127,248],{"class":222},[161,1129,1130],{"class":226}," pd.read_parquet(",[161,1132,1133],{"class":170},"\"cache\u002F\"",[161,1135,257],{"class":226},[161,1137,1138],{"class":260},"columns",[161,1140,248],{"class":222},[161,1142,462],{"class":226},[161,1144,482],{"class":170},[161,1146,257],{"class":226},[161,1148,1043],{"class":170},[161,1150,257],{"class":226},[161,1152,512],{"class":170},[161,1154,1155],{"class":226},"])\n",[10,1157,1158],{},"DuckDB will query the folder in SQL without loading it, which suits ad-hoc analysis:",[151,1160,1162],{"className":213,"code":1161,"language":215,"meta":156,"style":156},"import duckdb\n\nduckdb.sql(\"SELECT region, sum(revenue) FROM 'cache\u002F*.parquet' GROUP BY region\").show()\n",[158,1163,1164,1171,1175],{"__ignoreMap":156},[161,1165,1166,1168],{"class":58,"line":163},[161,1167,223],{"class":222},[161,1169,1170],{"class":226}," duckdb\n",[161,1172,1173],{"class":58,"line":187},[161,1174,239],{"emptyLinePlaceholder":238},[161,1176,1177,1180,1183],{"class":58,"line":242},[161,1178,1179],{"class":226},"duckdb.sql(",[161,1181,1182],{"class":170},"\"SELECT region, sum(revenue) FROM 'cache\u002F*.parquet' GROUP BY region\"",[161,1184,1185],{"class":226},").show()\n",[143,1187,1189],{"id":1188},"partition-when-the-archive-gets-large","Partition when the archive gets large",[10,1191,1192],{},"For a multi-year archive, write partitioned output so a query for one month reads one directory rather than everything:",[151,1194,1196],{"className":213,"code":1195,"language":215,"meta":156,"style":156},"import pandas as pd\n\ndf = pd.read_parquet(\"cache\u002F\")\ndf.to_parquet(\"warehouse\u002F\", partition_cols=[\"year\", \"region\"], compression=\"zstd\")\n",[158,1197,1198,1208,1212,1224],{"__ignoreMap":156},[161,1199,1200,1202,1204,1206],{"class":58,"line":163},[161,1201,223],{"class":222},[161,1203,227],{"class":226},[161,1205,230],{"class":222},[161,1207,233],{"class":226},[161,1209,1210],{"class":58,"line":187},[161,1211,239],{"emptyLinePlaceholder":238},[161,1213,1214,1216,1218,1220,1222],{"class":58,"line":242},[161,1215,245],{"class":226},[161,1217,248],{"class":222},[161,1219,1130],{"class":226},[161,1221,1133],{"class":170},[161,1223,269],{"class":226},[161,1225,1226,1228,1231,1233,1236,1238,1240,1243,1245,1247,1249,1251,1253,1255],{"class":58,"line":272},[161,1227,275],{"class":226},[161,1229,1230],{"class":170},"\"warehouse\u002F\"",[161,1232,257],{"class":226},[161,1234,1235],{"class":260},"partition_cols",[161,1237,248],{"class":222},[161,1239,462],{"class":226},[161,1241,1242],{"class":170},"\"year\"",[161,1244,257],{"class":226},[161,1246,482],{"class":170},[161,1248,524],{"class":226},[161,1250,283],{"class":260},[161,1252,248],{"class":222},[161,1254,288],{"class":170},[161,1256,269],{"class":226},[10,1258,1259,1260,1263],{},"That produces ",[158,1261,1262],{},"warehouse\u002Fyear=2026\u002Fregion=EMEA\u002F…",", and every reader — pandas, Polars, DuckDB — understands the layout and skips directories a filter excludes. Do not over-partition: thousands of tiny files are slower than a few large ones, so partition on the columns you actually filter by, usually a period.",[143,1265,1267],{"id":1266},"give-people-back-a-workbook","Give people back a workbook",[10,1269,1270],{},"Parquet is for machines. When a person needs the numbers, write a workbook from the cache — which is now a fast operation, because the slow part happened once:",[151,1272,1274],{"className":213,"code":1273,"language":215,"meta":156,"style":156},"import polars as pl\n\n(pl.scan_parquet(\"cache\u002F*.parquet\")\n   .group_by(\"region\")\n   .agg(revenue=pl.col(\"revenue\").sum())\n   .sort(\"revenue\", descending=True)\n   .collect()\n   .write_excel(\"regional_summary.xlsx\", autofit=True,\n                table_style=\"Table Style Medium 9\",\n                column_formats={\"revenue\": \"#,##0.00\"}))\n",[158,1275,1276,1286,1290,1299,1308,1323,1340,1345,1364,1376],{"__ignoreMap":156},[161,1277,1278,1280,1282,1284],{"class":58,"line":163},[161,1279,223],{"class":222},[161,1281,316],{"class":226},[161,1283,230],{"class":222},[161,1285,321],{"class":226},[161,1287,1288],{"class":58,"line":187},[161,1289,239],{"emptyLinePlaceholder":238},[161,1291,1292,1295,1297],{"class":58,"line":242},[161,1293,1294],{"class":226},"(pl.scan_parquet(",[161,1296,1015],{"class":170},[161,1298,269],{"class":226},[161,1300,1301,1304,1306],{"class":58,"line":272},[161,1302,1303],{"class":226},"   .group_by(",[161,1305,482],{"class":170},[161,1307,269],{"class":226},[161,1309,1310,1313,1315,1317,1319,1321],{"class":58,"line":407},[161,1311,1312],{"class":226},"   .agg(",[161,1314,135],{"class":260},[161,1316,248],{"class":222},[161,1318,1057],{"class":226},[161,1320,512],{"class":170},[161,1322,1062],{"class":226},[161,1324,1325,1328,1330,1332,1334,1336,1338],{"class":58,"line":419},[161,1326,1327],{"class":226},"   .sort(",[161,1329,512],{"class":170},[161,1331,257],{"class":226},[161,1333,1074],{"class":260},[161,1335,248],{"class":222},[161,1337,673],{"class":298},[161,1339,269],{"class":226},[161,1341,1342],{"class":58,"line":454},[161,1343,1344],{"class":226},"   .collect()\n",[161,1346,1347,1350,1353,1355,1358,1360,1362],{"class":58,"line":471},[161,1348,1349],{"class":226},"   .write_excel(",[161,1351,1352],{"class":170},"\"regional_summary.xlsx\"",[161,1354,257],{"class":226},[161,1356,1357],{"class":260},"autofit",[161,1359,248],{"class":222},[161,1361,673],{"class":298},[161,1363,404],{"class":226},[161,1365,1366,1369,1371,1374],{"class":58,"line":476},[161,1367,1368],{"class":260},"                table_style",[161,1370,248],{"class":222},[161,1372,1373],{"class":170},"\"Table Style Medium 9\"",[161,1375,404],{"class":226},[161,1377,1378,1381,1383,1385,1387,1389,1392],{"class":58,"line":507},[161,1379,1380],{"class":260},"                column_formats",[161,1382,248],{"class":222},[161,1384,427],{"class":226},[161,1386,512],{"class":170},[161,1388,433],{"class":226},[161,1390,1391],{"class":170},"\"#,##0.00\"",[161,1393,1394],{"class":226},"}))\n",[10,1396,1397,1398,18],{},"The formatting options are covered in ",[14,1399,1401],{"href":1400},"\u002Fadvanced-data-transformation-and-cleaning\u002Freading-excel-with-polars-and-arrow\u002Fwrite-a-polars-dataframe-to-excel-with-formatting\u002F","Write a Polars DataFrame to Excel with formatting",[143,1403,1405],{"id":1404},"validate-the-conversion-before-you-rely-on-it","Validate the conversion before you rely on it",[10,1407,1408],{},"A cache is only useful if it matches its source. Check the conversion the first time you run it — row counts, column names, and the totals of the numeric columns people actually quote:",[151,1410,1412],{"className":213,"code":1411,"language":215,"meta":156,"style":156},"\"\"\"Assert the Parquet copy agrees with the workbook it came from.\"\"\"\nimport polars as pl\n\nsrc = pl.read_excel(\"sales.xlsx\")\ncached = pl.read_parquet(\"sales.parquet\")\n\nassert src.height == cached.height, f\"{src.height} rows in, {cached.height} out\"\nassert src.columns == cached.columns, \"column set changed\"\n\nfor column in (\"revenue\", \"quantity\"):\n    a, b = src[column].sum(), cached[column].sum()\n    assert abs(a - b) \u003C 0.01, f\"{column}: {a} != {b}\"\n\nprint(f\"verified {cached.height:,} rows, {len(cached.columns)} columns\")\n",[158,1413,1414,1419,1429,1433,1447,1461,1465,1503,1518,1522,1544,1554,1610,1614],{"__ignoreMap":156},[161,1415,1416],{"class":58,"line":163},[161,1417,1418],{"class":170},"\"\"\"Assert the Parquet copy agrees with the workbook it came from.\"\"\"\n",[161,1420,1421,1423,1425,1427],{"class":58,"line":187},[161,1422,223],{"class":222},[161,1424,316],{"class":226},[161,1426,230],{"class":222},[161,1428,321],{"class":226},[161,1430,1431],{"class":58,"line":242},[161,1432,239],{"emptyLinePlaceholder":238},[161,1434,1435,1438,1440,1443,1445],{"class":58,"line":272},[161,1436,1437],{"class":226},"src ",[161,1439,248],{"class":222},[161,1441,1442],{"class":226}," pl.read_excel(",[161,1444,254],{"class":170},[161,1446,269],{"class":226},[161,1448,1449,1452,1454,1457,1459],{"class":58,"line":407},[161,1450,1451],{"class":226},"cached ",[161,1453,248],{"class":222},[161,1455,1456],{"class":226}," pl.read_parquet(",[161,1458,278],{"class":170},[161,1460,269],{"class":226},[161,1462,1463],{"class":58,"line":419},[161,1464,239],{"emptyLinePlaceholder":238},[161,1466,1467,1470,1473,1475,1478,1481,1483,1485,1488,1490,1493,1495,1498,1500],{"class":58,"line":454},[161,1468,1469],{"class":222},"assert",[161,1471,1472],{"class":226}," src.height ",[161,1474,1030],{"class":222},[161,1476,1477],{"class":226}," cached.height, ",[161,1479,1480],{"class":222},"f",[161,1482,726],{"class":170},[161,1484,427],{"class":729},[161,1486,1487],{"class":226},"src.height",[161,1489,735],{"class":729},[161,1491,1492],{"class":170}," rows in, ",[161,1494,427],{"class":729},[161,1496,1497],{"class":226},"cached.height",[161,1499,735],{"class":729},[161,1501,1502],{"class":170}," out\"\n",[161,1504,1505,1507,1510,1512,1515],{"class":58,"line":471},[161,1506,1469],{"class":222},[161,1508,1509],{"class":226}," src.columns ",[161,1511,1030],{"class":222},[161,1513,1514],{"class":226}," cached.columns, ",[161,1516,1517],{"class":170},"\"column set changed\"\n",[161,1519,1520],{"class":58,"line":476},[161,1521,239],{"emptyLinePlaceholder":238},[161,1523,1524,1526,1529,1531,1534,1536,1538,1541],{"class":58,"line":507},[161,1525,684],{"class":222},[161,1527,1528],{"class":226}," column ",[161,1530,690],{"class":222},[161,1532,1533],{"class":226}," (",[161,1535,512],{"class":170},[161,1537,257],{"class":226},[161,1539,1540],{"class":170},"\"quantity\"",[161,1542,1543],{"class":226},"):\n",[161,1545,1546,1549,1551],{"class":58,"line":537},[161,1547,1548],{"class":226},"    a, b ",[161,1550,248],{"class":222},[161,1552,1553],{"class":226}," src[column].sum(), cached[column].sum()\n",[161,1555,1556,1559,1562,1565,1567,1570,1573,1576,1578,1580,1582,1584,1587,1589,1591,1593,1595,1597,1600,1602,1605,1607],{"class":58,"line":542},[161,1557,1558],{"class":222},"    assert",[161,1560,1561],{"class":298}," abs",[161,1563,1564],{"class":226},"(a ",[161,1566,797],{"class":222},[161,1568,1569],{"class":226}," b) ",[161,1571,1572],{"class":222},"\u003C",[161,1574,1575],{"class":298}," 0.01",[161,1577,257],{"class":226},[161,1579,1480],{"class":222},[161,1581,726],{"class":170},[161,1583,427],{"class":729},[161,1585,1586],{"class":226},"column",[161,1588,735],{"class":729},[161,1590,433],{"class":170},[161,1592,427],{"class":729},[161,1594,14],{"class":226},[161,1596,735],{"class":729},[161,1598,1599],{"class":170}," != ",[161,1601,427],{"class":729},[161,1603,1604],{"class":226},"b",[161,1606,735],{"class":729},[161,1608,1609],{"class":170},"\"\n",[161,1611,1612],{"class":58,"line":551},[161,1613,239],{"emptyLinePlaceholder":238},[161,1615,1616,1618,1620,1622,1625,1627,1629,1632,1634,1637,1639,1642,1645,1647,1650],{"class":58,"line":769},[161,1617,545],{"class":298},[161,1619,696],{"class":226},[161,1621,1480],{"class":222},[161,1623,1624],{"class":170},"\"verified ",[161,1626,427],{"class":729},[161,1628,1497],{"class":226},[161,1630,1631],{"class":222},":,",[161,1633,735],{"class":729},[161,1635,1636],{"class":170}," rows, ",[161,1638,427],{"class":729},[161,1640,1641],{"class":298},"len",[161,1643,1644],{"class":226},"(cached.columns)",[161,1646,735],{"class":729},[161,1648,1649],{"class":170}," columns\"",[161,1651,269],{"class":226},[10,1653,1654,1655,18],{},"Comparing totals rather than every cell keeps the check fast on a large file while still catching the failures that matter: a truncated read, a column silently dropped, or a numeric column that became text and summed to something absurd. Keep it as a test over a small fixture workbook so a library upgrade cannot change the conversion behaviour unnoticed — the approach in ",[14,1656,1658],{"href":1657},"\u002Fautomating-reporting-workflows\u002Ftesting-and-packaging-excel-automation-scripts\u002Ftest-excel-output-with-pytest\u002F","Test Excel output with pytest",[20,1660,29,1665,29,1668,29,1671,29,1673,29,1676,29,1681,29,1687,29,1690,29,1693,29,1696,29,1700,29,1703,29,1706,29,1710,29,1715,29,1718,29,1721],{"viewBox":1661,"role":23,"ariaLabelledBy":1662,"xmlns":27,"style":28},"0 0 760 196",[1663,1664],"pq3-t","pq3-d",[31,1666,1667],{"id":1663},"Three assertions that catch a bad conversion",[35,1669,1670],{"id":1664},"Row count catches a truncated read, the column list catches a dropped or renamed column, and column totals catch a type change that turned numbers into text.",[39,1672],{"x":41,"y":41,"width":42,"height":48,"fill":44},[46,1674,1675],{"x":60,"y":856,"style":857},"Cheap checks, in order of what they catch",[39,1677],{"x":1678,"y":68,"width":853,"height":1679,"rx":1680,"fill":123,"stroke":63},"30","106","12",[46,1682,1686],{"x":1683,"y":1684,"style":1685},"141","80","font-size:12.5px;font-weight:700;fill:var(--brand-strong,#4338ca);text-anchor:middle","row count",[46,1688,1689],{"x":1683,"y":83,"style":100},"catches a truncated",[46,1691,1692],{"x":1683,"y":87,"style":100},"or partial read",[39,1694],{"x":1695,"y":68,"width":853,"height":1679,"rx":1680,"fill":107,"stroke":63},"268",[46,1697,1699],{"x":1698,"y":1684,"style":55},"379","column list",[46,1701,1702],{"x":1698,"y":83,"style":100},"catches a dropped",[46,1704,1705],{"x":1698,"y":87,"style":100},"or renamed column",[39,1707],{"x":1708,"y":68,"width":1709,"height":1679,"rx":1680,"fill":961,"stroke":63},"506","224",[46,1711,1714],{"x":1712,"y":1684,"style":1713},"618","font-size:12.5px;font-weight:700;fill:var(--gold-ink,#7a4e06);text-anchor:middle","column totals",[46,1716,1717],{"x":1712,"y":83,"style":100},"catches a numeric",[46,1719,1720],{"x":1712,"y":87,"style":100},"column read as text",[46,1722,1724],{"x":60,"y":1723,"style":874},"180","All three run in well under a second on a report-sized file",[143,1726,1728],{"id":1727},"common-pitfalls-and-gotchas","Common pitfalls and gotchas",[1730,1731,1732,1743,1749,1755,1764],"ul",{},[1733,1734,1735,1739,1740,1742],"li",{},[1736,1737,1738],"strong",{},"Writing the pandas index."," ",[158,1741,352],{}," unless you genuinely need it; otherwise every reader inherits a stray column.",[1733,1744,1745,1748],{},[1736,1746,1747],{},"Mixed types in one column across files."," A column that is text in January and numeric in March produces a schema clash when the folder is scanned. Pin the dtype at conversion time.",[1733,1750,1751,1754],{},[1736,1752,1753],{},"Assuming Parquet is human-readable."," It is binary. Keep a generated workbook or CSV for anyone who needs to look at it directly.",[1733,1756,1757,1760,1761,1763],{},[1736,1758,1759],{},"Converting into the same folder as the sources."," Keep the cache separate so a glob for ",[158,1762,875],{}," never picks up derived files.",[1733,1765,1766,1769,1770,18],{},[1736,1767,1768],{},"Forgetting the timezone."," A timezone-aware timestamp survives in Parquet but not in Excel, so decide which representation is authoritative — see ",[14,1771,1773],{"href":1772},"\u002Fadvanced-data-transformation-and-cleaning\u002Fworking-with-dates-and-times-in-excel-data\u002Fhandle-timezones-in-excel-timestamps-with-python\u002F","Handle timezones in Excel timestamps with Python",[143,1775,1777],{"id":1776},"performance-and-scale-notes","Performance and scale notes",[10,1779,1780,1781,1784,1785,18],{},"The conversion itself is bounded by the Excel parse, so use the fastest reader available — ",[158,1782,1783],{},"engine=\"calamine\""," in pandas, or Polars, which uses it by default. After conversion, expect reads to be an order of magnitude quicker and the files several times smaller. Memory behaves better too: a columnar read materialises only the columns requested, so a job that needs three of forty columns no longer pays for the other thirty-seven. For workbooks too large to convert in one pass, read them in chunks and append row groups — the chunking approach is in ",[14,1786,1788],{"href":1787},"\u002Fadvanced-data-transformation-and-cleaning\u002Fworking-with-large-excel-files-in-python\u002Fread-large-excel-file-in-chunks-with-pandas\u002F","Read a large Excel file in chunks with pandas",[143,1790,1792],{"id":1791},"conclusion","Conclusion",[10,1794,1795,1796,1798],{},"Convert once, read many times. A short conversion step with correct types turns every downstream report into a columnar read that keeps its schema, loads in a fraction of the time, and costs a fraction of the disk. Keep the conversion idempotent with an ",[158,1797,837],{}," check, partition only by the column you filter on, and generate a workbook from the cache whenever a human needs to see the numbers.",[143,1800,1802],{"id":1801},"frequently-asked-questions","Frequently asked questions",[10,1804,1805,1808],{},[1736,1806,1807],{},"Why convert at all — can't I just read the Excel file each time?","\nYou can, but you pay the parse every run. Parquet stores columns in a typed binary layout, so a later read costs a fraction of the time, uses less memory, and needs no type inference because the schema travels with the file.",[10,1810,1811,1814],{},[1736,1812,1813],{},"Does Parquet keep my column types?","\nYes, that is the main reason to use it. Strings stay strings, dates stay dates, and a column of identifiers with leading zeros keeps them — provided you set the types correctly during the conversion.",[10,1816,1817,1820,1821,1824,1825,1828,1829,1832],{},[1736,1818,1819],{},"Can Excel open a Parquet file?","\nModern Excel can import Parquet through Power Query on some plans, but treat Parquet as the machine-readable cache and keep a generated ",[158,1822,1823],{},".xlsx"," for people. Convert back with a single ",[158,1826,1827],{},"write_excel"," or ",[158,1830,1831],{},"to_excel"," call when a human needs it.",[10,1834,1835,1838,1839,1841],{},[1736,1836,1837],{},"How much smaller are the files?","\nTypically several times smaller than the equivalent ",[158,1840,1823],{},", because columnar storage compresses repeated values extremely well. A folder of monthly workbooks usually shrinks dramatically once converted.",[10,1843,1844,1847],{},[1736,1845,1846],{},"Do I need Spark or a data warehouse to use Parquet?","\nNo. pandas, Polars and DuckDB all read Parquet from a local folder. It is a file format, not a platform.",[143,1849,1851],{"id":1850},"related","Related",[1730,1853,1854,1862,1869,1874,1881],{},[1733,1855,1856,1739,1859,1861],{},[1736,1857,1858],{},"Up:",[14,1860,17],{"href":16}," — the toolkit this conversion completes.",[1733,1863,1864,1868],{},[14,1865,1867],{"href":1866},"\u002Fadvanced-data-transformation-and-cleaning\u002Freading-excel-with-polars-and-arrow\u002Fspeed-up-pandas-excel-reads-with-the-calamine-engine\u002F","Speed up pandas Excel reads with the calamine engine"," — making the one-off conversion itself faster.",[1733,1870,1871,1873],{},[14,1872,1401],{"href":1400}," — turning the cache back into something a person can read.",[1733,1875,1876,1880],{},[14,1877,1879],{"href":1878},"\u002Fadvanced-data-transformation-and-cleaning\u002Fworking-with-large-excel-files-in-python\u002Fconvert-excel-to-csv-with-python\u002F","Convert Excel to CSV with Python"," — the simpler, lossier conversion, and when it is enough.",[1733,1882,1883,1887],{},[14,1884,1886],{"href":1885},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fload-an-excel-file-into-a-sql-database-with-pandas\u002F","Load an Excel file into a SQL database with pandas"," — the other durable destination for spreadsheet data.",[1889,1890,1891],"style",{},"html pre.shiki code .sMTad, html code.shiki .sMTad{--shiki-default:#6F42C1;--shiki-dark:#FFB757}html pre.shiki code .srMev, html code.shiki .srMev{--shiki-default:#032F62;--shiki-dark:#ADDCFF}html pre.shiki code .s-wDw, html code.shiki .s-wDw{--shiki-default:#6A737D;--shiki-dark:#BDC4CC}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .s-kum, html code.shiki .s-kum{--shiki-default:#D73A49;--shiki-dark:#FF9492}html pre.shiki code .skGVy, html code.shiki .skGVy{--shiki-default:#24292E;--shiki-dark:#F0F3F6}html pre.shiki code .sa561, html code.shiki .sa561{--shiki-default:#E36209;--shiki-dark:#FFB757}html pre.shiki code .sP0c6, html code.shiki .sP0c6{--shiki-default:#005CC5;--shiki-dark:#91CBFF}html pre.shiki code .sSjpA, html code.shiki .sSjpA{--shiki-default:#005CC5;--shiki-dark:#FF9492}",{"title":156,"searchDepth":187,"depth":187,"links":1893},[1894,1895,1896,1897,1898,1899,1900,1901,1902,1903,1904,1905,1906],{"id":145,"depth":187,"text":146},{"id":206,"depth":187,"text":207},{"id":364,"depth":187,"text":365},{"id":583,"depth":187,"text":584},{"id":974,"depth":187,"text":975},{"id":1188,"depth":187,"text":1189},{"id":1266,"depth":187,"text":1267},{"id":1404,"depth":187,"text":1405},{"id":1727,"depth":187,"text":1728},{"id":1776,"depth":187,"text":1777},{"id":1791,"depth":187,"text":1792},{"id":1801,"depth":187,"text":1802},{"id":1850,"depth":187,"text":1851},"2026-08-27","Turn slow, repeated Excel reads into fast columnar ones: convert workbooks to Parquet with pandas or Polars, keep types, partition by period, and refresh incrementally.","md",[1911,1913,1915,1917,1919],{"q":1807,"a":1912},"You can, but you pay the parse every run. Parquet stores columns in a typed binary layout, so a later read costs a fraction of the time, uses less memory, and needs no type inference because the schema travels with the file.",{"q":1813,"a":1914},"Yes, that is the main reason to use it. Strings stay strings, dates stay dates, and a column of identifiers with leading zeros keeps them — provided you set the types correctly during the conversion.",{"q":1819,"a":1916},"Modern Excel can import Parquet through Power Query on some plans, but treat Parquet as the machine-readable cache and keep a generated xlsx for people. Convert back with a single write_excel or to_excel call when a human needs it.",{"q":1837,"a":1918},"Typically several times smaller than the equivalent xlsx, because columnar storage compresses repeated values extremely well. A folder of monthly workbooks usually shrinks dramatically once converted.",{"q":1846,"a":1920},"No. pandas, Polars and DuckDB all read Parquet from a local folder. It is a file format, not a platform.",{},"\u002Fadvanced-data-transformation-and-cleaning\u002Freading-excel-with-polars-and-arrow\u002Fconvert-excel-files-to-parquet-with-python",{"title":1924,"description":1925},"Convert Excel to Parquet in Python","Convert xlsx to Parquet with pandas or Polars: preserve dtypes, choose compression, partition a folder of monthly workbooks, refresh only changed files, and query lazily.","convert-excel-files-to-parquet-with-python","advanced-data-transformation-and-cleaning\u002Freading-excel-with-polars-and-arrow\u002Fconvert-excel-files-to-parquet-with-python\u002Findex","how-to","MvoseYdXfTJeb2zP2m1TjMxxNTgITS8F7w8mHK6tWwY",[1931,1934],{"title":17,"path":1932,"stem":1933,"children":-1},"\u002Fadvanced-data-transformation-and-cleaning\u002Freading-excel-with-polars-and-arrow","advanced-data-transformation-and-cleaning\u002Freading-excel-with-polars-and-arrow\u002Findex",{"title":1935,"path":1936,"stem":1937,"children":-1},"Read an Excel File with polars.read_excel","\u002Fadvanced-data-transformation-and-cleaning\u002Freading-excel-with-polars-and-arrow\u002Fread-an-excel-file-with-polars-read-excel","advanced-data-transformation-and-cleaning\u002Freading-excel-with-polars-and-arrow\u002Fread-an-excel-file-with-polars-read-excel\u002Findex",1788710154260]