[{"data":1,"prerenderedAt":1899},["ShallowReactive",2],{"doc:\u002Fgetting-started-with-python-excel-automation\u002Fchoosing-a-python-excel-library\u002Fexcel-vs-csv-vs-parquet-for-python-data-pipelines":3,"surround:\u002Fgetting-started-with-python-excel-automation\u002Fchoosing-a-python-excel-library\u002Fexcel-vs-csv-vs-parquet-for-python-data-pipelines":1891},{"id":4,"title":5,"body":6,"dateModified":1862,"datePublished":1862,"description":1863,"extension":1864,"faq":1865,"meta":1876,"navigation":231,"path":1884,"seo":1885,"slug":1887,"stem":1888,"type":1889,"__hash__":1890},"docs\u002Fgetting-started-with-python-excel-automation\u002Fchoosing-a-python-excel-library\u002Fexcel-vs-csv-vs-parquet-for-python-data-pipelines\u002Findex.md","Excel vs CSV vs Parquet for Python Data Pipelines",{"type":7,"value":8,"toc":1848},"minimark",[9,19,144,149,180,190,194,197,203,420,423,427,509,515,737,741,750,828,968,976,980,983,986,1090,1105,1109,1112,1222,1225,1508,1516,1520,1526,1534,1538,1652,1656,1664,1667,1760,1763,1767,1773,1777,1784,1790,1796,1802,1808,1812,1844],[10,11,12,13,18],"p",{},"Excel is a superb delivery format and a poor interchange format. It is the only one of the three\nthat a person can open, sort and annotate, and it is the slowest to parse, the largest on disk and\nthe easiest to corrupt with a stray manual edit. The productive question is not which format is\nbest but where each belongs in a pipeline. This guide, part of\n",[14,15,17],"a",{"href":16},"\u002Fgetting-started-with-python-excel-automation\u002Fchoosing-a-python-excel-library\u002F","Choosing a Python Excel Library",",\ndraws that line.",[20,21,29,30,29,34,29,38,29,45,29,55,29,62,29,69,29,74,29,78,29,82,29,86,29,91,29,96,29,100,29,103,29,106,29,109,29,112,29,117,29,122,29,126,29,129,29,132,29,135,29,138],"svg",{"viewBox":22,"role":23,"ariaLabelledBy":24,"xmlns":27,"style":28},"0 0 760 232","img",[25,26],"fmt-three-t","fmt-three-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;max-width:760px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif","\n  ",[31,32,33],"title",{"id":25},"What each format keeps and what it throws away",[35,36,37],"desc",{"id":26},"Excel keeps sheets, formats and formulas but is slow and large. CSV keeps only text, losing every type. Parquet keeps declared column types and compresses hard, but no person opens one by hand.",[39,40],"rect",{"x":41,"y":41,"width":42,"height":43,"fill":44},"0","760","232","#ffffff",[39,46],{"x":47,"y":48,"width":49,"height":50,"rx":51,"fill":52,"stroke":53,"style":54},"20.0","26","226.7","164","14","#fdefd8","var(--gold,#b4740a)","stroke-width:2px",[56,57,61],"text",{"x":58,"y":59,"style":60},"133.35","52","font-size:13px;font-weight:700;fill:var(--gold-ink,#7a4e06);text-anchor:middle",".xlsx",[63,64],"line",{"x1":65,"y1":66,"x2":67,"y2":66,"stroke":53,"style":68},"36.0","62","230.7","stroke-width:1px",[56,70,73],{"x":58,"y":71,"style":72},"84","font-size:11.5px;font-weight:400;fill:var(--text,#172033);text-anchor:middle","sheets and formulas",[56,75,77],{"x":58,"y":76,"style":72},"107","styles and charts",[56,79,81],{"x":58,"y":80,"style":72},"130","slowest to parse",[56,83,85],{"x":58,"y":84,"style":72},"153","a person can open it",[39,87],{"x":88,"y":48,"width":49,"height":50,"rx":51,"fill":89,"stroke":90,"style":54},"266.7","#e7ebef","var(--line,#cdd5e6)",[56,92,95],{"x":93,"y":59,"style":94},"380.04999999999995","font-size:13px;font-weight:700;fill:var(--muted,#5b6780);text-anchor:middle",".csv",[63,97],{"x1":98,"y1":66,"x2":99,"y2":66,"stroke":90,"style":68},"282.7","477.4",[56,101,102],{"x":93,"y":71,"style":72},"text and commas",[56,104,105],{"x":93,"y":76,"style":72},"no types at all",[56,107,108],{"x":93,"y":80,"style":72},"universal support",[56,110,111],{"x":93,"y":84,"style":72},"diffable in git",[39,113],{"x":114,"y":48,"width":49,"height":50,"rx":51,"fill":115,"stroke":116,"style":54},"513.4","#d9f4f1","var(--teal,#0f9488)",[56,118,121],{"x":119,"y":59,"style":120},"626.75","font-size:13px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle",".parquet",[63,123],{"x1":124,"y1":66,"x2":125,"y2":66,"stroke":116,"style":68},"529.4","724.0999999999999",[56,127,128],{"x":119,"y":71,"style":72},"typed columns",[56,130,131],{"x":119,"y":76,"style":72},"5-20x smaller",[56,133,134],{"x":119,"y":80,"style":72},"fastest to read",[56,136,137],{"x":119,"y":84,"style":72},"machines only",[56,139,143],{"x":140,"y":141,"style":142},"380.0","218","font-size:12.5px;font-weight:400;fill:var(--muted,#5b6780);text-anchor:middle","each is good at one job and poor at the others",[145,146,148],"h2",{"id":147},"prerequisites","Prerequisites",[150,151,156],"pre",{"className":152,"code":153,"language":154,"meta":155,"style":155},"language-bash shiki shiki-themes github-light github-dark-high-contrast","pip install pandas pyarrow openpyxl\n","bash","",[157,158,159],"code",{"__ignoreMap":155},[160,161,163,167,171,174,177],"span",{"class":63,"line":162},1,[160,164,166],{"class":165},"sMTad","pip",[160,168,170],{"class":169},"srMev"," install",[160,172,173],{"class":169}," pandas",[160,175,176],{"class":169}," pyarrow",[160,178,179],{"class":169}," openpyxl\n",[10,181,182,185,186,189],{},[157,183,184],{},"pyarrow"," is what gives pandas its Parquet support; without it ",[157,187,188],{},"to_parquet"," raises an import error\nnaming the missing engine.",[145,191,193],{"id":192},"what-each-format-actually-stores","What each format actually stores",[10,195,196],{},"The differences that matter are about types and structure, not size.",[10,198,199,200,202],{},"An ",[157,201,61],{}," file is a zip of XML that stores a value, a type flag and a style index per cell, plus\nsheets, formulas, charts, merged ranges and everything else Excel understands. A CSV stores text\nand commas — no types, no sheets, no formatting, and an encoding you have to guess if nobody wrote\nit down. A Parquet file stores columns, each with a declared type, compressed and accompanied by\nstatistics that let a reader skip whole blocks it does not need.",[150,204,208],{"className":205,"code":206,"language":207,"meta":155,"style":155},"language-python shiki shiki-themes github-light github-dark-high-contrast","import pandas as pd\n\nframe = pd.DataFrame({\n    \"Account\": [\"00123\", \"00456\"],      # leading zeros matter\n    \"Ordered\": pd.to_datetime([\"2026-01-04\", \"2026-02-11\"]),\n    \"Revenue\": [12400.5, 9800.25],\n})\n\nframe.to_excel(\"data.xlsx\", index=False)\nframe.to_csv(\"data.csv\", index=False)\nframe.to_parquet(\"data.parquet\", index=False)\n\nprint(pd.read_csv(\"data.csv\").dtypes)          # Account -> int64, zeros gone\nprint(pd.read_parquet(\"data.parquet\").dtypes)  # types preserved exactly\n","python",[157,209,210,226,233,245,270,290,310,316,321,344,363,382,387,404],{"__ignoreMap":155},[160,211,212,216,220,223],{"class":63,"line":162},[160,213,215],{"class":214},"s-kum","import",[160,217,219],{"class":218},"skGVy"," pandas ",[160,221,222],{"class":214},"as",[160,224,225],{"class":218}," pd\n",[160,227,229],{"class":63,"line":228},2,[160,230,232],{"emptyLinePlaceholder":231},true,"\n",[160,234,236,239,242],{"class":63,"line":235},3,[160,237,238],{"class":218},"frame ",[160,240,241],{"class":214},"=",[160,243,244],{"class":218}," pd.DataFrame({\n",[160,246,248,251,254,257,260,263,266],{"class":63,"line":247},4,[160,249,250],{"class":169},"    \"Account\"",[160,252,253],{"class":218},": [",[160,255,256],{"class":169},"\"00123\"",[160,258,259],{"class":218},", ",[160,261,262],{"class":169},"\"00456\"",[160,264,265],{"class":218},"],      ",[160,267,269],{"class":268},"s-wDw","# leading zeros matter\n",[160,271,273,276,279,282,284,287],{"class":63,"line":272},5,[160,274,275],{"class":169},"    \"Ordered\"",[160,277,278],{"class":218},": pd.to_datetime([",[160,280,281],{"class":169},"\"2026-01-04\"",[160,283,259],{"class":218},[160,285,286],{"class":169},"\"2026-02-11\"",[160,288,289],{"class":218},"]),\n",[160,291,293,296,298,302,304,307],{"class":63,"line":292},6,[160,294,295],{"class":169},"    \"Revenue\"",[160,297,253],{"class":218},[160,299,301],{"class":300},"sP0c6","12400.5",[160,303,259],{"class":218},[160,305,306],{"class":300},"9800.25",[160,308,309],{"class":218},"],\n",[160,311,313],{"class":63,"line":312},7,[160,314,315],{"class":218},"})\n",[160,317,319],{"class":63,"line":318},8,[160,320,232],{"emptyLinePlaceholder":231},[160,322,324,327,330,332,336,338,341],{"class":63,"line":323},9,[160,325,326],{"class":218},"frame.to_excel(",[160,328,329],{"class":169},"\"data.xlsx\"",[160,331,259],{"class":218},[160,333,335],{"class":334},"sa561","index",[160,337,241],{"class":214},[160,339,340],{"class":300},"False",[160,342,343],{"class":218},")\n",[160,345,347,350,353,355,357,359,361],{"class":63,"line":346},10,[160,348,349],{"class":218},"frame.to_csv(",[160,351,352],{"class":169},"\"data.csv\"",[160,354,259],{"class":218},[160,356,335],{"class":334},[160,358,241],{"class":214},[160,360,340],{"class":300},[160,362,343],{"class":218},[160,364,366,369,372,374,376,378,380],{"class":63,"line":365},11,[160,367,368],{"class":218},"frame.to_parquet(",[160,370,371],{"class":169},"\"data.parquet\"",[160,373,259],{"class":218},[160,375,335],{"class":334},[160,377,241],{"class":214},[160,379,340],{"class":300},[160,381,343],{"class":218},[160,383,385],{"class":63,"line":384},12,[160,386,232],{"emptyLinePlaceholder":231},[160,388,390,393,396,398,401],{"class":63,"line":389},13,[160,391,392],{"class":300},"print",[160,394,395],{"class":218},"(pd.read_csv(",[160,397,352],{"class":169},[160,399,400],{"class":218},").dtypes)          ",[160,402,403],{"class":268},"# Account -> int64, zeros gone\n",[160,405,407,409,412,414,417],{"class":63,"line":406},14,[160,408,392],{"class":300},[160,410,411],{"class":218},"(pd.read_parquet(",[160,413,371],{"class":169},[160,415,416],{"class":218},").dtypes)  ",[160,418,419],{"class":268},"# types preserved exactly\n",[10,421,422],{},"That round trip is the whole argument in five lines. CSV silently destroys the account code and\nturns the date into a string; Parquet returns exactly what it was given; Excel keeps the value but\ndepends on how the cell was formatted when it was written.",[145,424,426],{"id":425},"speed-and-size-on-the-same-data","Speed and size on the same data",[20,428,29,434,29,437,29,440,29,444,29,450,29,456,29,463,29,469,29,473,29,475,29,479,29,484,29,488,29,491,29,495,29,500,29,504],{"viewBox":429,"role":23,"ariaLabelledBy":430,"xmlns":27,"style":433},"0 0 720 196",[431,432],"fmt-cost-t","fmt-cost-d","width:100%;max-width:720px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif",[31,435,436],{"id":431},"Read time and file size for the same 200,000-row table",[35,438,439],{"id":432},"Excel is the slowest to read and the largest on disk, CSV sits in the middle on both, and Parquet is the fastest and smallest by a wide margin.",[39,441],{"x":41,"y":41,"width":442,"height":443,"fill":44},"720","196",[56,445,449],{"x":446,"y":447,"style":448},"20","56","font-size:12px;font-weight:600;fill:var(--text,#172033);text-anchor:start",".xlsx read",[39,451],{"x":452,"y":453,"width":454,"height":48,"rx":455,"fill":89,"stroke":90},"200","40","364.7","6",[39,457],{"x":458,"y":459,"width":460,"height":461,"rx":462,"fill":52,"stroke":53},"201","41","362.7","24","5",[56,464,468],{"x":465,"y":466,"style":467},"576.7","58","font-size:12px;font-weight:700;fill:var(--gold-ink,#7a4e06);text-anchor:start","largest, slowest",[56,470,472],{"x":446,"y":471,"style":448},"100",".csv read",[39,474],{"x":452,"y":71,"width":454,"height":48,"rx":455,"fill":89,"stroke":90},[39,476],{"x":458,"y":477,"width":478,"height":461,"rx":462,"fill":89,"stroke":90},"85","109.7",[56,480,483],{"x":465,"y":481,"style":482},"102","font-size:12px;font-weight:700;fill:var(--muted,#5b6780);text-anchor:start","middle on both",[56,485,487],{"x":446,"y":486,"style":448},"144",".parquet read",[39,489],{"x":452,"y":490,"width":454,"height":48,"rx":455,"fill":89,"stroke":90},"128",[39,492],{"x":458,"y":493,"width":494,"height":461,"rx":462,"fill":115,"stroke":116},"129","28",[56,496,499],{"x":465,"y":497,"style":498},"146","font-size:12px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:start","smallest, fastest",[56,501,503],{"x":446,"y":446,"style":502},"font-size:11.5px;font-weight:600;fill:var(--muted,#5b6780);text-anchor:start","relative cost",[56,505,508],{"x":506,"y":507,"style":142},"360.0","186","ratios move with the data; the ordering does not",[10,510,511,512,514],{},"The ratios move with the data, but the ordering is consistent: Parquet is the fastest to read and\nthe smallest on disk, CSV is fast to write and bulky, and ",[157,513,61],{}," is the slowest and largest of the\nthree because every cell carries structure a plain table does not need.",[150,516,518],{"className":205,"code":517,"language":207,"meta":155,"style":155},"import os\nimport time\nimport pandas as pd\n\nframe = pd.read_excel(\"bench.xlsx\")\nfor label, writer, reader, path in [\n    (\"xlsx\", frame.to_excel, pd.read_excel, \"out.xlsx\"),\n    (\"csv\", frame.to_csv, pd.read_csv, \"out.csv\"),\n    (\"parquet\", frame.to_parquet, pd.read_parquet, \"out.parquet\"),\n]:\n    writer(path, index=False)\n    start = time.perf_counter()\n    reader(path)\n    size = os.path.getsize(path) \u002F 1_048_576\n    print(f\"{label:>8}: read {time.perf_counter() - start:5.2f}s   {size:6.1f} MB\")\n",[157,519,520,527,534,544,548,562,576,593,608,623,628,641,651,656,672],{"__ignoreMap":155},[160,521,522,524],{"class":63,"line":162},[160,523,215],{"class":214},[160,525,526],{"class":218}," os\n",[160,528,529,531],{"class":63,"line":228},[160,530,215],{"class":214},[160,532,533],{"class":218}," time\n",[160,535,536,538,540,542],{"class":63,"line":235},[160,537,215],{"class":214},[160,539,219],{"class":218},[160,541,222],{"class":214},[160,543,225],{"class":218},[160,545,546],{"class":63,"line":247},[160,547,232],{"emptyLinePlaceholder":231},[160,549,550,552,554,557,560],{"class":63,"line":272},[160,551,238],{"class":218},[160,553,241],{"class":214},[160,555,556],{"class":218}," pd.read_excel(",[160,558,559],{"class":169},"\"bench.xlsx\"",[160,561,343],{"class":218},[160,563,564,567,570,573],{"class":63,"line":292},[160,565,566],{"class":214},"for",[160,568,569],{"class":218}," label, writer, reader, path ",[160,571,572],{"class":214},"in",[160,574,575],{"class":218}," [\n",[160,577,578,581,584,587,590],{"class":63,"line":312},[160,579,580],{"class":218},"    (",[160,582,583],{"class":169},"\"xlsx\"",[160,585,586],{"class":218},", frame.to_excel, pd.read_excel, ",[160,588,589],{"class":169},"\"out.xlsx\"",[160,591,592],{"class":218},"),\n",[160,594,595,597,600,603,606],{"class":63,"line":318},[160,596,580],{"class":218},[160,598,599],{"class":169},"\"csv\"",[160,601,602],{"class":218},", frame.to_csv, pd.read_csv, ",[160,604,605],{"class":169},"\"out.csv\"",[160,607,592],{"class":218},[160,609,610,612,615,618,621],{"class":63,"line":323},[160,611,580],{"class":218},[160,613,614],{"class":169},"\"parquet\"",[160,616,617],{"class":218},", frame.to_parquet, pd.read_parquet, ",[160,619,620],{"class":169},"\"out.parquet\"",[160,622,592],{"class":218},[160,624,625],{"class":63,"line":346},[160,626,627],{"class":218},"]:\n",[160,629,630,633,635,637,639],{"class":63,"line":365},[160,631,632],{"class":218},"    writer(path, ",[160,634,335],{"class":334},[160,636,241],{"class":214},[160,638,340],{"class":300},[160,640,343],{"class":218},[160,642,643,646,648],{"class":63,"line":384},[160,644,645],{"class":218},"    start ",[160,647,241],{"class":214},[160,649,650],{"class":218}," time.perf_counter()\n",[160,652,653],{"class":63,"line":389},[160,654,655],{"class":218},"    reader(path)\n",[160,657,658,661,663,666,669],{"class":63,"line":406},[160,659,660],{"class":218},"    size ",[160,662,241],{"class":214},[160,664,665],{"class":218}," os.path.getsize(path) ",[160,667,668],{"class":214},"\u002F",[160,670,671],{"class":300}," 1_048_576\n",[160,673,675,678,681,684,687,691,694,697,700,703,705,708,711,714,717,719,722,724,727,730,732,735],{"class":63,"line":674},15,[160,676,677],{"class":300},"    print",[160,679,680],{"class":218},"(",[160,682,683],{"class":214},"f",[160,685,686],{"class":169},"\"",[160,688,690],{"class":689},"sSjpA","{",[160,692,693],{"class":218},"label",[160,695,696],{"class":214},":>8",[160,698,699],{"class":689},"}",[160,701,702],{"class":169},": read ",[160,704,690],{"class":689},[160,706,707],{"class":218},"time.perf_counter() ",[160,709,710],{"class":214},"-",[160,712,713],{"class":218}," start",[160,715,716],{"class":214},":5.2f",[160,718,699],{"class":689},[160,720,721],{"class":169},"s   ",[160,723,690],{"class":689},[160,725,726],{"class":218},"size",[160,728,729],{"class":214},":6.1f",[160,731,699],{"class":689},[160,733,734],{"class":169}," MB\"",[160,736,343],{"class":218},[145,738,740],{"id":739},"where-each-one-belongs","Where each one belongs",[10,742,743,744,746,747,749],{},"The pattern that survives contact with production is a three-stage split. Source data arrives in\nwhatever the upstream system produces — often ",[157,745,61],{},", because a person exported it. The pipeline\nconverts it once to Parquet and does all subsequent work against that. The final human-facing\nartefact is generated as ",[157,748,61],{}," at the end, formatted for reading rather than for parsing.",[20,751,29,755,29,758,29,761,29,763,29,768,29,775,29,781,29,786,29,792,29,797,29,800,29,804,29,807,29,811,29,814,29,817,29,821,29,824],{"viewBox":22,"role":23,"ariaLabelledBy":752,"xmlns":27,"style":28},[753,754],"fmt-stage-t","fmt-stage-d",[31,756,757],{"id":753},"Where each format belongs in a reporting pipeline",[35,759,760],{"id":754},"A workbook arrives from an upstream system, is converted once to Parquet for all computation, and a formatted Excel file is generated at the end for the person who reads it.",[39,762],{"x":41,"y":41,"width":42,"height":43,"fill":44},[56,764,767],{"x":140,"y":765,"style":766},"32","font-size:13px;font-weight:600;fill:var(--muted,#5b6780);text-anchor:middle","convert once, deliver last",[39,769],{"x":770,"y":771,"width":772,"height":773,"rx":774,"fill":52,"stroke":53,"style":54},"24.0","74","208.0","96","12",[56,776,780],{"x":777,"y":778,"style":779},"128.0","114","font-size:14px;font-weight:700;fill:var(--gold-ink,#7a4e06);text-anchor:middle","source .xlsx",[56,782,785],{"x":777,"y":783,"style":784},"136","font-size:11.5px;font-weight:400;fill:var(--muted,#5b6780);text-anchor:middle","someone exported it",[63,787],{"x1":788,"y1":789,"x2":790,"y2":789,"stroke":791,"style":54},"237.0","122.0","269.0","var(--brand,#5b5cf0)",[793,794],"polygon",{"points":795,"fill":796},"269.0,122.0 260.0,117.0 260.0,127.0","#5b5cf0",[39,798],{"x":799,"y":771,"width":772,"height":773,"rx":774,"fill":115,"stroke":116,"style":54},"276.0",[56,801,803],{"x":140,"y":778,"style":802},"font-size:14px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","Parquet cache",[56,805,806],{"x":140,"y":783,"style":784},"every step runs here",[63,808],{"x1":809,"y1":789,"x2":810,"y2":789,"stroke":791,"style":54},"489.0","521.0",[793,812],{"points":813,"fill":796},"521.0,122.0 512.0,117.0 512.0,127.0",[39,815],{"x":816,"y":771,"width":772,"height":773,"rx":774,"fill":52,"stroke":53,"style":54},"528.0",[56,818,820],{"x":819,"y":778,"style":779},"632.0","delivered .xlsx",[56,822,823],{"x":819,"y":783,"style":784},"formatted for a reader",[56,825,827],{"x":140,"y":826,"style":142},"210","compute in the middle format; deliver in the one people open",[150,829,831],{"className":205,"code":830,"language":207,"meta":155,"style":155},"import pandas as pd\nfrom pathlib import Path\n\ndef cached_frame(source: Path) -> pd.DataFrame:\n    cache = source.with_suffix(\".parquet\")\n    if not cache.exists() or cache.stat().st_mtime \u003C source.stat().st_mtime:\n        pd.read_excel(source, engine=\"calamine\").to_parquet(cache, index=False)\n    return pd.read_parquet(cache)\n\nframe = cached_frame(Path(\"monthly-export.xlsx\"))\nprint(frame.shape)\n",[157,832,833,843,856,860,872,887,910,934,942,946,961],{"__ignoreMap":155},[160,834,835,837,839,841],{"class":63,"line":162},[160,836,215],{"class":214},[160,838,219],{"class":218},[160,840,222],{"class":214},[160,842,225],{"class":218},[160,844,845,848,851,853],{"class":63,"line":228},[160,846,847],{"class":214},"from",[160,849,850],{"class":218}," pathlib ",[160,852,215],{"class":214},[160,854,855],{"class":218}," Path\n",[160,857,858],{"class":63,"line":235},[160,859,232],{"emptyLinePlaceholder":231},[160,861,862,865,869],{"class":63,"line":247},[160,863,864],{"class":214},"def",[160,866,868],{"class":867},"s_Opv"," cached_frame",[160,870,871],{"class":218},"(source: Path) -> pd.DataFrame:\n",[160,873,874,877,879,882,885],{"class":63,"line":272},[160,875,876],{"class":218},"    cache ",[160,878,241],{"class":214},[160,880,881],{"class":218}," source.with_suffix(",[160,883,884],{"class":169},"\".parquet\"",[160,886,343],{"class":218},[160,888,889,892,895,898,901,904,907],{"class":63,"line":292},[160,890,891],{"class":214},"    if",[160,893,894],{"class":214}," not",[160,896,897],{"class":218}," cache.exists() ",[160,899,900],{"class":214},"or",[160,902,903],{"class":218}," cache.stat().st_mtime ",[160,905,906],{"class":214},"\u003C",[160,908,909],{"class":218}," source.stat().st_mtime:\n",[160,911,912,915,918,920,923,926,928,930,932],{"class":63,"line":312},[160,913,914],{"class":218},"        pd.read_excel(source, ",[160,916,917],{"class":334},"engine",[160,919,241],{"class":214},[160,921,922],{"class":169},"\"calamine\"",[160,924,925],{"class":218},").to_parquet(cache, ",[160,927,335],{"class":334},[160,929,241],{"class":214},[160,931,340],{"class":300},[160,933,343],{"class":218},[160,935,936,939],{"class":63,"line":318},[160,937,938],{"class":214},"    return",[160,940,941],{"class":218}," pd.read_parquet(cache)\n",[160,943,944],{"class":63,"line":323},[160,945,232],{"emptyLinePlaceholder":231},[160,947,948,950,952,955,958],{"class":63,"line":346},[160,949,238],{"class":218},[160,951,241],{"class":214},[160,953,954],{"class":218}," cached_frame(Path(",[160,956,957],{"class":169},"\"monthly-export.xlsx\"",[160,959,960],{"class":218},"))\n",[160,962,963,965],{"class":63,"line":365},[160,964,392],{"class":300},[160,966,967],{"class":218},"(frame.shape)\n",[10,969,970,971,975],{},"The modification-time check is what makes this safe to leave in a scheduled job: a refreshed source\ninvalidates the cache automatically, and an unchanged one is never parsed twice.\n",[14,972,974],{"href":973},"\u002Fadvanced-data-transformation-and-cleaning\u002Freading-excel-with-polars-and-arrow\u002Fconvert-excel-files-to-parquet-with-python\u002F","Convert Excel Files to Parquet with Python","\ncovers the conversion in more depth, including partitioned output.",[145,977,979],{"id":978},"when-csv-is-still-the-right-answer","When CSV is still the right answer",[10,981,982],{},"CSV keeps winning for reasons that have nothing to do with its technical merits. Every system reads\nit, including ones written before Parquet existed; it streams line by line, so a 40 GB file can be\nprocessed without a columnar reader; and it is diffable, which makes it the only one of the three\nthat behaves sensibly in version control.",[10,984,985],{},"If you do use it as an interchange format, remove the guesswork on the way back in:",[150,987,989],{"className":205,"code":988,"language":207,"meta":155,"style":155},"import pandas as pd\n\nframe = pd.read_csv(\n    \"export.csv\",\n    dtype={\"Account\": \"string\", \"Postcode\": \"string\"},   # keep leading zeros\n    parse_dates=[\"Ordered\"],\n    encoding=\"utf-8-sig\",                                # strip the BOM Excel writes\n)\n",[157,990,991,1001,1005,1014,1022,1055,1070,1086],{"__ignoreMap":155},[160,992,993,995,997,999],{"class":63,"line":162},[160,994,215],{"class":214},[160,996,219],{"class":218},[160,998,222],{"class":214},[160,1000,225],{"class":218},[160,1002,1003],{"class":63,"line":228},[160,1004,232],{"emptyLinePlaceholder":231},[160,1006,1007,1009,1011],{"class":63,"line":235},[160,1008,238],{"class":218},[160,1010,241],{"class":214},[160,1012,1013],{"class":218}," pd.read_csv(\n",[160,1015,1016,1019],{"class":63,"line":247},[160,1017,1018],{"class":169},"    \"export.csv\"",[160,1020,1021],{"class":218},",\n",[160,1023,1024,1027,1029,1031,1034,1037,1040,1042,1045,1047,1049,1052],{"class":63,"line":272},[160,1025,1026],{"class":334},"    dtype",[160,1028,241],{"class":214},[160,1030,690],{"class":218},[160,1032,1033],{"class":169},"\"Account\"",[160,1035,1036],{"class":218},": ",[160,1038,1039],{"class":169},"\"string\"",[160,1041,259],{"class":218},[160,1043,1044],{"class":169},"\"Postcode\"",[160,1046,1036],{"class":218},[160,1048,1039],{"class":169},[160,1050,1051],{"class":218},"},   ",[160,1053,1054],{"class":268},"# keep leading zeros\n",[160,1056,1057,1060,1062,1065,1068],{"class":63,"line":292},[160,1058,1059],{"class":334},"    parse_dates",[160,1061,241],{"class":214},[160,1063,1064],{"class":218},"[",[160,1066,1067],{"class":169},"\"Ordered\"",[160,1069,309],{"class":218},[160,1071,1072,1075,1077,1080,1083],{"class":63,"line":312},[160,1073,1074],{"class":334},"    encoding",[160,1076,241],{"class":214},[160,1078,1079],{"class":169},"\"utf-8-sig\"",[160,1081,1082],{"class":218},",                                ",[160,1084,1085],{"class":268},"# strip the BOM Excel writes\n",[160,1087,1088],{"class":63,"line":318},[160,1089,343],{"class":218},[10,1091,1092,1095,1096,1099,1100,1104],{},[157,1093,1094],{},"utf-8-sig"," deserves the callout: Excel writes a byte-order mark when it saves a CSV as UTF-8, and\nreading it as plain ",[157,1097,1098],{},"utf-8"," leaves an invisible prefix on the first column name, which then fails\nevery subsequent lookup by name. ",[14,1101,1103],{"href":1102},"\u002Fadvanced-data-transformation-and-cleaning\u002Fworking-with-large-excel-files-in-python\u002Fconvert-excel-to-csv-with-python\u002F","Convert Excel to CSV with Python","\ncovers the rest of the encoding and quoting decisions.",[145,1106,1108],{"id":1107},"handing-a-schema-across-the-boundary","Handing a schema across the boundary",[10,1110,1111],{},"The reason a Parquet handoff is safe and a CSV handoff is not comes down to one thing: Parquet\ncarries its schema and CSV does not. That difference shows up on the second run rather than the\nfirst, when an upstream column changes type and every downstream step quietly reinterprets it.",[150,1113,1115],{"className":205,"code":1114,"language":207,"meta":155,"style":155},"import pandas as pd\nimport pyarrow.parquet as pq\n\npq.write_table(\n    __import__(\"pyarrow\").Table.from_pandas(frame, preserve_index=False),\n    \"orders.parquet\",\n    compression=\"zstd\",\n)\n\nschema = pq.read_schema(\"orders.parquet\")\nprint(schema)                                  # types, exactly as written\n",[157,1116,1117,1127,1139,1143,1148,1170,1177,1189,1193,1197,1212],{"__ignoreMap":155},[160,1118,1119,1121,1123,1125],{"class":63,"line":162},[160,1120,215],{"class":214},[160,1122,219],{"class":218},[160,1124,222],{"class":214},[160,1126,225],{"class":218},[160,1128,1129,1131,1134,1136],{"class":63,"line":228},[160,1130,215],{"class":214},[160,1132,1133],{"class":218}," pyarrow.parquet ",[160,1135,222],{"class":214},[160,1137,1138],{"class":218}," pq\n",[160,1140,1141],{"class":63,"line":235},[160,1142,232],{"emptyLinePlaceholder":231},[160,1144,1145],{"class":63,"line":247},[160,1146,1147],{"class":218},"pq.write_table(\n",[160,1149,1150,1153,1155,1158,1161,1164,1166,1168],{"class":63,"line":272},[160,1151,1152],{"class":300},"    __import__",[160,1154,680],{"class":218},[160,1156,1157],{"class":169},"\"pyarrow\"",[160,1159,1160],{"class":218},").Table.from_pandas(frame, ",[160,1162,1163],{"class":334},"preserve_index",[160,1165,241],{"class":214},[160,1167,340],{"class":300},[160,1169,592],{"class":218},[160,1171,1172,1175],{"class":63,"line":292},[160,1173,1174],{"class":169},"    \"orders.parquet\"",[160,1176,1021],{"class":218},[160,1178,1179,1182,1184,1187],{"class":63,"line":312},[160,1180,1181],{"class":334},"    compression",[160,1183,241],{"class":214},[160,1185,1186],{"class":169},"\"zstd\"",[160,1188,1021],{"class":218},[160,1190,1191],{"class":63,"line":318},[160,1192,343],{"class":218},[160,1194,1195],{"class":63,"line":323},[160,1196,232],{"emptyLinePlaceholder":231},[160,1198,1199,1202,1204,1207,1210],{"class":63,"line":346},[160,1200,1201],{"class":218},"schema ",[160,1203,241],{"class":214},[160,1205,1206],{"class":218}," pq.read_schema(",[160,1208,1209],{"class":169},"\"orders.parquet\"",[160,1211,343],{"class":218},[160,1213,1214,1216,1219],{"class":63,"line":365},[160,1215,392],{"class":300},[160,1217,1218],{"class":218},"(schema)                                  ",[160,1220,1221],{"class":268},"# types, exactly as written\n",[10,1223,1224],{},"If the pipeline has to stay on CSV, write the schema next to it and assert against it on the way\nback in. Ten lines of contract prevents the class of bug where a report is wrong but nothing\nerrored:",[150,1226,1228],{"className":205,"code":1227,"language":207,"meta":155,"style":155},"import json\nimport pandas as pd\n\nEXPECTED = {\"Account\": \"string\", \"Ordered\": \"datetime64[ns]\", \"Revenue\": \"float64\"}\njson.dump(EXPECTED, open(\"orders.schema.json\", \"w\"))\n\nframe = pd.read_csv(\"orders.csv\", dtype={\"Account\": \"string\"}, parse_dates=[\"Ordered\"])\nactual = {name: str(dtype) for name, dtype in frame.dtypes.items()}\nmissing = [c for c in EXPECTED if c not in actual]\nwrong = {c: (EXPECTED[c], actual[c]) for c in EXPECTED if c in actual and actual[c] != EXPECTED[c]}\nassert not missing and not wrong, f\"schema drift — missing {missing}, wrong {wrong}\"\n",[157,1229,1230,1237,1247,1251,1290,1314,1318,1360,1386,1420,1466],{"__ignoreMap":155},[160,1231,1232,1234],{"class":63,"line":162},[160,1233,215],{"class":214},[160,1235,1236],{"class":218}," json\n",[160,1238,1239,1241,1243,1245],{"class":63,"line":228},[160,1240,215],{"class":214},[160,1242,219],{"class":218},[160,1244,222],{"class":214},[160,1246,225],{"class":218},[160,1248,1249],{"class":63,"line":235},[160,1250,232],{"emptyLinePlaceholder":231},[160,1252,1253,1256,1259,1262,1264,1266,1268,1270,1272,1274,1277,1279,1282,1284,1287],{"class":63,"line":247},[160,1254,1255],{"class":300},"EXPECTED",[160,1257,1258],{"class":214}," =",[160,1260,1261],{"class":218}," {",[160,1263,1033],{"class":169},[160,1265,1036],{"class":218},[160,1267,1039],{"class":169},[160,1269,259],{"class":218},[160,1271,1067],{"class":169},[160,1273,1036],{"class":218},[160,1275,1276],{"class":169},"\"datetime64[ns]\"",[160,1278,259],{"class":218},[160,1280,1281],{"class":169},"\"Revenue\"",[160,1283,1036],{"class":218},[160,1285,1286],{"class":169},"\"float64\"",[160,1288,1289],{"class":218},"}\n",[160,1291,1292,1295,1297,1299,1302,1304,1307,1309,1312],{"class":63,"line":272},[160,1293,1294],{"class":218},"json.dump(",[160,1296,1255],{"class":300},[160,1298,259],{"class":218},[160,1300,1301],{"class":300},"open",[160,1303,680],{"class":218},[160,1305,1306],{"class":169},"\"orders.schema.json\"",[160,1308,259],{"class":218},[160,1310,1311],{"class":169},"\"w\"",[160,1313,960],{"class":218},[160,1315,1316],{"class":63,"line":292},[160,1317,232],{"emptyLinePlaceholder":231},[160,1319,1320,1322,1324,1327,1330,1332,1335,1337,1339,1341,1343,1345,1348,1351,1353,1355,1357],{"class":63,"line":312},[160,1321,238],{"class":218},[160,1323,241],{"class":214},[160,1325,1326],{"class":218}," pd.read_csv(",[160,1328,1329],{"class":169},"\"orders.csv\"",[160,1331,259],{"class":218},[160,1333,1334],{"class":334},"dtype",[160,1336,241],{"class":214},[160,1338,690],{"class":218},[160,1340,1033],{"class":169},[160,1342,1036],{"class":218},[160,1344,1039],{"class":169},[160,1346,1347],{"class":218},"}, ",[160,1349,1350],{"class":334},"parse_dates",[160,1352,241],{"class":214},[160,1354,1064],{"class":218},[160,1356,1067],{"class":169},[160,1358,1359],{"class":218},"])\n",[160,1361,1362,1365,1367,1370,1373,1376,1378,1381,1383],{"class":63,"line":318},[160,1363,1364],{"class":218},"actual ",[160,1366,241],{"class":214},[160,1368,1369],{"class":218}," {name: ",[160,1371,1372],{"class":300},"str",[160,1374,1375],{"class":218},"(dtype) ",[160,1377,566],{"class":214},[160,1379,1380],{"class":218}," name, dtype ",[160,1382,572],{"class":214},[160,1384,1385],{"class":218}," frame.dtypes.items()}\n",[160,1387,1388,1391,1393,1396,1398,1401,1403,1406,1409,1411,1414,1417],{"class":63,"line":323},[160,1389,1390],{"class":218},"missing ",[160,1392,241],{"class":214},[160,1394,1395],{"class":218}," [c ",[160,1397,566],{"class":214},[160,1399,1400],{"class":218}," c ",[160,1402,572],{"class":214},[160,1404,1405],{"class":300}," EXPECTED",[160,1407,1408],{"class":214}," if",[160,1410,1400],{"class":218},[160,1412,1413],{"class":214},"not",[160,1415,1416],{"class":214}," in",[160,1418,1419],{"class":218}," actual]\n",[160,1421,1422,1425,1427,1430,1432,1435,1437,1439,1441,1443,1445,1447,1449,1452,1455,1458,1461,1463],{"class":63,"line":346},[160,1423,1424],{"class":218},"wrong ",[160,1426,241],{"class":214},[160,1428,1429],{"class":218}," {c: (",[160,1431,1255],{"class":300},[160,1433,1434],{"class":218},"[c], actual[c]) ",[160,1436,566],{"class":214},[160,1438,1400],{"class":218},[160,1440,572],{"class":214},[160,1442,1405],{"class":300},[160,1444,1408],{"class":214},[160,1446,1400],{"class":218},[160,1448,572],{"class":214},[160,1450,1451],{"class":218}," actual ",[160,1453,1454],{"class":214},"and",[160,1456,1457],{"class":218}," actual[c] ",[160,1459,1460],{"class":214},"!=",[160,1462,1405],{"class":300},[160,1464,1465],{"class":218},"[c]}\n",[160,1467,1468,1471,1473,1476,1478,1480,1483,1485,1488,1490,1493,1495,1498,1500,1503,1505],{"class":63,"line":365},[160,1469,1470],{"class":214},"assert",[160,1472,894],{"class":214},[160,1474,1475],{"class":218}," missing ",[160,1477,1454],{"class":214},[160,1479,894],{"class":214},[160,1481,1482],{"class":218}," wrong, ",[160,1484,683],{"class":214},[160,1486,1487],{"class":169},"\"schema drift — missing ",[160,1489,690],{"class":689},[160,1491,1492],{"class":218},"missing",[160,1494,699],{"class":689},[160,1496,1497],{"class":169},", wrong ",[160,1499,690],{"class":689},[160,1501,1502],{"class":218},"wrong",[160,1504,699],{"class":689},[160,1506,1507],{"class":169},"\"\n",[10,1509,1510,1511,1515],{},"The same reasoning drives the column checks in\n",[14,1512,1514],{"href":1513},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fvalidate-excel-columns-before-import-with-pandas\u002F","Validate Excel Columns Before Import with Pandas",":\nthe cheapest place to catch a type change is the moment the data crosses into your code.",[145,1517,1519],{"id":1518},"what-a-person-actually-needs-from-the-file","What a person actually needs from the file",[10,1521,1522,1523,1525],{},"It is worth being honest about why ",[157,1524,61],{}," keeps winning despite everything above. A recipient can\nopen it without installing anything, sort a column, filter it, add a note in the margin and send it\nback. No other format on this list does that, and a pipeline that delivers Parquet to a finance\nteam has optimised the wrong end.",[10,1527,1528,1529,1533],{},"That is why the recommendation is a split rather than a replacement. The moment the data stops\nmoving between programs and starts being read by a person, the format should change — and it should\nchange into a workbook that is formatted for reading, with frozen headers, number formats and\nsensible column widths, as\n",[14,1530,1532],{"href":1531},"\u002Fformatting-and-charting-excel-reports-with-python\u002Fbuilding-excel-reports-with-xlsxwriter\u002Fwrite-a-formatted-excel-report-with-xlsxwriter\u002F","Write a Formatted Excel Report with XlsxWriter"," shows.",[145,1535,1537],{"id":1536},"common-pitfalls","Common pitfalls",[1539,1540,1541,1557],"table",{},[1542,1543,1544],"thead",{},[1545,1546,1547,1551,1554],"tr",{},[1548,1549,1550],"th",{},"Symptom",[1548,1552,1553],{},"Cause",[1548,1555,1556],{},"Fix",[1558,1559,1560,1579,1597,1611,1625,1639],"tbody",{},[1545,1561,1562,1566,1572],{},[1563,1564,1565],"td",{},"Leading zeros gone after a CSV round trip",[1563,1567,1568,1569],{},"CSV has no types; pandas inferred ",[157,1570,1571],{},"int64",[1563,1573,1574,1575,1578],{},"Pass ",[157,1576,1577],{},"dtype={\"col\": \"string\"}"," on read",[1545,1580,1581,1589,1592],{},[1563,1582,1583,1586,1587],{},[157,1584,1585],{},"ImportError: Unable to find a usable engine"," on ",[157,1588,188],{},[1563,1590,1591],{},"No Parquet library installed",[1563,1593,1594],{},[157,1595,1596],{},"pip install pyarrow",[1545,1598,1599,1602,1605],{},[1563,1600,1601],{},"First column name has a strange prefix",[1563,1603,1604],{},"Excel wrote a UTF-8 BOM",[1563,1606,1607,1608],{},"Read with ",[157,1609,1610],{},"encoding=\"utf-8-sig\"",[1545,1612,1613,1616,1619],{},[1563,1614,1615],{},"Parquet file cannot be opened by a colleague",[1563,1617,1618],{},"It is not meant to be opened by hand",[1563,1620,1621,1622,1624],{},"Deliver ",[157,1623,61],{},"; keep Parquet inside the pipeline",[1545,1626,1627,1630,1633],{},[1563,1628,1629],{},"Dates arrive as text from CSV",[1563,1631,1632],{},"No type information in the file",[1563,1634,1635,1638],{},[157,1636,1637],{},"parse_dates=[...]",", or store Parquet instead",[1545,1640,1641,1646,1649],{},[1563,1642,1643,1645],{},[157,1644,61],{}," write fails past a million rows",[1563,1647,1648],{},"The format's own row limit is 1,048,576",[1563,1650,1651],{},"Split across sheets or files, or deliver CSV\u002FParquet",[145,1653,1655],{"id":1654},"performance-and-scale","Performance and scale",[10,1657,1658,1659,1663],{},"The row limit is the constraint people meet first and plan for last. A worksheet holds 1,048,576\nrows and 16,384 columns; exceed either and the write fails or silently truncates depending on the\nlibrary. CSV and Parquet have no such limit, which by itself decides the format for any dataset\napproaching that scale — see ",[14,1660,1662],{"href":1661},"\u002Fadvanced-data-transformation-and-cleaning\u002Fworking-with-large-excel-files-in-python\u002F","Working with Large Excel Files in Python","\nfor the strategies when the deliverable still has to be a spreadsheet.",[10,1665,1666],{},"Parquet also brings predicate and column pushdown, which no spreadsheet format can offer. A query\nthat needs three columns of a fifty-column dataset reads three columns from disk:",[150,1668,1670],{"className":205,"code":1669,"language":207,"meta":155,"style":155},"import pandas as pd\n\nnorth = pd.read_parquet(\n    \"orders.parquet\",\n    columns=[\"Region\", \"Revenue\", \"Ordered\"],\n    filters=[(\"Region\", \"==\", \"North\")],\n)\nprint(north.shape)\n",[157,1671,1672,1682,1686,1696,1702,1724,1749,1753],{"__ignoreMap":155},[160,1673,1674,1676,1678,1680],{"class":63,"line":162},[160,1675,215],{"class":214},[160,1677,219],{"class":218},[160,1679,222],{"class":214},[160,1681,225],{"class":218},[160,1683,1684],{"class":63,"line":228},[160,1685,232],{"emptyLinePlaceholder":231},[160,1687,1688,1691,1693],{"class":63,"line":235},[160,1689,1690],{"class":218},"north ",[160,1692,241],{"class":214},[160,1694,1695],{"class":218}," pd.read_parquet(\n",[160,1697,1698,1700],{"class":63,"line":247},[160,1699,1174],{"class":169},[160,1701,1021],{"class":218},[160,1703,1704,1707,1709,1711,1714,1716,1718,1720,1722],{"class":63,"line":272},[160,1705,1706],{"class":334},"    columns",[160,1708,241],{"class":214},[160,1710,1064],{"class":218},[160,1712,1713],{"class":169},"\"Region\"",[160,1715,259],{"class":218},[160,1717,1281],{"class":169},[160,1719,259],{"class":218},[160,1721,1067],{"class":169},[160,1723,309],{"class":218},[160,1725,1726,1729,1731,1734,1736,1738,1741,1743,1746],{"class":63,"line":292},[160,1727,1728],{"class":334},"    filters",[160,1730,241],{"class":214},[160,1732,1733],{"class":218},"[(",[160,1735,1713],{"class":169},[160,1737,259],{"class":218},[160,1739,1740],{"class":169},"\"==\"",[160,1742,259],{"class":218},[160,1744,1745],{"class":169},"\"North\"",[160,1747,1748],{"class":218},")],\n",[160,1750,1751],{"class":63,"line":312},[160,1752,343],{"class":218},[160,1754,1755,1757],{"class":63,"line":318},[160,1756,392],{"class":300},[160,1758,1759],{"class":218},"(north.shape)\n",[10,1761,1762],{},"That is not a faster parser doing the same work — it is less work. It is the reason a converted\npipeline keeps getting faster as the dataset grows rather than slower.",[145,1764,1766],{"id":1765},"conclusion","Conclusion",[10,1768,1769,1770,1772],{},"Treat Excel as the format you deliver in, not the one you compute in. Convert an incoming workbook\nto Parquet once and every later step becomes faster, smaller and type-safe; keep CSV for the\ninterfaces that demand universality, and read it defensively because it carries no types of its\nown. The ",[157,1771,61],{}," file then reappears exactly where it earns its cost: at the end, formatted for a\nperson who is going to open it.",[145,1774,1776],{"id":1775},"frequently-asked-questions","Frequently asked questions",[10,1778,1779,1783],{},[1780,1781,1782],"strong",{},"Is CSV really lossy?","\nYes, in the ways that bite. It has no types, so a leading-zero account code comes back as an integer and a date comes back as whatever the reader guesses. It has no sheets, no formats and no formulas. What it does have is universal support, which is why it survives.",[10,1785,1786,1789],{},[1780,1787,1788],{},"Does Parquet need a special reader?","\nIt needs pyarrow or fastparquet installed, both a single pip install. Nobody opens a Parquet file by double-clicking it, which is exactly the point: it is a machine-to-machine format, not a deliverable.",[10,1791,1792,1795],{},[1780,1793,1794],{},"How much smaller is Parquet than Excel?","\nTypically five to twenty times, because it stores columns rather than rows and compresses each one with a codec suited to its type. A repetitive string column — region names, SKUs — compresses especially well through dictionary encoding.",[10,1797,1798,1801],{},[1780,1799,1800],{},"Can I keep using Excel as the delivery format but not the storage format?","\nThat is the recommended pattern. Store and move data as Parquet between automated steps, and generate the .xlsx only at the moment a person is going to open it. The spreadsheet becomes an output rather than a database.",[10,1803,1804,1807],{},[1780,1805,1806],{},"What about Feather, ORC or Avro?","\nFeather is Arrow's on-disk format and is faster to write than Parquet but compresses less, which makes it a good scratch format between steps of one job. ORC and Avro belong to the Hadoop lineage and rarely appear in a Python reporting pipeline unless the warehouse already uses them.",[145,1809,1811],{"id":1810},"related","Related",[1813,1814,1815,1822,1827,1832,1837],"ul",{},[1816,1817,1818,1819,1821],"li",{},"Up one level: ",[14,1820,17],{"href":16}," — which library to use once the format is settled.",[1816,1823,1824,1826],{},[14,1825,974],{"href":973}," — the conversion step, including partitioning and schema control.",[1816,1828,1829,1831],{},[14,1830,1103],{"href":1102}," — encodings, quoting and multi-sheet exports.",[1816,1833,1834,1836],{},[14,1835,1662],{"href":1661}," — what to do when the deliverable must stay a spreadsheet.",[1816,1838,1839,1843],{},[14,1840,1842],{"href":1841},"\u002Fgetting-started-with-python-excel-automation\u002Fchoosing-a-python-excel-library\u002Fbenchmark-python-excel-read-and-write-speed\u002F","Benchmark Python Excel Read and Write Speed"," — measure the difference on your own data.",[1845,1846,1847],"style",{},"html pre.shiki code .sMTad, html code.shiki .sMTad{--shiki-default:#6F42C1;--shiki-dark:#FFB757}html pre.shiki code .srMev, html code.shiki .srMev{--shiki-default:#032F62;--shiki-dark:#ADDCFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .s-kum, html code.shiki .s-kum{--shiki-default:#D73A49;--shiki-dark:#FF9492}html pre.shiki code .skGVy, html code.shiki .skGVy{--shiki-default:#24292E;--shiki-dark:#F0F3F6}html pre.shiki code .s-wDw, html code.shiki .s-wDw{--shiki-default:#6A737D;--shiki-dark:#BDC4CC}html pre.shiki code .sP0c6, html code.shiki .sP0c6{--shiki-default:#005CC5;--shiki-dark:#91CBFF}html pre.shiki code .sa561, html code.shiki .sa561{--shiki-default:#E36209;--shiki-dark:#FFB757}html pre.shiki code .sSjpA, html code.shiki .sSjpA{--shiki-default:#005CC5;--shiki-dark:#FF9492}html pre.shiki code .s_Opv, html code.shiki .s_Opv{--shiki-default:#6F42C1;--shiki-dark:#DBB7FF}",{"title":155,"searchDepth":228,"depth":228,"links":1849},[1850,1851,1852,1853,1854,1855,1856,1857,1858,1859,1860,1861],{"id":147,"depth":228,"text":148},{"id":192,"depth":228,"text":193},{"id":425,"depth":228,"text":426},{"id":739,"depth":228,"text":740},{"id":978,"depth":228,"text":979},{"id":1107,"depth":228,"text":1108},{"id":1518,"depth":228,"text":1519},{"id":1536,"depth":228,"text":1537},{"id":1654,"depth":228,"text":1655},{"id":1765,"depth":228,"text":1766},{"id":1775,"depth":228,"text":1776},{"id":1810,"depth":228,"text":1811},"2026-09-04","Excel is a delivery format, not a storage format. Compare types, size and read speed, and use Parquet in the middle of the pipeline with .xlsx only at the end.","md",[1866,1868,1870,1872,1874],{"q":1782,"a":1867},"Yes, in the ways that bite. It has no types, so a leading-zero account code comes back as an integer and a date comes back as whatever the reader guesses. It has no sheets, no formats and no formulas. What it does have is universal support, which is why it survives.",{"q":1788,"a":1869},"It needs pyarrow or fastparquet installed, both a single pip install. Nobody opens a Parquet file by double-clicking it, which is exactly the point: it is a machine-to-machine format, not a deliverable.",{"q":1794,"a":1871},"Typically five to twenty times, because it stores columns rather than rows and compresses each one with a codec suited to its type. A repetitive string column — region names, SKUs — compresses especially well through dictionary encoding.",{"q":1800,"a":1873},"That is the recommended pattern. Store and move data as Parquet between automated steps, and generate the .xlsx only at the moment a person is going to open it. The spreadsheet becomes an output rather than a database.",{"q":1806,"a":1875},"Feather is Arrow's on-disk format and is faster to write than Parquet but compresses less, which makes it a good scratch format between steps of one job. ORC and Avro belong to the Hadoop lineage and rarely appear in a Python reporting pipeline unless the warehouse already uses them.",{"breadcrumb":1877},[1878,1880,1883],{"name":1879,"item":668},"Home",{"name":1881,"item":1882},"Getting Started with Python Excel Automation","\u002Fgetting-started-with-python-excel-automation\u002F",{"name":17,"item":16},"\u002Fgetting-started-with-python-excel-automation\u002Fchoosing-a-python-excel-library\u002Fexcel-vs-csv-vs-parquet-for-python-data-pipelines",{"title":5,"description":1886},"Compare .xlsx, .csv and .parquet on types, size and read speed, then use each where it belongs: Parquet for computation, Excel for the file a person opens.","excel-vs-csv-vs-parquet-for-python-data-pipelines","getting-started-with-python-excel-automation\u002Fchoosing-a-python-excel-library\u002Fexcel-vs-csv-vs-parquet-for-python-data-pipelines\u002Findex","how-to","aeFprlvChMqjZNAbLrR7e2TufH2wtM_AvkxdyVp5B9c",[1892,1895],{"title":1842,"path":1893,"stem":1894,"children":-1},"\u002Fgetting-started-with-python-excel-automation\u002Fchoosing-a-python-excel-library\u002Fbenchmark-python-excel-read-and-write-speed","getting-started-with-python-excel-automation\u002Fchoosing-a-python-excel-library\u002Fbenchmark-python-excel-read-and-write-speed\u002Findex",{"title":1896,"path":1897,"stem":1898,"children":-1},"openpyxl vs pandas for Excel Automation","\u002Fgetting-started-with-python-excel-automation\u002Fchoosing-a-python-excel-library\u002Fopenpyxl-vs-pandas-for-excel-automation","getting-started-with-python-excel-automation\u002Fchoosing-a-python-excel-library\u002Fopenpyxl-vs-pandas-for-excel-automation\u002Findex",1788710154573]