Python Excel AutomationPractical guides for developers
Start learning
Guide
Advanced Data Transformation And CleaningDeep dive

Convert Excel Files to Parquet with Python

Turn slow, repeated Excel reads into fast columnar ones: convert workbooks to Parquet with pandas or Polars, keep types, partition by period, and refresh incrementally.

If the same workbook is read more than once, parsing it more than once is wasted work. Converting to Parquet — a typed, columnar, compressed format — turns a slow Excel read into a fast columnar one and carries the schema with the data, so no later job has to guess whether an order number is text. This guide converts single files and whole folders, keeps types intact, refreshes incrementally, and queries the result lazily. It completes the toolkit in Reading Excel with Polars and Arrow.

Row storage versus columnar storage An xlsx stores cells row by row inside XML, so any read parses everything; Parquet stores each column contiguously with its type, so a query reads only the columns it needs. xlsx — rows of XML cells Parquet — typed columns row 1: date, region, product, revenue row 2: date, region, product, revenue row 3: date, region, product, revenue every read parses every cell date region product revenue a query touches only the columns it names

Prerequisites

Either library will do the conversion; pick whichever your project already uses:

Bash
pip install pandas openpyxl pyarrow        # pandas route
pip install "polars[excel]"                # polars route (pyarrow optional)

pyarrow is what actually reads and writes the Parquet format in the pandas route.

Convert one workbook

The pandas version is two lines:

Python
import pandas as pd

df = pd.read_excel("sales.xlsx", engine="calamine")
df.to_parquet("sales.parquet", compression="zstd", index=False)

The Polars version is one:

Python
import polars as pl

pl.read_excel("sales.xlsx").write_parquet("sales.parquet", compression="zstd")

index=False matters in pandas: writing the index adds a column that every later read has to ignore. zstd compresses better than the default snappy at a small cost in write time, which is the right trade for a file written once and read many times.

Fix the types before you freeze them

Parquet stores the schema, so whatever types you convert with are the types every downstream job inherits. That is a feature — provided the types are right at the moment of conversion:

Python
import pandas as pd

df = pd.read_excel(
    "sales.xlsx",
    engine="calamine",
    dtype={"order_id": "string", "postcode": "string"},   # keep leading zeros
    parse_dates=["order_date"],
)
df["region"] = df["region"].astype("category")            # small and fast to group
df["revenue"] = pd.to_numeric(df["revenue"], errors="coerce")

print(df.dtypes)
df.to_parquet("sales.parquet", compression="zstd", index=False)

Converting with sloppy types bakes the sloppiness in permanently, and the mistake is invisible until a join misses. The identifier problem in particular is worth being deliberate about — see Convert Excel text columns to numbers with pandas.

Convert a folder, and record where each row came from

Monthly exports are the common case. Convert each file once, tag its origin, and keep the outputs beside each other so they can be scanned as a set:

Python
"""Convert every workbook in a folder, skipping ones already up to date."""
from pathlib import Path

import polars as pl

SRC = Path("monthly")
DST = Path("cache")
DST.mkdir(exist_ok=True)

for src in sorted(SRC.glob("*.xlsx")):
    dst = DST / f"{src.stem}.parquet"
    if dst.exists() and dst.stat().st_mtime >= src.stat().st_mtime:
        continue                                    # already converted, unchanged
    (pl.read_excel(src)
       .with_columns(source_file=pl.lit(src.name), period=pl.lit(src.stem[-7:]))
       .write_parquet(dst, compression="zstd"))
    print("converted", src.name)

The mtime comparison makes the script idempotent: run it as often as you like and it only does work for files that actually changed. That property is what lets it sit at the top of a nightly job without doubling the run time.

Convert once, then query the cache every run Incoming workbooks are converted to Parquet only when they change, and every downstream report scans the Parquet cache instead of re-parsing spreadsheets. The Excel parse happens once per file, ever incoming *.xlsx changed since last convert? yes parse and write no skip cache/*.parquet Reports scan the cache; a rerun after a failure costs almost nothing

Query the cache instead of the spreadsheets

Once converted, reads get dramatically cheaper — especially through Polars' lazy scan, which reads only the columns and row groups a query needs:

Python
import polars as pl

top = (
    pl.scan_parquet("cache/*.parquet")
      .filter(pl.col("region") == "EMEA")
      .group_by("product")
      .agg(revenue=pl.col("revenue").sum())
      .sort("revenue", descending=True)
      .head(20)
      .collect()
)

pandas reads the same files, and can push a column selection down too:

Python
import pandas as pd

df = pd.read_parquet("cache/", columns=["region", "product", "revenue"])

DuckDB will query the folder in SQL without loading it, which suits ad-hoc analysis:

Python
import duckdb

duckdb.sql("SELECT region, sum(revenue) FROM 'cache/*.parquet' GROUP BY region").show()

Partition when the archive gets large

For a multi-year archive, write partitioned output so a query for one month reads one directory rather than everything:

Python
import pandas as pd

df = pd.read_parquet("cache/")
df.to_parquet("warehouse/", partition_cols=["year", "region"], compression="zstd")

That produces warehouse/year=2026/region=EMEA/…, and every reader — pandas, Polars, DuckDB — understands the layout and skips directories a filter excludes. Do not over-partition: thousands of tiny files are slower than a few large ones, so partition on the columns you actually filter by, usually a period.

Give people back a workbook

Parquet is for machines. When a person needs the numbers, write a workbook from the cache — which is now a fast operation, because the slow part happened once:

Python
import polars as pl

(pl.scan_parquet("cache/*.parquet")
   .group_by("region")
   .agg(revenue=pl.col("revenue").sum())
   .sort("revenue", descending=True)
   .collect()
   .write_excel("regional_summary.xlsx", autofit=True,
                table_style="Table Style Medium 9",
                column_formats={"revenue": "#,##0.00"}))

The formatting options are covered in Write a Polars DataFrame to Excel with formatting.

Validate the conversion before you rely on it

A cache is only useful if it matches its source. Check the conversion the first time you run it — row counts, column names, and the totals of the numeric columns people actually quote:

Python
"""Assert the Parquet copy agrees with the workbook it came from."""
import polars as pl

src = pl.read_excel("sales.xlsx")
cached = pl.read_parquet("sales.parquet")

assert src.height == cached.height, f"{src.height} rows in, {cached.height} out"
assert src.columns == cached.columns, "column set changed"

for column in ("revenue", "quantity"):
    a, b = src[column].sum(), cached[column].sum()
    assert abs(a - b) < 0.01, f"{column}: {a} != {b}"

print(f"verified {cached.height:,} rows, {len(cached.columns)} columns")

Comparing totals rather than every cell keeps the check fast on a large file while still catching the failures that matter: a truncated read, a column silently dropped, or a numeric column that became text and summed to something absurd. Keep it as a test over a small fixture workbook so a library upgrade cannot change the conversion behaviour unnoticed — the approach in Test Excel output with pytest.

Three assertions that catch a bad conversion Row count catches a truncated read, the column list catches a dropped or renamed column, and column totals catch a type change that turned numbers into text. Cheap checks, in order of what they catch row count catches a truncated or partial read column list catches a dropped or renamed column column totals catches a numeric column read as text All three run in well under a second on a report-sized file

Common pitfalls and gotchas

  • Writing the pandas index. index=False unless you genuinely need it; otherwise every reader inherits a stray column.
  • Mixed types in one column across files. A column that is text in January and numeric in March produces a schema clash when the folder is scanned. Pin the dtype at conversion time.
  • Assuming Parquet is human-readable. It is binary. Keep a generated workbook or CSV for anyone who needs to look at it directly.
  • Converting into the same folder as the sources. Keep the cache separate so a glob for *.xlsx never picks up derived files.
  • Forgetting the timezone. A timezone-aware timestamp survives in Parquet but not in Excel, so decide which representation is authoritative — see Handle timezones in Excel timestamps with Python.

Performance and scale notes

The conversion itself is bounded by the Excel parse, so use the fastest reader available — engine="calamine" in pandas, or Polars, which uses it by default. After conversion, expect reads to be an order of magnitude quicker and the files several times smaller. Memory behaves better too: a columnar read materialises only the columns requested, so a job that needs three of forty columns no longer pays for the other thirty-seven. For workbooks too large to convert in one pass, read them in chunks and append row groups — the chunking approach is in Read a large Excel file in chunks with pandas.

Conclusion

Convert once, read many times. A short conversion step with correct types turns every downstream report into a columnar read that keeps its schema, loads in a fraction of the time, and costs a fraction of the disk. Keep the conversion idempotent with an mtime check, partition only by the column you filter on, and generate a workbook from the cache whenever a human needs to see the numbers.

Frequently asked questions

Why convert at all — can't I just read the Excel file each time? You can, but you pay the parse every run. Parquet stores columns in a typed binary layout, so a later read costs a fraction of the time, uses less memory, and needs no type inference because the schema travels with the file.

Does Parquet keep my column types? Yes, that is the main reason to use it. Strings stay strings, dates stay dates, and a column of identifiers with leading zeros keeps them — provided you set the types correctly during the conversion.

Can Excel open a Parquet file? Modern Excel can import Parquet through Power Query on some plans, but treat Parquet as the machine-readable cache and keep a generated .xlsx for people. Convert back with a single write_excel or to_excel call when a human needs it.

How much smaller are the files? Typically several times smaller than the equivalent .xlsx, because columnar storage compresses repeated values extremely well. A folder of monthly workbooks usually shrinks dramatically once converted.

Do I need Spark or a data warehouse to use Parquet? No. pandas, Polars and DuckDB all read Parquet from a local folder. It is a file format, not a platform.