Python Excel AutomationPractical guides for developers
How-to
Advanced Data Transformation And CleaningDeep dive

Scan and Filter Many Excel Files Lazily with Polars

Polars has scan_csv and scan_parquet but no scan_excel, so a folder of workbooks cannot be queried lazily as-is. This page reads only the columns you need from each file with the calamine engine, tags rows with their source file, builds one LazyFrame for filtering and aggregation, and adds a Parquet cache that turns every later run into a true lazy scan with projection and predicate pushdown.
Polars & ArrowLarge files & performance

A year of daily sales exports is 250 workbooks in a folder, and the question is narrow: total revenue for one region, by month. With CSV or Parquet files, Polars would answer it with scan_csv or scan_parquet, reading only the columns and rows the query needs. With Excel it cannot: Polars has no scan_excel, because an .xlsx file is a zip archive of XML that has to be decompressed and parsed before any row can be filtered. That does not mean every run has to read every byte of every workbook, though. This page shows the fastest way to read a folder of workbooks into one lazy query — selecting columns at read time, tagging rows with their file — and then a small Parquet cache that makes repeat runs genuinely lazy. It belongs to Reading Excel with Polars and Arrow.

Prerequisites

Bash
pip install polars fastexcel

The exports/ folder holds twelve monthly workbooks of 8,000 orders each, with eight columns. Polars' read_excel uses the calamine engine (from the fastexcel package) by default, which is by far the fastest Excel reader available to Python. The examples were run with Polars 1.44.

Step 1 — Why there is no lazy scan for Excel

A lazy scan works because the file format lets the reader skip work: Parquet stores each column separately with statistics per row group, so a query for two columns where Region == "North" reads two columns and skips row groups whose statistics rule North out. CSV at least lets the reader stream rows and stop early. An .xlsx sheet is one XML document inside a zip archive, with cells stored row by row; to find the Region values the reader must decompress and parse the whole sheet. So Polars offers only read_excel, which returns an eager DataFrame:

Python
import polars as pl

print("scan_excel available:", hasattr(pl, "scan_excel"))
one = pl.read_excel("exports/sales_2026-01.xlsx")
print(one.shape, one.columns)

The lesson is not "Excel is slow" — calamine parses these files quickly — but that every query starts by reading whole sheets, so the levers are how little to materialise from each read and how rarely to read the workbooks at all.

What a lazy scan can skip, by format Parquet stores columns separately with statistics, so a lazy scan reads only the needed columns and skips row groups. CSV can be streamed row by row. An xlsx sheet is a single compressed XML document stored row by row, so the whole sheet must be parsed before filtering. Parquet column chunks + statistics projection pushdown row groups skipped by filter CSV streamed row by row scan_csv exists all columns still parsed xlsx one zipped XML per sheet no scan_excel parse first, then filter

Step 2 — Read only the columns you need, tagged with their file

read_excel accepts columns= and include_file_paths=. Selecting columns at read time means the other columns are never turned into Polars data, and the file path column records where each row came from:

Python
import time
from pathlib import Path

files = sorted(Path("exports").glob("sales_*.xlsx"))
needed = ["Date", "Region", "Qty", "Unit price"]

t0 = time.perf_counter()
frames = [pl.read_excel(f, columns=needed, include_file_paths="source") for f in files]
sales = pl.concat(frames, how="vertical")
print(f"read {len(files)} files: {sales.shape} in {time.perf_counter() - t0:.2f}s")
print(sales.head(3))

Four of eight columns are read from each file, and the source column holds the path. Look closely at the types: Unit price came back as i64, not a float. The prices in these files happen to be whole numbers (24.0, 89.0 …), Excel stores them as plain numbers, and calamine infers integers from the first rows it samples. If a later file contains 24.50, that file's column is a float and the concatenation fails on mismatched types. Pin the types you care about with schema_overrides={"Unit price": pl.Float64} so every file agrees, rather than relying on inference. pl.concat(how="vertical") requires matching schemas; if exports from different months might differ — a column added mid-year, a column typed differently — use how="diagonal_relaxed", which aligns columns by name, fills missing ones with nulls and widens types where needed. Reading files in a loop also makes it easy to skip or log a corrupt workbook instead of failing the whole run.

Step 3 — Query the combined data lazily

Once the rows are in memory, switching to a LazyFrame lets Polars optimise the whole query — filter, derived columns and aggregation — as one plan:

Python
query = (
    sales.lazy()
    .filter(pl.col("Region") == "North")
    .with_columns((pl.col("Qty") * pl.col("Unit price")).alias("Revenue"),
                  pl.col("Date").dt.truncate("1mo").alias("Month"))
    .group_by("Month")
    .agg(pl.col("Revenue").sum().round(2), pl.len().alias("Orders"))
    .sort("Month")
)
t0 = time.perf_counter()
north = query.collect()
print(f"query: {time.perf_counter() - t0:.3f}s")
print(north.head(4))

The query itself takes about two milliseconds; reading the twelve workbooks took around a fifth of a second — calamine is fast, but still a hundred times slower than the query. That is the key fact about Polars and Excel: the lazy engine helps with the processing, and to speed up the whole job you have to read less Excel. More on Polars aggregation in Clean and Aggregate Excel Data with Polars.

Step 4 — A Parquet cache for genuinely lazy repeat runs

Most folders of exports are append-only: last month's workbook does not change. Convert each workbook to Parquet once, re-convert only when the workbook is newer than its cached copy, and run every query with scan_parquet:

Python
CACHE = Path("cache")
CACHE.mkdir(exist_ok=True)

def refresh_cache(files):
    converted = 0
    for f in files:
        target = CACHE / f"{f.stem}.parquet"
        if not target.exists() or target.stat().st_mtime < f.stat().st_mtime:
            pl.read_excel(f, include_file_paths="source").write_parquet(target)
            converted += 1
    return converted

t0 = time.perf_counter()
print("first run converted:", refresh_cache(files), f"files in {time.perf_counter() - t0:.2f}s")
t0 = time.perf_counter()
print("second run converted:", refresh_cache(files), f"files in {time.perf_counter() - t0:.3f}s")

The first run pays the full Excel cost once — every column, so later queries can use any of them. The second run finds every cached file up to date and does nothing. Comparing modification times is the simplest freshness test; for files copied from elsewhere, whose timestamps may not change, compare a content hash instead. The general conversion is covered in Convert Excel Files to Parquet with Python.

Now the same question as a true lazy scan:

Python
lazy = (
    pl.scan_parquet("cache/*.parquet")
    .filter(pl.col("Region") == "North")
    .with_columns((pl.col("Qty") * pl.col("Unit price")).alias("Revenue"),
                  pl.col("Date").dt.truncate("1mo").alias("Month"))
    .group_by("Month").agg(pl.col("Revenue").sum().round(2), pl.len().alias("Orders"))
    .sort("Month")
)
plan = lazy.explain()
print([line.strip() for line in plan.splitlines() if "PROJECT" in line or "SELECTION" in line][:3])
t0 = time.perf_counter()
cached = lazy.collect()
print(f"cached query: {time.perf_counter() - t0:.3f}s | same result: {cached.equals(north)}")

explain() shows the optimised plan: the Parquet scan reads only the four columns the query uses (projection pushdown) and applies the Region filter inside the scan (predicate pushdown). The whole query — reading twelve files and aggregating 96,000 rows — runs in a small fraction of the time the Excel reads took, and gives exactly the same result. Every later question about this data now costs milliseconds instead of a full re-read of the workbooks.

Excel once, Parquet ever after Each workbook is read once with calamine and written to a Parquet cache, re-converted only when the workbook changes. Queries then use scan_parquet over the cache, where Polars reads only the needed columns and filters inside the scan. Workbooks one per month read_excel once only if newer than cache Parquet cache one file per workbook scan_parquet projection + filter pushdown Where the time goes (one run, 12 files × 8,000 rows) Reading the twelve workbooks with calamine took about 0.19 seconds and the lazy query on the combined data about 0.002 seconds. Building the Parquet cache took about 0.22 seconds once, a cache check with nothing to convert took almost no time, and the cached scan_parquet query took about 0.003 seconds. read 12 workbooks ≈ 0.19 s query in memory ≈ 0.002 s build Parquet cache (once) ≈ 0.22 s scan_parquet query ≈ 0.003 s

When to use which approach

The numbers above are small because the example is small; they scale roughly with the number of cells, so a folder of 250 workbooks with 50,000 rows each — over a hundred times more data — takes tens of seconds to read on every run, while the cached query still takes well under a second. For a one-off question, Steps 2 and 3 are all you need: read the needed columns, combine, query. For a folder that is queried repeatedly — a dashboard refresh, several reports from the same exports, an analyst exploring — the cache pays for itself on the second run. And if you control the export, the best fix is upstream: ask for Parquet or CSV alongside the Excel file, and keep Excel for people.

Common pitfalls

  • Looking for scan_excel. It does not exist; Excel must be read eagerly.
  • Reading every column. Pass columns= when you only need some.
  • concat failing on schema differences. Use how="diagonal_relaxed" for mixed exports.
  • Stale cache files. Re-convert when the workbook is newer, or compare hashes.
  • Caching only the columns of today's query. Cache all columns so future queries can use them.

Conclusion

Polars cannot scan Excel files lazily, because a sheet has to be parsed in full before it can be filtered. Read each workbook with read_excel(columns=..., include_file_paths=...), concatenate, and switch to .lazy() for an optimised query. For folders you query more than once, convert each workbook to Parquet once and re-convert only when it changes; from then on scan_parquet gives true laziness, reading only the columns a query needs and filtering inside the scan.

Frequently asked questions

Does Polars have scan_excel? No. Polars offers scan_csv, scan_parquet and others, but Excel files can only be read eagerly with read_excel, because an xlsx sheet must be parsed in full before rows can be filtered.

How do I read many Excel files into one Polars DataFrame? Loop over the files with read_excel, optionally passing columns and include_file_paths, and combine them with pl.concat, using how="diagonal_relaxed" if their columns differ.

How can I make repeated queries over Excel files faster? Convert each workbook to Parquet once, re-converting only when the workbook changes, and query the Parquet files with scan_parquet.

How do I know which file a row came from in Polars? Pass include_file_paths with a column name to read_excel, which adds the source path to every row.

More on polars & arrow

More on polars & arrow →