Guide

Advanced Data Transformation And CleaningDeep dive

Speed Up pandas Excel Reads with the calamine Engine

Swap openpyxl for calamine in pandas.read_excel: install it, measure the difference, understand the trade-offs, and know which reads it will not help.

Most of the time a pandas Excel read spends is not in pandas at all — it is in openpyxl, parsing XML in Python, cell by cell. Swapping the engine for calamine, a Rust parser exposed through python-calamine, removes that cost without touching a line of your DataFrame code. This guide installs it, measures the difference honestly, and sets out the handful of behaviours that differ so a switch does not change your numbers. It sits in Reading Excel with Polars and Arrow, because it is the same parser Polars uses by default.

Prerequisites

Bash

pip install "pandas>=2.2" python-calamine

pandas gained the calamine engine in 2.2, so check your version before assuming the argument is available:

Python

import pandas as pd
print(pd.__version__)

Make the switch

One argument:

Python

import pandas as pd

df = pd.read_excel("sales.xlsx", engine="calamine")

Everything else — sheet_name, usecols, skiprows, nrows, header, dtype — behaves as before. That is the point: the engine is an implementation detail of the read, not a different API. It handles .xlsx, .xlsm, .xls, .xlsb and .ods, so it also removes the need to install and select a different engine per format:

Python

for name in ("modern.xlsx", "legacy.xls", "binary.xlsb", "open.ods"):
    print(name, pd.read_excel(name, engine="calamine").shape)

That last property is worth as much as the speed in a pipeline that ingests whatever a business system produces — no more engine table, no more ImportError when a .xlsb shows up unannounced. The engine-per-format problem is catalogued in Fix "Excel file format cannot be determined" in pandas.

Measure it on your own file

Published numbers are measured on other people's spreadsheets. Run this against yours, three times, and take the best of each:

Python

"""Compare engines on the file your job actually reads."""
import time

import pandas as pd

PATH = "sales.xlsx"

def timed(engine: str, repeats: int = 3) -> float:
    best = float("inf")
    for _ in range(repeats):
        start = time.perf_counter()
        df = pd.read_excel(PATH, engine=engine)
        best = min(best, time.perf_counter() - start)
    print(f"{engine:10s} {best:6.2f}s  {df.shape[0]:,} rows x {df.shape[1]} cols")
    return best

openpyxl_time = timed("openpyxl")
calamine_time = timed("calamine")
print(f"speed-up: {openpyxl_time / calamine_time:.1f}x")

Memory is the other half of the story, and often the more important one in a container with a hard limit:

Python

import tracemalloc

import pandas as pd

for engine in ("openpyxl", "calamine"):
    tracemalloc.start()
    pd.read_excel("sales.xlsx", engine=engine)
    _current, peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    print(f"{engine:10s} peak {peak / 1e6:7.1f} MB")

Check the dtypes did not move

The read is a different implementation, so inference can differ at the margins. Compare before trusting a switch in a job that produces numbers people act on:

Python

import pandas as pd

a = pd.read_excel("sales.xlsx", engine="openpyxl")
b = pd.read_excel("sales.xlsx", engine="calamine")

print(a.dtypes.compare(b.dtypes) if not a.dtypes.equals(b.dtypes) else "dtypes identical")
print("values identical:", a.equals(b))

Where they differ, pin the type rather than depending on either engine's guess:

Python

df = pd.read_excel(
    "sales.xlsx",
    engine="calamine",
    dtype={"order_id": "string", "postcode": "string"},
    parse_dates=["order_date"],
)

Identifier columns are the usual offenders — the leading-zero problem covered in Convert Excel text columns to numbers with pandas.

Know what it will not do

calamine reads values. It does not read styles, merged-cell metadata, comments, charts or defined names, and it cannot write. That means it is a drop-in for the ingest side of a job and irrelevant to the output side:

Task	calamine	Use instead
Read values fast, any format	Yes	—
Read cell colours or comments	No	openpyxl
Detect merged regions	No	openpyxl
Write or edit a workbook	No	openpyxl, xlsxwriter
Read a password-protected file	No	msoffcrypto-tool, then any engine

If your read depends on formatting — for example detecting a highlighted row — you need openpyxl regardless of speed, as in Highlight invalid cells in Excel with Python.

Roll it out safely across a codebase

Changing an engine everywhere at once is a needless risk. A small indirection makes the switch reversible, per environment, without touching call sites:

Python

"""One switch for the whole codebase, overridable per deployment."""
import os

import pandas as pd

EXCEL_ENGINE = os.getenv("EXCEL_ENGINE", "calamine")

def read_excel(path, **kwargs):
    """Project-wide read: engine chosen centrally, arguments passed through."""
    kwargs.setdefault("engine", EXCEL_ENGINE)
    return pd.read_excel(path, **kwargs)

Deploy with EXCEL_ENGINE=openpyxl if a job misbehaves, and the rollback is an environment variable rather than a release. Add a regression test that reads a small fixture workbook and asserts on the shape and dtypes, so the engine change is covered the same way any other dependency change would be:

Python

"""tests/test_read.py — guard the ingest against an engine regression."""
import pandas as pd

def test_fixture_reads_consistently():
    df = pd.read_excel("tests/fixtures/sales.xlsx", engine="calamine")
    assert list(df.columns) == ["order_date", "region", "product", "revenue"]
    assert str(df["revenue"].dtype) == "float64"
    assert len(df) == 250

That test costs milliseconds and catches both an engine behaviour change and a fixture that quietly moved. Building it into the wider test suite is covered in Test Excel output with pytest.

Combine it with the arguments that matter more

An engine swap is multiplicative with the two arguments that decide how much work the read does at all. Narrowing the columns and capping the rows often saves more than the parser does:

Python

import pandas as pd

df = pd.read_excel(
    "sales.xlsx",
    engine="calamine",
    usecols=["order_date", "region", "revenue"],   # skip 30 columns you never use
    nrows=None,                                     # set a number while developing
    dtype={"region": "category"},                   # smaller in memory, faster to group
)

usecols is the strongest of the three: a report that reads three columns from a forty-column export does roughly a tenth of the work regardless of engine. category dtype on a low-cardinality text column both shrinks memory and speeds up later groupby calls, which is exactly the shape of a regional summary.

Common pitfalls and gotchas

pandas older than 2.2 does not know the engine name and raises a ValueError about an unsupported engine.
Installing the wrong package. The distribution is python-calamine; the engine string is "calamine".
Assuming it fixes a slow report. If the workbook is small and the aggregation is heavy, the parse was never the bottleneck. Measure before and after.
Losing merged-cell behaviour. openpyxl-based reads and calamine can differ on how a merged title cell fills — check the top rows of a styled report after switching.
Forgetting it in the deployment image. A local pip install that never reaches requirements.txt produces a job that fails only in production.

Performance and scale notes

The gain is bounded by how much of the read was parsing. For a wide sheet of mostly text, parsing dominates and the improvement is large; for a narrow sheet already read with usecols, less of the file is touched and the difference narrows. Two complementary techniques usually beat any engine change on their own: read fewer columns, and stop re-reading. usecols cuts the work at the source, and converting a workbook you read repeatedly into Parquet removes the Excel parse entirely from every subsequent run — see Convert Excel files to Parquet with Python and Speed up openpyxl with read-only mode for the openpyxl-side equivalent.

Conclusion

engine="calamine" is the cheapest performance change available to a pandas Excel job: one argument, one dependency, no code restructuring, and one engine that reads every spreadsheet format you are likely to receive. Measure it on your own workbook, compare dtypes before you commit, and keep openpyxl for anything that touches formatting or writes a file. When the same workbook is read on a schedule, follow the switch with a Parquet cache and the parse disappears altogether.

Frequently asked questions

What is calamine? A spreadsheet parser written in Rust, exposed to Python by the python-calamine package. pandas can use it as a read engine, and Polars uses it by default. It reads .xlsx, .xlsm, .xls, .xlsb and .ods through one code path.

How much faster is it really? On typical report-sized workbooks expect a multiple rather than a percentage — the parse is where nearly all the time goes, and a compiled parser removes most of it. Measure on your own file, since sheet shape and cell types matter more than raw row count.

Does it change the DataFrame I get back? Mostly no, but type inference can differ at the edges — a column of mixed values or an ambiguous date may land as a different dtype. Compare dtypes when you switch, and pin the ones that matter with dtype= or an astype.

Can calamine write Excel files? No. It is a reader only. Keep openpyxl or xlsxwriter for writing, and use calamine purely to speed up the ingest side.

Is it safe for production? Yes, with the usual caveat of pinning the version. Because it is a single dependency doing one job, upgrades are low-risk, but a read engine change is still worth validating against a known workbook in CI.

Up: Reading Excel with Polars and Arrow — the wider Arrow-backed toolkit this engine belongs to.
Read an Excel file with polars.read_excel — the same parser, with a query engine on top.
Convert Excel files to Parquet with Python — removing the Excel parse from repeat reads entirely.
Read a large Excel file in chunks with pandas — what to do when the file will not fit in memory at all.
Read specific columns from Excel with pandas — narrowing the read, which compounds with a faster engine.