Watch a Folder and Process New Excel Files with Python

A drop folder is the simplest integration there is: someone — or some other system — saves a workbook into a folder, and it gets processed. A scheduled job that scans the folder every few minutes works, but a watcher can react within a second of the file arriving. The watchdog package receives file system events from the operating system and calls your code. The details that make it reliable are all about what an "event" really means: Excel's lock files, files still being copied, the same file reported twice, and files that arrived while the watcher was not running. This page is part of Scheduling Python Excel Scripts with Cron.

Prerequisites

pip install watchdog pandas openpyxl

The examples use three folders: dropbox where files arrive, and processed and failed where they go afterwards. One file is already waiting in dropbox from before the watcher started.

Step 1 — Decide which paths are workbooks

Plenty of files appear in a folder that holds workbooks without being workbooks you should process. Excel creates a lock file named ~$ plus the workbook name while someone has it open; saving creates a temporary file with a random name and then renames it; copy tools create partial files with their own suffixes. Filter on the name before anything else:

from pathlib import Path

EXTENSIONS = {".xlsx", ".xlsm"}

def is_workbook(path):
    p = Path(path)
    return (p.suffix.lower() in EXTENSIONS
            and not p.name.startswith(("~$", ".~", "."))
            and p.parent.name == "dropbox")

for name in ["dropbox/sales_w39.xlsx", "dropbox/~$sales_w39.xlsx", "dropbox/7A3F2B10.tmp",
             "dropbox/.~lock.sales_w39.xlsx#", "dropbox/notes.txt", "processed/sales_w38.xlsx"]:
    print(f"{name:34} {is_workbook(name)}")

Only the real workbook passes. Checking the parent folder matters if the watcher is recursive or if the processed folder sits inside the drop folder — otherwise moving a file after processing looks like a new file arriving. LibreOffice's lock files start with .~lock., which the check also skips.

Step 2 — Wait until the file has finished arriving

The created event fires when a file appears, not when it is complete. A large workbook copied over a network appears at 0 bytes and grows for seconds. Wait until its size has been stable for a moment and it can actually be opened as a workbook:

import time
import zipfile

def wait_until_complete(path, stable_for=1.0, timeout=60.0, poll=0.25):
    deadline = time.monotonic() + timeout
    last_size, stable_since = -1, None
    while time.monotonic() < deadline:
        try:
            size = Path(path).stat().st_size
        except FileNotFoundError:
            return False
        if size == last_size and size > 0:
            stable_since = stable_since or time.monotonic()
            if time.monotonic() - stable_since >= stable_for and zipfile.is_zipfile(path):
                return True
        else:
            last_size, stable_since = size, None
        time.sleep(poll)
    return False

An .xlsx is a zip archive, and a truncated one is not a valid zip, so zipfile.is_zipfile is a cheap final test that the copy is complete. If the file disappears while waiting — renamed or deleted by whoever put it there — the function returns False and the file is skipped.

Step 3 — Watch the folder and hand files to a worker

The event handler runs on watchdog's own thread and should return quickly. Put each workbook path on a queue; a single worker thread takes paths from the queue, waits for them to complete and processes them one at a time. Handle on_moved as well as on_created, because Excel and many copy tools write a temporary name and then rename it:

import queue
import shutil
import threading
import pandas as pd
from watchdog.events import FileSystemEventHandler
from watchdog.observers import Observer

work = queue.Queue()
log = []

class DropHandler(FileSystemEventHandler):
    def on_created(self, event):
        if not event.is_directory and is_workbook(event.src_path):
            work.put(event.src_path)

    def on_moved(self, event):
        if not event.is_directory and is_workbook(event.dest_path):
            work.put(event.dest_path)

def process(path):
    df = pd.read_excel(path)
    return f"{len(df)} rows, revenue {df['Revenue'].sum():,}"

def worker():
    seen = set()
    while (path := work.get()) is not None:
        if path in seen or not Path(path).exists():
            continue
        seen.add(path)
        if not wait_until_complete(path, timeout=10):
            log.append((Path(path).name, "gave up waiting"))
            continue
        try:
            result = process(path)
            shutil.move(path, Path("processed") / Path(path).name)
            log.append((Path(path).name, result))
        except Exception as exc:
            shutil.move(path, Path("failed") / Path(path).name)
            log.append((Path(path).name, f"failed: {type(exc).__name__}"))

The seen set drops duplicate events — some platforms report a created and a modified event, or two created events, for one copy. Moving each file out of the drop folder once handled makes the folder itself a to-do list: anything still in it has not been processed.

Step 4 — Start the watcher, including files already waiting

Files that arrived while the watcher was stopped produce no events. Queue everything already in the folder when the watcher starts, then begin watching:

import io

observer = Observer()
observer.schedule(DropHandler(), "dropbox", recursive=False)
thread = threading.Thread(target=worker, daemon=True)
thread.start()
observer.start()
for existing in sorted(Path("dropbox").iterdir()):
    if is_workbook(existing):
        work.put(str(existing))

pd.DataFrame({"Region": ["North", "South"], "Revenue": [48200, 39100]}).to_excel("dropbox/sales_w39.xlsx", index=False)
Path("dropbox/~$sales_w39.xlsx").write_bytes(b"lock")
buf = io.BytesIO()
pd.DataFrame({"Region": ["East"], "Revenue": [51750]}).to_excel(buf, index=False)
Path("dropbox/tmp_save.partial").write_bytes(buf.getvalue())
Path("dropbox/tmp_save.partial").rename("dropbox/sales_east.xlsx")
Path("dropbox/broken.xlsx").write_bytes(b"not a workbook at all")

time.sleep(6)
observer.stop(); observer.join()
work.put(None); thread.join()
for name, outcome in log:
    print(f"{name:28} {outcome}")
print("left in dropbox:", sorted(p.name for p in Path("dropbox").iterdir()))

The file left over from yesterday is processed first, from the startup scan. The new workbook arrives through a created event and the renamed one through a moved event; the lock file is ignored. broken.xlsx never becomes a valid zip, so the worker gives up waiting after its timeout and leaves it in place for a person to look at. The East file imitates a program that writes under a temporary name and renames the finished file into place — the pattern that makes on_moved necessary. In a long-running service the watcher runs until stopped; here it is stopped after a few seconds to show the result.

Network shares and other places events do not reach

Watchdog relies on the operating system's change notifications — inotify on Linux, FSEvents on macOS, ReadDirectoryChangesW on Windows. Changes made by another machine to a network share are frequently not reported through them at all, so a watcher on a mapped drive can sit silently while files pile up. For shares, use watchdog.observers.polling.PollingObserver, which lists the folder at an interval and compares, or skip the watcher and run a scheduled scan every few minutes with the same filter, wait and move logic. The startup scan in Step 4 is exactly that scan, so the code is already there.

Common pitfalls

• Processing inside the event handler. Queue the path and let a worker do the work.
• Reading files too early. Wait for a stable size and a valid zip before opening.
• Treating ~$ files as workbooks. Filter lock and temporary files by name.
• Only handling on_created. Saves and copies often finish with a rename; handle on_moved.
• No startup scan. Files that arrived while the watcher was down produce no events.
• Watching a network drive natively. Use PollingObserver or a scheduled scan.

Performance notes

A single worker keeps processing sequential and simple. If files arrive faster than one worker can handle, start several worker threads — or processes, for CPU-heavy work — reading from the same queue, and use per-file locks if more than one watcher could see the same folder.

Conclusion

Filter events to real workbooks, ignoring ~$ lock files, temporary saves and anything outside the drop folder. Queue each path from on_created and on_moved, and let one worker wait for the file to stop growing and become a valid zip before processing it, then move it to a processed or failed folder. Scan the folder at startup for files that arrived while the watcher was down, and use PollingObserver or a scheduled scan on network shares.

Frequently asked questions

How do I run a Python script when a new Excel file appears in a folder? Use the watchdog package: subclass FileSystemEventHandler, handle on_created and on_moved for .xlsx paths, and start an Observer on the folder. Put each new path on a queue and process it in a worker, rather than inside the event handler.

Why does my watcher try to read a file before it has finished copying? The created event fires when the file appears, not when it is complete. Wait until the size has stopped changing for a second or two and the file opens, then process it. Large copies over a network can take many seconds.

What are the ~$ files that appear next to Excel workbooks? They are lock files Excel creates while someone has the workbook open, such as ~$sales.xlsx. They are not workbooks and must be ignored, along with temporary files Excel writes while saving.

Does watchdog work on network drives? Often not. File system events from SMB or NFS shares are not reliably delivered. Use watchdog's PollingObserver, which checks the folder at intervals, or run a scheduled job that scans the folder every few minutes.

Related

• Up one level: Scheduling Python Excel Scripts with Cron — running jobs on a timer.
• Download Excel Attachments from an Email Inbox — filling the drop folder from email.
• Consolidate Timesheets from Many Workbooks — processing the files once they arrive.
• Validate an Uploaded Excel File in a Web App — checks worth running on each file.