[{"data":1,"prerenderedAt":2104},["ShallowReactive",2],{"doc:\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fload-excel-data-into-mongodb-with-python":3,"surround:\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fload-excel-data-into-mongodb-with-python":2070,"ancestors:\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fload-excel-data-into-mongodb-with-python":2079,"children:\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fload-excel-data-into-mongodb-with-python":2084,"related:\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fload-excel-data-into-mongodb-with-python":2085},{"id":4,"title":5,"body":6,"dateModified":2037,"datePublished":2037,"description":2038,"extension":2039,"faq":2040,"keyword":2049,"kind":2050,"meta":2051,"navigation":126,"path":2060,"seo":2061,"slug":2064,"stem":2065,"topics":2066,"type":2068,"__hash__":2069},"docs\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fload-excel-data-into-mongodb-with-python\u002Findex.md","Load Excel Data into MongoDB with Python",{"type":7,"value":8,"toc":2025},"minimark",[9,37,42,73,91,95,98,197,215,219,222,601,631,883,887,894,1131,1157,1161,1164,1395,1413,1562,1676,1680,1683,1879,1895,1899,1946,1950,1960,1964,1970,1976,1982,1988,1992,2021],[10,11,12,13,17,18,21,22,25,26,30,31,36],"p",{},"A product catalogue maintained in Excel needs to feed an application that stores its data in MongoDB. The obvious one-liner — ",[14,15,16],"code",{},"collection.insert_many(df.to_dict(\"records\"))"," — works the first time and causes trouble from then on: blank cells arrive as ",[14,19,20],{},"NaN"," values that the application cannot handle, dates are stored as strings or timestamps of the wrong type, a column like \"Supplier city\" stays flat when the application expects a nested ",[14,23,24],{},"supplier"," object, and running the import again duplicates every product. A good import shapes each row into the document the application expects, writes real types, leaves blanks out, and ",[27,28,29],"em",{},"upserts"," on a business key so that reloading the same or an updated file is safe. This page builds that import with pandas and PyMongo. It belongs to ",[32,33,35],"a",{"href":34},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002F","Moving Data Between Excel and Databases",".",[38,39,41],"h2",{"id":40},"prerequisites","Prerequisites",[43,44,49],"pre",{"className":45,"code":46,"language":47,"meta":48,"style":48},"language-bash shiki shiki-themes github-light github-dark-high-contrast","pip install pandas openpyxl pymongo\n","bash","",[14,50,51],{"__ignoreMap":48},[52,53,56,60,64,67,70],"span",{"class":54,"line":55},"line",1,[52,57,59],{"class":58},"sMTad","pip",[52,61,63],{"class":62},"srMev"," install",[52,65,66],{"class":62}," pandas",[52,68,69],{"class":62}," openpyxl",[52,71,72],{"class":62}," pymongo\n",[10,74,75,78,79,82,83,86,87,90],{},[14,76,77],{},"catalogue.xlsx"," lists five products with SKU, name, price, stock, launch date, semicolon-separated tags and supplier details in columns named ",[14,80,81],{},"supplier.name"," and ",[14,84,85],{},"supplier.city","; some cells are blank. The code uses PyMongo against a MongoDB server (",[14,88,89],{},"mongodb:\u002F\u002Flocalhost:27017"," here). The outputs on this page were produced against a throwaway local MongoDB 8.0 server started for the test; the code is identical against any server or Atlas cluster.",[38,92,94],{"id":93},"step-1-read-the-sheet-with-the-right-types","Step 1 — Read the sheet with the right types",[10,96,97],{},"Start by reading the sheet so that each column already has the type it should have in MongoDB:",[43,99,103],{"className":100,"code":101,"language":102,"meta":48,"style":48},"language-python shiki shiki-themes github-light github-dark-high-contrast","import pandas as pd\n\ndf = pd.read_excel(\"catalogue.xlsx\", dtype={\"sku\": str})\nprint(df.dtypes.to_dict())\nprint(df.head(2).to_dict(\"records\"))\n","python",[14,104,105,121,128,168,177],{"__ignoreMap":48},[52,106,107,111,115,118],{"class":54,"line":55},[52,108,110],{"class":109},"s-kum","import",[52,112,114],{"class":113},"skGVy"," pandas ",[52,116,117],{"class":109},"as",[52,119,120],{"class":113}," pd\n",[52,122,124],{"class":54,"line":123},2,[52,125,127],{"emptyLinePlaceholder":126},true,"\n",[52,129,131,134,137,140,143,146,150,152,155,158,161,165],{"class":54,"line":130},3,[52,132,133],{"class":113},"df ",[52,135,136],{"class":109},"=",[52,138,139],{"class":113}," pd.read_excel(",[52,141,142],{"class":62},"\"catalogue.xlsx\"",[52,144,145],{"class":113},", ",[52,147,149],{"class":148},"sa561","dtype",[52,151,136],{"class":109},[52,153,154],{"class":113},"{",[52,156,157],{"class":62},"\"sku\"",[52,159,160],{"class":113},": ",[52,162,164],{"class":163},"sP0c6","str",[52,166,167],{"class":113},"})\n",[52,169,171,174],{"class":54,"line":170},4,[52,172,173],{"class":163},"print",[52,175,176],{"class":113},"(df.dtypes.to_dict())\n",[52,178,180,182,185,188,191,194],{"class":54,"line":179},5,[52,181,173],{"class":163},[52,183,184],{"class":113},"(df.head(",[52,186,187],{"class":163},"2",[52,189,190],{"class":113},").to_dict(",[52,192,193],{"class":62},"\"records\"",[52,195,196],{"class":113},"))\n",[10,198,199,200,203,204,207,208,211,212,214],{},"SKUs are read as text so a code such as ",[14,201,202],{},"00123"," would keep its leading zeros. Prices are floats, the launch date column is a pandas datetime and the stock column, which has a blank, is a float — pandas has no integer type with missing values by default. Each of these needs a decision before it becomes a document: MongoDB stores dates as BSON dates, integers as ",[14,205,206],{},"int"," or ",[14,209,210],{},"long",", and has no use for ",[14,213,20],{}," in fields the application treats as optional.",[38,216,218],{"id":217},"step-2-shape-each-row-into-a-document","Step 2 — Shape each row into a document",[10,220,221],{},"Write one function that turns a row into exactly the document the application expects. Dotted column names become nested objects, blanks are left out, numbers and dates get proper types, and the tag string becomes an array:",[43,223,225],{"className":100,"code":224,"language":102,"meta":48,"style":48},"import math\nfrom datetime import datetime\n\ndef to_document(row):\n    doc = {}\n    for col, value in row.items():\n        if value is None or (isinstance(value, float) and math.isnan(value)) or value is pd.NaT or value == \"\":\n            continue                                   # blank cell: leave the field out\n        if isinstance(value, pd.Timestamp):\n            value = value.to_pydatetime()\n        elif col == \"stock\":\n            value = int(value)\n        elif col == \"tags\":\n            value = [t.strip() for t in value.split(\";\") if t.strip()]\n        target = doc\n        *parents, leaf = col.split(\".\")\n        for p in parents:\n            target = target.setdefault(p, {})\n        target[leaf] = value\n    return doc\n\ndocs = [to_document(r) for r in df.to_dict(\"records\")]\nprint(docs[0])\nprint(docs[3])\nprint(docs[4])\n",[14,226,227,234,247,251,263,273,288,350,360,371,382,398,411,425,457,468,488,502,513,524,532,537,563,577,589],{"__ignoreMap":48},[52,228,229,231],{"class":54,"line":55},[52,230,110],{"class":109},[52,232,233],{"class":113}," math\n",[52,235,236,239,242,244],{"class":54,"line":123},[52,237,238],{"class":109},"from",[52,240,241],{"class":113}," datetime ",[52,243,110],{"class":109},[52,245,246],{"class":113}," datetime\n",[52,248,249],{"class":54,"line":130},[52,250,127],{"emptyLinePlaceholder":126},[52,252,253,256,260],{"class":54,"line":170},[52,254,255],{"class":109},"def",[52,257,259],{"class":258},"s_Opv"," to_document",[52,261,262],{"class":113},"(row):\n",[52,264,265,268,270],{"class":54,"line":179},[52,266,267],{"class":113},"    doc ",[52,269,136],{"class":109},[52,271,272],{"class":113}," {}\n",[52,274,276,279,282,285],{"class":54,"line":275},6,[52,277,278],{"class":109},"    for",[52,280,281],{"class":113}," col, value ",[52,283,284],{"class":109},"in",[52,286,287],{"class":113}," row.items():\n",[52,289,291,294,297,300,303,306,309,312,315,318,321,324,327,330,332,334,337,339,341,344,347],{"class":54,"line":290},7,[52,292,293],{"class":109},"        if",[52,295,296],{"class":113}," value ",[52,298,299],{"class":109},"is",[52,301,302],{"class":163}," None",[52,304,305],{"class":109}," or",[52,307,308],{"class":113}," (",[52,310,311],{"class":163},"isinstance",[52,313,314],{"class":113},"(value, ",[52,316,317],{"class":163},"float",[52,319,320],{"class":113},") ",[52,322,323],{"class":109},"and",[52,325,326],{"class":113}," math.isnan(value)) ",[52,328,329],{"class":109},"or",[52,331,296],{"class":113},[52,333,299],{"class":109},[52,335,336],{"class":113}," pd.NaT ",[52,338,329],{"class":109},[52,340,296],{"class":113},[52,342,343],{"class":109},"==",[52,345,346],{"class":62}," \"\"",[52,348,349],{"class":113},":\n",[52,351,353,356],{"class":54,"line":352},8,[52,354,355],{"class":109},"            continue",[52,357,359],{"class":358},"s-wDw","                                   # blank cell: leave the field out\n",[52,361,363,365,368],{"class":54,"line":362},9,[52,364,293],{"class":109},[52,366,367],{"class":163}," isinstance",[52,369,370],{"class":113},"(value, pd.Timestamp):\n",[52,372,374,377,379],{"class":54,"line":373},10,[52,375,376],{"class":113},"            value ",[52,378,136],{"class":109},[52,380,381],{"class":113}," value.to_pydatetime()\n",[52,383,385,388,391,393,396],{"class":54,"line":384},11,[52,386,387],{"class":109},"        elif",[52,389,390],{"class":113}," col ",[52,392,343],{"class":109},[52,394,395],{"class":62}," \"stock\"",[52,397,349],{"class":113},[52,399,401,403,405,408],{"class":54,"line":400},12,[52,402,376],{"class":113},[52,404,136],{"class":109},[52,406,407],{"class":163}," int",[52,409,410],{"class":113},"(value)\n",[52,412,414,416,418,420,423],{"class":54,"line":413},13,[52,415,387],{"class":109},[52,417,390],{"class":113},[52,419,343],{"class":109},[52,421,422],{"class":62}," \"tags\"",[52,424,349],{"class":113},[52,426,428,430,432,435,438,441,443,446,449,451,454],{"class":54,"line":427},14,[52,429,376],{"class":113},[52,431,136],{"class":109},[52,433,434],{"class":113}," [t.strip() ",[52,436,437],{"class":109},"for",[52,439,440],{"class":113}," t ",[52,442,284],{"class":109},[52,444,445],{"class":113}," value.split(",[52,447,448],{"class":62},"\";\"",[52,450,320],{"class":113},[52,452,453],{"class":109},"if",[52,455,456],{"class":113}," t.strip()]\n",[52,458,460,463,465],{"class":54,"line":459},15,[52,461,462],{"class":113},"        target ",[52,464,136],{"class":109},[52,466,467],{"class":113}," doc\n",[52,469,471,474,477,479,482,485],{"class":54,"line":470},16,[52,472,473],{"class":109},"        *",[52,475,476],{"class":113},"parents, leaf ",[52,478,136],{"class":109},[52,480,481],{"class":113}," col.split(",[52,483,484],{"class":62},"\".\"",[52,486,487],{"class":113},")\n",[52,489,491,494,497,499],{"class":54,"line":490},17,[52,492,493],{"class":109},"        for",[52,495,496],{"class":113}," p ",[52,498,284],{"class":109},[52,500,501],{"class":113}," parents:\n",[52,503,505,508,510],{"class":54,"line":504},18,[52,506,507],{"class":113},"            target ",[52,509,136],{"class":109},[52,511,512],{"class":113}," target.setdefault(p, {})\n",[52,514,516,519,521],{"class":54,"line":515},19,[52,517,518],{"class":113},"        target[leaf] ",[52,520,136],{"class":109},[52,522,523],{"class":113}," value\n",[52,525,527,530],{"class":54,"line":526},20,[52,528,529],{"class":109},"    return",[52,531,467],{"class":113},[52,533,535],{"class":54,"line":534},21,[52,536,127],{"emptyLinePlaceholder":126},[52,538,540,543,545,548,550,553,555,558,560],{"class":54,"line":539},22,[52,541,542],{"class":113},"docs ",[52,544,136],{"class":109},[52,546,547],{"class":113}," [to_document(r) ",[52,549,437],{"class":109},[52,551,552],{"class":113}," r ",[52,554,284],{"class":109},[52,556,557],{"class":113}," df.to_dict(",[52,559,193],{"class":62},[52,561,562],{"class":113},")]\n",[52,564,566,568,571,574],{"class":54,"line":565},23,[52,567,173],{"class":163},[52,569,570],{"class":113},"(docs[",[52,572,573],{"class":163},"0",[52,575,576],{"class":113},"])\n",[52,578,580,582,584,587],{"class":54,"line":579},24,[52,581,173],{"class":163},[52,583,570],{"class":113},[52,585,586],{"class":163},"3",[52,588,576],{"class":113},[52,590,592,594,596,599],{"class":54,"line":591},25,[52,593,173],{"class":163},[52,595,570],{"class":113},[52,597,598],{"class":163},"4",[52,600,576],{"class":113},[10,602,603,82,605,607,608,610,611,614,615,618,619,622,623,626,627,630],{},[14,604,81],{},[14,606,85],{}," become a nested ",[14,609,24],{}," object — the dotted-name convention mirrors how MongoDB addresses nested fields in queries, so the spreadsheet column names read naturally. The rake has no stock figure, so its document has no ",[14,612,613],{},"stock"," field at all, instead of ",[14,616,617],{},"stock: NaN","; the ladder has no launch date and no supplier city, so those fields are absent too. Leaving fields out is the MongoDB-idiomatic way to say \"unknown\", and queries such as ",[14,620,621],{},"{\"stock\": {\"$exists\": False}}"," find such documents easily. Converting pandas ",[14,624,625],{},"Timestamp"," values to Python ",[14,628,629],{},"datetime"," objects makes PyMongo store them as BSON dates.",[632,633,641,642,641,646,641,650,641,657,641,666,641,670,641,677,641,681,641,685,641,689,641,694,641,696,641,700,641,702,641,705,641,708,641,711,641,713,641,717,641,719,641,722,641,725,641,728,641,730,641,732,641,737,641,740,641,743,641,745,641,748,641,750,641,753,641,756,641,760,641,762,641,765,641,767,641,770,641,773,641,777,641,779,641,781,641,783,641,786,641,790,641,795,641,799,641,802,641,807,641,811,641,813,641,816,641,819,641,822,641,824,641,827,641,830,641,833,641,835,641,838,641,840,641,843,641,845,641,848,641,854,641,858,641,865,641,870,641,878],"svg",{"viewBox":634,"role":635,"ariaLabelledBy":636,"xmlns":639,"style":640},"58 0 644 248","img",[637,638],"mg-shape-t","mg-shape-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;max-width:644px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif","\n  ",[643,644,645],"title",{"id":637},"One spreadsheet row becomes one document",[647,648,649],"desc",{"id":638},"The rake row has a blank stock cell, a launch date, a single tag and supplier columns named with dots. The document leaves out the blank stock, stores the launch date as a date, the tags as an array and the supplier as a nested object.",[651,652],"rect",{"x":653,"y":573,"width":654,"height":655,"fill":656},"58","644","248","#ffffff",[651,658],{"x":659,"y":660,"width":661,"height":662,"rx":573,"fill":663,"stroke":664,"style":665},"74.1","40.0","32.0","24.0","#e7ebef","var(--line,#cdd5e6)","stroke-width:1px",[651,667],{"x":668,"y":660,"width":669,"height":662,"rx":573,"fill":663,"stroke":664,"style":665},"106.1","91.8",[671,672,676],"text",{"x":673,"y":674,"style":675},"152.0","57.0","font-size:11px;font-weight:600;fill:var(--muted,#5b6780);text-anchor:middle","A",[651,678],{"x":679,"y":660,"width":680,"height":662,"rx":573,"fill":663,"stroke":664,"style":665},"197.9","89.4",[671,682,684],{"x":683,"y":674,"style":675},"242.6","B",[651,686],{"x":659,"y":687,"width":661,"height":688,"rx":573,"fill":663,"stroke":664,"style":665},"64.0","28.0",[671,690,693],{"x":691,"y":692,"style":675},"90.1","83.0","1",[651,695],{"x":668,"y":687,"width":669,"height":688,"rx":573,"fill":656,"stroke":664,"style":665},[671,697,699],{"x":673,"y":692,"style":698},"font-size:10.5px;font-weight:700;fill:var(--text,#172033);text-anchor:middle","column",[651,701],{"x":679,"y":687,"width":680,"height":688,"rx":573,"fill":656,"stroke":664,"style":665},[671,703,704],{"x":683,"y":692,"style":698},"value",[651,706],{"x":659,"y":707,"width":661,"height":688,"rx":573,"fill":663,"stroke":664,"style":665},"92.0",[671,709,187],{"x":691,"y":710,"style":675},"111.0",[651,712],{"x":668,"y":707,"width":669,"height":688,"rx":573,"fill":656,"stroke":664,"style":665},[671,714,716],{"x":673,"y":710,"style":715},"font-size:10.5px;font-weight:400;fill:var(--text,#172033);text-anchor:middle","sku",[651,718],{"x":679,"y":707,"width":680,"height":688,"rx":573,"fill":656,"stroke":664,"style":665},[671,720,721],{"x":683,"y":710,"style":715},"RAK-04",[651,723],{"x":659,"y":724,"width":661,"height":688,"rx":573,"fill":663,"stroke":664,"style":665},"120.0",[671,726,586],{"x":691,"y":727,"style":675},"139.0",[651,729],{"x":668,"y":724,"width":669,"height":688,"rx":573,"fill":656,"stroke":664,"style":665},[671,731,613],{"x":673,"y":727,"style":715},[651,733],{"x":679,"y":724,"width":680,"height":688,"rx":573,"fill":734,"stroke":735,"style":736},"#fce9e9","var(--danger,#dc2626)","stroke-width:1.5px",[651,738],{"x":659,"y":739,"width":661,"height":688,"rx":573,"fill":663,"stroke":664,"style":665},"148.0",[671,741,598],{"x":691,"y":742,"style":675},"167.0",[651,744],{"x":668,"y":739,"width":669,"height":688,"rx":573,"fill":656,"stroke":664,"style":665},[671,746,747],{"x":673,"y":742,"style":715},"launched",[651,749],{"x":679,"y":739,"width":680,"height":688,"rx":573,"fill":656,"stroke":664,"style":665},[671,751,752],{"x":683,"y":742,"style":715},"2022-09-01",[651,754],{"x":659,"y":755,"width":661,"height":688,"rx":573,"fill":663,"stroke":664,"style":665},"176.0",[671,757,759],{"x":691,"y":758,"style":675},"195.0","5",[651,761],{"x":668,"y":755,"width":669,"height":688,"rx":573,"fill":656,"stroke":664,"style":665},[671,763,764],{"x":673,"y":758,"style":715},"tags",[651,766],{"x":679,"y":755,"width":680,"height":688,"rx":573,"fill":656,"stroke":664,"style":665},[671,768,769],{"x":683,"y":758,"style":715},"garden",[651,771],{"x":659,"y":772,"width":661,"height":688,"rx":573,"fill":663,"stroke":664,"style":665},"204.0",[671,774,776],{"x":691,"y":775,"style":675},"223.0","6",[651,778],{"x":668,"y":772,"width":669,"height":688,"rx":573,"fill":656,"stroke":664,"style":665},[671,780,85],{"x":673,"y":775,"style":715},[651,782],{"x":679,"y":772,"width":680,"height":688,"rx":573,"fill":656,"stroke":664,"style":665},[671,784,785],{"x":683,"y":775,"style":715},"Bristol",[651,787],{"x":788,"y":660,"width":789,"height":688,"rx":573,"fill":663,"stroke":664,"style":736},"470.4","74.0",[671,791,794],{"x":792,"y":793,"style":698},"507.4","59.0","field",[651,796],{"x":797,"y":660,"width":798,"height":688,"rx":573,"fill":656,"stroke":664,"style":665},"544.4","141.5",[671,800,704],{"x":801,"y":793,"style":698},"615.1",[651,803],{"x":788,"y":804,"width":789,"height":688,"rx":573,"fill":805,"stroke":806,"style":736},"68.0","#d9f4f1","var(--teal,#0f9488)",[671,808,716],{"x":792,"y":809,"style":810},"87.0","font-size:10.5px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle",[651,812],{"x":797,"y":804,"width":798,"height":688,"rx":573,"fill":656,"stroke":664,"style":665},[671,814,815],{"x":801,"y":809,"style":715},"'RAK-04'",[651,817],{"x":788,"y":818,"width":789,"height":688,"rx":573,"fill":805,"stroke":806,"style":736},"96.0",[671,820,747],{"x":792,"y":821,"style":810},"115.0",[651,823],{"x":797,"y":818,"width":798,"height":688,"rx":573,"fill":656,"stroke":664,"style":665},[671,825,826],{"x":801,"y":821,"style":715},"ISODate(2022-09-01)",[651,828],{"x":788,"y":829,"width":789,"height":688,"rx":573,"fill":805,"stroke":806,"style":736},"124.0",[671,831,764],{"x":792,"y":832,"style":810},"143.0",[651,834],{"x":797,"y":829,"width":798,"height":688,"rx":573,"fill":656,"stroke":664,"style":665},[671,836,837],{"x":801,"y":832,"style":715},"['garden']",[651,839],{"x":788,"y":673,"width":789,"height":688,"rx":573,"fill":805,"stroke":806,"style":736},[671,841,24],{"x":792,"y":842,"style":810},"171.0",[651,844],{"x":797,"y":673,"width":798,"height":688,"rx":573,"fill":656,"stroke":664,"style":665},[671,846,847],{"x":801,"y":842,"style":715},"{name, city}",[671,849,853],{"x":850,"y":851,"style":852},"180.7","29.0","font-size:12.5px;font-weight:700;fill:var(--muted,#5b6780);text-anchor:middle","Excel row",[671,855,857],{"x":856,"y":851,"style":852},"578.1","MongoDB document",[54,859],{"x1":860,"y1":861,"x2":862,"y2":861,"stroke":863,"style":864},"297.3","110.0","456.4","var(--brand,#5b5cf0)","stroke-width:2px",[866,867],"polygon",{"points":868,"fill":869},"460.4,110.0 451.4,115.0 451.4,105.0","#5b5cf0",[651,871],{"x":872,"y":873,"width":874,"height":875,"rx":876,"fill":656,"stroke":664,"style":877},"321.6","77.0","114.5","22.0","11","stroke-width:1.2px",[671,879,882],{"x":880,"y":707,"style":881},"378.8","font-size:11.5px;font-weight:400;fill:var(--brand-strong,#4338ca);text-anchor:middle;font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace","to_document()",[38,884,886],{"id":885},"step-3-upsert-on-the-business-key","Step 3 — Upsert on the business key",[10,888,889,890,893],{},"Inserting every time duplicates products on the second run. Instead, update the document with the same SKU if it exists and insert it otherwise. ",[14,891,892],{},"bulk_write"," sends all the operations in one round trip, and a unique index on the SKU guards against duplicates from any other source:",[43,895,897],{"className":100,"code":896,"language":102,"meta":48,"style":48},"from pymongo import MongoClient, UpdateOne\n\nclient = MongoClient(\"mongodb:\u002F\u002Flocalhost:27017\")\nproducts = client[\"shop\"][\"products\"]\nproducts.create_index(\"sku\", unique=True)\n\ndef load(docs):\n    ops = [UpdateOne({\"sku\": d[\"sku\"]}, {\"$set\": d, \"$currentDate\": {\"updated_at\": True}}, upsert=True) for d in docs]\n    result = products.bulk_write(ops, ordered=False)\n    return {\"inserted\": result.upserted_count, \"updated\": result.modified_count, \"matched\": result.matched_count}\n\nprint(\"first load: \", load(docs))\nprint(\"second load:\", load(docs))\nprint(\"documents:\", products.count_documents({}))\n",[14,898,899,911,915,930,952,971,975,985,1046,1066,1091,1095,1108,1119],{"__ignoreMap":48},[52,900,901,903,906,908],{"class":54,"line":55},[52,902,238],{"class":109},[52,904,905],{"class":113}," pymongo ",[52,907,110],{"class":109},[52,909,910],{"class":113}," MongoClient, UpdateOne\n",[52,912,913],{"class":54,"line":123},[52,914,127],{"emptyLinePlaceholder":126},[52,916,917,920,922,925,928],{"class":54,"line":130},[52,918,919],{"class":113},"client ",[52,921,136],{"class":109},[52,923,924],{"class":113}," MongoClient(",[52,926,927],{"class":62},"\"mongodb:\u002F\u002Flocalhost:27017\"",[52,929,487],{"class":113},[52,931,932,935,937,940,943,946,949],{"class":54,"line":170},[52,933,934],{"class":113},"products ",[52,936,136],{"class":109},[52,938,939],{"class":113}," client[",[52,941,942],{"class":62},"\"shop\"",[52,944,945],{"class":113},"][",[52,947,948],{"class":62},"\"products\"",[52,950,951],{"class":113},"]\n",[52,953,954,957,959,961,964,966,969],{"class":54,"line":179},[52,955,956],{"class":113},"products.create_index(",[52,958,157],{"class":62},[52,960,145],{"class":113},[52,962,963],{"class":148},"unique",[52,965,136],{"class":109},[52,967,968],{"class":163},"True",[52,970,487],{"class":113},[52,972,973],{"class":54,"line":275},[52,974,127],{"emptyLinePlaceholder":126},[52,976,977,979,982],{"class":54,"line":290},[52,978,255],{"class":109},[52,980,981],{"class":258}," load",[52,983,984],{"class":113},"(docs):\n",[52,986,987,990,992,995,997,1000,1002,1005,1008,1011,1014,1017,1020,1022,1024,1027,1030,1032,1034,1036,1038,1041,1043],{"class":54,"line":352},[52,988,989],{"class":113},"    ops ",[52,991,136],{"class":109},[52,993,994],{"class":113}," [UpdateOne({",[52,996,157],{"class":62},[52,998,999],{"class":113},": d[",[52,1001,157],{"class":62},[52,1003,1004],{"class":113},"]}, {",[52,1006,1007],{"class":62},"\"$set\"",[52,1009,1010],{"class":113},": d, ",[52,1012,1013],{"class":62},"\"$currentDate\"",[52,1015,1016],{"class":113},": {",[52,1018,1019],{"class":62},"\"updated_at\"",[52,1021,160],{"class":113},[52,1023,968],{"class":163},[52,1025,1026],{"class":113},"}}, ",[52,1028,1029],{"class":148},"upsert",[52,1031,136],{"class":109},[52,1033,968],{"class":163},[52,1035,320],{"class":113},[52,1037,437],{"class":109},[52,1039,1040],{"class":113}," d ",[52,1042,284],{"class":109},[52,1044,1045],{"class":113}," docs]\n",[52,1047,1048,1051,1053,1056,1059,1061,1064],{"class":54,"line":362},[52,1049,1050],{"class":113},"    result ",[52,1052,136],{"class":109},[52,1054,1055],{"class":113}," products.bulk_write(ops, ",[52,1057,1058],{"class":148},"ordered",[52,1060,136],{"class":109},[52,1062,1063],{"class":163},"False",[52,1065,487],{"class":113},[52,1067,1068,1070,1073,1076,1079,1082,1085,1088],{"class":54,"line":373},[52,1069,529],{"class":109},[52,1071,1072],{"class":113}," {",[52,1074,1075],{"class":62},"\"inserted\"",[52,1077,1078],{"class":113},": result.upserted_count, ",[52,1080,1081],{"class":62},"\"updated\"",[52,1083,1084],{"class":113},": result.modified_count, ",[52,1086,1087],{"class":62},"\"matched\"",[52,1089,1090],{"class":113},": result.matched_count}\n",[52,1092,1093],{"class":54,"line":384},[52,1094,127],{"emptyLinePlaceholder":126},[52,1096,1097,1099,1102,1105],{"class":54,"line":400},[52,1098,173],{"class":163},[52,1100,1101],{"class":113},"(",[52,1103,1104],{"class":62},"\"first load: \"",[52,1106,1107],{"class":113},", load(docs))\n",[52,1109,1110,1112,1114,1117],{"class":54,"line":413},[52,1111,173],{"class":163},[52,1113,1101],{"class":113},[52,1115,1116],{"class":62},"\"second load:\"",[52,1118,1107],{"class":113},[52,1120,1121,1123,1125,1128],{"class":54,"line":427},[52,1122,173],{"class":163},[52,1124,1101],{"class":113},[52,1126,1127],{"class":62},"\"documents:\"",[52,1129,1130],{"class":113},", products.count_documents({}))\n",[10,1132,1133,1134,1137,1138,1141,1142,1145,1146,1149,1150,1152,1153,1156],{},"The first load inserts five documents; loading the same file again matches all five and inserts nothing, so the collection still holds five products. That is what makes the import safe to schedule or rerun after a failure. ",[14,1135,1136],{},"$set"," replaces only the fields present in the document, which matters for the next step; ",[14,1139,1140],{},"$currentDate"," stamps each document with the time it was last loaded. ",[14,1143,1144],{},"ordered=False"," lets MongoDB carry on with the remaining operations if one fails, and the result reports how many were upserted, matched and actually modified. Note that the second load reports five ",[27,1147,1148],{},"modified"," documents even though no product data changed: ",[14,1151,1140],{}," sets a new timestamp, so every matched document really is modified. Drop the timestamp if you want ",[14,1154,1155],{},"modified_count"," to show only genuine data changes.",[38,1158,1160],{"id":1159},"step-4-reload-an-edited-spreadsheet","Step 4 — Reload an edited spreadsheet",[10,1162,1163],{},"The real test is the next version of the file: a price changes, a stock figure is filled in, and a new product is added. Simulate that and load again:",[43,1165,1167],{"className":100,"code":1166,"language":102,"meta":48,"style":48},"df2 = df.copy()\ndf2.loc[df2[\"sku\"] == \"SAW-02\", \"price\"] = 26.0\ndf2.loc[df2[\"sku\"] == \"RAK-04\", \"stock\"] = 30\ndf2.loc[len(df2)] = [\"GLV-06\", \"Work gloves\", 6.5, 120, pd.Timestamp(\"2026-09-01\"), \"safety\", \"SafeHands\", \"Leeds\"]\nprint(\"edited load:\", load([to_document(r) for r in df2.to_dict(\"records\")]))\nprint(products.find_one({\"sku\": \"RAK-04\"}, {\"_id\": 0, \"sku\": 1, \"stock\": 1}))\nprint(products.find_one({\"sku\": \"SAW-02\"}, {\"_id\": 0, \"price\": 1}), \"| total:\", products.count_documents({}))\n",[14,1168,1169,1179,1206,1231,1289,1315,1358],{"__ignoreMap":48},[52,1170,1171,1174,1176],{"class":54,"line":55},[52,1172,1173],{"class":113},"df2 ",[52,1175,136],{"class":109},[52,1177,1178],{"class":113}," df.copy()\n",[52,1180,1181,1184,1186,1189,1191,1194,1196,1199,1201,1203],{"class":54,"line":123},[52,1182,1183],{"class":113},"df2.loc[df2[",[52,1185,157],{"class":62},[52,1187,1188],{"class":113},"] ",[52,1190,343],{"class":109},[52,1192,1193],{"class":62}," \"SAW-02\"",[52,1195,145],{"class":113},[52,1197,1198],{"class":62},"\"price\"",[52,1200,1188],{"class":113},[52,1202,136],{"class":109},[52,1204,1205],{"class":163}," 26.0\n",[52,1207,1208,1210,1212,1214,1216,1219,1221,1224,1226,1228],{"class":54,"line":130},[52,1209,1183],{"class":113},[52,1211,157],{"class":62},[52,1213,1188],{"class":113},[52,1215,343],{"class":109},[52,1217,1218],{"class":62}," \"RAK-04\"",[52,1220,145],{"class":113},[52,1222,1223],{"class":62},"\"stock\"",[52,1225,1188],{"class":113},[52,1227,136],{"class":109},[52,1229,1230],{"class":163}," 30\n",[52,1232,1233,1236,1239,1242,1244,1247,1250,1252,1255,1257,1260,1262,1265,1268,1271,1274,1277,1279,1282,1284,1287],{"class":54,"line":170},[52,1234,1235],{"class":113},"df2.loc[",[52,1237,1238],{"class":163},"len",[52,1240,1241],{"class":113},"(df2)] ",[52,1243,136],{"class":109},[52,1245,1246],{"class":113}," [",[52,1248,1249],{"class":62},"\"GLV-06\"",[52,1251,145],{"class":113},[52,1253,1254],{"class":62},"\"Work gloves\"",[52,1256,145],{"class":113},[52,1258,1259],{"class":163},"6.5",[52,1261,145],{"class":113},[52,1263,1264],{"class":163},"120",[52,1266,1267],{"class":113},", pd.Timestamp(",[52,1269,1270],{"class":62},"\"2026-09-01\"",[52,1272,1273],{"class":113},"), ",[52,1275,1276],{"class":62},"\"safety\"",[52,1278,145],{"class":113},[52,1280,1281],{"class":62},"\"SafeHands\"",[52,1283,145],{"class":113},[52,1285,1286],{"class":62},"\"Leeds\"",[52,1288,951],{"class":113},[52,1290,1291,1293,1295,1298,1301,1303,1305,1307,1310,1312],{"class":54,"line":179},[52,1292,173],{"class":163},[52,1294,1101],{"class":113},[52,1296,1297],{"class":62},"\"edited load:\"",[52,1299,1300],{"class":113},", load([to_document(r) ",[52,1302,437],{"class":109},[52,1304,552],{"class":113},[52,1306,284],{"class":109},[52,1308,1309],{"class":113}," df2.to_dict(",[52,1311,193],{"class":62},[52,1313,1314],{"class":113},")]))\n",[52,1316,1317,1319,1322,1324,1326,1329,1332,1335,1337,1339,1341,1343,1345,1347,1349,1351,1353,1355],{"class":54,"line":275},[52,1318,173],{"class":163},[52,1320,1321],{"class":113},"(products.find_one({",[52,1323,157],{"class":62},[52,1325,160],{"class":113},[52,1327,1328],{"class":62},"\"RAK-04\"",[52,1330,1331],{"class":113},"}, {",[52,1333,1334],{"class":62},"\"_id\"",[52,1336,160],{"class":113},[52,1338,573],{"class":163},[52,1340,145],{"class":113},[52,1342,157],{"class":62},[52,1344,160],{"class":113},[52,1346,693],{"class":163},[52,1348,145],{"class":113},[52,1350,1223],{"class":62},[52,1352,160],{"class":113},[52,1354,693],{"class":163},[52,1356,1357],{"class":113},"}))\n",[52,1359,1360,1362,1364,1366,1368,1371,1373,1375,1377,1379,1381,1383,1385,1387,1390,1393],{"class":54,"line":290},[52,1361,173],{"class":163},[52,1363,1321],{"class":113},[52,1365,157],{"class":62},[52,1367,160],{"class":113},[52,1369,1370],{"class":62},"\"SAW-02\"",[52,1372,1331],{"class":113},[52,1374,1334],{"class":62},[52,1376,160],{"class":113},[52,1378,573],{"class":163},[52,1380,145],{"class":113},[52,1382,1198],{"class":62},[52,1384,160],{"class":113},[52,1386,693],{"class":163},[52,1388,1389],{"class":113},"}), ",[52,1391,1392],{"class":62},"\"| total:\"",[52,1394,1130],{"class":113},[10,1396,1397,1398,1400,1401,1404,1405,1408,1409,1412],{},"The new gloves are inserted, the saw's price and the rake's stock are updated, and nothing is duplicated — six products in total. One subtlety: ",[14,1399,1136],{}," never ",[27,1402,1403],{},"removes"," a field, so if someone clears a cell in the spreadsheet, the old value stays in MongoDB. If cleared cells should delete fields, add an ",[14,1406,1407],{},"$unset"," for every expected field the row is missing, or replace whole documents with ",[14,1410,1411],{},"ReplaceOne"," instead — the right choice depends on whether other systems also write to these documents.",[632,1414,641,1420,641,1423,641,1426,641,1430,641,1438,641,1449,641,1454,641,1458,641,1462,641,1465,641,1468,641,1475,641,1480,641,1486,641,1493,641,1497,641,1502,641,1504,641,1507,641,1511,641,1514,641,1517,641,1524,641,1528,641,1531,641,1533,641,1536,641,1540,641,1543,641,1548,641,1555,641,1559],{"viewBox":1415,"role":635,"ariaLabelledBy":1416,"xmlns":639,"style":1419},"4 0 752 272",[1417,1418],"mg-write-t","mg-write-d","width:100%;max-width:752px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif",[643,1421,1422],{"id":1417},"Which write for each import?",[647,1424,1425],{"id":1418},"Insert many duplicates documents on a rerun. Update with upsert and set changes the fields present and keeps others, so it is safe to rerun but never removes cleared fields. Replace with upsert writes the whole document from the row, removing fields that are blank in the sheet but overwriting fields other systems set.",[651,1427],{"x":598,"y":573,"width":1428,"height":1429,"fill":656},"752","272",[651,1431],{"x":1432,"y":1433,"width":1434,"height":1435,"rx":1436,"fill":1437,"stroke":863,"style":864},"240.0","16.0","280.0","75.5","12","#f0f4ff",[1439,1440,1442],"g",{"transform":1441},"translate(368.0,29.0) scale(1.000)",[1443,1444],"path",{"d":1445,"fill":1446,"stroke":1447,"style":1448},"M11 4a7 7 0 1 0 .01 0z M16 16l5 5","none","var(--brand-strong,#4338ca)","stroke-width:1.80px;stroke-linecap:round;stroke-linejoin:round",[671,1450,1453],{"x":1451,"y":1435,"style":1452},"380.0","font-size:13.5px;font-weight:700;fill:var(--brand-strong,#4338ca);text-anchor:middle","How should a reload behave?",[54,1455],{"x1":1451,"y1":1456,"x2":1451,"y2":1457,"stroke":863,"style":864},"91.5","111.5",[54,1459],{"x1":1460,"y1":1457,"x2":1461,"y2":1457,"stroke":863,"style":864},"134.7","625.3",[54,1463],{"x1":1460,"y1":1457,"x2":1460,"y2":1464,"stroke":863,"style":864},"157.5",[866,1466],{"points":1467,"fill":869},"134.7,161.5 129.7,152.5 139.7,152.5",[651,1469],{"x":1470,"y":1471,"width":1472,"height":1473,"rx":1474,"fill":656,"stroke":664,"style":877},"89.2","121.5","90.9","21.0","10",[671,1476,1479],{"x":1460,"y":1477,"style":1478},"136.0","font-size:11.5px;font-weight:700;fill:var(--muted,#5b6780);text-anchor:middle","one-off load",[651,1481],{"x":1482,"y":1483,"width":1484,"height":1485,"rx":1436,"fill":734,"stroke":735,"style":864},"20.0","163.5","229.3","93.0",[1439,1487,1489],{"transform":1488},"translate(122.7,176.5) scale(1.000)",[1443,1490],{"d":1491,"fill":1446,"stroke":1492,"style":1448},"M12 3.5l9.5 17h-19z M12 10v5 M12 17.8v.4","var(--accent-ink,#be185d)",[671,1494,1496],{"x":1460,"y":775,"style":1495},"font-size:13.5px;font-weight:700;fill:var(--accent-ink,#be185d);text-anchor:middle","insert_many",[671,1498,1501],{"x":1460,"y":1499,"style":1500},"242.5","font-size:11.5px;font-weight:400;fill:var(--text,#172033);text-anchor:middle","duplicates on rerun",[54,1503],{"x1":1451,"y1":1457,"x2":1451,"y2":1464,"stroke":863,"style":864},[866,1505],{"points":1506,"fill":869},"380.0,161.5 375.0,152.5 385.0,152.5",[651,1508],{"x":1509,"y":1471,"width":1510,"height":1473,"rx":1474,"fill":656,"stroke":664,"style":877},"288.1","183.8",[671,1512,1513],{"x":1451,"y":1477,"style":1478},"safe rerun, keep extra fields",[651,1515],{"x":1516,"y":1483,"width":1484,"height":1485,"rx":1436,"fill":805,"stroke":806,"style":864},"265.3",[1439,1518,1520],{"transform":1519},"translate(368.0,176.5) scale(1.000)",[1443,1521],{"d":1522,"fill":1446,"stroke":1523,"style":1448},"M20 11a8 8 0 1 0-2.3 5.7 M20 4v7h-7","var(--teal-ink,#0b6157)",[671,1525,1527],{"x":1451,"y":775,"style":1526},"font-size:13.5px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","UpdateOne + $set",[671,1529,1530],{"x":1451,"y":1499,"style":1500},"cleared cells not removed",[54,1532],{"x1":1461,"y1":1457,"x2":1461,"y2":1464,"stroke":863,"style":864},[866,1534],{"points":1535,"fill":869},"625.3,161.5 620.3,152.5 630.3,152.5",[651,1537],{"x":1538,"y":1471,"width":1539,"height":1473,"rx":1474,"fill":656,"stroke":664,"style":877},"547.2","156.2",[671,1541,1542],{"x":1461,"y":1477,"style":1478},"sheet is the whole truth",[651,1544],{"x":1545,"y":1483,"width":1484,"height":1485,"rx":1436,"fill":1546,"stroke":1547,"style":864},"510.7","#fdefd8","var(--gold,#b4740a)",[1439,1549,1551],{"transform":1550},"translate(613.3,176.5) scale(1.000)",[1443,1552],{"d":1553,"fill":1446,"stroke":1554,"style":1448},"M12 3l9 5-9 5-9-5z M3 13l9 5 9-5 M3 17l9 5 9-5","var(--gold-ink,#7a4e06)",[671,1556,1558],{"x":1461,"y":775,"style":1557},"font-size:13.5px;font-weight:700;fill:var(--gold-ink,#7a4e06);text-anchor:middle","ReplaceOne upsert",[671,1560,1561],{"x":1461,"y":1499,"style":1500},"blanks remove fields",[632,1563,641,1568,641,1571,641,1574,641,1577,641,1581,641,1588,641,1594,641,1598,641,1603,641,1606,641,1609,641,1615,641,1619,641,1623,641,1627,641,1631,641,1634,641,1637,641,1643,641,1647,641,1650,641,1653,641,1657,641,1660,641,1663,641,1669,641,1673],{"viewBox":1564,"role":635,"ariaLabelledBy":1565,"xmlns":639,"style":1419},"4 0 752 142",[1566,1567],"mg-flow-t","mg-flow-d",[643,1569,1570],{"id":1566},"An import that is safe to rerun",[647,1572,1573],{"id":1567},"Read the sheet with deliberate types, shape every row into a document, upsert each document on its SKU with bulk_write against a unique index, then compare the keys in the sheet and the collection.",[651,1575],{"x":598,"y":573,"width":1428,"height":1576,"fill":656},"142",[651,1578],{"x":1482,"y":1433,"width":1579,"height":1580,"rx":1436,"fill":663,"stroke":664,"style":864},"153.0","110.5",[1439,1582,1584],{"transform":1583},"translate(84.5,37.8) scale(1.000)",[1443,1585],{"d":1586,"fill":1446,"stroke":1587,"style":1448},"M4 4h16v16H4z M4 9h16 M4 14h16 M10 4v16","var(--text,#172033)",[671,1589,1593],{"x":1590,"y":1591,"style":1592},"96.5","84.2","font-size:13.5px;font-weight:700;fill:var(--text,#172033);text-anchor:middle","Read typed",[671,1595,1597],{"x":1590,"y":1596,"style":1500},"103.8","SKU as text, real dates",[54,1599],{"x1":1600,"y1":1601,"x2":1602,"y2":1601,"stroke":863,"style":864},"178.0","71.2","200.0",[866,1604],{"points":1605,"fill":869},"204.0,71.2 195.0,76.2 195.0,66.2",[651,1607],{"x":1608,"y":1433,"width":1579,"height":1580,"rx":1436,"fill":1437,"stroke":863,"style":864},"209.0",[1439,1610,1612],{"transform":1611},"translate(273.5,29.0) scale(1.000)",[1443,1613],{"d":1614,"fill":1446,"stroke":1447,"style":1448},"M9 8l-4 4 4 4 M15 8l4 4-4 4",[671,1616,1618],{"x":1617,"y":1435,"style":1452},"285.5","Shape documents",[671,1620,1622],{"x":1617,"y":1621,"style":1500},"95.0","nested, arrays, no",[671,1624,1626],{"x":1617,"y":1625,"style":1500},"112.5","blanks",[54,1628],{"x1":1629,"y1":1601,"x2":1630,"y2":1601,"stroke":863,"style":864},"367.0","389.0",[866,1632],{"points":1633,"fill":869},"393.0,71.2 384.0,76.2 384.0,66.2",[651,1635],{"x":1636,"y":1433,"width":1579,"height":1580,"rx":1436,"fill":805,"stroke":806,"style":864},"398.0",[1439,1638,1640],{"transform":1639},"translate(462.5,29.0) scale(1.000)",[1443,1641],{"d":1642,"fill":1446,"stroke":1523,"style":1448},"M5 6c0-1.7 3.1-3 7-3s7 1.3 7 3-3.1 3-7 3-7-1.3-7-3z M5 6v12c0 1.7 3.1 3 7 3s7-1.3 7-3V6 M5 12c0 1.7 3.1 3 7 3s7-1.3 7-3",[671,1644,1646],{"x":1645,"y":1435,"style":1526},"474.5","Upsert",[671,1648,1649],{"x":1645,"y":1621,"style":1500},"bulk_write + unique",[671,1651,1652],{"x":1645,"y":1625,"style":1500},"index",[54,1654],{"x1":1655,"y1":1601,"x2":1656,"y2":1601,"stroke":863,"style":864},"556.0","578.0",[866,1658],{"points":1659,"fill":869},"582.0,71.2 573.0,76.2 573.0,66.2",[651,1661],{"x":1662,"y":1433,"width":1579,"height":1580,"rx":1436,"fill":1546,"stroke":1547,"style":864},"587.0",[1439,1664,1666],{"transform":1665},"translate(651.5,37.8) scale(1.000)",[1443,1667],{"d":1668,"fill":1446,"stroke":1554,"style":1448},"M5 12.5l4.5 4.5L19 7",[671,1670,1672],{"x":1671,"y":1591,"style":1557},"663.5","Compare keys",[671,1674,1675],{"x":1671,"y":1596,"style":1500},"sheet vs collection",[38,1677,1679],{"id":1678},"step-5-check-what-landed","Step 5 — Check what landed",[10,1681,1682],{},"Finish with a summary that compares the collection against the spreadsheet, so a broken import is noticed:",[43,1684,1686],{"className":100,"code":1685,"language":102,"meta":48,"style":48},"expected = set(df2[\"sku\"])\nstored = {d[\"sku\"] for d in products.find({}, {\"sku\": 1})}\nprint(\"missing from MongoDB:\", sorted(expected - stored), \"| not in sheet:\", sorted(stored - expected))\nprint(\"no stock figure:\", [d[\"sku\"] for d in products.find({\"stock\": {\"$exists\": False}}, {\"sku\": 1})])\nprint(\"launched before 2023:\", [d[\"sku\"] for d in products.find({\"launched\": {\"$lt\": datetime(2023, 1, 1)}}, {\"sku\": 1})])\n",[14,1687,1688,1705,1737,1775,1823],{"__ignoreMap":48},[52,1689,1690,1693,1695,1698,1701,1703],{"class":54,"line":55},[52,1691,1692],{"class":113},"expected ",[52,1694,136],{"class":109},[52,1696,1697],{"class":163}," set",[52,1699,1700],{"class":113},"(df2[",[52,1702,157],{"class":62},[52,1704,576],{"class":113},[52,1706,1707,1710,1712,1715,1717,1719,1721,1723,1725,1728,1730,1732,1734],{"class":54,"line":123},[52,1708,1709],{"class":113},"stored ",[52,1711,136],{"class":109},[52,1713,1714],{"class":113}," {d[",[52,1716,157],{"class":62},[52,1718,1188],{"class":113},[52,1720,437],{"class":109},[52,1722,1040],{"class":113},[52,1724,284],{"class":109},[52,1726,1727],{"class":113}," products.find({}, {",[52,1729,157],{"class":62},[52,1731,160],{"class":113},[52,1733,693],{"class":163},[52,1735,1736],{"class":113},"})}\n",[52,1738,1739,1741,1743,1746,1748,1751,1754,1757,1760,1763,1765,1767,1770,1772],{"class":54,"line":130},[52,1740,173],{"class":163},[52,1742,1101],{"class":113},[52,1744,1745],{"class":62},"\"missing from MongoDB:\"",[52,1747,145],{"class":113},[52,1749,1750],{"class":163},"sorted",[52,1752,1753],{"class":113},"(expected ",[52,1755,1756],{"class":109},"-",[52,1758,1759],{"class":113}," stored), ",[52,1761,1762],{"class":62},"\"| not in sheet:\"",[52,1764,145],{"class":113},[52,1766,1750],{"class":163},[52,1768,1769],{"class":113},"(stored ",[52,1771,1756],{"class":109},[52,1773,1774],{"class":113}," expected))\n",[52,1776,1777,1779,1781,1784,1787,1789,1791,1793,1795,1797,1800,1802,1804,1807,1809,1811,1814,1816,1818,1820],{"class":54,"line":170},[52,1778,173],{"class":163},[52,1780,1101],{"class":113},[52,1782,1783],{"class":62},"\"no stock figure:\"",[52,1785,1786],{"class":113},", [d[",[52,1788,157],{"class":62},[52,1790,1188],{"class":113},[52,1792,437],{"class":109},[52,1794,1040],{"class":113},[52,1796,284],{"class":109},[52,1798,1799],{"class":113}," products.find({",[52,1801,1223],{"class":62},[52,1803,1016],{"class":113},[52,1805,1806],{"class":62},"\"$exists\"",[52,1808,160],{"class":113},[52,1810,1063],{"class":163},[52,1812,1813],{"class":113},"}}, {",[52,1815,157],{"class":62},[52,1817,160],{"class":113},[52,1819,693],{"class":163},[52,1821,1822],{"class":113},"})])\n",[52,1824,1825,1827,1829,1832,1834,1836,1838,1840,1842,1844,1846,1849,1851,1854,1857,1860,1862,1864,1866,1868,1871,1873,1875,1877],{"class":54,"line":179},[52,1826,173],{"class":163},[52,1828,1101],{"class":113},[52,1830,1831],{"class":62},"\"launched before 2023:\"",[52,1833,1786],{"class":113},[52,1835,157],{"class":62},[52,1837,1188],{"class":113},[52,1839,437],{"class":109},[52,1841,1040],{"class":113},[52,1843,284],{"class":109},[52,1845,1799],{"class":113},[52,1847,1848],{"class":62},"\"launched\"",[52,1850,1016],{"class":113},[52,1852,1853],{"class":62},"\"$lt\"",[52,1855,1856],{"class":113},": datetime(",[52,1858,1859],{"class":163},"2023",[52,1861,145],{"class":113},[52,1863,693],{"class":163},[52,1865,145],{"class":113},[52,1867,693],{"class":163},[52,1869,1870],{"class":113},")}}, {",[52,1872,157],{"class":62},[52,1874,160],{"class":113},[52,1876,693],{"class":163},[52,1878,1822],{"class":113},[10,1880,1881,1882,1885,1886,1889,1890,1894],{},"Every SKU in the sheet is in the collection and vice versa; the query for documents without a stock field finds none now that the rake's stock was filled in; and a date-range query works because the launch dates were stored as real dates — with strings, ",[14,1883,1884],{},"$lt"," would compare text. Documents in MongoDB that are ",[27,1887,1888],{},"not"," in the sheet any more are reported rather than deleted: whether a product dropped from the spreadsheet should disappear from the application is a business decision, not something an import should do silently. Exporting the collection back to Excel follows the reverse path, flattening nested fields into dotted columns — see ",[32,1891,1893],{"href":1892},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-sql-query-results-to-excel-with-python\u002F","Export SQL Query Results to Excel with Python"," for the writing side.",[38,1896,1898],{"id":1897},"common-pitfalls","Common pitfalls",[1900,1901,1902,1912,1920,1926,1932],"ul",{},[1903,1904,1905,1911],"li",{},[1906,1907,1908,36],"strong",{},[14,1909,1910],{},"insert_many(df.to_dict(\"records\"))"," NaN values, wrong types and duplicates on every rerun.",[1903,1913,1914,1917,1918,36],{},[1906,1915,1916],{},"Storing dates as strings."," Range queries then compare text; convert to ",[14,1919,629],{},[1903,1921,1922,1925],{},[1906,1923,1924],{},"Flat columns for nested data."," Use dotted column names and build nested objects.",[1903,1927,1928,1931],{},[1906,1929,1930],{},"No unique index on the key."," Duplicates can still arrive from other writers.",[1903,1933,1934,1940,1941,207,1943,1945],{},[1906,1935,1936,1937,1939],{},"Expecting ",[14,1938,1136],{}," to clear fields."," Use ",[14,1942,1407],{},[14,1944,1411],{}," if blanks should remove values.",[38,1947,1949],{"id":1948},"conclusion","Conclusion",[10,1951,1952,1953,1955,1956,1959],{},"To load Excel data into MongoDB, read the sheet with deliberate types, shape each row into the document the application expects — nested objects from dotted column names, lists from delimited text, real dates and integers, blank cells left out — and write with ",[14,1954,892],{}," of ",[14,1957,1958],{},"UpdateOne(..., upsert=True)"," keyed on a business identifier with a unique index. The import is then safe to rerun, picks up edits and new rows, and can be checked by comparing the keys in the sheet and the collection.",[38,1961,1963],{"id":1962},"frequently-asked-questions","Frequently asked questions",[10,1965,1966,1969],{},[1906,1967,1968],{},"How do I import an Excel file into MongoDB with Python?","\nRead the sheet with pandas, convert each row into a dictionary shaped like the target document, and write the documents with PyMongo, ideally with bulk_write and UpdateOne upserts on a key field.",[10,1971,1972,1975],{},[1906,1973,1974],{},"How do I stop NaN values being stored in MongoDB?","\nSkip blank cells when building each document so the field is left out, rather than passing NaN from pandas into the document.",[10,1977,1978,1981],{},[1906,1979,1980],{},"How do I avoid duplicate documents when re-importing an Excel file?","\nUse UpdateOne with upsert=True filtered on a business key such as the SKU, and create a unique index on that key.",[10,1983,1984,1987],{},[1906,1985,1986],{},"How do I create nested MongoDB fields from Excel columns?","\nName the columns with dots, such as supplier.name and supplier.city, and split each name on the dot to build nested dictionaries.",[38,1989,1991],{"id":1990},"related","Related",[1900,1993,1994,2000,2007,2014],{},[1903,1995,1996,1997,1999],{},"Up one level: ",[32,1998,35],{"href":34}," — getting data in and out.",[1903,2001,2002,2006],{},[32,2003,2005],{"href":2004},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fupsert-excel-rows-into-a-database-table\u002F","Upsert Excel Rows into a Database Table"," — the same idea for SQL databases.",[1903,2008,2009,2013],{},[32,2010,2012],{"href":2011},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fvalidate-excel-rows-with-pydantic\u002F","Validate Excel Rows with Pydantic"," — checking rows before they are loaded.",[1903,2015,2016,2020],{},[32,2017,2019],{"href":2018},"\u002Fgetting-started-with-python-excel-automation\u002Fhandling-excel-file-formats-and-conversions\u002Fconvert-an-excel-sheet-to-json-with-python\u002F","Convert an Excel Sheet to JSON with Python"," — documents without a database.",[2022,2023,2024],"style",{},"html pre.shiki code .sMTad, html code.shiki .sMTad{--shiki-default:#6F42C1;--shiki-dark:#FFB757}html pre.shiki code .srMev, html code.shiki .srMev{--shiki-default:#032F62;--shiki-dark:#ADDCFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .s-kum, html code.shiki .s-kum{--shiki-default:#D73A49;--shiki-dark:#FF9492}html pre.shiki code .skGVy, html code.shiki .skGVy{--shiki-default:#24292E;--shiki-dark:#F0F3F6}html pre.shiki code .sa561, html code.shiki .sa561{--shiki-default:#E36209;--shiki-dark:#FFB757}html pre.shiki code .sP0c6, html code.shiki .sP0c6{--shiki-default:#005CC5;--shiki-dark:#91CBFF}html pre.shiki code .s_Opv, html code.shiki .s_Opv{--shiki-default:#6F42C1;--shiki-dark:#DBB7FF}html pre.shiki code .s-wDw, html code.shiki .s-wDw{--shiki-default:#6A737D;--shiki-dark:#BDC4CC}",{"title":48,"searchDepth":123,"depth":123,"links":2026},[2027,2028,2029,2030,2031,2032,2033,2034,2035,2036],{"id":40,"depth":123,"text":41},{"id":93,"depth":123,"text":94},{"id":217,"depth":123,"text":218},{"id":885,"depth":123,"text":886},{"id":1159,"depth":123,"text":1160},{"id":1678,"depth":123,"text":1679},{"id":1897,"depth":123,"text":1898},{"id":1948,"depth":123,"text":1949},{"id":1962,"depth":123,"text":1963},{"id":1990,"depth":123,"text":1991},"2026-10-02","Turn spreadsheet rows into well-shaped MongoDB documents with pandas and PyMongo: nested fields from dotted column names, real dates and numbers instead of strings, blanks left out rather than stored as NaN, upserts on a business key with bulk_write so reloading a file is safe, a unique index, and a check of what was inserted, updated and skipped.","md",[2041,2043,2045,2047],{"q":1968,"a":2042},"Read the sheet with pandas, convert each row into a dictionary shaped like the target document, and write the documents with PyMongo, ideally with bulk_write and UpdateOne upserts on a key field.",{"q":1974,"a":2044},"Skip blank cells when building each document so the field is left out, rather than passing NaN from pandas into the document.",{"q":1980,"a":2046},"Use UpdateOne with upsert=True filtered on a business key such as the SKU, and create a unique index on that key.",{"q":1986,"a":2048},"Name the columns with dots, such as supplier.name and supplier.city, and split each name on the dot to build nested dictionaries.","load excel data into mongodb with python",null,{"breadcrumb":2052},[2053,2056,2059],{"name":2054,"item":2055},"Home","\u002F",{"name":2057,"item":2058},"Advanced Data Transformation and Cleaning for Excel Automation","\u002Fadvanced-data-transformation-and-cleaning\u002F",{"name":35,"item":34},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fload-excel-data-into-mongodb-with-python",{"title":2062,"description":2063},"Load Excel Data into MongoDB with Python and PyMongo","Import Excel rows into MongoDB: dotted columns to nested documents, typed dates and numbers, omit blanks instead of NaN, idempotent upserts with bulk_write and UpdateOne, a unique index and an import summary.","load-excel-data-into-mongodb-with-python","advanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fload-excel-data-into-mongodb-with-python\u002Findex",[2067],"data-sources","how-to","HkCVkCKK8cXlI69675Uu_t0Hx_0toVum5s8XuMLkGeI",[2071,2075],{"title":2072,"path":2073,"stem":2074,"children":-1},"Load an Excel File into a SQL Database with pandas","\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fload-an-excel-file-into-a-sql-database-with-pandas","advanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fload-an-excel-file-into-a-sql-database-with-pandas\u002Findex",{"title":2076,"path":2077,"stem":2078,"children":-1},"Load Excel into SQLite with a Defined Schema","\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fload-excel-into-sqlite-with-a-defined-schema","advanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fload-excel-into-sqlite-with-a-defined-schema\u002Findex",[2080,2082],{"path":2081,"title":2057},"\u002Fadvanced-data-transformation-and-cleaning",{"path":2083,"title":35},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases",[],[2086,2089,2092,2095,2098,2101],{"path":2087,"title":2088,"kind":2068},"\u002Fautomating-reporting-workflows\u002Fpublishing-excel-reports-to-cloud-storage\u002Fread-an-excel-file-directly-from-amazon-s3-with-pandas","Read an Excel File Directly from Amazon S3 with pandas",{"path":2090,"title":2091,"kind":2068},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fcache-api-responses-when-building-excel-reports","Cache API Responses When Building Excel Reports",{"path":2093,"title":2094,"kind":2068},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fcompare-an-excel-file-with-a-database-table","Compare an Excel File with a Database Table",{"path":2096,"title":2097,"kind":2068},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fdownload-new-excel-files-from-an-sftp-server","Download New Excel Files from an SFTP Server",{"path":2099,"title":2100,"kind":2068},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-a-large-database-table-to-excel-in-chunks","Export a Large Database Table to Excel in Chunks",{"path":2102,"title":2103,"kind":2068},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-database-lookup-tables-as-excel-dropdown-lists","Export Database Lookup Tables as Excel Dropdown Lists",1790972096791]