[{"data":1,"prerenderedAt":2086},["ShallowReactive",2],{"doc:\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Ffetch-paginated-api-data-into-excel":3,"surround:\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Ffetch-paginated-api-data-into-excel":2054,"ancestors:\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Ffetch-paginated-api-data-into-excel":2062,"children:\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Ffetch-paginated-api-data-into-excel":2067,"related:\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Ffetch-paginated-api-data-into-excel":2068},{"id":4,"title":5,"body":6,"dateModified":2021,"datePublished":2021,"description":2022,"extension":2023,"faq":2024,"keyword":2033,"kind":2034,"meta":2035,"navigation":131,"path":2044,"seo":2045,"slug":2048,"stem":2049,"topics":2050,"type":2052,"__hash__":2053},"docs\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Ffetch-paginated-api-data-into-excel\u002Findex.md","Fetch Paginated API Data into Excel",{"type":7,"value":8,"toc":2009},"minimark",[9,24,29,60,90,94,101,442,471,669,673,680,903,913,992,996,1002,1188,1207,1211,1218,1455,1482,1676,1680,1683,1865,1876,1880,1918,1922,1944,1948,1954,1960,1966,1972,1976,2005],[10,11,12,13,17,18,23],"p",{},"Almost every API that returns lists — orders from a shop platform, tickets from a helpdesk, invoices from an accounting system — returns them a page at a time: 50 or 100 records per request, with some way to ask for the next page. A script that calls the endpoint once and writes the result to Excel silently exports only the first page, and the report looks complete. Fetching everything means looping until the API says there is nothing more, and doing it robustly means coping with the three things that go wrong on long runs: rate limits, temporary server errors, and records that shift between pages while you read. This page builds a fetcher for the common pagination styles with ",[14,15,16],"code",{},"requests",", flattens the JSON with pandas, and checks the result before writing the workbook. It belongs to ",[19,20,22],"a",{"href":21},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002F","Moving Data Between Excel and Databases",".",[25,26,28],"h2",{"id":27},"prerequisites","Prerequisites",[30,31,36],"pre",{"className":32,"code":33,"language":34,"meta":35,"style":35},"language-bash shiki shiki-themes github-light github-dark-high-contrast","pip install requests pandas openpyxl\n","bash","",[14,37,38],{"__ignoreMap":35},[39,40,43,47,51,54,57],"span",{"class":41,"line":42},"line",1,[39,44,46],{"class":45},"sMTad","pip",[39,48,50],{"class":49},"srMev"," install",[39,52,53],{"class":49}," requests",[39,55,56],{"class":49}," pandas",[39,58,59],{"class":49}," openpyxl\n",[10,61,62,63,66,67,70,71,74,75,78,79,82,83,86,87,89],{},"The examples call an orders API with 237 orders, exposed in two styles: ",[14,64,65],{},"\u002Fv1\u002Forders"," with page numbers and a total count, and ",[14,68,69],{},"\u002Fv2\u002Forders"," with cursor pagination and a ",[14,72,73],{},"next"," link. To keep the page self-contained, the outputs were produced against a small local stand-in for the API (its address is in ",[14,76,77],{},"BASE",") that also returns one ",[14,80,81],{},"429 Too Many Requests"," and one ",[14,84,85],{},"503"," along the way, as real APIs do. Replace ",[14,88,77],{}," and add your authentication header to use the code with a real service.",[25,91,93],{"id":92},"step-1-a-session-that-retries-politely","Step 1 — A session that retries politely",[10,95,96,97,100],{},"Every request goes through one ",[14,98,99],{},"requests.Session"," with a timeout, and a small wrapper that waits and retries when the server says \"too many requests\" or has a temporary failure:",[30,102,106],{"className":103,"code":104,"language":105,"meta":35,"style":35},"language-python shiki shiki-themes github-light github-dark-high-contrast","import time\nimport requests\n\nsession = requests.Session()\nsession.headers.update({\"Accept\": \"application\u002Fjson\"})      # add Authorization here\n\ndef get_json(url, params=None, retries=5):\n    for attempt in range(retries):\n        resp = session.get(url, params=params, timeout=30)\n        if resp.status_code in (429, 500, 502, 503, 504):\n            wait = float(resp.headers.get(\"Retry-After\", 2 ** attempt))\n            print(f\"  {resp.status_code} — waiting {wait:.0f}s before retry {attempt + 1}\")\n            time.sleep(wait)\n            continue\n        resp.raise_for_status()\n        return resp.json()\n    raise RuntimeError(f\"gave up on {url} after {retries} attempts\")\n","python",[14,107,108,118,126,133,145,167,172,202,220,251,290,318,377,383,389,395,404],{"__ignoreMap":35},[39,109,110,114],{"class":41,"line":42},[39,111,113],{"class":112},"s-kum","import",[39,115,117],{"class":116},"skGVy"," time\n",[39,119,121,123],{"class":41,"line":120},2,[39,122,113],{"class":112},[39,124,125],{"class":116}," requests\n",[39,127,129],{"class":41,"line":128},3,[39,130,132],{"emptyLinePlaceholder":131},true,"\n",[39,134,136,139,142],{"class":41,"line":135},4,[39,137,138],{"class":116},"session ",[39,140,141],{"class":112},"=",[39,143,144],{"class":116}," requests.Session()\n",[39,146,148,151,154,157,160,163],{"class":41,"line":147},5,[39,149,150],{"class":116},"session.headers.update({",[39,152,153],{"class":49},"\"Accept\"",[39,155,156],{"class":116},": ",[39,158,159],{"class":49},"\"application\u002Fjson\"",[39,161,162],{"class":116},"})      ",[39,164,166],{"class":165},"s-wDw","# add Authorization here\n",[39,168,170],{"class":41,"line":169},6,[39,171,132],{"emptyLinePlaceholder":131},[39,173,175,178,182,185,187,191,194,196,199],{"class":41,"line":174},7,[39,176,177],{"class":112},"def",[39,179,181],{"class":180},"s_Opv"," get_json",[39,183,184],{"class":116},"(url, params",[39,186,141],{"class":112},[39,188,190],{"class":189},"sP0c6","None",[39,192,193],{"class":116},", retries",[39,195,141],{"class":112},[39,197,198],{"class":189},"5",[39,200,201],{"class":116},"):\n",[39,203,205,208,211,214,217],{"class":41,"line":204},8,[39,206,207],{"class":112},"    for",[39,209,210],{"class":116}," attempt ",[39,212,213],{"class":112},"in",[39,215,216],{"class":189}," range",[39,218,219],{"class":116},"(retries):\n",[39,221,223,226,228,231,235,237,240,243,245,248],{"class":41,"line":222},9,[39,224,225],{"class":116},"        resp ",[39,227,141],{"class":112},[39,229,230],{"class":116}," session.get(url, ",[39,232,234],{"class":233},"sa561","params",[39,236,141],{"class":112},[39,238,239],{"class":116},"params, ",[39,241,242],{"class":233},"timeout",[39,244,141],{"class":112},[39,246,247],{"class":189},"30",[39,249,250],{"class":116},")\n",[39,252,254,257,260,262,265,268,271,274,276,279,281,283,285,288],{"class":41,"line":253},10,[39,255,256],{"class":112},"        if",[39,258,259],{"class":116}," resp.status_code ",[39,261,213],{"class":112},[39,263,264],{"class":116}," (",[39,266,267],{"class":189},"429",[39,269,270],{"class":116},", ",[39,272,273],{"class":189},"500",[39,275,270],{"class":116},[39,277,278],{"class":189},"502",[39,280,270],{"class":116},[39,282,85],{"class":189},[39,284,270],{"class":116},[39,286,287],{"class":189},"504",[39,289,201],{"class":116},[39,291,293,296,298,301,304,307,309,312,315],{"class":41,"line":292},11,[39,294,295],{"class":116},"            wait ",[39,297,141],{"class":112},[39,299,300],{"class":189}," float",[39,302,303],{"class":116},"(resp.headers.get(",[39,305,306],{"class":49},"\"Retry-After\"",[39,308,270],{"class":116},[39,310,311],{"class":189},"2",[39,313,314],{"class":112}," **",[39,316,317],{"class":116}," attempt))\n",[39,319,321,324,327,330,333,337,340,343,346,348,351,354,356,359,361,364,367,370,372,375],{"class":41,"line":320},12,[39,322,323],{"class":189},"            print",[39,325,326],{"class":116},"(",[39,328,329],{"class":112},"f",[39,331,332],{"class":49},"\"  ",[39,334,336],{"class":335},"sSjpA","{",[39,338,339],{"class":116},"resp.status_code",[39,341,342],{"class":335},"}",[39,344,345],{"class":49}," — waiting ",[39,347,336],{"class":335},[39,349,350],{"class":116},"wait",[39,352,353],{"class":112},":.0f",[39,355,342],{"class":335},[39,357,358],{"class":49},"s before retry ",[39,360,336],{"class":335},[39,362,363],{"class":116},"attempt ",[39,365,366],{"class":112},"+",[39,368,369],{"class":189}," 1",[39,371,342],{"class":335},[39,373,374],{"class":49},"\"",[39,376,250],{"class":116},[39,378,380],{"class":41,"line":379},13,[39,381,382],{"class":116},"            time.sleep(wait)\n",[39,384,386],{"class":41,"line":385},14,[39,387,388],{"class":112},"            continue\n",[39,390,392],{"class":41,"line":391},15,[39,393,394],{"class":116},"        resp.raise_for_status()\n",[39,396,398,401],{"class":41,"line":397},16,[39,399,400],{"class":112},"        return",[39,402,403],{"class":116}," resp.json()\n",[39,405,407,410,413,415,417,420,422,425,427,430,432,435,437,440],{"class":41,"line":406},17,[39,408,409],{"class":112},"    raise",[39,411,412],{"class":189}," RuntimeError",[39,414,326],{"class":116},[39,416,329],{"class":112},[39,418,419],{"class":49},"\"gave up on ",[39,421,336],{"class":335},[39,423,424],{"class":116},"url",[39,426,342],{"class":335},[39,428,429],{"class":49}," after ",[39,431,336],{"class":335},[39,433,434],{"class":116},"retries",[39,436,342],{"class":335},[39,438,439],{"class":49}," attempts\"",[39,441,250],{"class":116},[10,443,444,446,447,450,451,454,455,458,459,462,463,466,467,23],{},[14,445,267],{}," means the client is calling too fast; the ",[14,448,449],{},"Retry-After"," header says how many seconds to wait, and honouring it is both polite and the fastest way to get going again. The ",[14,452,453],{},"5xx"," codes in the list are usually temporary, so they get the same treatment, with exponential back-off when no header is given. Every other error — ",[14,456,457],{},"401"," for a bad token, ",[14,460,461],{},"404"," for a wrong URL — is raised immediately by ",[14,464,465],{},"raise_for_status()",", because retrying will not fix it. The timeout prevents a stalled connection from hanging the job forever. A general retry pattern for whole jobs is in ",[19,468,470],{"href":469},"\u002Fautomating-reporting-workflows\u002Ferror-handling-and-logging-in-excel-automation\u002Fretry-a-failed-excel-report-job-in-python\u002F","Retry a Failed Excel Report Job in Python",[472,473,481,482,481,486,481,490,481,498,481,508,481,519,481,525,481,529,481,533,481,536,481,541,481,550,481,555,481,563,481,570,481,575,481,580,481,583,481,586,481,590,481,592,481,597,481,604,481,608,481,611,481,614,481,617,481,621,481,623,481,626,481,632,481,635,481,638,481,640,481,643,481,647,481,650,481,655,481,662,481,666],"svg",{"viewBox":474,"role":475,"ariaLabelledBy":476,"xmlns":479,"style":480},"4 0 752 272","img",[477,478],"pg-status-t","pg-status-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;max-width:752px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif","\n  ",[483,484,485],"title",{"id":477},"What to do with each response",[487,488,489],"desc",{"id":478},"A 200 response is used. A 429 means wait for the Retry-After seconds and retry. A 500, 502, 503 or 504 means back off and retry a few times. Any other error such as 401 or 404 fails immediately because retrying will not help.",[491,492],"rect",{"x":493,"y":494,"width":495,"height":496,"fill":497},"4","0","752","272","#ffffff",[491,499],{"x":500,"y":501,"width":502,"height":503,"rx":504,"fill":505,"stroke":506,"style":507},"240.0","16.0","280.0","75.5","12","#f0f4ff","var(--brand,#5b5cf0)","stroke-width:2px",[509,510,512],"g",{"transform":511},"translate(368.0,29.0) scale(1.000)",[513,514],"path",{"d":515,"fill":516,"stroke":517,"style":518},"M8 7l-5 5 5 5 M16 7l5 5-5 5 M14 4l-4 16","none","var(--brand-strong,#4338ca)","stroke-width:1.80px;stroke-linecap:round;stroke-linejoin:round",[520,521,524],"text",{"x":522,"y":503,"style":523},"380.0","font-size:13.5px;font-weight:700;fill:var(--brand-strong,#4338ca);text-anchor:middle","Response status?",[41,526],{"x1":522,"y1":527,"x2":522,"y2":528,"stroke":506,"style":507},"91.5","111.5",[41,530],{"x1":531,"y1":528,"x2":532,"y2":528,"stroke":506,"style":507},"104.0","656.0",[41,534],{"x1":531,"y1":528,"x2":531,"y2":535,"stroke":506,"style":507},"157.5",[537,538],"polygon",{"points":539,"fill":540},"104.0,161.5 99.0,152.5 109.0,152.5","#5b5cf0",[491,542],{"x":543,"y":544,"width":545,"height":546,"rx":547,"fill":497,"stroke":548,"style":549},"83.1","121.5","41.9","21.0","10","var(--line,#cdd5e6)","stroke-width:1.2px",[520,551,554],{"x":531,"y":552,"style":553},"136.0","font-size:11.5px;font-weight:700;fill:var(--muted,#5b6780);text-anchor:middle","200",[491,556],{"x":557,"y":558,"width":559,"height":560,"rx":504,"fill":561,"stroke":562,"style":507},"20.0","163.5","168.0","93.0","#d9f4f1","var(--teal,#0f9488)",[509,564,566],{"transform":565},"translate(92.0,176.5) scale(1.000)",[513,567],{"d":568,"fill":516,"stroke":569,"style":518},"M5 12.5l4.5 4.5L19 7","var(--teal-ink,#0b6157)",[520,571,574],{"x":531,"y":572,"style":573},"223.0","font-size:13.5px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","use the page",[520,576,579],{"x":531,"y":577,"style":578},"242.5","font-size:11.5px;font-weight:400;fill:var(--text,#172033);text-anchor:middle","continue the loop",[41,581],{"x1":582,"y1":528,"x2":582,"y2":535,"stroke":506,"style":507},"288.0",[537,584],{"points":585,"fill":540},"288.0,161.5 283.0,152.5 293.0,152.5",[491,587],{"x":588,"y":544,"width":589,"height":546,"rx":547,"fill":497,"stroke":548,"style":549},"267.2","41.6",[520,591,267],{"x":582,"y":552,"style":553},[491,593],{"x":594,"y":558,"width":559,"height":560,"rx":504,"fill":595,"stroke":596,"style":507},"204.0","#fdefd8","var(--gold,#b4740a)",[509,598,600],{"transform":599},"translate(276.0,176.5) scale(1.000)",[513,601],{"d":602,"fill":516,"stroke":603,"style":518},"M12 3a9 9 0 1 0 .01 0z M12 7v5l3.5 2","var(--gold-ink,#7a4e06)",[520,605,607],{"x":582,"y":572,"style":606},"font-size:13.5px;font-weight:700;fill:var(--gold-ink,#7a4e06);text-anchor:middle","wait Retry-After",[520,609,610],{"x":582,"y":577,"style":578},"then retry",[41,612],{"x1":613,"y1":528,"x2":613,"y2":535,"stroke":506,"style":507},"472.0",[537,615],{"points":616,"fill":540},"472.0,161.5 467.0,152.5 477.0,152.5",[491,618],{"x":619,"y":544,"width":620,"height":546,"rx":547,"fill":497,"stroke":548,"style":549},"452.2","39.5",[520,622,453],{"x":613,"y":552,"style":553},[491,624],{"x":625,"y":558,"width":559,"height":560,"rx":504,"fill":505,"stroke":506,"style":507},"388.0",[509,627,629],{"transform":628},"translate(460.0,176.5) scale(1.000)",[513,630],{"d":631,"fill":516,"stroke":517,"style":518},"M20 11a8 8 0 1 0-2.3 5.7 M20 4v7h-7",[520,633,634],{"x":613,"y":572,"style":523},"back off and retry",[520,636,637],{"x":613,"y":577,"style":578},"up to 5 attempts",[41,639],{"x1":532,"y1":528,"x2":532,"y2":535,"stroke":506,"style":507},[537,641],{"points":642,"fill":540},"656.0,161.5 651.0,152.5 661.0,152.5",[491,644],{"x":645,"y":544,"width":646,"height":546,"rx":547,"fill":497,"stroke":548,"style":549},"611.3","89.3",[520,648,649],{"x":532,"y":552,"style":553},"401 \u002F 404 …",[491,651],{"x":652,"y":558,"width":559,"height":560,"rx":504,"fill":653,"stroke":654,"style":507},"572.0","#fce9e9","var(--danger,#dc2626)",[509,656,658],{"transform":657},"translate(644.0,176.5) scale(1.000)",[513,659],{"d":660,"fill":516,"stroke":661,"style":518},"M6 6l12 12 M18 6L6 18","var(--accent-ink,#be185d)",[520,663,665],{"x":532,"y":572,"style":664},"font-size:13.5px;font-weight:700;fill:var(--accent-ink,#be185d);text-anchor:middle","fail now",[520,667,668],{"x":532,"y":577,"style":578},"retrying won't help",[25,670,672],{"id":671},"step-2-page-number-pagination","Step 2 — Page-number pagination",[10,674,675,676,679],{},"The simplest style takes a ",[14,677,678],{},"page"," number and a page size, and often reports the total. Loop until a page comes back short or empty:",[30,681,683],{"className":103,"code":682,"language":105,"meta":35,"style":35},"def fetch_pages(url, per_page=50):\n    records, page = [], 1\n    while True:\n        body = get_json(url, params={\"page\": page, \"per_page\": per_page})\n        batch = body[\"data\"]\n        records.extend(batch)\n        print(f\"page {page}: {len(batch)} records\")\n        if len(batch) \u003C per_page:\n            return records, body.get(\"meta\", {}).get(\"total\")\n        page += 1\n\norders_v1, reported_total = fetch_pages(f\"{BASE}\u002Fv1\u002Forders\")\nprint(len(orders_v1), \"records fetched; API reports\", reported_total)\n",[14,684,685,702,715,726,754,770,775,810,826,845,856,860,885],{"__ignoreMap":35},[39,686,687,689,692,695,697,700],{"class":41,"line":42},[39,688,177],{"class":112},[39,690,691],{"class":180}," fetch_pages",[39,693,694],{"class":116},"(url, per_page",[39,696,141],{"class":112},[39,698,699],{"class":189},"50",[39,701,201],{"class":116},[39,703,704,707,709,712],{"class":41,"line":120},[39,705,706],{"class":116},"    records, page ",[39,708,141],{"class":112},[39,710,711],{"class":116}," [], ",[39,713,714],{"class":189},"1\n",[39,716,717,720,723],{"class":41,"line":128},[39,718,719],{"class":112},"    while",[39,721,722],{"class":189}," True",[39,724,725],{"class":116},":\n",[39,727,728,731,733,736,738,740,742,745,748,751],{"class":41,"line":135},[39,729,730],{"class":116},"        body ",[39,732,141],{"class":112},[39,734,735],{"class":116}," get_json(url, ",[39,737,234],{"class":233},[39,739,141],{"class":112},[39,741,336],{"class":116},[39,743,744],{"class":49},"\"page\"",[39,746,747],{"class":116},": page, ",[39,749,750],{"class":49},"\"per_page\"",[39,752,753],{"class":116},": per_page})\n",[39,755,756,759,761,764,767],{"class":41,"line":147},[39,757,758],{"class":116},"        batch ",[39,760,141],{"class":112},[39,762,763],{"class":116}," body[",[39,765,766],{"class":49},"\"data\"",[39,768,769],{"class":116},"]\n",[39,771,772],{"class":41,"line":169},[39,773,774],{"class":116},"        records.extend(batch)\n",[39,776,777,780,782,784,787,789,791,793,795,797,800,803,805,808],{"class":41,"line":174},[39,778,779],{"class":189},"        print",[39,781,326],{"class":116},[39,783,329],{"class":112},[39,785,786],{"class":49},"\"page ",[39,788,336],{"class":335},[39,790,678],{"class":116},[39,792,342],{"class":335},[39,794,156],{"class":49},[39,796,336],{"class":335},[39,798,799],{"class":189},"len",[39,801,802],{"class":116},"(batch)",[39,804,342],{"class":335},[39,806,807],{"class":49}," records\"",[39,809,250],{"class":116},[39,811,812,814,817,820,823],{"class":41,"line":204},[39,813,256],{"class":112},[39,815,816],{"class":189}," len",[39,818,819],{"class":116},"(batch) ",[39,821,822],{"class":112},"\u003C",[39,824,825],{"class":116}," per_page:\n",[39,827,828,831,834,837,840,843],{"class":41,"line":222},[39,829,830],{"class":112},"            return",[39,832,833],{"class":116}," records, body.get(",[39,835,836],{"class":49},"\"meta\"",[39,838,839],{"class":116},", {}).get(",[39,841,842],{"class":49},"\"total\"",[39,844,250],{"class":116},[39,846,847,850,853],{"class":41,"line":253},[39,848,849],{"class":116},"        page ",[39,851,852],{"class":112},"+=",[39,854,855],{"class":189}," 1\n",[39,857,858],{"class":41,"line":292},[39,859,132],{"emptyLinePlaceholder":131},[39,861,862,865,867,870,872,874,876,878,880,883],{"class":41,"line":320},[39,863,864],{"class":116},"orders_v1, reported_total ",[39,866,141],{"class":112},[39,868,869],{"class":116}," fetch_pages(",[39,871,329],{"class":112},[39,873,374],{"class":49},[39,875,336],{"class":335},[39,877,77],{"class":189},[39,879,342],{"class":335},[39,881,882],{"class":49},"\u002Fv1\u002Forders\"",[39,884,250],{"class":116},[39,886,887,890,892,894,897,900],{"class":41,"line":379},[39,888,889],{"class":189},"print",[39,891,326],{"class":116},[39,893,799],{"class":189},[39,895,896],{"class":116},"(orders_v1), ",[39,898,899],{"class":49},"\"records fetched; API reports\"",[39,901,902],{"class":116},", reported_total)\n",[10,904,905,906,908,909,912],{},"Five pages of up to 50 records cover all 237 orders, and the loop stops when the fifth page returns only 37. Stopping on a short page is more reliable than computing the number of pages from the total up front, because the total can change while you fetch. The ",[14,907,267],{}," from the stand-in appears in the output as a wait and a retry, and the fetch carries on. When the API offers it, request the largest page size it allows: fewer requests mean fewer chances to be rate-limited and a faster export. Offset-based APIs (",[14,910,911],{},"offset=0, 50, 100 …",") work the same way, with the offset increasing by the page size instead of the page number by one.",[472,914,481,919,481,922,481,925,481,928,481,933,481,940,481,946,481,952,481,957,481,961,481,965,481,968,481,974,481,979,481,983,481,986,481,989],{"viewBox":915,"role":475,"ariaLabelledBy":916,"xmlns":479,"style":480},"4 0 752 207",[917,918],"pg-styles-t","pg-styles-d",[483,920,921],{"id":917},"The two common pagination styles",[487,923,924],{"id":918},"Page or offset pagination asks for page numbers or offsets and stops on a short page; records can be skipped or repeated if data changes while fetching. Cursor pagination follows a next link or after-id cursor until has_more is false and stays consistent when records are added.",[491,926],{"x":493,"y":494,"width":495,"height":927,"fill":497},"207",[491,929],{"x":557,"y":501,"width":930,"height":931,"rx":932,"fill":595,"stroke":596,"style":507},"351.0","175.0","14",[509,934,936],{"transform":935},"translate(182.5,29.0) scale(1.083)",[513,937],{"d":938,"fill":516,"stroke":603,"style":939},"M5 9h14 M4 15h14 M10 4l-2 16 M16 4l-2 16","stroke-width:1.66px;stroke-linecap:round;stroke-linejoin:round",[520,941,945],{"x":942,"y":943,"style":944},"195.5","78.0","font-size:14px;font-weight:700;fill:var(--gold-ink,#7a4e06);text-anchor:middle","page \u002F offset",[41,947],{"x1":948,"y1":949,"x2":950,"y2":949,"stroke":596,"style":951},"36.0","91.0","355.0","stroke-width:1px",[520,953,956],{"x":942,"y":954,"style":955},"111.0","font-size:12px;font-weight:400;fill:var(--text,#172033);text-anchor:middle","?page=3&per_page=50",[520,958,960],{"x":942,"y":959,"style":955},"137.0","stop on a short page",[520,962,964],{"x":942,"y":963,"style":955},"163.0","inserts shift records between pages",[491,966],{"x":967,"y":501,"width":930,"height":931,"rx":932,"fill":561,"stroke":562,"style":507},"389.0",[509,969,971],{"transform":970},"translate(551.5,29.0) scale(1.083)",[513,972],{"d":973,"fill":516,"stroke":569,"style":939},"M10 14a4 4 0 0 0 5.7 0l3-3a4 4 0 0 0-5.7-5.7l-1 1 M14 10a4 4 0 0 0-5.7 0l-3 3a4 4 0 0 0 5.7 5.7l1-1",[520,975,978],{"x":976,"y":943,"style":977},"564.5","font-size:14px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","cursor \u002F next link",[41,980],{"x1":981,"y1":949,"x2":982,"y2":949,"stroke":562,"style":951},"405.0","724.0",[520,984,985],{"x":976,"y":954,"style":955},"?starting_after=5149",[520,987,988],{"x":976,"y":959,"style":955},"follow 'next' until has_more is false",[520,990,991],{"x":976,"y":963,"style":955},"stable while data changes",[25,993,995],{"id":994},"step-3-cursor-pagination-with-next-links","Step 3 — Cursor pagination with next links",[10,997,998,999,1001],{},"Many modern APIs use cursors instead: each response includes a token or a full ",[14,1000,73],{}," URL pointing to the page after the last record you received. Follow it until the API says there is no more:",[30,1003,1005],{"className":103,"code":1004,"language":105,"meta":35,"style":35},"def fetch_cursor(first_url):\n    records, url = [], first_url\n    while url:\n        body = get_json(url if url.startswith(\"http\") else BASE + url)\n        records.extend(body[\"data\"])\n        url = body.get(\"next\") if body.get(\"has_more\") else None\n    return records\n\norders_v2 = fetch_cursor(f\"{BASE}\u002Fv2\u002Forders?limit=100\")\nprint(len(orders_v2), \"records via cursor; first id\", orders_v2[0][\"id\"], \"last id\", orders_v2[-1][\"id\"])\n",[14,1006,1007,1017,1027,1034,1067,1077,1106,1114,1118,1143],{"__ignoreMap":35},[39,1008,1009,1011,1014],{"class":41,"line":42},[39,1010,177],{"class":112},[39,1012,1013],{"class":180}," fetch_cursor",[39,1015,1016],{"class":116},"(first_url):\n",[39,1018,1019,1022,1024],{"class":41,"line":120},[39,1020,1021],{"class":116},"    records, url ",[39,1023,141],{"class":112},[39,1025,1026],{"class":116}," [], first_url\n",[39,1028,1029,1031],{"class":41,"line":128},[39,1030,719],{"class":112},[39,1032,1033],{"class":116}," url:\n",[39,1035,1036,1038,1040,1043,1046,1049,1052,1055,1058,1061,1064],{"class":41,"line":135},[39,1037,730],{"class":116},[39,1039,141],{"class":112},[39,1041,1042],{"class":116}," get_json(url ",[39,1044,1045],{"class":112},"if",[39,1047,1048],{"class":116}," url.startswith(",[39,1050,1051],{"class":49},"\"http\"",[39,1053,1054],{"class":116},") ",[39,1056,1057],{"class":112},"else",[39,1059,1060],{"class":189}," BASE",[39,1062,1063],{"class":112}," +",[39,1065,1066],{"class":116}," url)\n",[39,1068,1069,1072,1074],{"class":41,"line":147},[39,1070,1071],{"class":116},"        records.extend(body[",[39,1073,766],{"class":49},[39,1075,1076],{"class":116},"])\n",[39,1078,1079,1082,1084,1087,1090,1092,1094,1096,1099,1101,1103],{"class":41,"line":169},[39,1080,1081],{"class":116},"        url ",[39,1083,141],{"class":112},[39,1085,1086],{"class":116}," body.get(",[39,1088,1089],{"class":49},"\"next\"",[39,1091,1054],{"class":116},[39,1093,1045],{"class":112},[39,1095,1086],{"class":116},[39,1097,1098],{"class":49},"\"has_more\"",[39,1100,1054],{"class":116},[39,1102,1057],{"class":112},[39,1104,1105],{"class":189}," None\n",[39,1107,1108,1111],{"class":41,"line":174},[39,1109,1110],{"class":112},"    return",[39,1112,1113],{"class":116}," records\n",[39,1115,1116],{"class":41,"line":204},[39,1117,132],{"emptyLinePlaceholder":131},[39,1119,1120,1123,1125,1128,1130,1132,1134,1136,1138,1141],{"class":41,"line":222},[39,1121,1122],{"class":116},"orders_v2 ",[39,1124,141],{"class":112},[39,1126,1127],{"class":116}," fetch_cursor(",[39,1129,329],{"class":112},[39,1131,374],{"class":49},[39,1133,336],{"class":335},[39,1135,77],{"class":189},[39,1137,342],{"class":335},[39,1139,1140],{"class":49},"\u002Fv2\u002Forders?limit=100\"",[39,1142,250],{"class":116},[39,1144,1145,1147,1149,1151,1154,1157,1160,1162,1165,1168,1171,1174,1176,1179,1182,1184,1186],{"class":41,"line":253},[39,1146,889],{"class":189},[39,1148,326],{"class":116},[39,1150,799],{"class":189},[39,1152,1153],{"class":116},"(orders_v2), ",[39,1155,1156],{"class":49},"\"records via cursor; first id\"",[39,1158,1159],{"class":116},", orders_v2[",[39,1161,494],{"class":189},[39,1163,1164],{"class":116},"][",[39,1166,1167],{"class":49},"\"id\"",[39,1169,1170],{"class":116},"], ",[39,1172,1173],{"class":49},"\"last id\"",[39,1175,1159],{"class":116},[39,1177,1178],{"class":112},"-",[39,1180,1181],{"class":189},"1",[39,1183,1164],{"class":116},[39,1185,1167],{"class":49},[39,1187,1076],{"class":116},[10,1189,1190,1191,1195,1196,1199,1200,1202,1203,1206],{},"Cursor pagination is the more reliable style: because each request asks for records ",[1192,1193,1194],"em",{},"after"," the last one seen, a new order inserted while you fetch does not shift the remaining pages, and nothing is skipped or repeated. Some APIs return the next link in the HTTP ",[14,1197,1198],{},"Link"," header instead of the body; ",[14,1201,16],{}," parses that into ",[14,1204,1205],{},"resp.links[\"next\"][\"url\"]",". Either way the loop is the same: follow the pointer until there is none.",[25,1208,1210],{"id":1209},"step-4-flatten-the-json-into-a-table","Step 4 — Flatten the JSON into a table",[10,1212,1213,1214,1217],{},"API records are nested — here, a total with amount and currency, a customer object and a list of order lines. ",[14,1215,1216],{},"pd.json_normalize"," flattens nested objects into dotted columns; lists need a decision:",[30,1219,1221],{"className":103,"code":1220,"language":105,"meta":35,"style":35},"import pandas as pd\n\ndf = pd.json_normalize(orders_v2)\ndf[\"created\"] = pd.to_datetime(df[\"created\"]).dt.tz_convert(\"Europe\u002FBerlin\").dt.tz_localize(None)\ndf[\"line count\"] = df[\"lines\"].str.len()\ndf = df.drop(columns=\"lines\").rename(columns={\"total.amount\": \"amount\", \"total.currency\": \"currency\",\n                                              \"customer.id\": \"customer\", \"customer.country\": \"country\"})\nprint(df.dtypes.to_dict())\nprint(df.head(3).to_string(index=False))\n\nlines = pd.json_normalize(orders_v2, record_path=\"lines\", meta=[\"id\"])\nprint(lines.head(3).to_string(index=False))\n",[14,1222,1223,1236,1240,1250,1281,1301,1347,1370,1377,1401,1405,1436],{"__ignoreMap":35},[39,1224,1225,1227,1230,1233],{"class":41,"line":42},[39,1226,113],{"class":112},[39,1228,1229],{"class":116}," pandas ",[39,1231,1232],{"class":112},"as",[39,1234,1235],{"class":116}," pd\n",[39,1237,1238],{"class":41,"line":120},[39,1239,132],{"emptyLinePlaceholder":131},[39,1241,1242,1245,1247],{"class":41,"line":128},[39,1243,1244],{"class":116},"df ",[39,1246,141],{"class":112},[39,1248,1249],{"class":116}," pd.json_normalize(orders_v2)\n",[39,1251,1252,1255,1258,1261,1263,1266,1268,1271,1274,1277,1279],{"class":41,"line":135},[39,1253,1254],{"class":116},"df[",[39,1256,1257],{"class":49},"\"created\"",[39,1259,1260],{"class":116},"] ",[39,1262,141],{"class":112},[39,1264,1265],{"class":116}," pd.to_datetime(df[",[39,1267,1257],{"class":49},[39,1269,1270],{"class":116},"]).dt.tz_convert(",[39,1272,1273],{"class":49},"\"Europe\u002FBerlin\"",[39,1275,1276],{"class":116},").dt.tz_localize(",[39,1278,190],{"class":189},[39,1280,250],{"class":116},[39,1282,1283,1285,1288,1290,1292,1295,1298],{"class":41,"line":147},[39,1284,1254],{"class":116},[39,1286,1287],{"class":49},"\"line count\"",[39,1289,1260],{"class":116},[39,1291,141],{"class":112},[39,1293,1294],{"class":116}," df[",[39,1296,1297],{"class":49},"\"lines\"",[39,1299,1300],{"class":116},"].str.len()\n",[39,1302,1303,1305,1307,1310,1313,1315,1317,1320,1322,1324,1326,1329,1331,1334,1336,1339,1341,1344],{"class":41,"line":169},[39,1304,1244],{"class":116},[39,1306,141],{"class":112},[39,1308,1309],{"class":116}," df.drop(",[39,1311,1312],{"class":233},"columns",[39,1314,141],{"class":112},[39,1316,1297],{"class":49},[39,1318,1319],{"class":116},").rename(",[39,1321,1312],{"class":233},[39,1323,141],{"class":112},[39,1325,336],{"class":116},[39,1327,1328],{"class":49},"\"total.amount\"",[39,1330,156],{"class":116},[39,1332,1333],{"class":49},"\"amount\"",[39,1335,270],{"class":116},[39,1337,1338],{"class":49},"\"total.currency\"",[39,1340,156],{"class":116},[39,1342,1343],{"class":49},"\"currency\"",[39,1345,1346],{"class":116},",\n",[39,1348,1349,1352,1354,1357,1359,1362,1364,1367],{"class":41,"line":174},[39,1350,1351],{"class":49},"                                              \"customer.id\"",[39,1353,156],{"class":116},[39,1355,1356],{"class":49},"\"customer\"",[39,1358,270],{"class":116},[39,1360,1361],{"class":49},"\"customer.country\"",[39,1363,156],{"class":116},[39,1365,1366],{"class":49},"\"country\"",[39,1368,1369],{"class":116},"})\n",[39,1371,1372,1374],{"class":41,"line":204},[39,1373,889],{"class":189},[39,1375,1376],{"class":116},"(df.dtypes.to_dict())\n",[39,1378,1379,1381,1384,1387,1390,1393,1395,1398],{"class":41,"line":222},[39,1380,889],{"class":189},[39,1382,1383],{"class":116},"(df.head(",[39,1385,1386],{"class":189},"3",[39,1388,1389],{"class":116},").to_string(",[39,1391,1392],{"class":233},"index",[39,1394,141],{"class":112},[39,1396,1397],{"class":189},"False",[39,1399,1400],{"class":116},"))\n",[39,1402,1403],{"class":41,"line":253},[39,1404,132],{"emptyLinePlaceholder":131},[39,1406,1407,1410,1412,1415,1418,1420,1422,1424,1427,1429,1432,1434],{"class":41,"line":292},[39,1408,1409],{"class":116},"lines ",[39,1411,141],{"class":112},[39,1413,1414],{"class":116}," pd.json_normalize(orders_v2, ",[39,1416,1417],{"class":233},"record_path",[39,1419,141],{"class":112},[39,1421,1297],{"class":49},[39,1423,270],{"class":116},[39,1425,1426],{"class":233},"meta",[39,1428,141],{"class":112},[39,1430,1431],{"class":116},"[",[39,1433,1167],{"class":49},[39,1435,1076],{"class":116},[39,1437,1438,1440,1443,1445,1447,1449,1451,1453],{"class":41,"line":320},[39,1439,889],{"class":189},[39,1441,1442],{"class":116},"(lines.head(",[39,1444,1386],{"class":189},[39,1446,1389],{"class":116},[39,1448,1392],{"class":233},[39,1450,141],{"class":112},[39,1452,1397],{"class":189},[39,1454,1400],{"class":116},[10,1456,1457,1460,1461,1464,1465,1468,1469,1472,1473,1476,1477,1481],{},[14,1458,1459],{},"json_normalize"," turns ",[14,1462,1463],{},"total.amount"," and ",[14,1466,1467],{},"customer.country"," into ordinary columns. The order lines are a list per order, which cannot sit in one cell sensibly, so the orders table keeps only a count, and a second call with ",[14,1470,1471],{},"record_path=\"lines\""," produces a separate lines table with the order ID repeated — the shape you would put on a second sheet. Timestamps arrive in UTC with a ",[14,1474,1475],{},"Z"," suffix; converting them to local time and dropping the zone is necessary because Excel has no time zones and openpyxl refuses zone-aware datetimes. Nested JSON in general is covered in ",[19,1478,1480],{"href":1479},"\u002Fgetting-started-with-python-excel-automation\u002Fhandling-excel-file-formats-and-conversions\u002Fconvert-an-excel-sheet-to-json-with-python\u002F","Convert an Excel Sheet to JSON with Python",", which goes the other way.",[472,1483,481,1489,481,1492,481,1495,481,1499,481,1505,481,1509,481,1515,481,1519,481,1523,481,1527,481,1530,481,1532,481,1536,481,1538,481,1541,481,1544,481,1546,481,1548,481,1552,481,1554,481,1557,481,1560,481,1563,481,1565,481,1568,481,1570,481,1573,481,1576,481,1579,481,1581,481,1584,481,1586,481,1589,481,1592,481,1595,481,1597,481,1600,481,1602,481,1605,481,1610,481,1615,481,1619,481,1622,481,1625,481,1630,481,1632,481,1635,481,1638,481,1642,481,1644,481,1647,481,1653,481,1657,481,1662,481,1665,481,1672],{"viewBox":1484,"role":475,"ariaLabelledBy":1485,"xmlns":479,"style":1488},"3 0 753 220",[1486,1487],"pg-flat-t","pg-flat-d","width:100%;max-width:753px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif",[483,1490,1491],{"id":1486},"A nested order becomes rows on two sheets",[487,1493,1494],{"id":1487},"Each order has nested total and customer objects and a list of lines. json_normalize turns the nested objects into columns of the orders table and the lines into a separate table that repeats the order id.",[491,1496],{"x":1386,"y":494,"width":1497,"height":1498,"fill":497},"753","220",[491,1500],{"x":557,"y":1501,"width":1502,"height":1503,"rx":494,"fill":1504,"stroke":548,"style":951},"40.0","32.0","24.0","#e7ebef",[491,1506],{"x":1507,"y":1501,"width":1508,"height":1503,"rx":494,"fill":1504,"stroke":548,"style":951},"52.0","73.2",[520,1510,1514],{"x":1511,"y":1512,"style":1513},"88.6","57.0","font-size:11px;font-weight:600;fill:var(--muted,#5b6780);text-anchor:middle","A",[491,1516],{"x":1517,"y":1501,"width":1518,"height":1503,"rx":494,"fill":1504,"stroke":548,"style":951},"125.2","126.6",[520,1520,1522],{"x":1521,"y":1512,"style":1513},"188.5","B",[491,1524],{"x":557,"y":1525,"width":1502,"height":1526,"rx":494,"fill":1504,"stroke":548,"style":951},"64.0","28.0",[520,1528,1181],{"x":948,"y":1529,"style":1513},"83.0",[491,1531],{"x":1507,"y":1525,"width":1508,"height":1526,"rx":494,"fill":497,"stroke":548,"style":951},[520,1533,1535],{"x":1511,"y":1529,"style":1534},"font-size:10.5px;font-weight:700;fill:var(--text,#172033);text-anchor:middle","key",[491,1537],{"x":1517,"y":1525,"width":1518,"height":1526,"rx":494,"fill":497,"stroke":548,"style":951},[520,1539,1540],{"x":1521,"y":1529,"style":1534},"value",[491,1542],{"x":557,"y":1543,"width":1502,"height":1526,"rx":494,"fill":1504,"stroke":548,"style":951},"92.0",[520,1545,311],{"x":948,"y":954,"style":1513},[491,1547],{"x":1507,"y":1543,"width":1508,"height":1526,"rx":494,"fill":497,"stroke":548,"style":951},[520,1549,1551],{"x":1511,"y":954,"style":1550},"font-size:10.5px;font-weight:400;fill:var(--text,#172033);text-anchor:middle","id",[491,1553],{"x":1517,"y":1543,"width":1518,"height":1526,"rx":494,"fill":497,"stroke":548,"style":951},[520,1555,1556],{"x":1521,"y":954,"style":1550},"5000",[491,1558],{"x":557,"y":1559,"width":1502,"height":1526,"rx":494,"fill":1504,"stroke":548,"style":951},"120.0",[520,1561,1386],{"x":948,"y":1562,"style":1513},"139.0",[491,1564],{"x":1507,"y":1559,"width":1508,"height":1526,"rx":494,"fill":497,"stroke":548,"style":951},[520,1566,1567],{"x":1511,"y":1562,"style":1550},"total",[491,1569],{"x":1517,"y":1559,"width":1518,"height":1526,"rx":494,"fill":497,"stroke":548,"style":951},[520,1571,1572],{"x":1521,"y":1562,"style":1550},"{amount, currency}",[491,1574],{"x":557,"y":1575,"width":1502,"height":1526,"rx":494,"fill":1504,"stroke":548,"style":951},"148.0",[520,1577,493],{"x":948,"y":1578,"style":1513},"167.0",[491,1580],{"x":1507,"y":1575,"width":1508,"height":1526,"rx":494,"fill":497,"stroke":548,"style":951},[520,1582,1583],{"x":1511,"y":1578,"style":1550},"customer",[491,1585],{"x":1517,"y":1575,"width":1518,"height":1526,"rx":494,"fill":497,"stroke":548,"style":951},[520,1587,1588],{"x":1521,"y":1578,"style":1550},"{id, country}",[491,1590],{"x":557,"y":1591,"width":1502,"height":1526,"rx":494,"fill":1504,"stroke":548,"style":951},"176.0",[520,1593,198],{"x":948,"y":1594,"style":1513},"195.0",[491,1596],{"x":1507,"y":1591,"width":1508,"height":1526,"rx":494,"fill":497,"stroke":548,"style":951},[520,1598,1599],{"x":1511,"y":1594,"style":1550},"lines",[491,1601],{"x":1517,"y":1591,"width":1518,"height":1526,"rx":494,"fill":497,"stroke":548,"style":951},[520,1603,1604],{"x":1521,"y":1594,"style":1550},"[{sku, qty}]",[491,1606],{"x":1607,"y":1501,"width":1608,"height":1526,"rx":494,"fill":1504,"stroke":548,"style":1609},"444.1","59.6","stroke-width:1.5px",[520,1611,1614],{"x":1612,"y":1613,"style":1534},"473.9","59.0","sheet",[491,1616],{"x":1617,"y":1501,"width":1618,"height":1526,"rx":494,"fill":497,"stroke":548,"style":951},"503.7","236.3",[520,1620,1312],{"x":1621,"y":1613,"style":1534},"621.8",[491,1623],{"x":1607,"y":1624,"width":1608,"height":1526,"rx":494,"fill":561,"stroke":562,"style":1609},"68.0",[520,1626,1629],{"x":1612,"y":1627,"style":1628},"87.0","font-size:10.5px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","Orders",[491,1631],{"x":1617,"y":1624,"width":1618,"height":1526,"rx":494,"fill":497,"stroke":548,"style":951},[520,1633,1634],{"x":1621,"y":1627,"style":1550},"id, amount, currency, customer, country",[491,1636],{"x":1607,"y":1637,"width":1608,"height":1526,"rx":494,"fill":561,"stroke":562,"style":1609},"96.0",[520,1639,1641],{"x":1612,"y":1640,"style":1628},"115.0","Lines",[491,1643],{"x":1617,"y":1637,"width":1618,"height":1526,"rx":494,"fill":497,"stroke":548,"style":951},[520,1645,1646],{"x":1621,"y":1640,"style":1550},"id, sku, qty",[520,1648,1652],{"x":1649,"y":1650,"style":1651},"135.9","29.0","font-size:12.5px;font-weight:700;fill:var(--muted,#5b6780);text-anchor:middle","JSON record",[520,1654,1656],{"x":1655,"y":1650,"style":1651},"592.0","tables",[41,1658],{"x1":1659,"y1":1660,"x2":1661,"y2":1660,"stroke":506,"style":507},"261.8","82.0","430.1",[537,1663],{"points":1664,"fill":540},"434.1,82.0 425.1,87.0 425.1,77.0",[491,1666],{"x":1667,"y":1668,"width":1669,"height":1670,"rx":1671,"fill":497,"stroke":548,"style":549},"287.1","49.0","121.8","22.0","11",[520,1673,1459],{"x":1674,"y":1525,"style":1675},"348.0","font-size:11.5px;font-weight:400;fill:var(--brand-strong,#4338ca);text-anchor:middle;font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace",[25,1677,1679],{"id":1678},"step-5-check-completeness-and-write-the-workbook","Step 5 — Check completeness and write the workbook",[10,1681,1682],{},"Before writing, check the export is whole: no duplicate IDs, and as many records as the API reported. Then write the two tables:",[30,1684,1686],{"className":103,"code":1685,"language":105,"meta":35,"style":35},"dupes = df[\"id\"].duplicated().sum()\nprint(\"duplicates:\", dupes, \"| fetched:\", len(df), \"| expected:\", reported_total)\nassert dupes == 0 and len(df) == reported_total, \"incomplete or inconsistent export\"\n\nwith pd.ExcelWriter(\"orders_from_api.xlsx\", engine=\"openpyxl\") as xw:\n    df.to_excel(xw, sheet_name=\"Orders\", index=False)\n    lines.to_excel(xw, sheet_name=\"Lines\", index=False)\nprint(\"written\", len(df), \"orders and\", len(lines), \"lines\")\n",[14,1687,1688,1702,1729,1759,1763,1791,1814,1836],{"__ignoreMap":35},[39,1689,1690,1693,1695,1697,1699],{"class":41,"line":42},[39,1691,1692],{"class":116},"dupes ",[39,1694,141],{"class":112},[39,1696,1294],{"class":116},[39,1698,1167],{"class":49},[39,1700,1701],{"class":116},"].duplicated().sum()\n",[39,1703,1704,1706,1708,1711,1714,1717,1719,1721,1724,1727],{"class":41,"line":120},[39,1705,889],{"class":189},[39,1707,326],{"class":116},[39,1709,1710],{"class":49},"\"duplicates:\"",[39,1712,1713],{"class":116},", dupes, ",[39,1715,1716],{"class":49},"\"| fetched:\"",[39,1718,270],{"class":116},[39,1720,799],{"class":189},[39,1722,1723],{"class":116},"(df), ",[39,1725,1726],{"class":49},"\"| expected:\"",[39,1728,902],{"class":116},[39,1730,1731,1734,1737,1740,1743,1746,1748,1751,1753,1756],{"class":41,"line":128},[39,1732,1733],{"class":112},"assert",[39,1735,1736],{"class":116}," dupes ",[39,1738,1739],{"class":112},"==",[39,1741,1742],{"class":189}," 0",[39,1744,1745],{"class":112}," and",[39,1747,816],{"class":189},[39,1749,1750],{"class":116},"(df) ",[39,1752,1739],{"class":112},[39,1754,1755],{"class":116}," reported_total, ",[39,1757,1758],{"class":49},"\"incomplete or inconsistent export\"\n",[39,1760,1761],{"class":41,"line":135},[39,1762,132],{"emptyLinePlaceholder":131},[39,1764,1765,1768,1771,1774,1776,1779,1781,1784,1786,1788],{"class":41,"line":147},[39,1766,1767],{"class":112},"with",[39,1769,1770],{"class":116}," pd.ExcelWriter(",[39,1772,1773],{"class":49},"\"orders_from_api.xlsx\"",[39,1775,270],{"class":116},[39,1777,1778],{"class":233},"engine",[39,1780,141],{"class":112},[39,1782,1783],{"class":49},"\"openpyxl\"",[39,1785,1054],{"class":116},[39,1787,1232],{"class":112},[39,1789,1790],{"class":116}," xw:\n",[39,1792,1793,1796,1799,1801,1804,1806,1808,1810,1812],{"class":41,"line":169},[39,1794,1795],{"class":116},"    df.to_excel(xw, ",[39,1797,1798],{"class":233},"sheet_name",[39,1800,141],{"class":112},[39,1802,1803],{"class":49},"\"Orders\"",[39,1805,270],{"class":116},[39,1807,1392],{"class":233},[39,1809,141],{"class":112},[39,1811,1397],{"class":189},[39,1813,250],{"class":116},[39,1815,1816,1819,1821,1823,1826,1828,1830,1832,1834],{"class":41,"line":174},[39,1817,1818],{"class":116},"    lines.to_excel(xw, ",[39,1820,1798],{"class":233},[39,1822,141],{"class":112},[39,1824,1825],{"class":49},"\"Lines\"",[39,1827,270],{"class":116},[39,1829,1392],{"class":233},[39,1831,141],{"class":112},[39,1833,1397],{"class":189},[39,1835,250],{"class":116},[39,1837,1838,1840,1842,1845,1847,1849,1851,1854,1856,1858,1861,1863],{"class":41,"line":204},[39,1839,889],{"class":189},[39,1841,326],{"class":116},[39,1843,1844],{"class":49},"\"written\"",[39,1846,270],{"class":116},[39,1848,799],{"class":189},[39,1850,1723],{"class":116},[39,1852,1853],{"class":49},"\"orders and\"",[39,1855,270],{"class":116},[39,1857,799],{"class":189},[39,1859,1860],{"class":116},"(lines), ",[39,1862,1297],{"class":49},[39,1864,250],{"class":116},[10,1866,1867,1868,1871,1872,1875],{},"The duplicate check catches the classic page-number problem — a record that moved from page 2 to page 3 between requests and was fetched twice — and if duplicates are possible, ",[14,1869,1870],{},"drop_duplicates(\"id\", keep=\"last\")"," keeps the most recent copy. Comparing the count with the API's reported total catches the opposite problem, a skipped page. Failing loudly is better than writing a workbook that silently misses records, because the people reading it have no way to tell. For daily exports, store the newest ",[14,1873,1874],{},"created"," timestamp and ask the API only for records after it next time, so each run fetches the changes rather than everything.",[25,1877,1879],{"id":1878},"common-pitfalls","Common pitfalls",[1881,1882,1883,1891,1900,1906,1912],"ul",{},[1884,1885,1886,1890],"li",{},[1887,1888,1889],"strong",{},"Calling the endpoint once."," You get the first page only, and nothing tells you.",[1884,1892,1893,1896,1897,1899],{},[1887,1894,1895],{},"Ignoring 429 responses."," Honour ",[14,1898,449],{}," instead of retrying immediately.",[1884,1901,1902,1905],{},[1887,1903,1904],{},"Retrying 401 or 404."," These will not succeed on retry; fail fast.",[1884,1907,1908,1911],{},[1887,1909,1910],{},"Page numbers on changing data."," Records can be skipped or repeated; prefer cursors and de-duplicate.",[1884,1913,1914,1917],{},[1887,1915,1916],{},"Zone-aware timestamps."," Convert to local time and drop the zone before writing Excel.",[25,1919,1921],{"id":1920},"conclusion","Conclusion",[10,1923,1924,1925,1927,1928,1931,1932,1934,1935,1937,1938,1940,1941,1943],{},"To get a complete paginated API result into Excel, loop until the API says there is no more — stopping on a short page for page and offset styles, following the ",[14,1926,73],{}," link or cursor until ",[14,1929,1930],{},"has_more"," is false for cursor styles — through a session that honours ",[14,1933,449],{}," on ",[14,1936,267],{}," and retries temporary ",[14,1939,453],{}," errors. Flatten the records with ",[14,1942,1459],{},", put nested lists on their own sheet, convert timestamps to local naive datetimes, and check for duplicates and against the API's total before writing the workbook.",[25,1945,1947],{"id":1946},"frequently-asked-questions","Frequently asked questions",[10,1949,1950,1953],{},[1887,1951,1952],{},"How do I get all pages from an API in Python?","\nRequest the first page, then keep requesting the next one — by page number, offset or the next link the API returns — until a page is short, empty, or the API says there are no more results.",[10,1955,1956,1959],{},[1887,1957,1958],{},"How do I handle API rate limits when exporting to Excel?","\nWhen the API returns 429 Too Many Requests, wait for the number of seconds in the Retry-After header before retrying, and use the largest page size allowed to make fewer requests.",[10,1961,1962,1965],{},[1887,1963,1964],{},"How do I turn nested API JSON into Excel columns?","\nUse pandas json_normalize, which turns nested objects into dotted columns. Put nested lists into a separate table with the record_path option.",[10,1967,1968,1971],{},[1887,1969,1970],{},"Why does my API export have duplicate rows?","\nWith page-number pagination, records can move between pages while you fetch. Prefer cursor pagination if available and de-duplicate on the record ID.",[25,1973,1975],{"id":1974},"related","Related",[1881,1977,1978,1984,1991,1998],{},[1884,1979,1980,1981,1983],{},"Up one level: ",[19,1982,22],{"href":21}," — getting data in and out.",[1884,1985,1986,1990],{},[19,1987,1989],{"href":1988},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Ffetch-api-data-into-excel-with-python-requests\u002F","Fetch API Data into Excel with Python requests"," — the single-request basics.",[1884,1992,1993,1997],{},[19,1994,1996],{"href":1995},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fcache-api-responses-when-building-excel-reports\u002F","Cache API Responses When Building Excel Reports"," — avoiding repeat fetches.",[1884,1999,2000,2004],{},[19,2001,2003],{"href":2002},"\u002Fautomating-reporting-workflows\u002Fautomating-excel-online-with-microsoft-graph\u002Fhandle-microsoft-graph-throttling-for-excel\u002F","Handle Microsoft Graph Throttling for Excel"," — rate limits in one specific API.",[2006,2007,2008],"style",{},"html pre.shiki code .sMTad, html code.shiki .sMTad{--shiki-default:#6F42C1;--shiki-dark:#FFB757}html pre.shiki code .srMev, html code.shiki .srMev{--shiki-default:#032F62;--shiki-dark:#ADDCFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .s-kum, html code.shiki .s-kum{--shiki-default:#D73A49;--shiki-dark:#FF9492}html pre.shiki code .skGVy, html code.shiki .skGVy{--shiki-default:#24292E;--shiki-dark:#F0F3F6}html pre.shiki code .s-wDw, html code.shiki .s-wDw{--shiki-default:#6A737D;--shiki-dark:#BDC4CC}html pre.shiki code .s_Opv, html code.shiki .s_Opv{--shiki-default:#6F42C1;--shiki-dark:#DBB7FF}html pre.shiki code .sP0c6, html code.shiki .sP0c6{--shiki-default:#005CC5;--shiki-dark:#91CBFF}html pre.shiki code .sa561, html code.shiki .sa561{--shiki-default:#E36209;--shiki-dark:#FFB757}html pre.shiki code .sSjpA, html code.shiki .sSjpA{--shiki-default:#005CC5;--shiki-dark:#FF9492}",{"title":35,"searchDepth":120,"depth":120,"links":2010},[2011,2012,2013,2014,2015,2016,2017,2018,2019,2020],{"id":27,"depth":120,"text":28},{"id":92,"depth":120,"text":93},{"id":671,"depth":120,"text":672},{"id":994,"depth":120,"text":995},{"id":1209,"depth":120,"text":1210},{"id":1678,"depth":120,"text":1679},{"id":1878,"depth":120,"text":1879},{"id":1920,"depth":120,"text":1921},{"id":1946,"depth":120,"text":1947},{"id":1974,"depth":120,"text":1975},"2026-10-02","Pull every record from a REST API that returns results a page at a time and write them to Excel: page-number, offset and cursor pagination, following next links, handling 429 rate limits with Retry-After, retrying transient errors, flattening nested JSON with json_normalize, de-duplicating across pages, and checking you received as many records as the API says exist.","md",[2025,2027,2029,2031],{"q":1952,"a":2026},"Request the first page, then keep requesting the next one — by page number, offset or the next link the API returns — until a page is short, empty, or the API says there are no more results.",{"q":1958,"a":2028},"When the API returns 429 Too Many Requests, wait for the number of seconds in the Retry-After header before retrying, and use the largest page size allowed to make fewer requests.",{"q":1964,"a":2030},"Use pandas json_normalize, which turns nested objects into dotted columns. Put nested lists into a separate table with the record_path option.",{"q":1970,"a":2032},"With page-number pagination, records can move between pages while you fetch. Prefer cursor pagination if available and de-duplicate on the record ID.","fetch paginated api data into excel",null,{"breadcrumb":2036},[2037,2040,2043],{"name":2038,"item":2039},"Home","\u002F",{"name":2041,"item":2042},"Advanced Data Transformation and Cleaning for Excel Automation","\u002Fadvanced-data-transformation-and-cleaning\u002F",{"name":22,"item":21},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Ffetch-paginated-api-data-into-excel",{"title":2046,"description":2047},"Fetch Paginated REST API Data into Excel with Python","Loop through paginated API results with requests: page, offset and cursor styles, next links, 429 Retry-After handling, retries, json_normalize for nested JSON, de-duplication and a total-count check before writing Excel.","fetch-paginated-api-data-into-excel","advanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Ffetch-paginated-api-data-into-excel\u002Findex",[2051],"data-sources","how-to","UdNpD9GlixsZgOqziYziKOPk0ePzKe4ktLvzJf1j_pk",[2055,2058],{"title":1989,"path":2056,"stem":2057,"children":-1},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Ffetch-api-data-into-excel-with-python-requests","advanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Ffetch-api-data-into-excel-with-python-requests\u002Findex",{"title":2059,"path":2060,"stem":2061,"children":-1},"Generate SQL INSERT Statements from an Excel Sheet","\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fgenerate-sql-insert-statements-from-an-excel-sheet","advanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fgenerate-sql-insert-statements-from-an-excel-sheet\u002Findex",[2063,2065],{"path":2064,"title":2041},"\u002Fadvanced-data-transformation-and-cleaning",{"path":2066,"title":22},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases",[],[2069,2072,2074,2077,2080,2083],{"path":2070,"title":2071,"kind":2052},"\u002Fautomating-reporting-workflows\u002Fpublishing-excel-reports-to-cloud-storage\u002Fread-an-excel-file-directly-from-amazon-s3-with-pandas","Read an Excel File Directly from Amazon S3 with pandas",{"path":2073,"title":1996,"kind":2052},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fcache-api-responses-when-building-excel-reports",{"path":2075,"title":2076,"kind":2052},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fcompare-an-excel-file-with-a-database-table","Compare an Excel File with a Database Table",{"path":2078,"title":2079,"kind":2052},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fdownload-new-excel-files-from-an-sftp-server","Download New Excel Files from an SFTP Server",{"path":2081,"title":2082,"kind":2052},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-a-large-database-table-to-excel-in-chunks","Export a Large Database Table to Excel in Chunks",{"path":2084,"title":2085,"kind":2052},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-database-lookup-tables-as-excel-dropdown-lists","Export Database Lookup Tables as Excel Dropdown Lists",1790972096725]