Guide 10 of 12

How to Scrape Multiple Pages Automatically

Scrape paginated listings automatically: next-link detection, custom next-page selectors, page limits, polite delays, merging results and removing duplicates.

10 min read · Updated · By the Instant Data Scraper team

How to Scrape Multiple Pages Automatically

Almost every useful listing is split across pages: 24 products per page, 20 jobs per page, 50 table rows per page. Scraping only page one gives you a fraction of the data. Doing it page by page by hand is slow and error-prone.

The Instant Data Scraper tool can follow the "Next" link for you, wait politely between pages, and merge all rows into one dataset. This guide explains how pagination works, how to set it up, how to handle unusual pagination, and how to clean the combined result.

Types of pagination

Next / Previous links

The most common pattern: a "Next ›" link at the bottom of the page. Fully supported.

Numbered pages

Links 1, 2, 3 … 10 — usually with a Next link as well. Supported through the Next link.

URL parameters

Pages like ?page=2 or /page/2/. Supported because the Next link points to them.

"Load more" buttons

A button that loads more items with JavaScript. Not followed automatically — look for a numbered version of the page or a larger page size.

Infinite scroll

Items load as you scroll. Not followed by server-side tools. Many infinite-scroll sites still offer ?page= URLs or a "view all" link.

How next-page detection works

When you leave the Next-page selector empty, the tool checks, in order:

  1. A <link rel="next"> tag in the page head, or <a rel="next">.
  2. Links whose text is "Next", "Next page", "›", "»", ">", "Older posts" or "More results".
  3. Links whose class, title or aria-label contains "next".

The first valid candidate that points to a different page is used.

Setting up pagination

1. Test page one

Run a single-page extraction and make sure the right dataset is selected and the columns look correct.

2. Enable pagination

Open Extraction options, tick Support pagination.

3. Set maximum pages

Choose how many pages to follow. Your plan sets the upper limit. Start with 2–3 to test.

4. Add a Next-page selector if needed

If automatic detection picks the wrong link, right-click the Next link, choose Inspect, and write a selector such as:

  • a.next
  • li.pagination-next a
  • a[aria-label="Next page"]
  • a:contains("Next")

5. Run and watch progress

The progress bar shows which page is being fetched and how many rows have been collected. You can cancel at any time.

Built-in safeguards

  • Same website only — pagination stops if the next link points to another domain.
  • Loop protection — a page that was already visited is never fetched again.
  • Polite delay — at least one second between pages, longer when robots.txt sets a Crawl-delay (up to five seconds).
  • robots.txt — pagination stops at disallowed pages.
  • Limits — maximum pages per job and rows per dataset.
  • Cancel — stops between pages or even during a download.

How results are merged

Datasets are matched across pages by their structure — the same table headers or the same repeating-item signature. Page 2's products are appended to page 1's products. If a later page contains a field that earlier pages didn't (for example a "Sale" badge), a new column is added automatically and earlier rows are left blank in that column.

Cleaning multi-page results

Remove duplicates

Featured or sponsored items often repeat on every page. Use the item URL as the key in Data → Remove Duplicates.

Check the last page

The final page may contain fewer items or extra footer content.

Record page order if needed

The row number column (#) in the preview reflects the order of collection — useful for ranking analysis.

Tips for large crawls

  • Use the site's "show more per page" option to reduce the number of pages.
  • Split big catalogues by category instead of one giant crawl.
  • Run during off-peak hours for the website.
  • Only collect what you need.

Working with URL patterns

Look at the page URLs as you click through a listing:

https://shop.example.com/lamps?page=1
https://shop.example.com/lamps?page=2
https://shop.example.com/lamps?page=3

Patterns like this tell you a few useful things:

  • You can often add &per_page=96 or &limit=100 to reduce the number of pages — try it in your browser first.
  • You can start pagination from any page, for example page 5, by extracting that URL.
  • You can split a long crawl into ranges by starting at different page numbers.

Next-page selector recipes

Pagination HTMLSelector
<a class="next">a.next
<li class="pagination-next"><a>li.pagination-next a
<a rel="next">a[rel="next"]
<a aria-label="Next page">a[aria-label="Next page"]
Arrow icon inside a linkSelect the link: .pagination a:last-child
Text "Next" in a linka:contains("Next")

If your selector matches an element inside the link (such as an icon), the tool climbs up to the surrounding link automatically.

Pagination vs. multi-level crawling

Pagination collects the same kind of page repeatedly (page 1, 2, 3 of a listing). Multi-level crawling visits each item's detail page to collect more fields. Instant Data Scraper focuses on pagination, which covers most needs and keeps load on websites low. If you need detail-page fields:

  1. Extract the listing with pagination to get every item URL.
  2. Decide whether the detail fields are really needed.
  3. If they are, extract a small number of detail pages individually, or use a platform built for multi-level crawls and respect the site's limits.

Planning a large multi-page extraction

  • Estimate size — items per page × number of pages. Check this against your plan's row limit.
  • Estimate time — each page takes the fetch time plus the polite delay; 20 pages usually take well under a minute.
  • Split by category or filter — smaller, focused extractions are easier to verify.
  • Check the end — make sure the last page of results was reached by comparing the final rows with the website.

Troubleshooting pagination

  • Same page repeated — the Next link may use JavaScript; look for a URL-based alternative.
  • Next link found on page 1 only — later pages use different markup; use a broader selector.
  • New columns appear on later pages — normal; earlier rows are blank in those columns.
  • Different dataset chosen on later pages — pages with very different layouts may not merge; extract them separately.

Walkthrough: collecting a 10-page job board

  1. Open the first results page and run a single-page extraction to confirm the job-card dataset.
  2. Inspect the pagination area: the Next link is <a class="pager-next">.
  3. Tick Support pagination, set maximum pages to 10 and enter a.pager-next as the Next-page selector.
  4. Start the extraction and watch the counter move from page 1 to page 10.
  5. The job finishes with 200 rows in one dataset. Export to Excel.
  6. Remove duplicate "featured" jobs using the job URL and sort by posting date.

Multi-page checklist

  • Page one tested on its own first.
  • Page limit within your plan and as low as practical.
  • Next selector set only when needed.
  • Last page reached (compare final rows with the website).
  • Duplicates removed using a unique URL.
  • Extraction date recorded.
  • Website's robots.txt and terms respected.

Glossary

  • Pagination — splitting results over several pages.
  • Next link — the link that goes to the following page of results.
  • Crawl delay — the waiting time between page requests.
  • Infinite scroll — loading more items as the user scrolls, without separate pages.
  • Deduplication — removing rows that appear more than once.

Why polite pagination matters

Every page you request uses a little of the website's server capacity. A person browsing a listing might view a page every few seconds; an aggressive scraper can request dozens per second. That difference is why many websites rate-limit or block automated traffic, and why Instant Data Scraper always waits between pages and follows any Crawl-delay the site specifies in robots.txt.

Polite pagination is also in your own interest. Websites that are not under pressure respond consistently, so you get complete pages instead of errors and partial results. Tools that hammer servers often end up blocked half-way through, leaving you with an incomplete dataset and a site that now refuses your requests. A steady pace, sensible page limits and focused extractions — one category at a time rather than the whole catalogue — produce better data with less trouble.

If you find yourself needing thousands of pages on a regular schedule, that is a strong signal to contact the website about an official data feed or API.

Typical multi-page projects

Job-market research collects all postings for a role across a board's result pages. Product research gathers an entire category from a shop. Event planning compiles every listing from a public calendar. Content inventories list every article in a blog archive. In each case, a single extraction with pagination enabled replaces dozens of manual copy-and-paste rounds, and the merged dataset is ready for analysis immediately.

Common pagination mistakes

  • Setting a huge page limit first — start small and confirm the data.
  • Not checking the final page — make sure the crawl reached the end you expected.
  • Ignoring duplicates — featured items often repeat on every page.
  • Expecting infinite scroll to work — look for numbered pages instead.

Key takeaways

  • Enable Support pagination and set a sensible maximum.
  • Let the tool detect the Next link, or provide a simple selector.
  • Built-in safeguards keep crawls on the same site, polite and loop-free.
  • Rows from all pages are merged into one dataset.
  • Deduplicate by URL after exporting.

Conclusion

Pagination turns a one-page sample into a complete dataset. Test the first page, enable pagination with a modest page limit, let the tool detect the Next link or give it a simple selector, and then deduplicate and verify the merged result. Built-in safeguards keep every crawl on the same website, avoid loops, respect robots.txt and wait politely between pages. Use larger page sizes and focused categories to reduce requests, and contact the website about an official feed if you need very large volumes regularly. With this approach, collecting dozens of pages becomes as easy as collecting one.

Step-by-step instructions

  1. Test one page

    Confirm the dataset and columns in the tool.

  2. Tick Support pagination

    Inside Extraction options.

  3. Set maximum pages

    Start small, then increase within your plan limit.

  4. Add a Next selector if needed

    Only when automatic detection picks the wrong link.

  5. Start and monitor

    Watch the page counter; cancel if something looks wrong.

  6. Export and deduplicate

    Remove duplicates by item URL.

Example

Page 1: 24 rows  →  Next ›  →  Page 2: 24 rows  →  Next ›  →  Page 3: 18 rows (no Next link)
Result: one dataset with 66 rows

Best practices

  • Always test with 2–3 pages first.
  • Prefer larger page sizes over more pages.
  • Use a custom selector only when needed.
  • Deduplicate on a unique URL.
  • Keep crawls small and polite.
Try it on a real page

Open the Instant Data Scraper tool and extract your first dataset in seconds.

Start Extracting

Frequently asked questions