Skip to content
Advertisement

Tag: web-crawler

How to limit Apify web crawler scope to first three list pages?

I have written the following web scraper in Apify (jQuery), but I am struggling to limit it to only look at certain list pages. The crawler scrapes articles I have published at https://www.beet.tv/author/randrews, a page which contains 102 paginated index pages, each containing 20 article links. The crawler works fine when executed manually and in full; it gets everything, 2,000+

Advertisement