I have written a small web scraper using puppeteer, but I can’t seem to properly extract the information I want. Could you please help me find the mistake? Background: I want to scrape a website that indicates how much of premium the city allows a landlord to add to rest controlled apartments (e.g. for …
Tag: web-scraping
Why does headless need to be false for Puppeteer to work?
I’m creating a web api that scrapes a given url and sends that back. I am using Puppeteer to do this. I asked this question: Puppeteer not behaving like in Developer Console and recieved an answer that suggested it would only work if headless was set to be false. I don’t want to be constantly open…
Scrape data inside a JavaScript function by using python selenium
Please let me know if there is any option available to scrape data inside a JavaScript function below using Selenium page https://www.bayut.com/index/sale-prices-apartments-abu-dhabi.html input code … Output Date Value 2010-01-31 1443 2010-02-28 1407 Tried but not working My output file should have date and v…
How to Inject Axios in Puppeteer
I want to inject axios in puppeteer in order to open a browser session, but sending requests and receiving responses through axios: So the question is, is it possible? Here’s my attempt, but it is not working since axios doesn’t intercepts the website responses I’m moving my first steps in a…
Getting index.html content while trying to scrape a react website
when i try to scrape a reactjs website using nodejs i am getting the content of index.html file only not the tags that were used in the website. Here is what i have tried – What should i do to get the whole of tags that were used in react website. And do tell i can scrape the hackernoon website
Unable to import data from webpage to google sheets
I know how to use the importhtml function of google spreadsheets but there is one webpage which refuses to load the table. Link given below. https://www.dream11.com/cricket/fantasy-scorecard/1027/14047 I even tried the below code in google developer console to check the table index but this also did not solve…
Scrape a page to get encoded javascript link
I am working on a task in which I have to get details from a web page using python request library. I have successfully got the details from the page but it has a button “Show Details” which fetches more details using ajax call, now I need to fetch those extra details also. can anyone help me in d…
How to call JavaScript function using BeautifulSoup and Python
I am performing web scraping to grab data from a website as part of my project. I can make the request and grab the data which is present in the dom. However, some data is getting rendered on javascript onClick function. One way could be, using the selenium to click on the link (which calls the javascript fun…
Scrapy + splash: can’t select element
I’m learning to use scrapy with splash. As an exercise, I’m trying to visit https://www.ubereats.com/stores/, click on the address text box, enter a location and then press the Enter button to move to next page containing the restaurants available for that location. I have the following lua code: …
How to scroll down with Phantomjs to load dynamic content
I am trying to scrape links from a page that generates content dynamically as the user scroll down to the bottom (infinite scrolling). I have tried doing different things with Phantomjs but not able to gather links beyond first page. Let say the element at the bottom which loads content has class .has-more-it…