Skip to content

Is there any way to crawl dynamic website without library?

I’m using ‘axios’ to crawl data from web sites(https://movie.daum.net/premovie/released)

I need data of li tags under ol tag, but li tags aren’t shown in source.

element tab in dev tool

source tab in dev tool

I guess the site renders li tags dynamically using JavaScript or etc., but I don’t know how to crawl them after all tags are loaded.

The point is, I wanna do it without any library related to crawling like puppeteer.

Please let me know how to do it if you know it. Thank you.

==============

Found the way to solve it!

[웹 크롤링] JavaScript로 된 html 불러오기

It is written in Korean, but was really helpful.

Answer

Found the way to solve it!

For my case, I found the XHR exists in network tab of dev tools.
So I requested the ‘get’ axios method to XHR url, not the url of web page.

XHR tab in dev tools

[웹 크롤링] JavaScript로 된 html 불러오기

I found it in here!
It is written in Korean, but was really helpful.