I am looking for way to read javascript json data loaded into one of a script tag of this page. I have tried various re
patterns posted on google and stackoveflow but got nothing.
The Json Formatter shows an Invalid (RFC 8259).
Here is a code
import requests,json from scrapy.selector import Selector headers = {'Content-Type': 'application/json', 'Accept-Language': 'en-US,en;q=0.5', 'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B179 Safari/7534.48.3'} url = 'https://www.zocdoc.com/doctor/andrew-fagelman-md-7363?insuranceCarrier=-1&insurancePlan=-1' response = requests.get(url,headers = headers) sel = Selector(text = response.text) profile_data = sel.css('script:contains(APOLLO_STATE)::text').get('{}').split('__REDUX_STATE__ = JSON.parse(')[-1].split(');n window.ZD = {')[0] profile_json = json.loads(profile_data) print(type(profile_json))
The problem seems an invalid json format. The type of profile_json
is string while a little amendments in above code shows below error stack
>>> profile_data = sel.css('script:contains(APOLLO_STATE)::text').get('{}').split('__REDUX_STATE__ = JSON.parse("')[-1].split('");n window.ZD = {')[0].replace("\","") >>> profile_json = json.loads(profile_data) Traceback (most recent call last): File "/usr/lib/python3.6/code.py", line 91, in runcode exec(code, self.locals) File "<console>", line 1, in <module> File "/usr/lib/python3.6/json/__init__.py", line 354, in loads return _default_decoder.decode(s) File "/usr/lib/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.6/json/decoder.py", line 355, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 41316 (char 41315)
Error in output are highlighted here:
The original HTML contains this (heavily trimmed):
<script> ... window.__REDUX_STATE__ = JSON.parse("{"routing": ... "awards":["Journal of Urology - \"Efficacy, Safety, and Use of Viagra in Clinical Practice.\"","Critical Care Resident of the Year - 2003"], ...
The same string extracted by scrapy is this:
"awards":[ "Journal of Urology - ""Efficacy", "Safety", "and Use of Viagra in Clinical Practice.""", "Critical Care Resident of the Year - 2003" ],
It appears the backslashes are removed from it, making the JSON invalid.
Advertisement
Answer
I don’t know if this is an efficient way of handling the problem but below code resolved my problem.
>>> import js2xml >>> profile_data = sel.css('script:contains(APOLLO_STATE)::text').get('{}') >>> parsed = js2xml.parse(profile_data) >>> js = json.loads(parsed.xpath("//string[contains(text(),'routing')]/text()")[0])