Skip to content

How to parse JavaScript Json into Python dict type, effeciently

I am looking for way to read javascript json data loaded into one of a script tag of this page. I have tried various re patterns posted on google and stackoveflow but got nothing.

The Json Formatter shows an Invalid (RFC 8259).

Here is a code

import requests,json
from scrapy.selector import Selector

headers = {'Content-Type': 'application/json', 'Accept-Language': 'en-US,en;q=0.5', 'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B179 Safari/7534.48.3'}

url = ''

response = requests.get(url,headers = headers)

sel = Selector(text = response.text)

profile_data = sel.css('script:contains(APOLLO_STATE)::text').get('{}').split('__REDUX_STATE__ = JSON.parse(')[-1].split(');n          window.ZD = {')[0]
profile_json = json.loads(profile_data)

The problem seems an invalid json format. The type of profile_json is string while a little amendments in above code shows below error stack

>>> profile_data = sel.css('script:contains(APOLLO_STATE)::text').get('{}').split('__REDUX_STATE__ = JSON.parse("')[-1].split('");n          window.ZD = {')[0].replace("\","")
>>> profile_json = json.loads(profile_data)
Traceback (most recent call last):
  File "/usr/lib/python3.6/", line 91, in runcode
    exec(code, self.locals)
  File "<console>", line 1, in <module>
  File "/usr/lib/python3.6/json/", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.6/json/", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 41316 (char 41315)

Error in output are highlighted here:


The original HTML contains this (heavily trimmed):

   window.__REDUX_STATE__ = JSON.parse("{"routing": ...
   "awards":["Journal of Urology - \"Efficacy, Safety, and Use of Viagra in Clinical Practice.\"","Critical Care Resident of the Year - 2003"],

The same string extracted by scrapy is this:

               "Journal of Urology - ""Efficacy",
               "and Use of Viagra in Clinical Practice.""",
               "Critical Care Resident of the Year - 2003"

It appears the backslashes are removed from it, making the JSON invalid.


I don’t know if this is an efficient way of handling the problem but below code resolved my problem.

>>> import js2xml
>>> profile_data = sel.css('script:contains(APOLLO_STATE)::text').get('{}')
>>> parsed = js2xml.parse(profile_data)
>>> js = json.loads(parsed.xpath("//string[contains(text(),'routing')]/text()")[0])