Accessing Data from Javascript API call [closed]

I am trying to access the data shown on this website: Link using either python requests or requests-html. The site calls this API: API

I’ve tried copying the POST request using Insomnia (Copying cURL). I get a response code 200 but an error saying “Error getting base data: Error parsing xmlsquery”

How can i get this data into a pandas dataframe? i would like to avoid using Selenium.

Thanks!

Answer

I have no idea what you used to send request
but when I use module requests then problem makes + in query.

requests converts + to %2B but server needs +.

It needs to manually convert data using urllib.parse with safe="+"

data = {
    "xmlquery": "<post>n<param+name="Exchange"+value="NMF"/>n<param+name="SubSystem"+value="Prices"/>n<param+name="Action"+value="GetMarket"/>n<param+name="inst__a"+value="0,1,2,5,21,23"/>n<param+name="ext_xslt"+value="/nordicV3/paging_inst_table.xsl"/>n<param+name="Market"+value="GITS:CO:CPHCB,GITS:CO:CPHBB,M:GITS:CO:CPHTA,GITS:CO:CPHAU,GITS:CO:CPHSA"/>n<param+name="RecursiveMarketElement"+value="True"/>n<param+name="XPath"+value="//inst[@itid='2'+or+@itid='3']"/>n<param+name="ext_xslt_lang"+value="en"/>n<param+name="ext_xslt_tableId"+value="bondsSearchDKTable"/>n<param+name="ext_xslt_options"+value=",noflag,"/>n<param+name="ext_xslt_hiddenattrs"+value=",fnm,isrid,dlt,tp,bb,ib,cpt,rps,os,lt,st,itid,lists,it,mkt,"/>n<param+name="ext_xslt_notlabel"+value=",fnm"/>n<param+name="ext_xslt_jspcbk"+value="doPaging"/>n<param+name="ext_xslt_jsscbk"+value="doSortPager"/>n<param+name="ext_xslt_sorder"+value="descending"/>n<param+name="ext_xslt_sattr"+value="chp"/>n<param+name="ext_xslt_start"+value="0"/>n<param+name="ext_xslt_size"+value="100"/>n<param+name="inst__an"+value="id,nm,fnm,isin,cpnrt,bp,ap,lsp,chp,atap,ed,dlt,cr,isrid,tp,bb,ib,cpt,rps,os,lt,st,itid,lists,it,mkt"/>n<param+name="app"+value="/obligationer/danmark"/>n</post>"
}

data_str = urllib.parse.urlencode(data, safe="+")

---

Full working code:

In code you can see all needed headers to access server.

import requests
import urllib.parse

data = {
    "xmlquery": "<post>n<param+name="Exchange"+value="NMF"/>n<param+name="SubSystem"+value="Prices"/>n<param+name="Action"+value="GetMarket"/>n<param+name="inst__a"+value="0,1,2,5,21,23"/>n<param+name="ext_xslt"+value="/nordicV3/paging_inst_table.xsl"/>n<param+name="Market"+value="GITS:CO:CPHCB,GITS:CO:CPHBB,M:GITS:CO:CPHTA,GITS:CO:CPHAU,GITS:CO:CPHSA"/>n<param+name="RecursiveMarketElement"+value="True"/>n<param+name="XPath"+value="//inst[@itid='2'+or+@itid='3']"/>n<param+name="ext_xslt_lang"+value="en"/>n<param+name="ext_xslt_tableId"+value="bondsSearchDKTable"/>n<param+name="ext_xslt_options"+value=",noflag,"/>n<param+name="ext_xslt_hiddenattrs"+value=",fnm,isrid,dlt,tp,bb,ib,cpt,rps,os,lt,st,itid,lists,it,mkt,"/>n<param+name="ext_xslt_notlabel"+value=",fnm"/>n<param+name="ext_xslt_jspcbk"+value="doPaging"/>n<param+name="ext_xslt_jsscbk"+value="doSortPager"/>n<param+name="ext_xslt_sorder"+value="descending"/>n<param+name="ext_xslt_sattr"+value="chp"/>n<param+name="ext_xslt_start"+value="0"/>n<param+name="ext_xslt_size"+value="100"/>n<param+name="inst__an"+value="id,nm,fnm,isin,cpnrt,bp,ap,lsp,chp,atap,ed,dlt,cr,isrid,tp,bb,ib,cpt,rps,os,lt,st,itid,lists,it,mkt"/>n<param+name="app"+value="/obligationer/danmark"/>n</post>"
}

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0', 
    'X-Requested-With': 'XMLHttpRequest',
    'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',    
}

url = 'http://www.nasdaqomxnordic.com/webproxy/DataFeedProxy.aspx'

data_str = urllib.parse.urlencode(data, safe="+")

response = requests.post(url, data=data_str, headers=headers)

#print(response.text)

import pandas as pd

all_tables = pd.read_html(response.text)
df = all_tables[1]

print(df)

---

How I found +:

Python code was giving "Error getting base data: Error parsing xmlsquery" – so problem was in xmlsquery.

I copied working xmlsquery from request in browser using DevTool (tab: Network) in Chrome/Firefox
and compared (char-by-char) with response.request.body from Python code.

correct = "xmlquery=...from browser..."
wrong   = response.request.body

for char_a, char_b in zip(correct, wrong):
    if char_a != char_b:
       print(char_a, char_b)