Currently, I have used Selenium to extract text from a table on a website. Following is the code:
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
# Using Chrome to access web
browser = webdriver.Chrome(ChromeDriverManager().install())
# Open the website
browser.get('https://launchstudio.bluetooth.com/Listings/Search')
element = browser.find_element_by_id('searchButton').click()
table_text = browser.find_element_by_class_name('table').text
while len(table_text) < 80:
table_text = browser.find_element_by_class_name('table').text
print(table_text)
browser.close()
However, I am trying to find a way to do the same with Requests/Beautiful soup or any other library where I can schedule this as a task in windows and store the result in a table at every x interval. Obviously, since I want all this to happen in the background and then trigger a notification etc.
What I Want is- Open this website, click on the search button (or trigger the corresponding javascript), and then export the table as a Dataframe or whatever.
Can you please guide me here?
thanks in advance!!
Advertisement
Answer
If you go to Network Tab you will get the API . You can use this post request to get all the value.Using max result field you can limit the results as well.
https://platformapi.bluetooth.com/api/platform/Listings/Search
import requests
import pandas as pd
data={
"searchString" : "",
"searchQualificationsAndDesigns": True,
"searchDeclarationOnly": True,
"bqaApprovalStatusId" : -1,
"bqaLockStatusId" : -1,
"layers" : [],
"listingDateEarliest" : "",
"listingDateLatest" : "",
"maxResults": 5000,
"memberId": "",
"productTypeId" : 0,
"searchDeclarationOnly" : True,
"searchEndProductList" : False,
"searchMyCompany" : False,
"searchPRDProductList" : True,
"searchQualificationsAndDesigns" : True,
"searchString" : "",
"specName": 0,
"userId" : 0
}
headers = {'User-Agent':
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36'}
url="https://platformapi.bluetooth.com/api/platform/Listings/Search"
response=requests.post(url,headers=headers,data=data).json()
df=pd.DataFrame(response)
print(df)
You can import to csv file.
df.to_csv("testresult.csv")
