Question wrt to scraping a list of all symbols
synergiator opened this issue · comments
Is there a way to scrape all symbols? I was not able to identify this feature in yfinance after some research; in case it's available, kindly advise - otherwise I hope this can be a valid feature request.
Desired output should be a sorted dictionary, similar to: {"AAPL", "ACME", ..... "ZZZ"}
What I could find so far, there is a standalone script which however as of today can't overcome the cookie accept form on the site. Side question - how does yfinance manage this?
- Original project, as it seems
- Most recent version I could find with symbols list as of 2023-12-31
Workaround: use Selenium in Docker to accept the cookies form and scrape the page.
Something like the following:
driver = webdriver.Chrome()
driver.get(url)
accept_button = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, '//button[text()="Accept all"]'))
)
accept_button.click()
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, 'lookup-page'))
)
html_content = driver.page_source
yf already uses cookies