pandas.read_html tables not found

Question

I'm trying to get a list of the major world indices in Yahoo Finance at this URL: https://finance.yahoo.com/world-indices. I tried first to get the indices in a table by just running

major_indices=pd.read_html("https://finance.yahoo.com/world-indices")[0]

In this case the error was:

ValueError: No tables found

So I read a solution using selenium at pandas read_html - no tables found the solution they came up with is (with some adjustment):

from selenium import webdriver
import pandas as pd
from selenium.webdriver.common.keys import Keys
from webdrivermanager.chrome import ChromeDriverManager


driver = webdriver.Chrome(ChromeDriverManager().download_and_install())
driver.get("https://finance.yahoo.com/world-indices")

html = driver.page_source

tables = pd.read_html(html)
data = tables[1]

Again this code gave me another error:

ValueError: No tables found

I don't know whether to keep using selenium or the pd.read_html is just fine. Either way I'm trying to get this data and don't know how to procede. Can anyone help me?

Answer 1

You don't need Selenium here, you just have to set the euConsentId cookie:

import pandas as pd
import requests
import uuid

url = 'https://finance.yahoo.com/world-indices'
cookies = {'euConsentId': str(uuid.uuid4())}

html = requests.get(url, cookies=cookies).content
df = pd.read_html(html)[0]

Output:

>>> df
       Symbol                       Name  Last Price   Change % Change    Volume  Intraday High/Low  52 Week Range  Day Chart
0       ^GSPC                    S&P 500     4023.89    93.81   +2.39%    2.545B                NaN            NaN        NaN
1        ^DJI                     Dow 30    32196.66   466.36   +1.47%  388.524M                NaN            NaN        NaN
2       ^IXIC                     Nasdaq    11805.00   434.04   +3.82%     5.15B                NaN            NaN        NaN
3        ^NYA        NYSE COMPOSITE (DJ)    15257.36   326.26   +2.19%         0                NaN            NaN        NaN
4        ^XAX  NYSE AMEX COMPOSITE INDEX     4025.81   122.66   +3.14%         0                NaN            NaN        NaN
5    ^BUK100P                Cboe UK 100      739.68    17.83   +2.47%         0                NaN            NaN        NaN
6        ^RUT               Russell 2000     1792.67    53.28   +3.06%         0                NaN            NaN        NaN
7        ^VIX      CBOE Volatility Index       28.87    -2.90   -9.13%         0                NaN            NaN        NaN
8       ^FTSE                   FTSE 100     7418.15   184.81   +2.55%         0                NaN            NaN        NaN
9      ^GDAXI      DAX PERFORMANCE-INDEX    14027.93   288.29   +2.10%         0                NaN            NaN        NaN
10      ^FCHI                     CAC 40     6362.68   156.42   +2.52%         0                NaN            NaN        NaN
11  ^STOXX50E             ESTX 50 PR.EUR     3703.42    89.99   +2.49%         0                NaN            NaN        NaN
12      ^N100         Euronext 100 Index     1211.74    28.89   +2.44%         0                NaN            NaN        NaN
13       ^BFX                     BEL 20     3944.56    14.35   +0.37%         0                NaN            NaN        NaN
14   IMOEX.ME          MOEX Russia Index     2307.50     9.61   +0.42%         0                NaN            NaN        NaN
15      ^N225                 Nikkei 225    26427.65   678.93   +2.64%         0                NaN            NaN        NaN
16       ^HSI            HANG SENG INDEX    19898.77   518.43   +2.68%         0                NaN            NaN        NaN
17  000001.SS        SSE Composite Index     3084.28    29.29   +0.96%    3.109B                NaN            NaN        NaN
18  399001.SZ         Shenzhen Component    11159.79    64.92   +0.59%     3.16B                NaN            NaN        NaN
19       ^STI                  STI Index     3191.16    25.98   +0.82%         0                NaN            NaN        NaN
20      ^AXJO                S&P/ASX 200     7075.10   134.10   +1.93%         0                NaN            NaN        NaN
21      ^AORD             ALL ORDINARIES     7307.70   141.10   +1.97%         0                NaN            NaN        NaN
22     ^BSESN             S&P BSE SENSEX    52793.62  -136.69   -0.26%         0                NaN            NaN        NaN
23      ^JKSE    Jakarta Composite Index     6597.99    -1.85   -0.03%         0                NaN            NaN        NaN
24      ^KLSE   FTSE Bursa Malaysia KLCI     1544.41     5.61   +0.36%         0                NaN            NaN        NaN
25      ^NZ50     S&P/NZX 50 INDEX GROSS    11168.18    -9.18   -0.08%         0                NaN            NaN        NaN
26      ^KS11      KOSPI Composite Index     2604.24    54.16   +2.12%    788539                NaN            NaN        NaN
27      ^TWII        TSEC weighted index    15832.54   215.86   +1.38%         0                NaN            NaN        NaN
28    ^GSPTSE    S&P/TSX Composite index    20099.81   400.76   +2.03%  294.637M                NaN            NaN        NaN
29      ^BVSP                   IBOVESPA   106924.18  1236.54   +1.17%         0                NaN            NaN        NaN
30       ^MXX                 IPC MEXICO    49579.90   270.58   +0.55%  212.868M                NaN            NaN        NaN
31      ^IPSA               S&P/CLX IPSA     5058.88     0.00    0.00%         0                NaN            NaN        NaN
32      ^MERV                     MERVAL    38390.84   233.89   +0.61%         0                NaN            NaN        NaN
33  ^TA125.TA                     TA-125     1964.95    23.38   +1.20%         0                NaN            NaN        NaN
34    ^CASE30  EGX 30 Price Return Index    10642.40  -213.50   -1.97%   36.837M                NaN            NaN        NaN
35   ^JN0U.JO   Top 40 USD Net TRI Index     4118.19    65.63   +1.62%         0                NaN            NaN        NaN