How to extract station data from the web page

Utpal Kumar   3 minute read      

In this post, we will use python to extract the station information from a web page. This can save a lot of time in comparison to manually copying the data. We will obtain the data into pandas dataframe and to save it into csv file.

Key idea — pd.read_html scrapes every HTML table for you in one line. You don’t need requests + BeautifulSoup for tabular data. pandas.read_html(url) fetches the page, finds all <table> elements, and returns them as a list of DataFrames. Your only real work is picking the right table out of that list and tidying its header rows — then it’s plain pandas from there (filter, save to CSV, plot).

Scraping a table of stations into a map pandas read_html pulls every table from a web page into a list of DataFrames; you select and clean the right one, save it to CSV, and plot the stations with PyGMT. Web page HTML tables pd.read_html list of DataFrames Select + clean pick the right table station CSV to_csv PyGMT map plot stations read every table, keep the one you want, then map it
The end-to-end path: scrape all tables, isolate the station table, save it, and plot the stations.

Extract table data into pandas dataframe

import pandas as pd

url = "https://bats.earth.sinica.edu.tw/Station/BATS_Stn_Summary.html"

htmlPage = pd.read_html(url)

# print(htmlPage)
print(f"total # of tables {len(htmlPage)}")


df = htmlPage[1]
columns = df.iloc[1, :].values
print(columns)

dict_list = []
for idx in range(2, df.shape[0]-3):
    _dict = {}
    for icol, col in enumerate(columns):
        _dict.update({col: df.iloc[idx, icol]})
    dict_list.append(_dict)

new_df = pd.DataFrame(dict_list)
print(new_df.head())

# save into csv file
new_df.to_csv('station_list.txt', index=False)

Notice the two-step cleanup: htmlPage[1] picks the second table on the page (tables are 0-indexed, so always print(len(htmlPage)) first to see how many there are), and the loop skips the header/footer rows (range(2, df.shape[0]-3)) to keep only the real station rows. Those magic numbers are specific to this page’s layout — inspect your own table and adjust them.

Two things that trip people up. (1) read_html needs a parser installed — lxml (fastest) or beautifulsoup4 + html5lib; pip install lxml if you get an ImportError. (2) Passing a URL to read_html still works in current pandas (2.x/3.x); only passing a literal HTML string is deprecated (wrap it in io.StringIO if you must). If a server blocks pandas’ default fetch, grab the page yourself with requests (setting a User-Agent header) and hand read_html the HTML text via StringIO.

Quick check: What does pd.read_html(url) return?

  • A single DataFrame of the first table
  • A list of DataFrames — one per <table> on the page — which is why we index htmlPage[1]
  • The raw HTML string of the page
  • A BeautifulSoup object you still have to parse

Plot stations

import numpy as np
import pygmt
import pandas as pd
np.random.seed(45)  # to get the same color at each run

df = pd.read_csv('station_list.txt')
print(df.head())

# get the list of networks
networks = list(set(df['Network'].tolist()))

dfs = []
for net in networks:
    df1 = df[df['Network'] == net]
    dfs.append(df1)

colorsList = []
for i in range(len(networks)):
    colorsList.append('#%06X' % np.random.randint(0, 0xFFFFFF))


minlon, maxlon = df['Long.'].min()-1, df['Long.'].max()+1
minlat, maxlat = df['Lat.'].min()-1, df['Lat.'].max()+1

# define etopo data file

topo_data = "@earth_relief_15s"

# Visualization
fig = pygmt.Figure()
# make color pallets
pygmt.makecpt(
    cmap='etopo1',
    series='-8000/5000/1000',
    continuous=True
)

# plot high res topography
fig.grdimage(
    grid=topo_data,
    region=[minlon, maxlon, minlat, maxlat],
    projection='M4i',
    shading=True,
    frame=True
)

# plot coastlines
fig.coast(
    region=[minlon, maxlon, minlat, maxlat],
    projection='M4i',
    shorelines=True,
    frame=True
)
leftjustify, rightoffset = "TL", "5p/-5p"
for idx, dff in enumerate(dfs):
    fig.plot(
        x=dff["Long."].values,
        y=dff["Lat."].values,
        style="i10p",
        color=colorsList[idx],
        pen="black",
        label=networks[idx]
    )

for snum in range(df.shape[0]):
    fig.text(
        x=df.loc[snum, 'Long.'],
        y=df.loc[snum, 'Lat.'],
        text=f"{df.loc[snum, 'Station']}",
        justify=leftjustify,
        angle=0,
        offset=rightoffset,
        fill="white",
        font=f"6p,Helvetica-Bold,black",
    )


fig.legend(position="JTR+jTR+o0.2c", box=True)

fig.savefig('station_map.png', crop=True, dpi=300)

PyGMT renamed color to fill. In the fig.plot(...) call above, the color=colorsList[idx] argument was deprecated in PyGMT v0.8 and later removed. On a current PyGMT, use fill=colorsList[idx] instead — the behavior is identical. (The @earth_relief_15s remote grid and the rest of the call are unchanged.)

Recap

  • read_html is one-line scraping for tables. It returns a list of DataFrames — one per HTML table — so no requests/BeautifulSoup boilerplate for tabular data.
  • The work is selection + cleanup. Index the right table (htmlPage[1]) and trim header/footer rows; those indices are page-specific, so inspect first.
  • Then it’s plain pandas. Filter by network, save with to_csv, and you have a reusable station list.
  • PyGMT maps it. Group by network for colors, plot with fig.plot (use fill= on modern PyGMT), and label each station with fig.text.

Where to go next

Disclaimer of liability

The information provided by the Earth Inversion is made available for educational purposes only.

Whilst we endeavor to keep the information up-to-date and correct. Earth Inversion makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability or availability with respect to the website or the information, products, services or related graphics content on the website for any purpose.

UNDER NO CIRCUMSTANCE SHALL WE HAVE ANY LIABILITY TO YOU FOR ANY LOSS OR DAMAGE OF ANY KIND INCURRED AS A RESULT OF THE USE OF THE SITE OR RELIANCE ON ANY INFORMATION PROVIDED ON THE SITE. ANY RELIANCE YOU PLACED ON SUCH MATERIAL IS THEREFORE STRICTLY AT YOUR OWN RISK.

Leave a comment