Python Web Scraping Tutorial: Build a Local Scraper

A small web scraper can teach you how Python retrieves a web page, reads its HTML, and extracts useful information. The safest way to learn is with a page you created yourself: you can inspect the markup, control the requests, and avoid scraping sites without permission. This Python web scraping tutorial builds a simple scraper for a local test page and shows how to handle common errors responsibly.

Key takeaways

Use a page you own or have explicit permission to scrape.
Serve a small HTML fixture locally so the exercise does not depend on a third-party website.
Use requests to fetch the page and Beautiful Soup to parse its HTML.
Set a timeout, check the HTTP response, and handle missing elements.
Do not use a scraper to bypass logins, paywalls, or anti-bot protections.

1. Create a local test page

Make a new folder for the exercise. Inside it, create a file named test_page.html with a few sample cards:

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>Local Scraper Test</title>
</head>
<body>
  <main>
    <article class="card">
      <h2>Python Basics</h2>
      <p>A short introduction to Python.</p>
    </article>
    <article class="card">
      <h2>Working with HTML</h2>
      <p>Practice reading structured page content.</p>
    </article>
  </main>
</body>
</html>

The article.card selector gives the scraper a predictable way to find each item. Real sites can have more complicated or changing markup, but starting with a page you control makes it easier to understand what the parser is doing.

2. Install the libraries and serve the page

This example uses requests for the HTTP request and Beautiful Soup for HTML parsing. Install them in your project environment:

python -m pip install requests beautifulsoup4

From the folder containing test_page.html, start Python's local development server:

python -m http.server 8000 --bind 127.0.0.1

Keep that terminal open. Binding to 127.0.0.1 serves the test page on your own computer rather than making it available to other machines on the network. The example scraper will request http://127.0.0.1:8000/test_page.html.

3. Fetch and parse the HTML

Create scrape_local.py in the same folder. This example requests only the fixed local test URL, checks for HTTP errors, uses a timeout, and handles expected connection failures:

import requests
from bs4 import BeautifulSoup

URL = "http://127.0.0.1:8000/test_page.html"


def scrape_cards(url: str) -> list[dict[str, str]]:
    try:
        response = requests.get(url, timeout=(2, 5))
        response.raise_for_status()
    except requests.exceptions.Timeout:
        print("The local server did not respond before the timeout.")
        return []
    except requests.exceptions.ConnectionError:
        print("Could not connect. Is the local server running?")
        return []
    except requests.exceptions.HTTPError as error:
        print(f"The server returned an HTTP error: {error}")
        return []
    except requests.exceptions.RequestException as error:
        print(f"The request failed: {error}")
        return []

    soup = BeautifulSoup(response.text, "html.parser")
    cards = []

    for card in soup.select("article.card"):
        heading = card.select_one("h2")
        paragraph = card.select_one("p")

        # Skip incomplete cards instead of assuming every field exists.
        if heading is None or paragraph is None:
            continue

        cards.append({
            "title": heading.get_text(" ", strip=True),
            "description": paragraph.get_text(" ", strip=True),
        })

    return cards


if __name__ == "__main__":
    results = scrape_cards(URL)

    if not results:
        print("No complete cards were found.")
    else:
        for item in results:
            print(f"Title: {item['title']}")
            print(f"Description: {item['description']}")
            print()

In a second terminal, run the scraper:

python scrape_local.py

You should see the two titles and descriptions from the fixture. If the server is stopped, the script reports a connection problem rather than failing with an unhandled exception.

What the code does

requests.get() makes an HTTP GET request. The timeout tuple sets limits for connecting and waiting for a response; timeouts help prevent a program from waiting indefinitely.
raise_for_status() treats unsuccessful HTTP status codes as errors instead of parsing an error page as though it were the expected content.
BeautifulSoup(response.text, "html.parser") parses the returned HTML using Python's built-in HTML parser.
select() and select_one() find elements using CSS selectors. get_text(" ", strip=True) returns readable text with surrounding whitespace removed.

4. Adapt the selectors carefully

Selectors depend on the HTML structure. If you change the fixture to use <h3> instead of <h2>, update the scraper's select_one("h2") selector as well. If the page has no matching article.card elements, the scraper returns an empty list and prints a message.

When adapting this exercise to a website you are authorized to access:

Read the site's terms and any applicable access rules before collecting data.
Inspect the page's HTML and identify stable, meaningful elements rather than relying on fragile positions such as “the third div.”
Request only the pages and fields you need.
Keep request volume low and add appropriate delays if you are collecting multiple pages and are permitted to do so.
Store only information you are allowed to collect, and follow applicable privacy and data-handling requirements.

A local parsing exercise does not need to contact an outside site. If you later change the program to fetch external URLs, a URL parser or a check of the hostname alone is not a complete security control. Network access can involve redirects, DNS changes, and private or otherwise restricted addresses. Do not turn a beginner scraper into a service that fetches arbitrary URLs supplied by other people; secure server-side fetching needs dedicated controls.

Common mistakes and troubleshooting

The connection is refused

Confirm that the local server is still running, that it is serving the folder containing test_page.html, and that the port in the command matches the port in URL.

The response is a 404 error

Check the filename and the path. With the commands above, the page should be available at /test_page.html.

The script finds no cards

Compare the HTML with the selectors in the code. The fixture needs elements matching article.card, with an h2 and a p inside each card. Remember that CSS class names and tag names must match the markup.

The page looks different in a browser

Some websites add content with JavaScript after the initial HTML loads. A basic requests call retrieves the server's HTTP response; it does not run the page's JavaScript. This tutorial does not cover browser automation, and you should not use automation to evade access restrictions or anti-bot measures.

Responsible scraping checklist

Scrape only your own test page or a page you are explicitly authorized to access.
Respect terms, access restrictions, and reasonable request limits.
Do not bypass authentication, paywalls, CAPTCHAs, or anti-bot protections.
Use timeouts and handle request failures.
Avoid collecting sensitive personal information.
Keep the target URL fixed and controlled in this beginner example; do not accept arbitrary URLs from untrusted users.

Conclusion

A small local scraper is a practical way to learn the basic workflow: serve HTML, request it with Python, parse it with CSS selectors, and handle missing data and HTTP errors. Once you understand those steps, you can improve the project with tests, structured output, and carefully controlled access to pages you are allowed to scrape.

Python Web Scraping Tutorial: Build a Small Local Scraper