WAi Forward Developer Blog

Getting Started with Web Scraping Using BeautifulSoup and Python

By: WAI Forward ltd on June 27, 2025

Web scraping is the practice of extracting data from websites using automation. Instead of copying and pasting information by hand, you write a short script that fetches pages and pulls out the bits you care about. It’s a powerful way to collect prices, headlines, tables, and other structured content at scale without hours of manual effort.

Python’s BeautifulSoup library is a popular choice because it turns messy HTML into a tree you can navigate with a few readable commands. In this beginner-friendly guide, you’ll go from installing the tools to parsing, extracting, and saving data, with a quick look at best practices and ethics so you scrape responsibly.

Why Use Web Scraping?

Scraping is handy whenever useful information lives on web pages instead of neat downloads or official APIs. Analysts use it to build datasets, founders use it to watch competitors, and researchers use it to track changes over time. In short: when the data is public but inconvenient to collect, scraping bridges the gap.

Data collection for prices, listings, articles, and more.
Market research across multiple sites to spot trends.
Automation that replaces repetitive copy-and-paste work.

Step 1: Install Required Libraries

You’ll need a way to download pages and a way to parse them. Install both:

pip install requests beautifulsoup4

Step 2: Import Necessary Modules

Start your script by bringing the libraries into scope:

import requests
from bs4 import BeautifulSoup

Step 3: Fetch a Web Page

Use requests to retrieve the HTML for any URL:

# Define the URL of the webpage you want to scrape
url = "https://example.com"

# Send a request to the website to fetch its content
response = requests.get(url)

# Check if the request was successful (status code 200 means success)
if response.status_code == 200:
    print("Successfully fetched the webpage!")
else:
    print(f"Failed to fetch webpage: {response.status_code}")

If the request succeeds, response.text holds the page’s HTML. If not, the status code tells you what went wrong.

Step 4: Parse HTML Content

BeautifulSoup turns raw HTML into a searchable structure:

# Parse the HTML content using BeautifulSoup
soup = BeautifulSoup(response.text, "html.parser")

# Print the formatted (prettified) HTML structure
print(soup.prettify())

Now you can navigate the page with tag names, attributes, and CSS-like queries.

Step 5: Extract Specific Data

1) Find all links on the page

# Extract all links from the page
links = soup.find_all("a")

# Print each link URL
for link in links:
    print(link.get("href"))

How it works: find_all("a") returns each anchor tag. get("href") grabs the URL.

2) Extract text from a specific class

# Extract text from all <h2> elements with class "title"
titles = soup.find_all("h2", class_="title")

for title in titles:
    print(title.text)

This filters elements to those with the class you care about and prints just the text.

3) Extract table data

# Find the first table on the page
table = soup.find("table")

# Extract all rows from the table
table_rows = table.find_all("tr")

# Loop through each row and extract cell data
for row in table_rows:
    cells = row.find_all("td")
    print([cell.text.strip() for cell in cells])

Tables are great for structured content. Strip whitespace so your data is clean when you save it.

Step 6: Save Scraped Data to a CSV File

Once you’ve collected items (e.g., titles), save them to a CSV for analysis:

import csv

with open("scraped_data.csv", "w", newline="", encoding="utf-8") as file:
    writer = csv.writer(file)
    writer.writerow(["Title"])  # header row
    for title in titles:
        writer.writerow([title.text])

CSV is a convenient format that opens in spreadsheets and most analytics tools.

Best Practices for Web Scraping

Scrape kindly. Check each site’s rules, avoid hammering servers, and identify your requests when appropriate.

Respect robots.txt: Review site policies before scraping.
Rate limit: Add short delays to prevent overload and bans.
Use headers: Mimic a browser with a User-Agent when needed.

Ethical Considerations

Public doesn’t always mean permitted. Make sure your use complies with laws and terms of service, and prefer official APIs when they exist.

Follow legal and privacy rules: Avoid scraping personal or copyrighted data.
Be considerate: Keep request volume modest.
Prefer APIs: They’re cleaner, faster, and more stable when available.

FAQs

Can I scrape any website?
Not always. Check robots.txt and the site’s terms. Some prohibit scraping.

What if a site blocks my scraper?
Add headers, manage sessions, and slow down requests. If scraping is disallowed, stop.

Is web scraping legal?
It depends on jurisdiction and the site’s terms. When in doubt, seek permission or legal advice.

How do I scrape JavaScript-rendered pages?
Use tools like Selenium or headless browsers; BeautifulSoup parses static HTML only.

How do I avoid IP bans?
Throttle requests, randomize timing, cache results, and respect site limits.

Conclusion

With requests and BeautifulSoup, you can turn unstructured pages into tidy datasets in just a few lines of Python. Start small, scrape responsibly, and build from there — the web is full of useful information once you know how to extract it.

Back to Blog