🐍 Lesson 46: Web Scraper Project (Python)
1. Project Overview
မြန်မာ → Web Scraping ဆိုတာ website တစ်ခုထဲက data (ဥပမာ – သတင်းခေါင်းစဉ်တွေ၊ ဈေးနှုန်းတွေ၊ အချက်အလက်စာရင်းတွေ) ကို Python script နဲ့ အလိုအလျောက် ယူနိုင်တဲ့ နည်းလမ်း။
English → Web scraping is the process of extracting data from websites automatically using Python scripts.
2. Required Libraries
requests→ website ကို request ပေးဖို့beautifulsoup4→ HTML ကို parse လုပ်ဖို့
Install: pip install requests beautifulsoup4
3. အကျဉ်းချုပ်
✅ Web Scraping = website data ကို Python နဲ့ ယူခြင်း
✅ Libraries → requests + BeautifulSoup
✅ Example → news headlines, product prices
✅ Error handling → network error, HTML changes
python
# ===== 1. Basic Web Scraper Setup =====
import requests
from bs4 import BeautifulSoup
# ===== 2. Example: Scraping News Headlines =====
print("===== Web Scraping Example =====")
# Note: This is a demonstration. In real use, replace URL with target site.
# URL = "https://news.ycombinator.com/"
# response = requests.get(URL)
# soup = BeautifulSoup(response.text, "html.parser")
# headlines = soup.find_all("a", class_="storylink")
#
# for i, headline in enumerate(headlines[:10], 1):
# print(f"{i}. {headline.text}")
print("\n===== Scraping Process =====")
print("1. Use requests.get(URL) to fetch HTML")
print("2. Parse HTML with BeautifulSoup")
print("3. Find elements with soup.find_all()")
print("4. Extract text/data from elements")
print("5. Save or process the data")
# ===== 3. Error Handling =====
print(f"\n===== Error Handling =====")
print("✅ Handle requests.exceptions.RequestException")
print("✅ Check response status code")
print("✅ Validate HTML structure changes")
# ===== 4. Best Practices =====
print(f"\n===== Best Practices =====")
print("✅ Check robots.txt before scraping")
print("✅ Use headers to identify your bot")
print("✅ Add delays between requests")
print("✅ Respect website terms of service")You should see
===== Web Scraping Example ===== ===== Scraping Process ===== 1. Use requests.get(URL) to fetch HTML 2. Parse HTML with BeautifulSoup 3. Find elements with soup.find_all() 4. Extract text/data from elements 5. Save or process the data ===== Error Handling ===== ✅ Handle requests.exceptions.RequestException ✅ Check response status code ✅ Validate HTML structure changes ===== Best Practices ===== ✅ Check robots.txt before scraping ✅ Use headers to identify your bot ✅ Add delays between requests ✅ Respect website terms of service