Web Scraping with Python Training

Throughout this course, you will learn how to extract, alter, and use data from websites efficiently. The training provides practical skills that quickly turn you into an expert web scraper.

What is web scraping?

Web scraping is the process of obtaining data from websites. It is also known as web harvesting or web data extraction. It entails accessing websites using automated tools or scripts, retrieving web page content, and parsing and extracting the needed information from that source. Web scraping is a technique often used to collect data from the internet for various reasons, including data analysis, research, content aggregation, price comparison, and more.

Roles and Responsibilities in Web Scraping

Project Manager:

Define the project’s goals and requirements. Scraping tasks should be planned and scheduled. Control the project’s budget. Coordinate team members’ communication. Ensure that all legal and ethical norms are followed.

Data Analyst/Scientist:

Determine the data sources and needs. Define the rules for data extraction and transformation. Analyze and interpret the data that has been extracted.

Present data-driven discoveries and insights.

Web Scraping Developer/Engineer:

Create scripts or code for web scraping. Create the scraping environment, which includes tools and libraries. Maintain and monitor the scraping process.

Handle scraping exceptions and errors.

Database Administrator:

Create and manage the infrastructure for data storage. Improve data storage efficiency and scalability. Maintain data security and control.

Syllabus of Web Scraping with Python

Part 1: Introduction 

• Introduction to BeautifulSoup
• Installing BeautifulSoup
• Running BeautifulSoup
• Connecting Reliably

Part 2: Starting to Crawl

• Traversing a Single Domain
• Crawling an Entire Site
• Collecting Data Across an Entire Site
• Crawling Across the Internet
• Crawling with Scrapy

Part 3: Storing Data

• Media Files
• Storing Data to CSV
• MySQL
• Installing MySQL
• Some Basic Commands
• Integrating with Python
• Database Techniques and Good Practice

Part 4: Reading Documents

• Document Encoding
• Text
• Text Encoding and the Global Internet
• CSV
• Reading CSV Files
• PDF
• Microsoft Word and .docx

Part 5: Cleaning Data

• Cleaning in Code
• Data Normalization
• Cleaning After the Fact
• OpenRefine

Part 6: Reading and Writing Natural Languages

• Summarizing Data
• Markov Models
• Six Degrees of Wikipedia: Conclusion
• Natural Language Toolkit
• Installation and Setup
• Statistical Analysis with NLTK
• Lexicographical Analysis with NLTK

Part 7: Crawling Through Forms and Logins

• Python Requests Library
• Submitting a Basic Form
• Radio Buttons, Checkboxes, and Other Inputs
• Submitting Files and Images
• Handling Logins and Cookies
• HTTP Basic Access Authentication

Part 8: Image Processing and Text Recognition

• Overview of Libraries
• Pillow
• Tesseract
• NumPy
• Processing Well-Formatted Text
• Scraping Text from Images on Websites
• Reading CAPTCHAs and Training Tesseract
• Training Tesseract
• Retrieving CAPTCHAs and Submitting Solutions