Skip to main content

Building a Complete Scrapy Project

In this tutorial, we will guide you through the process of building a complete Scrapy project from scratch. We will cover the following sections:

  • Setting up the environment
  • Creating a Scrapy project
  • Defining a Scrapy spider
  • Writing spider logic
  • Storing scraped data

1. Setting up the Environment​

Before we start, we need to install Scrapy. If you haven't installed Scrapy yet, you can do so by running the following command in your terminal:

pip install Scrapy

2. Creating a Scrapy Project​

Once Scrapy is installed, we can create our first Scrapy project. Navigate to the directory where you want to create the project and run:

scrapy startproject tutorial

This will create a new directory named "tutorial" with the structure of a Scrapy project.

3. Defining a Scrapy Spider​

Now, let's create a spider. In Scrapy, a spider is a class that defines how a certain site will be scraped. Navigate to the "spiders" directory and create a new Python file. For example, my_spider.py. Here's a basic spider:

import scrapy

class MySpider(scrapy.Spider):
    name = 'my_spider'

    def start_requests(self):
        urls = ['http://example.com']
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        pass  # parsing logic goes here

4. Writing Spider Logic​

The spider logic is defined in the parse method. This is where you extract the data from the website. For example, to scrape titles from a blog:

def parse(self, response):
    titles = response.css('h1::text').getall()  # get all H1 text
    for title in titles:
        yield {'title': title}

Here we used CSS selectors to extract the text inside H1 tags. getall() returns all matches in a list.

5. Storing Scraped Data​

Scrapy provides several methods to store scraped data. The simplest one is to output the data in JSON format. You can do this by providing an output file when running the spider:

scrapy crawl my_spider -o output.json

This will run the spider and store the scraped data in output.json.

In this tutorial, we have covered how to set up a Scrapy project, define a spider, write spider logic, and store scraped data. We hope you find it useful in your journey to mastering Scrapy. Happy scraping!