Tutorial: Gathering data with the Scrapy framework

UPDATE: On June 1, 2012, Washington State began allowing private-sector businesses to sell liquor and closed all state-owned stores. The Liquor Control Board web site I used in this example was closed along with them. I’ll leave this tutorial on the site, but only as a reference. If I get time to post a new Scrapy tutorial, I’ll post an update here as well.

This tutorial will walk you through a web-scraping project from scratch using Scrapy, a Python scraping framework. By the end, you’ll be able to:

• Create a spider from scratch using both GET and POST requests
• Handle the responses via Items and Pipelines
• Parse your data into multiple files (in this case – CSV)
• Use XPath selectors to find specific elements within a HTML document

What you’ll need:

• Scrapy (and required libraries – see Scrapy install guide)
• Tutorial files (or you can just go through the steps below)
• Python (I’m using 2.6, but 2.5 or 2.7 should work)
• A browser that makes viewing page markup easy, like Firefox (with Firebug) or Google Chrome

I won’t post any sort of install tutorials here, there is a good one on the Scrapy site already. If you are having trouble getting libraries to install (especially on a Mac), hit me up in the comments or email and I’ll do my best to help. Let’s get started.

Planning the data format

In Washington State, the Liquor Control Board site allows users to search all of the state-controlled liquor stores for specific brands, liquor types, prices, amounts, etc. This is the site we’ll use for our project. Sounds like some interesting data, no?

Before we start writing any code, let’s think about what data we’ll gather and how to save it. For our purposes we’ll focus on the ‘Product Availability‘ page of the site. This page has a list of brand categories in a dropdown list (Brandy, Vodka, Rum, etc.). When you select a category, the site loads a brands page (Smirnoff, Absolute, Grey Goose, etc). From there, you can click one of the ‘Find a store’ buttons to see where the product is in stock and how many bottles are left.

These three pages will be our focus of our project. We’ll break the data into three CSV files – brandCategoryTable.csv, brandsTable.csv, and inStockTable.csv. Here’s the format for each file:

brandCategoryTable.csv
brandCategoryId, brandCategoryName

brandsTable.csv
brandCategoryId, brandId, brandName, totalPrice, specialNote, size

storeStockTable.csv
brandId, stateStoreNumber, amountInStock

NOTE: This project may seem as if there is a database in its future, but that’s not part of this tutorial, so the data format is not a big focus. For now, we’re just focusing on scraping.

Creating a Scrapy project

Open a console and go to the location where you want to create your new project files. Then type:

This creates the basic structure for your project in a folder called “waliquor”. Before it will do anything, we need to define the spider, the items and the pipelines.

Building the spider

Spiders are user-written classes used to scrape information you’re after. They return Items.

Our spider will have three major steps:

1. Request the brand search page containing the list of brand categories. Once we’ve received the response containing all of the page data, write the brand category data to CSV and then…
2. Parse the data for each brand page and create a new request for each brand page. Once we receive the response for a brand page, write the brand data to CSV and then…
3. Parse the data for each store page that has this brand in stock and write the store data to CSV.

Create a new file in your /waliquor/spiders/ folder. Name it “waliquor_spider.py” and insert the following code:

This is the basic structure always needed for a Scrapy spider. Now add this function below the allowed domains, but above “SPIDER = WaliquorSpider()”.

Start_request() is the method called by Scrapy when the spider is opened for scraping when no particular URLs are specified with a default start_urls list.

Once the request is made and the server returns a response, the callback function (self.parseBrandCategories()) takes over. Add this to your spider below the start_requests() function:

Let’s review what’s happening: The function receives the response object from the initial request and creates a HtmlXPathSelector called “hxs”. HtmlXPathSelector gives us a way to look through the HTML and find the specific elements we’re after.

But before we can write an XPath statement, we have to look at the HTML by hand and figure out the structure. This is where a plugin like Firebug can come in very handy. Remember, in this step we’re after the list of brand categories. By looking at the page markup in Firebug, I can see that the brand categories are in a form, then a div, then a center tag, then a table, etc, etc. Eventually, you get to the actual options, where I’ve used the “[position()>1]” property to avoid grabbing the first item: “(Please select…)”. Next is a text() selector to grab only the text, not the tags around it.

ROOKIE MISTAKE: “If you are viewing page markup in Firefox, remember that the browser adds its own < tbody > tags to tables, but Scrapy will not receive those in the response. Including < tbody > in your Scrapy XPath will always break it.

Now that we have the full list of brand categories, we must step through them and create a new request for each – to load the brand pages. Unlike the previous request, these pages will require a POST request.

Again, Firefox/Firebug is a great tool. In this case, you can use it to load a page and then look through the response the browser is sending. Using that information, we added a form request. You may notice another callback function there – we’ll get to that soon. For now, don’t worry about it.

The last part of the parseBrandCategories() function in your spider handles sending the brand category data to its corresponding item. The code creates a BrandCategoryItem item, then the two pieces of data we’re after – brandCategoryId and brandCategoryName – are populated. Remember these fields from our “brandCategoryTable.csv” file?

Defining your items

Items are containers that will be loaded with the scraped data.

Open the items.py file in your project folder and overwrite everything with the following code:

That’s it! An item is a simple container for the values you just parsed via the spider. The last part is outputting your data to a CSV file. For this, we need a pipeline.

Creating pipelines

Pipelines receive the scraped data from the items and stores it in the way you specify.

In your project folder, there is a file called pipelines.py. Open the file and overwrite everything with the following:

The pipeline is using the built-in CSV abilities of Python to open a new file (or an existing one if it’s already been created) and then write a header row to the top. Next, when all the BrandCategoryItem instances are created by your spider, they will automatically be passed in to this pipeline where the process_item() class will add a row to “brandCategoryTable.csv”.

You’ve now gone from an initial request to outputting CSV file! We also completed the first step of our project – getting the brand category data. Don’t worry, the next two steps are very similar, so we’ll fly through them.

Rinse and repeat for the brand pages

Back in your spider class, add the following code at the top of the file:

Also, add this beneath the parseBrandCategories() function:

When the parseBrandCategories() function builds its requests for each brand page, this is the callback function that will receive the response. The process is almost identical to the previous function. It receives the response and creates a HtmlXPathSelector, but this time, instead of selecting strings we’re selecting table rows.

As the for loop steps through the list of rows, the various properties we want are again selected from the table rows using XPath and sent into the item – this time a BrandItem.

Want to learn more about XPath and selectors? Of course you do! Here’s a nice tutorial from Mozilla.

Next, open your items.py file and add the following below the BrandCategoryItem:

Now, we need to wire up the pipeline to take the item and store the data. But wait! We’re going to hit a slight snag here. If you remember, the pipelines.py file uses a function called process_item() to handle each item it receives. But in our case, we have three different items from different functions that need to write to different CSV files. Currently, our pipelines.py class will handle every item the same. Here’s how we’ll get around that:

With this change, the process item function handles the items differently, depending on their type. Now we have lots of data moving through to our CSVs, only one more step – the store page.

Last step, gather and store data

This process should be starting to make sense now. Let’s go back to the beginning – the spider class – one more time. Add the following code at the very beginning:

Then add this function below the parseBrandPage() function:

Next, we’ll add our StockItem to the item.py class:

Lastly, here’s what the complete pipelines.py file should look like:

Besides adding the StockItem in the pipelines class, you’ll noticed I also added a stack of try/excepts to the part handling the BrandItems. In some cases, the values in those fields are sometimes empty. These check make sure that the value will be returned as an empty string, not null in those cases.

Running your spider

If you haven’t already, you’re probably ready to test your project and scrape some data. To run your scraper, open your console and go to your project folder. Next, type:

You should begin to see lots of lines whizzing by as Scrapy makes requests, sends items, etc. If things don’t work the first time, don’t sweat it. Take a close look at any error messages you may be seeing. Python is usually good at pointing you to the file and line number where the error occurred. Also, the link to the working tutorial files is at the beginning of this post. Keep in mind that we’re scraping a lot of data from the site and making thousands of requests. Don’t be surprised if the spider takes 30-40 minutes to finish.

Want to speed up the spider for testing purposes? Go to the parseBrandCategories() function in the spider and change the for loop to read:

This will limit the brand categories you parse to just the first two in the list.