Dragline Tutorial¶

Creating a project¶

Let us think our spider name is pydocs.

Before you start scraping, you will have to set up a new Dragline project. Traverse into a directory where you’d like to store your spider and its associated code and then run the following command:

dragline-admin init pydocs

If you are a django developer then you might be familiar with above type of command.

This will create a pydocs directory with the following two files:

pydocs/
  main.py
  settings.py

These are basically:

• settings.py: the project’s settings file.
• main.py: file where we write our custom spider

Our first Spider¶

Spiders are user-written classes used to scrape information from a domain (or group of domains).

They define an initial list of URLs to download, how to follow links, and how to parse the contents of those pages to extract content.

To create a Spider, you must create a class named Spider in main.py define the three main, mandatory, attributes:

• name: identifies the Spider. It must be unique, that is, you can’t set the same name for different Spiders.

• start: is a list of Request where the Spider will begin to crawl from. So, the first pages downloaded will be those listed here. The subsequent URLs will be generated successively from data contained in the start URLs.

• allowed_domains: is list of the allowed domains through which the spider will crawl

• parse() is a method of the spider, which will be called with the downloaded Response object of each start URL. The response is passed to the method as the first and only argument.

  This method is responsible for parsing the response data and extracting scraped data and more URLs to follow.

This is the code for our first Spider; save it in a file named main.py under the pydocs directory:

from dragline.htmlparser import HtmlParser
from dragline.http import Request


class Spider:

    def __init__(self, *args, **kwargs):
        self.name = "pydocs"
        self.start = Request("https://docs.python.org/3/")

    def parse(self, response):
        html = HtmlParser(response)
        table = html.find('.//table')
        for url in table.extract_urls():
            yield Request(url, callback="parse_group")

    def parse_group(self, response):
        html = HtmlParser(response)
        for i in html.extract_urls('//a[@class="reference internal"]'):
            yield Request(i, callback="parse_page")

    def parse_page(self, response):
        html = HtmlParser(response)
        page = {
            'title': 'strip(//h1)',
            'subtitles': ['strip(//h2)']
        }
        print html.extract(page)