Crawley’s Documentation

Crawley is a pythonic Scraping / Crawling Framework intended to make easy the way you extract data from web pages into structured storages such as databases.

Features

• High Speed WebCrawler built on Eventlet.
• Supports databases engines like Postgre, Mysql, Oracle, Sqlite.
• Command line tools.
• Extract data using your favourite tool. XPath or Pyquery (A Jquery-like library for python).
• Cookie Handlers.
• Very easy to use (see the example).

Table of Contents

• Pypi Example
  • Models
  • Crawlers
  • Run crawley
  • Downloading the Code
• Settings File
  • Configure your database
  • Sqlite
  • MySql
  • Postgres
  • Oracle
• Reference
  • Crawlers
  • Scrapers
  • Persistance
  • Connector
  • Extractors
  • Utils
  • Manager
  • Commands
• More Examples