HEPcrawl

HEPcrawl is a harvesting library based on Scrapy (http://scrapy.org) for INSPIRE-HEP (http://inspirehep.net) that focuses on automatic and semi-automatic retrieval of new content from all the sources the site aggregates. In particular content from major and minor publishers in the field of High-Energy Physics.

The project is currently in early stage of development.

See full documentation at http://pythonhosted.org/hepcrawl

User’s Guide

This part of the documentation will show you how to get started in using HEPCrawl.

• What is HEPcrawl?
• Installation
  • Quick Installation
  • Installation for developers
  • Run a crawler
• Deployment
  • Install HEPCrawl
  • Setup Scrapyd
  • Run Scrapyd
  • Deploy Scrapy project
  • Schedule crawls
  • Pushing to remote servers
  • Install via PyPi
  • Enable Sentry
• Known issues
  • Sentry integration with Python 2.7.9
• Developers Guide
  • Spiders in HEPcrawl
  • Getting data with start_requests
  • Creating records in parse_node
  • Re-using common metadata handling using item loaders
  • Exporting the final record with item pipelines
• Useful tools
  • Testing your spider
  • Writing extraction code with scrapy shell

API Reference

If you are looking for information on a specific function, class or method, this part of the documentation is for you.

• API
  • Items
  • Spiders

Additional Notes

Notes on how to contribute, legal information and changes are here for the interested.

• Changelog
• Release Notes
• License
• Authors

Happy crawling!

INSPIRE Development Team

Email: feedback@inspirehep.net

Twitter: http://twitter.com/inspirehep

GitHub: http://github.com/inspirehep

URL: http://inspirehep.net