Common Crawl Job Library

CommonCrawlJob is a small module for simplifying common mapreduce workflows over Common Crawl

Author:Sang Han License:Apache Software License v2 Code:Github Codebase Release:v0.0.0 alpha

• Common Crawl Job Library
  • Installation
  • Compatibility
• Getting Started
  • Getting Familiar with the Dataset
• An Example Extractor
  • Running Locally
• Elastic Map Reduce - Configuration
  • Install Hadoop
  • Setup your requirements.txt file:
  • Configuring your own environment
  • Common Crawl Region
  • Create an Amazon EC2 Key Pair and PEM File
  • To create an Amazon EC2 key pair:
  • Configuring mrjob.conf
• Deploy on EMR
  • Start Hadoop
  • Call EMR
  • Sit back as EMR begins processing
  • Next Steps

Indices and tables

• Index
• Module Index
• Search Page