Spyda - Python Spider Tool and Library
spyda is a set of tools and a library written in the Python Programming Language for web crawling, article extraction entity matching and rdf graog geberatuib.
copyright: | CopyRight (C) 2012-2013 by James Mills |
---|
Crawler
Crawl a given url recursively for urls.
Parameters: |
|
---|---|
Returns: | A dict in the form {“error”: set(...), “urls”: set(...)} The errors set contains 2-item tuples of (status, url) The urls set contains 2-item tuples of (rel_url, abs_url) |
Return type: | dict |
Also in verbose mode each followed URL is printed in the form: <status> <reason> <type> <length> <link> <url>
Deprecated since version 0.0.2: The allowed_url parameter has been deprecated in favor of two additional parameters blacklist and whitelist.
New in version 0.0.2: blacklist and whitelist control a regex matched pattern of urls to traverse.
Web Extraction Tool
Entity Matching Tool
Utilities
Removes HTML or XML character references and entities from a text string.
Parameters: | text – The HTML (or XML) source text. |
---|---|
Returns: | The plain text, as a Unicode string, if necessary. |
Convert some common unicode characters to their plain text equivilent.
This includes for example left and right double quotes, left and right single quotes, etc.
Use SequenceMatcher to return list of close matches.
word is a sequence for which close matches are desired (typically a string).
possibilities is a list of sequences against which to match word (typically a list of strings).
Optional arg n (default 3) is the maximum number of close matches to return. n must be > 0.
Optional arg cutoff (default 0.6) is a float in [0.0, 1.0]. Possibilities that don’t score at least that similar to word are ignored.
The best (no more than n) matches among the possibilities are returned in a list, sorted by similarity score, most similar first.