| | |
- fetchUrl(url, httpMethod='GET', userAgent='SimpleHttpFetch 4.0.0', defaultEncoding='utf-8', headers=None)
- fetchUrl - Fetches the contents of a url.
Will follow redirects via Location header or 301 status.
@param httpMethod <str> - HTTP Method (default GET)
@param userAgent <str> - User agent to provide, defaults to SimpleHttpFetch <version>
@param defaultEncoding <str> - default utf-8. Encoding to use if one is not specified in headers. If set to "nodecode", the results will not be decoded regardless of headers (use for binary data)
@param headers <None/dict> - overrides to default headers to send. keys is header name, value is header value.
@return <str> - Web page contents
@raises SimpleHttpFetchBadStatus If page does not return status 200 (success)
- fetchUrlAsJson(url, httpMethod='GET', userAgent='SimpleHttpFetch 4.0.0', defaultEncoding='utf-8', headers=None)
- fetchUrl - Fetches the contents of a url and converts the JSON to a python dictionary.
Will follow redirects via Location header or 301 status.
@param httpMethod <str> - HTTP Method (default GET)
@param userAgent <str> - User agent to provide, defaults to SimpleHttpFetch <version>
@param defaultEncoding <str> - default utf-8. Encoding to use if one is not specified in headers. If set to "nodecode", the results will not be decoded regardless of headers (use for binary data)
@param headers <None/dict> - overrides to default headers to send. keys is header name, value is header value.
@return <dict> - Dictionary of parsed JSON on page
@raises ValueError if webpage contents are not JSON-compatible
@raises SimpleHttpFetchBadStatus If page does not return status 200 (success)
- fetchUrlRaw(url, httpMethod='GET', userAgent='SimpleHttpFetch 4.0.0', headers=None)
- fetchUrlRaw - Fetches the contents of a url without decoding the data.
Will follow redirects via Location header or 301 status.
@param httpMethod <str> - HTTP Method (default GET)
@param userAgent <str> - User agent to provide, defaults to SimpleHttpFetch <version>
@param headers <None/dict> - overrides to default headers to send. keys is header name, value is header value.
@return <bytes> - Web page contents, unencoded.
@raises SimpleHttpFetchBadStatus If page does not return status 200 (success)
- getConnection(url)
- getConnection - Get a connection object given a url. Supports http and https
@return - Connection
- getRequestData(connection, url, httpMethod='GET', userAgent='SimpleHttpFetch 4.0.0', defaultEncoding='utf-8', headers=None, _depth=None)
- getRequestData - Given a connection, fetch a URL and return a string of the contents. Use this to make multiple requests instead of fetchUrl to the same server, as it allows you to reuse a connection.
Will follow redirects via Location header or 301 status.
@param connection <obj> - return of getConnection function
@parma url <str> - Url to fetch
@param httpMethod <str> - An http method. Probably GET.
@param userAgent <str> - Your user agent. Defaults to SimpleHttpFetch <version>
@param defaultEncoding <str> - default utf-8. Encoding to use if one is not specified in headers. If set to "nodecode", the results will not be decoded regardless of headers (use for binary data)
@param headers <None/dict> - overrides to default headers to send. keys is header name, value is header value
@param _depth <None/list> - If you pass in a list, you can check that list after the call (the function will modify it) to see if any redirects (301's) were followed. Each url fetched will have an entry, so len(_depth) == 1 means no redirects were followed.
@return <str> - Web page contents
@raises SimpleHttpFetchBadStatus If page does not return status 200 (success)
- getRequestDataAsJson(connection, url, httpMethod='GET', userAgent='SimpleHttpFetch 4.0.0', defaultEncoding='utf-8', headers=None)
- getRequestDataAsJson - Given a connection, fetch a URL and return a string of the contents. Use this to make multiple requests to the same server instead of fetchUrlAsJson, as it allows you to reuse a connection.
Will follow relative redirects via Location header or 301 status.
@param connection <obj> - Return of getConnection function
@param url <str> - Url to fetch
@param httpMethod <str> - An http method. Probably GET.
@param userAgent <str> - Your user agent. Defaults to SimpleHttpFetch <version>
@param defaultEncoding <str> - default utf-8. Encoding to use if one is not specified in headers. If set to "nodecode", the results will not be decoded regardless of headers (use for binary data)
@param headers <None/dict> - overrides to default headers to send. keys is header name, value is header value.
@return <dict> - Dictionary of parsed JSON on page
@raises ValueError if webpage contents are not JSON-compatible
@raises SimpleHttpFetchBadStatus If page does not return status 200 (success)
- parseURL(url)
- parseURL - parses a url and returns a dictionary containing the pieces of information
@param url <string> - A full URL (ex: http://www.example.com/test)
@return - Dictionary describing url. Keys are:
protocol <string> - http or https
domain <string> - host domain/server (ex: example.com)
port <int> - TCP Port for request
rel_uri <string> - Relative URI of request (ex: /index.html)
|