Class HTTPWorker

     object --+    
              |    
pyworker.Worker --+
                  |
                 HTTPWorker

Gets a local copy of the resource at the URL in the JSON msg ('url') and simply
prints the first "line".

Handles an HTTP cache, so that multiple HTTPWorkers performing different tasks
on the same resource will not cause multiple calls to the same resource, unless
the E-Tag header is different.

If the E-Tag header is different, the updated resource is downloaded and passed
instead.

It is expected that self.endtask will be overwritten. 

If the tempfile option is set, remember to delete the temporary file 
as well as ack the msg! Eg -
------------------------------------------------------------
import os
class SolrFeeder(HTTPWorker):
    def endtask(self, msg, response):
        try:
            # do stuff with response.context['fd'], the file-descriptor for the resource
        finally:
            response.context['fd'].close()
            if self.context.get('tempfile', False):
               os.remove(response.context['tempfile'])
            self.queue_stdin.task_done()

s = SolrFeeder(queue_stdin, queue_stdout=None, tempfile = True)
------------------------------------------------------------    
If 'id' is passed in the message instead, then this is inserted into a template, set
by instantiating this worker with the parameter 'http_template'. Normal python
string formating applies ( template % id )

Requires configuration parameters:
    http_template = template for the URL to GET