parallel_sync Usage
===================

parallel_sync is a python package for uploading or downloading files using multiprocessing and md5 checks on Linux.
The files can be transferred from a remote linux host or a url.

Benefits:
* Very fast file transfer (parallelized)
* If the file exists and is not changed, it will not waste time copying it
* You can specify retries in case you have a bad connection
* It can handle large files

In most of the examples below, you can specify **parallelism** and **tries** which allow you to parallelize tasks and retry upon failure.
By default **parallelism** is set to 10 workers.


Upstream Example:
::
    from parallel_sync import rsync
    creds = {'user': 'myusername', 'key':'~/.ssh/id_rsa', 'host':'192.168.16.31'}
    rsync.upload('/tmp/local_dir', '/tmp/remote_dir', creds=creds)

Downstream Example:
::
    from parallel_sync import rsync
    creds = {'user': 'myusername', 'key':'~/.ssh/id_rsa', 'host':'192.168.16.31'}
    rsync.download('/tmp/remote_dir', '/tmp/local_dir', creds=creds)


File Download Example:
::
    from parallel_sync import wget
    urls = ['http://something.png', 'http://somthing.tar.gz', 'http://somthing.zip']
    wget.download('/tmp', urls, extract=True)

    # download on a remote machine:
    creds = {'user': 'myusername', 'key':'~/.ssh/id_rsa', 'host':'192.168.16.31'}
    wget.download('/tmp', urls, creds=creds)

    # To untar or unzip compressed files after download:
    wget.download('/tmp', urls, creds=creds, extract=True)


Example extracting a file on a remote host:
::
    creds = {'user': 'myusername', 'key':'~/.ssh/id_rsa', 'host':'192.168.16.31'}
    rom parallel_sync import compression
    compression.extract('/tmp/x.tar.gz', creds=creds)


Example checking that a files exists:
::
    from parallel_sync import executor
    if executor.path_exists(path, creds):
        print "yes"


Example finding files or directories:
::
    from parallel_sync import executor
    files = executor.find_files(dir_path, creds, include=['*.png', '*.jpg'])
    dirs = executor.find_dirs(dir_path, creds, include=['test'])

Note that if creds is None, then it will search on localhost


Example Running commands:
::
    from parallel_sync import executor
    cmds = ['mv /tmp/x /tmp/y', 'touch /tmp/z']
    executor.run(cmds, creds=creds, parallelism=len(cmds)):
    print executor.run('pwd', creds=creds, curr_dir='/tmp')


Example using parallel_sync within fabric:
::
    from fabric.api import env
    from parallel_sync import rsync
    rsync.upload('/tmp/x', '/tmp/y', creds=env)
    rsync.download('/tmp/y', '/tmp/z', creds=env)


.. _github-link: https://github.com/kouroshparsa/parallel_sync
If you come across any bugs, please report it on
   https://github.com/kouroshparsa/parallel_sync