dht Package

dht Package

An implementation of a Kademlia, a distributed hash table (DHT).

dht Module

models Module

Models JSON serializatin

class bytestag.dht.models.CollectionInfo(file_infos, comment=None, timestamp=None)

Bases: bytestag.dht.models.Serializable

Represents a collection of file infos

COMMENT = 'comment'

FILES = 'files'

HEADER = 'BytestagCollectionInfo'

TIMESTAMP = 'timestamp'

comment

file_infos

classmethod from_json_loadable(d)

timestamp

to_json_dumpable()

class bytestag.dht.models.FileInfo(file_hash, part_hashes, size=None, filename=None)

Bases: bytestag.dht.models.Serializable

Represents the hashes of a file and its parts.

FILENAME = 'filename'

HASH = 'hash'

HEADER = 'BytestagFileInfo'

PART_HASHES = 'parts'

SIZE = 'size'

file_hash

filename

classmethod from_json_loadable(d)

part_hashes

size

to_json_dumpable()

class bytestag.dht.models.JSONDumpable

Bases: builtins.object

classmethod from_json_loadable(o)

to_json_dumpable()

class bytestag.dht.models.JSONKeys

Bases: builtins.object

JSON keys

INDEX = 'index'

KEY = 'key'

NETWORK_ID = 'netid'

NODES = 'nodes'

NODE_ID = 'nodeid'

RPC = 'rpc'

class RPCs

Bases: builtins.object

FIND_NODE = 'findnode'

FIND_VALUE = 'findval'

GET_VALUE = 'getval'

PING = 'ping'

STORE = 'store'

JSONKeys.SIZE = 'size'

JSONKeys.TIMESTAMP = 'timestmp'

JSONKeys.TRANSFER_ID = 'xferid'

JSONKeys.VALUES = 'vals'

JSONKeys.VALUE_OFFSET = 'valofs'

class bytestag.dht.models.KVPExchangeInfo

Bases: bytestag.dht.models.KVPExchangeInfo, bytestag.dht.models.JSONDumpable

Description about a key-value used for storage decisions.

INDEX = 'index'

KEY = 'key'

SIZE = 'size'

TIMESTAMP = 'time'

classmethod from_json_loadable(dict_obj)

classmethod from_kvp_record(kvp_record)

to_json_dumpable()

class bytestag.dht.models.KVPExchangeInfoList

Bases: builtins.list, bytestag.dht.models.JSONDumpable

A list of KVPExchangeInfo

classmethod from_json_loadable(array)

Return a node list

Raises:

• TypeError – Didn’t get a dict
• ValueError – The list is invalid

classmethod from_kvp_record_list(kvp_record_list)

to_json_dumpable()

class bytestag.dht.models.NodeList

Bases: builtins.list, bytestag.dht.models.JSONDumpable

A list of nodes

A node list is an array of objects.

Each object must contain the host name, port number, and base64 encoded node ID.

HOST = 'host'

NODE_ID = 'id'

PORT = 'port'

classmethod from_json_loadable(array)

Return a node list

Raises:

• TypeError – Didn’t get a dict
• ValueError – The list is invalid

sort_distance(key)

Sort inplace the list by distance to given key.

The first item in the list is closest to the given key.

to_json_dumpable()

Return a list of dict

Return type:list

class bytestag.dht.models.Serializable

Bases: bytestag.dht.models.JSONDumpable

classmethod from_bytes(bytes_)

to_bytes()

models_test Module

class bytestag.dht.models_test.TestCollectionInfo(methodName='runTest')

Bases: unittest.case.TestCase

test_read_json()

It should read in json with basic info

class bytestag.dht.models_test.TestFileInfo(methodName='runTest')

Bases: unittest.case.TestCase

test_read_json()

It should read in a json with basic info

test_read_json_extended()

It should read in a json with extended info

class bytestag.dht.models_test.TestKVPExchangeInfo(methodName='runTest')

Bases: unittest.case.TestCase

test_json_support()

It should convert to json and back

class bytestag.dht.models_test.TestKVPExchangeInfoList(methodName='runTest')

Bases: unittest.case.TestCase

test_josn_support()

It should convert to json and back

class bytestag.dht.models_test.TestNodeList(methodName='runTest')

Bases: unittest.case.TestCase

test_josn_support()

It should convert to json and back

network Module

DHT networking protocols

class bytestag.dht.network.DHTNetwork(event_reactor, kvp_table, node_id=None, network=None, download_slot=None)

Bases: bytestag.events.EventReactorMixin

The distributed hash table network

Cvariables :

NETWORK_ID

The unique network id reserved only use in the Bytestag network.

MAX_VALUE_SIZE = 1048576

NETWORK_ID = 'BYTESTAG'

NETWORK_PARALLELISM = 3

TIME_EXPIRE = 86490

TIME_REFRESH = 3600

TIME_REPLICATE = 3600

TIME_REPUBLISH = 86400

address

The address of the server

Returns:A tuple holding host and port number. Return type:tuple

download_slot

The FnTaskSlot which holds ReadStoreFromNodeTask.

find_node_shortlist(key)

Return nodes close to a key

Return type:FindShortlistTask

find_nodes_from_node(node, key)

Find the closest nodes to a key

Return type:FindNodesFromNodeTask Returns:A future which returns a NodeList or None.

find_value_from_node(node, key, index=None)

Ask a node about values for a key

Parameters :

node: Node

The node to be contacted

key: KeyBytes

The key of the value

index: KeyBytes, None

If given, the request will be filtered to that given index.

Return type:

FindValueFromNodeTask

Returns:

A future which returns a FindValueFromNodeResult or None.

find_value_shortlist(key, index=None)

Return nodes close to a key and may have the value

Return type:FindShortlistTask

get_value(key, index)

get_value_from_node(node, key, index=None, offset=None)

Download, from a node, data value associated to the key

Return type:DownloadTask

join_network(address)

Join the network

Return type:JoinNetworkTask Returns:A future that returns bool. If True, the join was successful.

key

The node id

Return type:KeyBytes

node

The node info

Return type:Node

ping_address(address)

Ping an address

Return type:PingTask Returns:A future which returns bool or a tuple of (float, Node). If a tuple is returned, the ping was successful. The items represents the ping time and the node.

ping_node(node)

Ping a node

See :ping_address Return type:PingTask

routing_table

The routing table

Return type:RoutingTable

store_to_node(node, key, index, bytes_, timestamp)

Send data to node.

Return type:StoreToNodeTask

store_value(key, index)

Publish or replicate value to nodes.

Return type:StoreValueTask

class bytestag.dht.network.FindNodesFromNodeTask(*args, **kwargs)

Bases: bytestag.events.Task

run(controller, node, key)

class bytestag.dht.network.FindShortlistTask(*args, **kwargs)

Bases: bytestag.events.Task

Returns Shortlist

run(controller, key, index=None, find_nodes=True)

find x loop

class bytestag.dht.network.FindValueFromNodeResult

Bases: bytestag.dht.network.FindValueFromNodeResult

A named tuple representing key-value pair information or additional nodes.

class bytestag.dht.network.FindValueFromNodeTask(*args, **kwargs)

Bases: bytestag.events.Task

run(controller, node, key, index)

class bytestag.dht.network.GetValueTask(*args, **kwargs)

Bases: bytestag.events.Task

Returns the bytes

run(controller, key, index)

class bytestag.dht.network.JoinNetworkTask(*args, **kwargs)

Bases: bytestag.events.Task

run(controller, address)

class bytestag.dht.network.PingTask(*args, **kwargs)

Bases: bytestag.events.Task

run(address, controller)

class bytestag.dht.network.ReadStoreFromNodeTask(*args, **kwargs)

Bases: bytestag.network.DownloadTask

class bytestag.dht.network.Shortlist(key_obj, routing_table, server_node)

Bases: builtins.object

A shortlist containing close nodes to a key

add_nodes(node_list)

Add more possible nodes to contact

get_common_kvp_exchange_info(key, index)

get_nodes_for_contacting(count=3)

Pop nodes off uncontacted list

Return type:list

is_finished()

Return whether the shortlist is complete

Return type:bool

mark_node(node, active, useful=False, kvp_exchange_info_list=None)

Add or remove the node from the shortlist.

Parameters :

node: Node

The node

active: bool

Whether the node responded

useful: bool

If True, the node has the value

data_size: int

The size of the value

Call this using the nodes from get_nodes_for_contacting.

nodes

The nodes in the shortlist.

Return type:set

sorted_nodes

The nodes sorted by distance.

The first node is the closest.

Return type:list

useful_nodes

The nodes that may have the value associated to the key.

Return type:set

class bytestag.dht.network.StoreToNodeTask(*args, **kwargs)

Bases: bytestag.events.Task

Returns the number of bytes sent

run(controller, node, key, index, bytes_, timestamp)

class bytestag.dht.network.StoreValueTask(*args, **kwargs)

Bases: bytestag.events.Task

Stores a value to many nodes

run(controller, key, index)

store_to_node_task_observer

Observer for when StoreToNodeTask is started and finished.

The first argument of the callback is bool. If True, the task was created. Otherwise, the task was finished. The second argument is StoreToNodeTask

network_test Module

class bytestag.dht.network_test.TestNetworkControllerMultiNode(methodName='runTest')

Bases: unittest.case.TestCase

TIMEOUT = 5

join_event_reactors()

setup_nodes(count=2)

stop_event_reactors()

test_find_binary_value_size_from_node()

It should get the size of the data from the node

test_find_node_rpc()

It should get a list of nodes

test_get_value_from_other_node()

It should download the value from the other node

test_ping_address()

It should send a ping and receive a response

test_ping_rpc()

It should send a PING and reply with a PONG and both contacts are added to routing table

test_store_to_node()

It should store the data to another node

publishing Module

DHT publisher

This module includes classes that scan tables for values to publish or replicate.

class bytestag.dht.publishing.Publisher(event_reactor, dht_network, kvp_table, fn_task_slot)

Bases: bytestag.events.EventReactorMixin

Publishes values typically created by the client.

REPUBLISH_CHECK_INTERVAL = 3600

class bytestag.dht.publishing.Replicator(event_reactor, dht_network, kvp_table, fn_task_slot)

Bases: bytestag.events.EventReactorMixin

Replicates values typically stored into the cache by other nodes.

tables Module

Node tables

class bytestag.dht.tables.Bucket(number)

Bases: builtins.object

A bucket of nodes.

This class supports container methods with Node.

MAX_BUCKET_SIZE = 20

keep_new_node()

Keep the new node

The old node has not responded

keep_old_node()

Keep the old node

The old node has responded

last_update

Return the time the bucket was last updated

node_update(node)

Add or move the node to end of list

Once the bucket is full, the bucket is locked and any other nodes are ignored. To unlock the bucket, call either keep_old_node or keep_new_node.

Raises BucketFullError:  bucket is full

nodes

The nodes in the bucket

Return type:list Returns:A list of Node

number

The bucket number

Return type:int

exception bytestag.dht.tables.BucketFullError(bucket_number, node, bucket, *args)

Bases: builtins.Exception

Bucket is full

Ivariables :

bucket_number: int

The bucket number

bucket: Bucket

The Bucket

node: Node

The Node

class bytestag.dht.tables.Node(key, address)

Bases: builtins.object

An end-point connection.

This class implements equality and comparison operators using the key and address of the node.

address

Return the address of the node

Return type:tuple

key

Return the key of the node

Return type:KeyBytes

class bytestag.dht.tables.RoutingTable(key=None)

Bases: builtins.object

A list of buckets

buckets

The buckets

count_close(key)

Return the number of node closer than the given key

get_bucket(node)

Get the appropriate bucket for the node

Return type:Bucket

get_bucket_number(node)

Get the appropriate bucket number for the node

Return type:int

get_close_nodes(key, count=3)

Return the closest nodes to a key

Parameters :

key : KeyBytes

The target key

count: int

The maximum length of the list returned

Returns:

A list of Node

Return type:

list

node_update(node)

Call the appropriate bucket update

num_contacts

Return number of contacts

tables_test Module

Tables test

class bytestag.dht.tables_test.TestRoutingTable(methodName='runTest')

Bases: unittest.case.TestCase

test_add_node()

It should add a node

test_add_self()

It should not add a Node with a KeyBytes that is ours

test_full_bucket()

It should raise exception when the 8th bucket is full

test_node_equality()

It should be equal if the node’s id and address are equal

test_node_in_container()

It should return that the node is in the list with two instances of Node with same id and address