| | |
- builtins.object
-
- BackgroundIOPriority
- threading.Thread(builtins.object)
-
- BackgroundWriteProcess
class BackgroundIOPriority(builtins.object) |
| |
BackgroundIOPriority - Priority Profile for doing background writes.
See __init__ for fields
|
| |
Methods defined here:
- __getitem__(self, key)
- __init__(self, chainPollTime, defaultChunkSize, bandwidthPct, numChunksRateSmoothing=5)
- __init__ - Create a BackgroundIOPriority.
Some terms: throughput - Bandwidth out (Megs per second)
interactivity - CPU time available for other tasks (calculations, other I/O, etc)
@param chainPollTime - float > 0, When chaining, this is the sleep time between checking if prior is finished.
Too low and the polling takes up CPU time, too high and you'll lose a little time in between chained writes, while gaining interactivity elsewhere.
@param defaultChunkSize - integer > 0, When providing a straight string/bytes to bgwrite (instead of chunking yourself, or using bgwrite_chunk) this will
be used as the max size of each chunk. Each chunk is written and a flush is issued (if the stream supports it).
Increasing this increases throughput while decreasing interactivity
@param bandwidthPct - integer > 0 and < 100. This is the percentage of overall bandwidth that this task will attempt to use.
A high number means higher throughput at the cost of lest interactivity for other tasks, a low number means the opposite.
So, for example, a bandwidthPct of "50" will attempt to use "50%" of the available bandwidth. Note, this does not represent theroetical
max bandwidth, i.e. the max rate of the I/O device, but the amount of available bandwidth available to this application. For example,
if this is given "100%", no throttling is performed. If this is given "80%", then it calculates the average time to write a single chunk,
( see #numChunksRateSmoothing for how many chunks are used in evaluating this average ), and sleeps for then 20% of that time at the end
of every chunk.
@param numChunksRateSmoothing - integer >= 1 , Default 5. This is the number of chunks which are used in calculating the current throughput rate.
See #bandwidthPct for the other half of the story. The higher this number, the more "fair" your application will be against a constant
rate of I/O by other applications, but the less able it may be to play fair when the external I/O is spiking.
Also, consider that this is related to the #defaultChunkSize, as it is not a constant period of time. The default of "5" should be okay,
but you may want to tune it if you use really large or really small chunk sizes.
An "interactivity score" is defined to be (number of calculations) / (time to write data).
- __setitem__(self, key, value)
Data descriptors defined here:
- bandwidthPct
- chainPollTime
- defaultChunkSize
- numChunksRateSmoothing
|
class BackgroundWriteProcess(threading.Thread) |
| |
BackgroundWriteProcess - A thread and data store representing a background write task. You should probably use one of the bgwrite* methods and not this directly.
Attributes:
remainingData <deque> - A queue representing the data yet to be written
startedWriting <bool> - Starts False, changes to True when writing has started (thread has started and any pending prior chain has completed)
finished <bool> - Starts False, changes to True after writing has completed, and if closeWhenFinished is True the handle is also closed.
|
| |
- Method resolution order:
- BackgroundWriteProcess
- threading.Thread
- builtins.object
Methods defined here:
- __init__(self, fileObj, dataBlocks, closeWhenFinished=False, chainAfter=None, ioPrio=4)
- __init__ - Create the BackgroundWriteProcess thread. You should probably use bgwrite or bgwrite_chunk instead of calling this directly.
@param fileObj <stream> - A stream, like a file, to write into. Hopefully it supports flushing, but it is not a requirement.
@param dataBlocks <bytes/str/list<bytes/str>> - If a list of bytes/str, those are treated as the data blocks, written in order with heuristics for interactivity in between blocks. If bytes/str are provided not in a list form, they will be split based on the rules of the associated #ioPrio
@param closeWhenFinished <bool> - Default False. If True, the fileObj will be closed after writing has completed.
@param chainAfter <None/BackgroundWriteProcess> - If provided, will hold off writing until the provided BackgroundWriteProcess has completed (used for queueing multiple writes whilst retaining order)
@param ioPrio <int/BackgroundIOPriority> - If an integer (1-10), a predefined BackgroundIOPriority will be used. 1 is highest throughput, 10 is most interactivity. You can also pass in your own BackgroundIOPriority object if you want to define a custom profile.
@raises ValueError - If ioPrio is neither a BackgroundIOPriority nor integer 1-10 inclusive
- If chainAfter is not a BackgroundWriteProcess or None
- run(self)
- run - Starts the thread. bgwrite and bgwrite_chunk automatically start the thread.
Methods inherited from threading.Thread:
- __repr__(self)
- Return repr(self).
- getName(self)
- isAlive = is_alive(self)
- Return whether the thread is alive.
This method returns True just before the run() method starts until just
after the run() method terminates. The module function enumerate()
returns a list of all alive threads.
- isDaemon(self)
- is_alive(self)
- Return whether the thread is alive.
This method returns True just before the run() method starts until just
after the run() method terminates. The module function enumerate()
returns a list of all alive threads.
- join(self, timeout=None)
- Wait until the thread terminates.
This blocks the calling thread until the thread whose join() method is
called terminates -- either normally or through an unhandled exception
or until the optional timeout occurs.
When the timeout argument is present and not None, it should be a
floating point number specifying a timeout for the operation in seconds
(or fractions thereof). As join() always returns None, you must call
isAlive() after join() to decide whether a timeout happened -- if the
thread is still alive, the join() call timed out.
When the timeout argument is not present or None, the operation will
block until the thread terminates.
A thread can be join()ed many times.
join() raises a RuntimeError if an attempt is made to join the current
thread as that would cause a deadlock. It is also an error to join() a
thread before it has been started and attempts to do so raises the same
exception.
- setDaemon(self, daemonic)
- setName(self, name)
- start(self)
- Start the thread's activity.
It must be called at most once per thread object. It arranges for the
object's run() method to be invoked in a separate thread of control.
This method will raise a RuntimeError if called more than once on the
same thread object.
Data descriptors inherited from threading.Thread:
- __dict__
- dictionary for instance variables (if defined)
- __weakref__
- list of weak references to the object (if defined)
- daemon
- A boolean value indicating whether this thread is a daemon thread.
This must be set before start() is called, otherwise RuntimeError is
raised. Its initial value is inherited from the creating thread; the
main thread is not a daemon thread and therefore all threads created in
the main thread default to daemon = False.
The entire Python program exits when no alive non-daemon threads are
left.
- ident
- Thread identifier of this thread or None if it has not been started.
This is a nonzero integer. See the thread.get_ident() function. Thread
identifiers may be recycled when a thread exits and another thread is
created. The identifier is available even after the thread has exited.
- name
- A string used for identification purposes only.
It has no semantics. Multiple threads may be given the same name. The
initial name is set by the constructor.
| |