fandango.arrays module ¶

Contents

Description ¶

h1. fandango.arrays module

h2. Array Decimation

It contains many decimation methods, the one used by taurustrend is filter_array; which averages data at some given time intervals

Classes ¶

Grid ¶

class fandango.arrays.Grid(rowNames, colNames, *args, **kwds)[source]¶

Sat May 28 13:07:53 CEST 2005 You caught me in a good mood this morning. I woke to sunshine for the first time in many days, that might have something to do with it :-) Here is a dict subclass that extends __getitem__ and __setitem__ to allow setting an entire row. I included extensive doctests to show you what it does. Note: you can write d[‘A’,1] instead of d[(‘A’,1)], which looks a little cleaner. Kent

A two-dimensional array that can be accessed by row, by column, or by cell. Create with lists of row and column names plus any valid dict() constructorargs. >>> data = Grid( [‘A’, ‘B’], [1, 2] ) Row and column lists must not have any values in common. >>> data = Grid([1, 2], [2, 3]) Traceback (most recent call last): ValueError: Row and column lists must not have any values in common

CSVArray ¶

class fandango.arrays.CSVArray(filename=None, header=None, labels=None, comment=None, offset=0, trace=False)[source]¶

colByHead(head, header=None)[source]¶: Get the index of the column headed by ‘head’

fill(y=None, head=None, previous=[])[source]¶: Fill all the empty cells in a column with the last value introduced on it By default this method adds new values to a column only when the previous column remains unchanged To force all the values to be filled set previous=True

get(x=None, y=None, head=None, row=None, column=None, distinct=False, xsubset=[], ysubset=[])[source]¶: get(x=None,y=None,head=None,distinct=False):

getAsTree(root=0, xsubset=[], lastbranch=None, expand=False)[source]¶

This method returns the content of the array as recursive dicts It will produce the right output only if the array has been filled before! [self.fill(head=h) for h in self.getHeaders()]

headers={dict with headers as keys} for h in headers: fill with distinct keys on each column get the keys on first column

for each key, get the distinct keys in the next column matching lines

for each key, get the distinct keys in the next column matching lines

getd(row)[source]¶: This method returns a line as a dictionary using the headers as keys

resize(x, y)[source]¶: TODO: This method seems quite buggy, a refactoring should be done

set(x, y, val)[source]¶: def set(self,x,y,val):

updateFromTree(tree)[source]¶: This method takes a dictionary of type {level0:{level1:{level2:{}}}} and converts it into a grid like level0 level1 level2 WARNING!: This method deletes the actual content of the array!

DictFile ¶

class fandango.arrays.DictFile[source]¶

@TODO f = IndexedFile(‘/a/b/c.eps’,offset=0,split=’,;

‘,escape=’”’,comment=’#’,skip=True or ‘’,strip=True or ‘#.*’,case=True)

f.__getitem__(row,word=None)

exclusion regexp!!

()!()

changed()[source]¶: return True if file changed from last loading

get(row, word=None, split=True)[source]¶

Should understand both int or string indexes

For int, return row For str, return line that first word matches (strip key) Word = int, return word at column Word = regexp, return list with matching strings If both regexp, return all matchings in single list If both int, return single element

sections(begin, end)[source]¶

begin,end are regexps returns a dict with begins as keys and begin-end texts as values

txt =

<data 1>

da da da

</data> <data 2>

...

</data>

returns {‘<data 1>’:’da da da’,’<data 2>’:’...’}

It should work for texts like:

Header: bla Data:

da di do

End:

{‘Header:’:bla,’Data:’:’da

di do’}

update()[source]¶: reload from file (if changed)

TimedQueue ¶

class fandango.arrays.TimedQueue(arg=None)[source]¶

A FIFO that keeps all the values introduced at least for a given time. Applied to some device servers, to force States to be kept at least a minimum time. Previously named as device.StateQueue pop(): The value is removed only if delete_time has been reached. at least 1 value is always kept in the list

append(obj, keep=15)[source]¶: Inserts a tuple with (value,insert_time,delete_time=now+keep)

pop(index=0)[source]¶: Returns the indicated value, or the first one; but removes only if delete_time has been reached. All values are returned at least once. When the queue has only a value, it is not deleted.

Functions ¶

get_max_step ¶

fandango.arrays.get_max_step(data, index=False)[source]¶

array2dict ¶

fandango.arrays.array2dict(table)[source]¶: Converts a table in a dictionary of left-to-right nested data, unnested columns are lists

decimate_custom ¶

fandango.arrays.decimate_custom(seq, cmp=None, pops=None, keeptime=3960.0000000000005)[source]¶: @NOTE: Although faster, filter_array provides a better decimation for trends It will remove all values from a list that doesn’t provide information. In a set of X consecutive identical values it will remove all except the first and the last. A custom compare method can be passed as argument :param seq: a list of (timestamp,value) values

dict2array ¶

fandango.arrays.dict2array(dct)[source]¶: Converts a dictionary in a table showing nested data, lists are unnested columns

tree2table ¶

fandango.arrays.tree2table(tree)[source]¶

tree2table is different from dict2array, because allows list=row instead list=column {A:{AA:{1:{}},AB:{2:{}}},B:{BA:{3:{}},BB:{4:{}}}} should be represented like A AA 1

AB 2

B BA 3: BB 4

correlate_values ¶

fandango.arrays.correlate_values(values, stop=None, resolution=None, debug=False, rule=None, AUTO_SIZE=50000, MAX_SIZE=0, MIN_RESOLUTION=0.05)[source]¶: Correlates values to have all epochs in all columns :param values: {curve_name:[values]} :param resolution: two epochs with difference smaller than resolution will be considered equal :param stop: an end date for correlation :param rule: a method(tupleA,tupleB,epoch) like (min,max,median,average,last,etc...) that will take two last column (t,value) tuples and time and will return the tuple to keep

print_histogram ¶

fandango.arrays.print_histogram(data, n=20)[source]¶: Prints Groups of data in N steps

decimator ¶

fandango.arrays.decimator(data, min_inc=0.05, min_rel=None, max_size=2048, max_iter=1000)[source]¶: @NOTE, this is intensive when focusing on size, filter_array is better if you want to keep the shape over time++89+77 This size-focused decimator sequentially iterates until reducing the size of the array

logfloor ¶

fandango.arrays.logfloor(x)[source]¶

choose_max_value ¶

fandango.arrays.choose_max_value(v, w, t=0, tmin=-300)[source]¶: Args are v,w for values and t for point to calcullate; tmin is the min epoch to be considered valid

maxmin ¶

fandango.arrays.maxmin(*args)[source]¶: Returns timed ((t,max),(t,min)) values from a (t,v) dataset When used to filter an array the winndow will have to be doubled to allocate both values (or just keep the one with max absdiff from previous).

values2text ¶

fandango.arrays.values2text(table, order=None, sep='\t', eof='\n', header=True)[source]¶: Input is a dictionary of {variable:[(time,values)]}; it will convert the dictionary into a text table with a column for each variable It assumes that values in diferent columns have been already correlated

decimate_array ¶

fandango.arrays.decimate_array(data, fixed_size=0, keep_nones=True, fixed_inc=0, fixed_rate=0, fixed_time=0, logger=None)[source]¶: @NOTE: filter_array provides a better decimation for trends Decimates a [(time,numeric value)] buffer by size/rate/increment/time Repeated values are always decimated. keep_nones forces value-to-None steps to be kept fixed_size: smaller increments of data will be pruned until fitting in a fixed_size array fixed_inc: keeps all values with an absolute increment above fixed_inc; overrides fixed_size fixed_step keeps some random values by order, overrides fixed_inc and fixed_size fixed_time keeps some time-fixed values; it takes preference over the rest

logroof ¶

fandango.arrays.logroof(x)[source]¶

rms_value ¶

fandango.arrays.rms_value(*args)[source]¶

choose_last_value ¶

fandango.arrays.choose_last_value(v, w, t=0, tmin=-300)[source]¶: Args are v,w for values and t for point to calcullate; tmin is the min epoch to be considered valid

choose_last_max_value ¶

fandango.arrays.choose_last_max_value(v, w, t=0, tmin=-300)[source]¶: This method returns max value for epochs out of interval For epochs in interval, it returns latest Args are v,w for values and t for point to calcullate; tmin is the min epoch to be considered valid

filter_array ¶

fandango.arrays.filter_array(data, window=300, method=<function average at 0x7eff57531938>, begin=0, end=0, filling=0, trace=False)[source]¶

The array returned will contain @method applied to @data split in @window intervals First interval will be floor(data[0][0],window)+window, containing average of data[t0:t0+window] If begin,end intervals are passed, cut-off and filling methods are applied The value at floor(time) is the value that closes each interval

IF YOUR DATA SOURCE IS A PYTHON LIST; THIS METHOD IS AS FAST AS NUMPY CAN BE (crosschecked with 1e6 samples against the PyTangoArchiving.utils.decimate_array method using numpy)

average ¶

fandango.arrays.average(*args)[source]¶

choose_first_value ¶

fandango.arrays.choose_first_value(v, w, t=0, tmin=-300)[source]¶: Args are v,w for values and t for point to calcullate; tmin is the min epoch to be considered valid

get_histogram ¶

fandango.arrays.get_histogram(data, n=20, log=False)[source]¶: Groups data in N steps

maxdiff ¶

fandango.arrays.maxdiff(*args)[source]¶: Filter that maximizes changes (therefore, noise)

notnone ¶

fandango.arrays.notnone(*args)[source]¶

raw autodoc ¶

class fandango.arrays.Grid(rowNames, colNames, *args, **kwds)[source]

Bases: dict

class fandango.arrays.CSVArray(filename=None, header=None, labels=None, comment=None, offset=0, trace=False)[source]

Bases: object

Labeled(fun)[source]¶

colByHead(head, header=None)[source]: Get the index of the column headed by ‘head’

expandAll()[source]¶

fill(y=None, head=None, previous=[])[source]: Fill all the empty cells in a column with the last value introduced on it By default this method adds new values to a column only when the previous column remains unchanged To force all the values to be filled set previous=True

find(values)[source]¶

get(x=None, y=None, head=None, row=None, column=None, distinct=False, xsubset=[], ysubset=[])[source]: get(x=None,y=None,head=None,distinct=False):

getAsTree(root=0, xsubset=[], lastbranch=None, expand=False)[source]