Module dbf
[hide private]

Module dbf

source code


=========
Copyright
=========
    - Copyright: 2008-2012 Ad-Mail, Inc -- All rights reserved.
    - Author: Ethan Furman
    - Contact: ethanf@admailinc.com
    - Organization: Ad-Mail, Inc.
    - Version: 0.92.00 as of 10 May 2012

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
    - Redistributions of source code must retain the above copyright
      notice, this list of conditions and the following disclaimer.
    - Redistributions in binary form must reproduce the above copyright
      notice, this list of conditions and the following disclaimer in the
      documentation and/or other materials provided with the distribution.
    - Neither the name of Ad-Mail, Inc nor the
      names of its contributors may be used to endorse or promote products
      derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY Ad-Mail, Inc ''AS IS'' AND ANY
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL Ad-Mail, Inc BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

-------
Summary
-------

Python package for reading/writing dBase III and VFP 6 tables and memos

Goals:  programming style with databases
    - table = dbf.Table('table name' [, 'fielddesc[; fielddesc[; ....]]]')
        - fielddesc examples:  name C(30); age N(3,0); wisdom M; marriage D
    - record = [ table.current() | table[int] | table.append() | table.[next|prev|top|bottom|goto]() ]
    - record.field | record['field'] accesses the field

NOTE:   Of the VFP data types, auto-increment and variable-length character fields are not implemented.
        Record data is not written to disk until record.write_record() is called; there is a utility
        generator Write to help with this:

            for record in Write(table):
                record.some_field = some_data

        at the start of each loop the previous record is written to disk.

Examples:

    Create a test table:
        table = dbf.Table('temptable', 'name C(30); age N(3,0); birth D')

    Populate it:
        for datum in (
                ('John Doe', 31, dbf.Date(1979, 9,13)),
                ('Ethan Furman', 102, dbf.Date(1909, 4, 1)),
                ('Jane Smith', 57, dbf.Date(1954, 7, 2)),
                ('John Adams', 44, dbf.Date(1967, 1, 9)),
                ):
            table.append(datum)

    Export to csv:
        table.export(filename='filename', header=False)

    Iterate over it:
        for record in table:
            print "%s was born on %s, so s/he is %d years of age" % (record.name, record.birth, record.age)

    Create a new table from a csv file (all character fields now):
        table = dbf.from_csv('filename.csv') # this has field names of f0, f1, f2, etc
    or
        table = dbf.from_csv('filename.csv', field_names="name age birth".split())


    table = dbf.Table('temptable')  #reopen original file

    Sort it:
        name_index = table.create_index(lambda rec: rec.name)
        for record in name_index:
            print record.name

    Search using the sort:
        first = name_index.index(match=('John ',), partial=True) # or IndexError
    or
        first = name_index.find(match=('John ',), partial=True)  # or -1

    Delete a record:
        table[1].delete_record()

    Check if a record has been marked as deleted:
        record = table[1] # for example
        if record.has_been_deleted:
            print "table.pack() will physically remove this record! (and all other deleted records)"

    Ignore deleted records:
        table.use_deleted = False

    Write records:
        record.write_record()  # returns 1 if the record was actually written, 0 otherwise

    Specify data to write with write_record():
        record.write_record(age=39)

    Access record fields via attribute access:
        for record in table:
            print record.name
            record.name = 'The Black Knight was here!'      # not saved without a record.write_record() !
            print record.name

    or dictionary-style access:
        for record in table:
            print record['age']
            record['age'] = 29 # perpetual age of some ;)   # not saved without a record.write_record() !
            print record['age']

    and let's not forget index-style access:
        for record in table:
            print record[2]
            record[2] = dbf.Date.today() # just born?       # not saved without a record.write_record() !
            print record[2]

    can even use slices if you like:
        table = dbf.from_csv('filename.csv', field_names="name age birth".split())
        table[0][:2] = table[1][:2] # first record's first two fields are now the same as the second record's
        for record in table:
            print record, '
'

    Primitive SQL (work in progress):
        records = table.sql("select * where name[0] == 'J'")
        for rec in records:
            print rec, '
'

Field Types  -->  Python data types
  Dbf
    Character       unicode
    Date            datetime.date
    Logical         boolean
    Memo            unicode (same as character)
    Numeric         if N(x, 0) int; if N(x, 1+) float
  FP
    Float           same as Numeric
    General         binary data
    Photo           binary data
  VFP
    Currency        Decimal
    Double          float
    Integer         int
    DateTime        datetime.datetime
  Note: if any of the above are empty (nothing ever stored in that field) None is returned

Classes [hide private]
  NoneType
  Decimal
Floating point class for decimal arithmetic.
  bytes
str(object) -> string
  property
2.6 properties for 2.5-
  MutableDefault
Lives in the class, and on first access calls the supplied factory and maps the result into the instance it was called on
  DbfError
Fatal errors elicit this response.
  DataOverflow
Data too large for field
  BadData
bad data in table
  FieldMissing
Field does not exist in table
  NonUnicode
Data for table not in unicode
  DbfWarning
Normal operations elicit this response
  Eof
End of file reached
  Bof
Beginning of file reached
  DoNotIndex
Returned by indexing functions to suppress a record from becoming part of the index
  NullType
Null object -- any interaction returns Null
  Char
adds null capable str constructs
  Date
adds null capable datetime.date constructs
  DateTime
adds null capable datetime.datetime constructs
  Time
adds null capable datetime.time constructs
  Logical
return type for Logical fields; can take the values of True, False, or None/Null
  Quantum
return type for Logical fields; implements boolean algebra can take the values of True, False, or None/Null/Unknown/Other
  _DbfRecord
Provides routines to extract and save data within the fields of a dbf record.
  _DbfMemo
Provides access to memo fields as dictionaries must override _init, _get_memo, and _put_memo to store memo contents to disk
  _Db3Memo
Provides access to memo fields as dictionaries must override _init, _get_memo, and _put_memo to store memo contents to disk
  _VfpMemo
Provides access to memo fields as dictionaries must override _init, _get_memo, and _put_memo to store memo contents to disk
  DbfCsv
csv format for exporting tables
  FieldType
tuple with named attributes for representing a field's dbf type and python class
  DbfTable
Provides a framework for dbf style tables.
  Db3Table
Provides an interface for working with dBase III tables.
  FpTable
Provides an interface for working with FoxPro 2 tables
  VfpTable
Provides an interface for working with Visual FoxPro 6 tables
  List
list of Dbf records, with set-like behavior
  Index
  _Db4Table
  fake_module
Functions [hide private]
 
next(iterator) source code
 
all(iterable) source code
 
any(iterable) source code
 
none(*args, **kwargs) source code
 
packShortInt(value, bigendian=False)
Returns a two-bye integer from the value, or raises DbfError
source code
 
packLongInt(value, bigendian=False)
Returns a four-bye integer from the value, or raises DbfError
source code
 
packStr(string)
Returns an 11 byte, upper-cased, null padded string suitable for field names; raises DbfError if the string is bigger than 10 bytes
source code
 
unpackShortInt(bytes, bigendian=False)
Returns the value in the two-byte integer passed in
source code
 
unpackLongInt(bytes, bigendian=False)
Returns the value in the four-byte integer passed in
source code
 
unpackStr(chars)
Returns a normal, lower-cased string from a null-padded byte string
source code
 
unsupportedType(something, *ignore)
called if a data type is not supported for that style of table
source code
 
retrieveCharacter(bytes, fielddef, memo, decoder)
Returns the string in bytes as fielddef[CLASS] or fielddef[EMPTY]
source code
 
updateCharacter(string, fielddef, memo, decoder, encoder)
returns the string as bytes (not unicode) as fielddef[CLASS] or fielddef[EMPTY]
source code
 
retrieveCurrency(bytes, fielddef, *ignore)
returns the currency value in bytes
source code
 
updateCurrency(value, *ignore)
returns the value to be stored in the record's disk data
source code
 
retrieveDate(bytes, fielddef, *ignore)
Returns the ascii coded date as fielddef[CLASS] or fielddef[EMPTY]
source code
 
updateDate(moment, *ignore)
returns the Date or datetime.date object ascii-encoded (yyyymmdd)
source code
 
retrieveDouble(bytes, fielddef, *ignore)
Returns the double in bytes as fielddef[CLASS] ('default' == float)
source code
 
updateDouble(value, *ignore)
returns the value to be stored in the record's disk data
source code
 
retrieveInteger(bytes, fielddef, *ignore)
Returns the binary number stored in bytes in little-endian format as fielddef[CLASS]
source code
 
updateInteger(value, *ignore)
returns value in little-endian binary format
source code
 
retrieveLogical(bytes, fielddef, *ignore)
Returns True if bytes is 't', 'T', 'y', or 'Y', None if '?', and False otherwise
source code
 
updateLogical(data, *ignore)
Returns 'T' if logical is True, 'F' if False, '?' otherwise
source code
 
retrieveMemo(bytes, fielddef, memo, decoder)
Returns the block of data from a memo file
source code
 
updateMemo(string, fielddef, memo, decoder, encoder)
Writes string as a memo, returns the block number it was saved into
source code
 
retrieveNumeric(bytes, fielddef, *ignore)
Returns the number stored in bytes as integer if field spec for decimals is 0, float otherwise
source code
 
updateNumeric(value, fielddef, *ignore)
returns value as ascii representation, rounding decimal portion as necessary
source code
 
retrieveVfpDateTime(bytes, fielddef, *ignore)
returns the date/time stored in bytes; dates <= 01/01/1981 00:00:00 may not be accurate; BC dates are nulled.
source code
 
updateVfpDateTime(moment, *ignore)
sets the date/time stored in moment moment must have fields year, month, day, hour, minute, second, microsecond
source code
 
retrieveVfpMemo(bytes, fielddef, memo, decoder)
Returns the block of data from a memo file
source code
 
updateVfpMemo(string, fielddef, memo, decoder, encoder)
Writes string as a memo, returns the block number it was saved into
source code
 
addCharacter(format, flags) source code
 
addDate(format, flags) source code
 
addLogical(format, flags) source code
 
addMemo(format, flags) source code
 
addNumeric(format, flags) source code
 
addVfpCurrency(format, flags) source code
 
addVfpDateTime(format, flags) source code
 
addVfpDouble(format, flags) source code
 
addVfpInteger(format, flags) source code
 
addVfpMemo(format, flags) source code
 
addVfpNumeric(format, flags) source code
 
fieldSpecErrorText(format, flags)
generic routine for error text for the add...() functions
source code
 
ezip(*iters)
extends all iters to longest one, using last value from each as necessary
source code
 
sql_select(records, chosen_fields, condition, field_names) source code
 
sql_update(records, command, condition, field_names) source code
 
sql_delete(records, dead_fields, condition, field_names) source code
 
sql_recall(records, all_fields, condition, field_names) source code
 
sql_add(records, new_fields, condition, field_names) source code
 
sql_drop(records, dead_fields, condition, field_names) source code
 
sql_pack(records, command, condition, field_names) source code
 
sql_resize(records, fieldname_newsize, condition, field_names) source code
 
sql_criteria(records, criteria)
creates a function matching the sql criteria
source code
 
sql_cmd(command, field_names)
creates a function matching to apply command to each record in records
source code
 
sql(records, command)
recognized sql commands are SELECT, UPDATE | REPLACE, DELETE, RECALL, ADD, DROP
source code
 
_nop(value)
returns parameter unchanged
source code
 
_normalize_tuples(tuples, length, filler)
ensures each tuple is the same length, using filler[-missing] for the gaps
source code
 
_codepage_lookup(cp) source code
 
codepage(cp=None)
get/set default codepage for any new tables
source code
 
encoding(cp=None)
get/set default encoding for non-unicode strings passed into a table
source code
 
Write(records)
commits each record to disk before returning the next one
source code
 
Table(filename, field_specs='', memo_size=128, ignore_memos=False, read_only=False, keep_memos=False, meta_only=False, dbf_type=None, codepage=None, default_data_types={}, field_data_types={})
returns an open table of the correct dbf_type, or creates it if field_specs is given
source code
 
index(sequence)
returns integers 0 - len(sequence)
source code
 
guess_table_type(filename) source code
 
table_type(filename)
returns text representation of a table's dbf version
source code
 
add_fields(table_name, field_specs)
adds fields to an existing table
source code
 
delete_fields(table_name, field_names)
deletes fields from an existing table
source code
 
export(table_name, filename='', fields='', format='csv', header=True)
creates a csv or tab-delimited file from an existing table
source code
 
first_record(table_name)
prints the first record of a table
source code
 
from_csv(csvfile, to_disk=False, filename=None, field_names=None, extra_fields=None, dbf_type='db3', memo_size=64, min_field_size=1)
creates a Character table from a csv file to_disk will create a table with the same name filename will be used if provided field_names default to f0, f1, f2, etc, unless specified (list) extra_fields can be used to add additional fields -- should be normal field specifiers (list)
source code
 
get_fields(table_name)
returns the list of field names of a table
source code
 
info(table_name)
prints table info
source code
 
rename_field(table_name, oldfield, newfield)
renames a field in a table
source code
 
structure(table_name, field=None)
returns the definition of a field (or all fields)
source code
 
hex_dump(records)
just what it says ;)
source code
Variables [hide private]
  version = (0, 92, 2)
  LOGICAL_BAD_IS_FALSE = True
  input_decoding = 'cp1252'
  temp_dir = 'c:\\Temp'
  default_type = 'db3'
  sql_user_functions = {}
  SYSTEM = 1
  NULLABLE = 2
  BINARY = 4
  TYPE = 0
  START = 1
  LENGTH = 2
  END = 3
  DECIMALS = 4
  FLAGS = 5
  CLASS = 6
  EMPTY = 7
  NULL = 8
  FIELD_FLAGS = {1: 'system', 2: 'null', 4: 'binary', 'binary': ...
  YES_I_AM_SURE = 42
  Null = <null>
  NullDate = Date()
  NullDateTime = DateTime()
  NullTime = Time()
  Truth = Logical('T')
  Falsth = Logical('F')
  Unknown = Logical('?')
  On = Quantum('Y')
  Off = Quantum('N')
  Other = Quantum('?')
  VFPTIME = 1721425
  table_types = {'db3': <class 'dbf.Db3Table'>, 'dbf': <class 'd...
  version_map = {'\x02': 'FoxBASE', '\x03': 'dBase III Plus', '\...
  code_pages = {'\x00': ('ascii', 'plain ol\' ascii'), '\x01': (...
  default_codepage = None
hash(x)
  sql_functions = {'select': sql_select, 'update': sql_update, '...
  SEEK_CUR = 1
  SEEK_END = 2
  SEEK_SET = 0
  __package__ = None
hash(x)

Imports: codecs, csv, datetime, locale, os, struct, sys, time, traceback, unicodedata, weakref, array, bisect_left, bisect_right, decimal, floor, copyfileobj, ST_SIZE, defaultdict


Variables Details [hide private]

FIELD_FLAGS

Value:
{1: 'system',
 2: 'null',
 4: 'binary',
 'binary': 4,
 'nocptrans': 4,
 'null': 2}

table_types

Value:
{'db3': <class 'dbf.Db3Table'>,
 'dbf': <class 'dbf.DbfTable'>,
 'fp': <class 'dbf.FpTable'>,
 'vfp': <class 'dbf.VfpTable'>}

version_map

Value:
{'\x02': 'FoxBASE',
 '\x03': 'dBase III Plus',
 '\x04': 'dBase IV',
 '\x05': 'dBase V',
 '0': 'Visual FoxPro',
 '1': 'Visual FoxPro (auto increment field)',
 '2': 'Visual FoxPro (VarChar, VarBinary, or BLOB enabled)',
 'C': 'dBase IV SQL table files',
...

code_pages

Value:
{'\x00': ('ascii', 'plain ol\' ascii'),
 '\x01': ('cp437', 'U.S. MS-DOS'),
 '\x02': ('cp850', 'International MS-DOS'),
 '\x03': ('cp1252', 'Windows ANSI'),
 '\x04': ('mac_roman', 'Standard Macintosh'),
 '\x08': ('cp865', 'Danish OEM'),
 '\t': ('cp437', 'Dutch OEM'),
 '''
...

sql_functions

Value:
{'select': sql_select, 'update': sql_update, 'replace': sql_update, 'i\
nsert': None, 'delete': sql_delete, 'recall': sql_recall, 'add': sql_a\
dd, 'drop': sql_drop, 'count': None, 'pack': sql_pack, 'resize': sql_r\
esize,}