Package prest :: Module tknz :: Class ForwardTokenizer

Class ForwardTokenizer

object --+    
         |    
 Tokenizer --+
             |
            ForwardTokenizer

Tokenize a stream from the beginning to the end.

Class Hierarchy for ForwardTokenizer

Instance Methods

[hide private]

__init__(self, stream, lowercase=False, blankspaces=char.blankspaces, separators=char.separators)
Constructor of the ForwardTokenizer class.

source code

int

count_tokens(self)
Check the number of tokens left.

source code

int

count_chars(self)
Count the number of characters in the stream.

source code

bool

has_more_tokens(self)
Test if at least one token remains.

source code

str

next_token(self)
Retrieve the next token in the stream.

source code

float

progress(self)
Return the progress percentage.

source code

reset_stream(self)
Reset the offset to 0.

source code

Inherited from object: __delattr__, __format__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __sizeof__, __str__, __subclasshook__

Inherited from Tokenizer

bool

is_blankspace(self, char)
Test if a character is a blankspace.

source code

bool

is_separator(self, char)
Test if a character is a separator.

source code

Class Variables

[hide private]

Inherited from Tokenizer

__metaclass__ = abc.ABCMeta

Properties

[hide private]

Inherited from object: __class__

Method Details

[hide private]

init(self, stream, lowercase=False, blankspaces=char.blankspaces, separators=char.separators)
(Constructor)

source code

Constructor of the ForwardTokenizer class.

Parameters:

stream (str or io.IOBase) - The stream to tokenize. Can be a filename or any open IO stream.
blankspaces (str) - The characters that represent empty spaces.
separators (str) - The characters that separate token units (e.g. word boundaries).

Overrides: object.__init__

Warning: When passing IOBase type variable as stream parameter: the read() method is used to read the stream and it can be time consuming. Please don't pass IOBase during the prediction process!