Corpus internal format

Single corpus is stored as a directory. In the directory there are several files important for corpus structure.

File config

This file stores yaml formatted dict with properties of corpus. A typical config file has following properties:

chunk_size: 52428800
current_chunk: 0
encoding: utf-8
name: 50k Internet Corpus

Warning

you should not modify config file by yourself, unless you really know what you do.

chunk_size

max size (in bytes) of single corpus chunk

Note

single document must be stored within single chunk, so you cannot store documents larger that chunk_size.

current_chunk

number of current chunk that will be used when appending new document;

Note

chunks are numbered from 0.

encoding

internal chunk encoding; possibly always utf-8.

name

an optional name for corpus

File chunkN

Files like chunk0, chunk1, chunk2, ... contains raw texts and headers. Each chunk can have maximum size of chunk_size bytes config property.

Chunk file has a very simple internal format. Documents are stored sequentially (one after another). Each document is represented as yamled header dict and raw document text encoded with encoding defined in config file.

Internal format of chunk is:

[yamled header1]\n
[raw document1 text encoded]\n
[yamled header2]\n
[raw document2 text encoded]\n
...
[yamled headern]\n
[raw documentn text encoded]\n

Note

chunks are numbered from 0.

Note

single document must be stored within single chunk, so you cannot store documents larger that chunk_size.

An example of two documents long chunk:

id: 8
Prof. Wojciech Roszkowski jest oficjalnym kandydatem AWS na
prezesa Instytutu Pamięci Narodowej - zdecydowało prezydium
Klubu Parlamentarnego Akcji Wyborczej Solidarność.
Rzecznik klubu Piotr Żak przypomniał, że zgodnie z ustawą o IPN,
Sejm wybiera prezesa Instytutu większością 3/5. Do wyboru
Roszkowskiego konieczne jest zatem uzyskanie poparcia nie tylko
Unii Wolności, ale także Polskiego Stronnictwa Ludowego.
Politycy PSL, UW i SLD odmawiają deklaracji, czy ich ugrupowania
poprą kandydaturę prof. Roszkowskiego.

id: 20
Papieże Pius IX i Jan XXIII zostaną beatyfikowani 3 września -
ogłosił Watykan. Beatyfikacja obu papieży zbiegnie się z
uroczystościami Wielkiego Jubileuszu Roku 2000.

File idx

This file contains a list of documents descriptors (indexes in chunk file). This is a list, that contains a tuples like:

• chunk number
• offset of document start in chunk file
• length of header section (with additional \n )
• length of text section (with additional \n)

This file is managed by DB Berkeley Recno structure.

File ridx

This files stores a random access index. Basically it is a hashmap containing a mapping of document id to the index in idx list.

This file is managed by DB Berkeley Hashmap structure.