engine¶

TokenStream mixin with support for iteration and attributes cached as properties.

offset¶

Start and stop character offset.

payload¶

Payload bytes.

positionIncrement¶

Position relative to the previous token.

term¶

Term text.

type¶

Lexical type.

Bases: PythonTokenFilter, lupyne.engine.indexers.TokenStream

Create an iterable lucene TokenFilter from a TokenStream. Subclass and override incrementToken() or setattrs().

incrementToken()[source]¶

Advance to next token and return whether the stream is not empty.

Return a lucene Analyzer which chains together a tokenizer and filters.

static parse(query, field='', op='', version=None, parser=None, **attrs)[source]¶

Return parsed lucene Query.

Parameters:	query – query string field – default query field name, sequence of names, or boost mapping op – default query operator (‘or’, ‘and’) version – lucene Version parser – custom PythonQueryParser class attrs – additional attributes to set on the parser

tokens(text, field=None)[source]¶

Return lucene TokenStream from text.

Delegated lucene IndexReader, with a mapping interface of ids to document objects.

__len__()[source]¶

__contains__(id)[source]¶

__iter__()[source]¶

copy(dest, query=None, exclude=None, merge=0)[source]¶

Copy the index to the destination directory. Optimized to use hard links if the destination is a file system path.

Parameters:	dest – destination directory path or lucene Directory query – optional lucene Query to select documents exclude – optional lucene Query to exclude documents merge – optionally merge into maximum number of segments

directory¶

reader’s lucene Directory

docs(name, value, counts=False)[source]¶

Generate doc ids which contain given term, optionally with frequency counts.

morelikethis(doc, *fields, **attrs)[source]¶

Return MoreLikeThis query for document.

Parameters:	doc – document id or text fields – document fields to use, optional for termvectors attrs – additional attributes to set on the morelikethis object

names(**attrs)[source]¶

Return field names, given option description.

Changed in version 1.2: lucene requires FieldInfo filter attributes instead of option

numbers(name, step=0, type=<type 'int'>, counts=False)[source]¶

Generate decoded numeric term values, optionally with frequency counts.

Parameters:	name – field name step – precision step to select terms type – int or float counts – include frequency counts

path¶

FSDirectory path

positions(name, value, payloads=False, offsets=False)[source]¶

Generate doc ids and positions which contain given term, optionally with offsets, or only ones with payloads.

positionvector(id, field, offsets=False)[source]¶

Generate terms and positions for given doc id and field, optionally with character offsets.

readers¶

segment readers

segments¶

segment filenames with document counts

spans(query, positions=False, payloads=False)[source]¶

Generate docs with occurrence counts for a span query.

Parameters:	query – lucene SpanQuery positions – optionally include slice positions instead of counts payloads – optionally only include slice positions with payloads

terms(name, value='', stop=None, counts=False, distance=0)[source]¶

Generate a slice of term values, optionally with frequency counts. Supports a range of terms, wildcard terms, or fuzzy terms.

Parameters:	name – field name value – term prefix or lower bound for range terms stop – optional upper bound for range terms counts – include frequency counts distance – maximum edit distance for fuzzy terms

termvector(id, field, counts=False)[source]¶

Generate terms for given doc id and field, optionally with frequency counts.

timestamp¶

timestamp of reader’s last commit

Bases: IndexSearcher, lupyne.engine.indexers.IndexReader

Inherited lucene IndexSearcher, with a mixed-in IndexReader.

__getitem__(id)[source]¶

Return Document.

__del__()[source]¶

Closes index.

filters¶

Mapping of cached filters by field, also used for facet counts.

sorters¶

Mapping of cached sorters by field and associated parsers.

spellcheckers¶

Mapping of cached spellcheckers by field.

termsfilters¶

Set of registered termsfilters.

comparator(field, type='string', parser=None, multi=False)[source]¶

Return cache of field values suitable for sorting, using a cached SortField if available. Parsing values into an array is memory optimized. Map values into a list for speed optimization. Comparators are not thread-safe.

Parameters:	name – field name type – type object or name compatible with FieldCache parser – lucene FieldCache.Parser or callable applied to field values multi – retrieve multi-valued string terms as a tuple

correct(field, text, distance=2)[source]¶

Generate potential words ordered by increasing edit distance and decreasing frequency. For optimal performance only iterate the required slice size of corrections.

Parameters:	distance – the maximum edit distance to consider for enumeration

count(*query, **options)[source]¶

Return number of hits for given query or term.

Parameters:	query – search() compatible query, or optimally a name and value options – additional search() options

distances(lng, lat, lngfield, latfield)[source]¶

Return distance comparator computed from cached lat/lng fields.

facets(query, *keys)[source]¶

Return mapping of document counts for the intersection with each facet.

Changed in version 1.6: filters are no longer implicitly cached, a GroupingSearch is used instead

Parameters:	query – query string, lucene Query, or lucene Filter keys – field names, term tuples, or any keys to previously cached filters

get(id, *fields)[source]¶

Return Document with only selected fields loaded.

groupby(field, query, filter=None, count=None, start=0, **attrs)[source]¶

Return Hits grouped by field using a GroupingSearch.

highlighter(query, field, **kwargs)[source]¶

Return Highlighter or if applicable FastVectorHighlighter specific to searcher and query.

classmethod load(directory, analyzer=None)[source]¶

Open IndexSearcher with a lucene RAMDirectory, loading index into memory.

match(document, *queries)[source]¶

Generate scores for all queries against a given document mapping.

reopen(filters=False, sorters=False, spellcheckers=False)[source]¶

Return current IndexSearcher, only creating a new one if necessary. Any registered termsfilters are also refreshed.

Parameters:	filters – refresh cached facet filters sorters – refresh cached sorters with associated parsers spellcheckers – refresh cached spellcheckers

search(query=None, filter=None, count=None, sort=None, reverse=False, scores=False, maxscore=False, timeout=None, **parser)[source]¶

Run query and return Hits.

Changed in version 1.4: sort param for lucene only; use Hits.sorted with a callable

Parameters:	query – query string or lucene Query filter – lucene Filter count – maximum number of hits to retrieve sort – lucene Sort parameters reverse – reverse flag used with sort scores – compute scores for candidate results when sorting maxscore – compute maximum score of all results when sorting timeout – stop search after elapsed number of seconds parser – Analyzer.parse() options

sorter(field, type='string', parser=None, reverse=False)[source]¶

Return SortField with cached attributes if available.

spellchecker(field)[source]¶

Return and cache spellchecker for given field.

suggest(field, prefix, count=None)[source]¶

Return ordered suggested words for prefix.

termsfilter(field, values=())[source]¶

Return registered TermsFilter, which will be refreshed whenever the searcher is reopened.

New in version 1.7.

Note

This interface is experimental and might change in incompatible ways in the next release.

Bases: lupyne.engine.indexers.IndexSearcher

IndexSearcher with underlying lucene MultiReader.

Bases: IndexWriter

Inherited lucene IndexWriter. Supports setting fields parameters explicitly, so documents can be represented as dictionaries.

fields¶

Mapping of assigned fields. May be used directly, instead of set() method, for further customization.

__del__()[source]¶

Closes index.

__len__()¶

__iadd__(directory)[source]¶

Add directory (or reader, searcher, writer) to index.

add(document=(), **terms)[source]¶

Add document() to index with optional boost.

classmethod check(directory, fix=False)[source]¶

Check and optionally fix unlocked index, returning lucene CheckIndex.Status.

delete(*query, **options)[source]¶

Remove documents which match given query or term.

Parameters:	query – IndexSearcher.search() compatible query, or optimally a name and value options – additional Analyzer.parse() options

document(items=(), **terms)[source]¶

Return lucene Document from mapping of field names to one or multiple values.

set(name, cls=<class 'lupyne.engine.documents.Field'>, **settings)[source]¶

Assign settings to field name and return the field.

Parameters:	name – registered name of field cls – optional Field constructor settings – stored, indexed, etc. options compatible with Field

snapshot(*args, **kwds)[source]¶

Return context manager of an index commit snapshot.

Changed in version 1.4: lucene identifies snapshots by commit generation

update(name, value='', document=(), **terms)[source]¶

Atomically delete documents which match given term and add the new document().

Changed in version 1.7: update in-place if only DocValues are given

Bases: lupyne.engine.indexers.IndexWriter

An all-purpose interface to an index. Creates an IndexWriter with a delegated IndexSearcher.

commit(merge=False, **caches)[source]¶

Commit writes and refresh() searcher.

Parameters:	merge – merge segments with deletes, or optionally specify maximum number of segments

refresh(**caches)[source]¶

Store refreshed searcher with IndexSearcher.reopen() caches.

Bases: lupyne.engine.indexers.Indexer

Indexer which tracks a unique identifying field. Handles atomic updates of rapidly changing fields, managing termsfilters. Assign custom settings or cache custom sorter for primary field if necessary.

termsfilters¶

Mapping of filters to synchronized termsfilters.

refresh(**caches)[source]¶

Store refreshed searcher and synchronize termsfilters.

termsfilter(filter, *others)[source]¶

Return TermsFilter synced to given filter and optionally associated with other indexers.

update(value, document=(), **terms)[source]¶

Atomically update document based on unique field.

Bases: dict

Multimapping of field names to values, but default getters return the first value.

dict(*names, **defaults)[source]¶

Return dict representation of document.

Parameters:	names – names of multi-valued fields to return as a list defaults – include only given fields, using default values as necessary

getlist(name)[source]¶

Return list of all values for given field.

Bases: lupyne.engine.documents.Document

A Document from a search result, with id, score, and optional sort keys.

dict(*names, **defaults)[source]¶

Return dict representation of document with __id__, __score__, and any sort __keys__.

Search results: lazily evaluated and memory efficient. Provides a read-only sequence interface to hit objects.

__len__()[source]¶

__getitem__(index)[source]¶

filter(func)[source]¶

Return Hits filtered by function applied to doc ids.

groupby(func, count=None, docs=None)[source]¶

Return ordered list of Hits grouped by value of function applied to doc ids. Optionally limit the number of groups and docs per group.

items()[source]¶

Generate zipped ids and scores.

select(*fields)[source]¶

Only load selected fields.

sorted(key, reverse=False)[source]¶

Return Hits sorted by key function applied to doc ids.

Sequence of grouped Hits.

__len__()[source]¶

__getitem__(index)[source]¶

facets¶

mapping of field values and counts

select(*fields)¶

Only load selected fields.

Inherited lucene GroupingSearch with optimized faceting.

__len__()[source]¶

__iter__()[source]¶

search(searcher, query, filter=None, count=None, start=0)[source]¶

Run query and return Groups.

Saved parameters which can generate lucene Fields given values.

items(*values)[source]¶

Generate lucene Fields suitable for adding to a document.

settings¶

dict representation of settings

Bases: lupyne.engine.documents.Field

Field which applies a function across its values.

items(*values)[source]¶

Generate fields with mapped values.

Bases: lupyne.engine.documents.Field

Field which indexes every component into its own field. Original value may be stored for convenience.

items(*values)[source]¶

Generate indexed component fields.

prefix(value)[source]¶

Return prefix query of the closest possible prefixed field.

range(start, stop, lower=True, upper=False)[source]¶

Return range query of the closest possible prefixed field.

values(value)[source]¶

Generate component field values in order.

Bases: lupyne.engine.documents.Field

Field which stores a per-document values, used for efficient sorting.

items(*values)[source]¶

Generate lucene DocValuesFields suitable for adding to a document.

Bases: lupyne.engine.documents.Field

Field which indexes numbers in a prefix tree.

filter(*args, **kwargs)[source]¶

Deprecated since version 1.9: convert NumericRangeQuery with Query.filter instead

items(*values)[source]¶

Generate lucene NumericFields suitable for adding to a document.

range(start, stop, lower=True, upper=False)[source]¶

Return lucene NumericRangeQuery.

term(value)[source]¶

Return range query to match single term.

Bases: lupyne.engine.documents.NumericField

Field which indexes datetimes as a NumericField of timestamps. Supports datetimes, dates, and any prefix of time tuples.

duration(date, days=0, **delta)[source]¶

Return date range query within time span of date.

Parameters:	date – origin date or tuple days,delta – timedelta parameters

items(*dates)[source]¶

Generate lucene NumericFields of timestamps.

prefix(date)[source]¶

Return range query which matches the date prefix.

range(start, stop, lower=True, upper=False)[source]¶

Return NumericRangeQuery of timestamps.

classmethod timestamp(date)[source]¶

Return utc timestamp from date or time tuple.

within(days=0, weeks=0, utc=True, **delta)[source]¶

Return date range query within current time and delta. If the delta is an exact number of days, then dates will be used.

Parameters:	days,weeks – number of days to offset from today utc – optionally use utc instead of local time delta – additional timedelta parameters

Inherited lucene Query, with dynamic base class acquisition. Uses class methods and operator overloading for convenient query construction.

__and__(other)[source]¶

BooleanQuery +self +other>

__or__(other)[source]¶

BooleanQuery self other>

__sub__(other)[source]¶

BooleanQuery self -other>

classmethod all(*queries, **terms)[source]¶

Return BooleanQuery (AND) from queries and terms.

classmethod any(*queries, **terms)[source]¶

Return BooleanQuery (OR) from queries and terms.

constant()[source]¶

Return lucene ConstantScoreQuery.

classmethod disjunct(multiplier, *queries, **terms)[source]¶

Return lucene DisjunctionMaxQuery from queries and terms.

static filter(cache=True)[source]¶

Return query as a filter, as specifically matching as possible, but defaulting to QueryWrapperFilter.

classmethod fuzzy(name, value, *args)[source]¶

Return lucene FuzzyQuery.

classmethod multiphrase(name, *values)[source]¶

Return lucene MultiPhraseQuery. None may be used as a placeholder.

classmethod near(name, *values, **kwargs)[source]¶

Return SpanNearQuery from terms. Term values which supply another field name will be masked.

classmethod phrase(name, *values, **attrs)[source]¶

Return lucene PhraseQuery. None may be used as a placeholder.

classmethod prefix(name, value)[source]¶

Return lucene PrefixQuery.

classmethod range(name, start, stop, lower=True, upper=False)[source]¶

Return lucene RangeQuery, by default with a half-open interval.

classmethod regexp(name, value, *args)[source]¶

Return lucene RegexpQuery.

classmethod span(*term)[source]¶

Return SpanQuery from term name and value or a MultiTermQuery.

classmethod term(name, value, **attrs)[source]¶

Return lucene TermQuery.

static terms()[source]¶

Generate set of query term items.

classmethod wildcard(name, value)[source]¶

Return lucene WildcardQuery.

Bases: lupyne.engine.queries.Query

Inherited lucene BooleanQuery with sequence interface to clauses.

__len__()[source]¶

__iter__()[source]¶

__getitem__(index)[source]¶

__iand__(other)[source]¶

add +other

__ior__(other)[source]¶

add other

__isub__(other)[source]¶

add -other

Bases: lupyne.engine.queries.Query

Inherited lucene SpanQuery with additional span constructors.

__getitem__(slc)[source]¶

<SpanFirstQuery: spanFirst(self, other.stop)>

__sub__(other)[source]¶

<SpanNotQuery: spanNot(self, other)>

__or__(*spans_)[source]¶

<SpanOrQuery: spanOr(spans)>

mask(name)[source]¶

Return lucene FieldMaskingSpanQuery, which allows combining span queries from different fields.

near(*spans_, **kwargs)[source]¶

Return lucene SpanNearQuery from SpanQueries.

Parameters:	slop – default 0 inOrder – default True collectPayloads – default True

payload(*values)[source]¶

Return lucene SpanPayloadCheckQuery from payload values.

Bases: CachingWrapperFilter

Caching filter based on a unique field and set of matching values. Optimized for many terms and docs, with support for incremental updates. Suitable for searching external metadata associated with indexed identifiers. Call refresh() to cache a new (or reopened) searcher.

add(*values)[source]¶

Add a few term values.

discard(*values)[source]¶

Discard a few term values.

filter(values, cache=True)[source]¶

Return lucene TermsFilter, optionally using the FieldCache.

refresh(searcher)[source]¶

Refresh cached bitsets of current values for new segments of searcher.

update(values, op='or', cache=True)[source]¶

Update allowed values and corresponding cached bitsets.

Parameters:	values – additional term values op – set operation used to combine terms and docs: and, or, andNot cache – optionally cache all term values using FieldCache

Bases: SortField

Inherited lucene SortField used for caching FieldCache parsers.

comparator(searcher, multi=False)[source]¶

Return indexed values from default FieldCache using the given searcher.

filter(start, stop, lower=True, upper=False)[source]¶

Return lucene FieldCacheRangeFilter based on field and type.

terms(filter, *readers)[source]¶

Generate field cache terms from docs which match filter from all segments.

Bases: Highlighter

Inherited lucene Highlighter with stored analysis options.

fragments(doc, count=1)[source]¶

Return highlighted text fragments.

Parameters:	doc – text string or doc id to be highlighted count – maximum number of fragments

Bases: FastVectorHighlighter

Inherited lucene FastVectorHighlighter with stored query. Fields must be stored and have term vectors with offsets and positions.

fragments(id, count=1, size=100)[source]¶

Return highlighted text fragments.

Parameters:	id – document id count – maximum number of fragments size – maximum number of characters in fragment

Bases: dict

Correct spellings and suggest words for queries. Supply a vocabulary mapping words to (reverse) sort keys, such as document frequencies.

correct(word)[source]¶

Generate ordered sets of words by increasing edit distance.

edits(word, length=0)[source]¶

Return set of potential words one edit distance away, mapped to valid prefix lengths.

suggest(prefix, count=None)[source]¶

Return ordered suggested words for prefix.

Inherited lucene QueryParser which corrects spelling. Assign a searcher attribute or override correct() implementation.

searcher¶

IndexSearcher

correct(term)[source]¶

Return term with text replaced as necessary.

rewrite(query)[source]¶

Return term or phrase query with corrected terms substituted.

Bases: tuple

Geodetic coordinates (EPSG:4326) in Spherical Mercator (EPSG:900913) format.

coords¶

Geodetic coordinates (EPSG:4326)

distance(other)[source]¶

Return euclidean distance between points in meters.

tile(zoom)[source]¶

Return enclosing Tile at given zoom level.

within(distance, zoom)[source]¶

Generate sets of tiles with increasing zoom which are within distance of point.

Bases: str

TMS tile coordinates in QuadTree format.

coords¶

TMS tile coordinates

distance(point)[source]¶

Return euclidean distance between tile and point in meters.

points¶

lower-left and upper-right Point corners

walk(other)[source]¶

Generate all tiles between corners.

Bases: lupyne.engine.documents.NumericField

Geospatial points, which create a tiered index of tiles. Points must still be stored if exact distances are required upon retrieval.

items(*points)[source]¶

Generate tiles from points (lng, lat).

near(lng, lat, precision=None)[source]¶

Return prefix query for point at given precision.

prefix(tile)[source]¶

Return range query which is equivalent to the prefix of the tile.

ranges(tiles)[source]¶

Generate range queries by grouping adjacent tiles.

within(lng, lat, distance, limit=4)[source]¶

Return range queries for any tiles which could be within distance of given point.

Parameters:	lng,lat – point distance – search radius in meters limit – maximum number of tiles to consider

Bases: lupyne.engine.spatial.PointField

PointField which implicitly supports polygons (technically linear rings of points). Differs from points in that all necessary tiles are included to match the points’ boundary. As with PointField, the tiered tiles are a search optimization, not a distance calculator.

items(*polygons)[source]¶

Generate all covered tiles from polygons.

Bases: object

Distance comparator computed from cached lat/lngs.

__getitem__(id)[source]¶