metaStorage

meta storage for macsserver

author:Tobias Kölling

The meta storage provides an abstraction for a database used to store metadata about the processing of raw data to products.

Current backend implementations include:

• DictMetaStorage: Using a python dict as in-memory storage. This storage can only be used in one process and looses all contents after program termination. It is useful for e.g. for unittests.
• LevelDBMetaStorage: Using a LevelDB as on-disk storage. This storage can only be used in one process at a time, but keeps information after program termination. It is intended for interactive testing.
• MongoDBMetaStorage: Using a MongoDB as storage. This storage can be used by multiple processes concurrently and is intended for production use.

class runmacs.processor.metaStorage.DictMetaStorage

MetaStorage with dict backend.

class runmacs.processor.metaStorage.EnhancedIterable(it)

Iterable which behaves more like a MongoDB-Cursor.

close()

EnhancedIterable is Garbage Collected, explicit closing is not needed.

class runmacs.processor.metaStorage.KVBaseMetaStorage

Base class for MetaStorage using key-value based backends.

class runmacs.processor.metaStorage.LevelDBMetaStorage(path)

MetaStorage with LevelDB backend.

class runmacs.processor.metaStorage.MetaStorage

Base class for every meta storage implementation.

class runmacs.processor.metaStorage.MongoDBMetaStorage(url, dbname='macsServer')

MetaStorage with MongoDB backend.

query(query, noTimeout=False, projection=None)

Performs a query

Parameters:

• query – the query in MongoDB syntax
• noTimeout – if you have a very slow process, you can set ths

Note:

setting noTimeout will only release the database cursor if the iteration is done until end

runmacs.processor.metaStorage.enhancedFilter(query, it)

Filters many datasets using a query.

Creates an EnhancedIterable from and iterable of mongodb-like datasets. The result will only contain datasets which match the given query.

Parameters:

• query – A mongodb-style query.
• it – An iterable over dict/list trees representing mongodb-like data.

Returns:

An EnhancedIterable with matching datasets only.

runmacs.processor.metaStorage.filterQuery(query, data, context=None)

Checks if data matches query.

Parameters:

• query – A mongodb-style query.
• data – A dict/list tree representing mongodb-like data.

Returns:

True if query matches the data, False otherwise.