interfaces – Core gensim interfaces¶Basic interfaces used across the whole Gensim package.
These interfaces are used for building corpora, model transformation and similarity queries.
The interfaces are realized as abstract base classes. This means some functionality is already provided in the interface itself, and subclasses should inherit from these interfaces and implement the missing methods.
gensim.interfaces.CorpusABC¶Bases: gensim.utils.SaveLoad
Interface for corpus classes from gensim.corpora.
Corpus is simply an iterable object, where each iteration step yields one document:
>>> from gensim.corpora import MmCorpus # this is inheritor of CorpusABC class
>>> from gensim.test.utils import datapath
>>>
>>> corpus = MmCorpus(datapath("testcorpus.mm"))
>>> for doc in corpus:
... pass # do something with the doc...
A document represented in bag-of-word (BoW) format, i.e. list of (attr_id, attr_value),
like [(1, 0.2), (4, 0.6), ...].
>>> from gensim.corpora import MmCorpus # this is inheritor of CorpusABC class
>>> from gensim.test.utils import datapath
>>>
>>> corpus = MmCorpus(datapath("testcorpus.mm"))
>>> doc = next(iter(corpus))
>>> print(doc)
[(0, 1.0), (1, 1.0), (2, 1.0)]
Remember, that save/load methods save only corpus class (not corpus as data itself), for save/load functionality, please use this pattern :
>>> from gensim.corpora import MmCorpus # this is inheritor of CorpusABC class
>>> from gensim.test.utils import datapath, get_tmpfile
>>>
>>> corpus = MmCorpus(datapath("testcorpus.mm"))
>>> tmp_path = get_tmpfile("temp_corpus.mm")
>>>
>>> MmCorpus.serialize(tmp_path, corpus) # serialize corpus to disk in MmCorpus format
>>> # MmCorpus.save_corpus(tmp_path, corpus) # this variant also possible, but if serialize availbe - call it.
>>> loaded_corpus = MmCorpus(tmp_path) # load corpus through constructor
>>> for (doc_1, doc_2) in zip(corpus, loaded_corpus):
... assert doc_1 == doc_2 # check that corpuses exactly same
See also
gensim.corporaCorpuses in different formats
load(fname, mmap=None)¶Load an object previously saved using save() from a file.
fname (str) – Path to file that contains needed object.
mmap (str, optional) – Memory-map option. If the object was saved with large arrays stored separately, you can load these arrays via mmap (shared memory) using mmap=’r’. If the file being loaded is compressed (either ‘.gz’ or ‘.bz2’), then `mmap=None must be set.
See also
save()Save object to file.
Object loaded from fname.
object
AttributeError – When called on an object instance instead of class (this is a class method).
save(*args, **kwargs)¶Saves corpus in-memory state.
Warning
This save only the “state” of a corpus class, not the corpus data!
For saving data use the serialize method of the output format you’d like to use
(e.g. gensim.corpora.mmcorpus.MmCorpus.serialize()).
save_corpus(fname, corpus, id2word=None, metadata=False)¶Save corpus to disk.
Some formats support saving the dictionary (feature_id -> word mapping), which can be provided by the optional id2word parameter.
Notes
Some corpora also support random access via document indexing, so that the documents on disk
can be accessed in O(1) time (see the gensim.corpora.indexedcorpus.IndexedCorpus base class).
In this case, save_corpus() is automatically called internally by
serialize(), which does save_corpus() plus saves the index
at the same time.
Calling serialize() is preferred to calling :meth:`gensim.interfaces.CorpusABC.save_corpus().
fname (str) – Path to output file.
corpus (iterable of list of (int, number)) – Corpus in BoW format.
id2word (Dictionary, optional) – Dictionary of corpus.
metadata (bool, optional) – Write additional metadata to a separate too?
gensim.interfaces.SimilarityABC(corpus)¶Bases: gensim.utils.SaveLoad
Interface for similarity search over a corpus.
In all instances, there is a corpus against which we want to perform the similarity search. For each similarity search, the input is a document or a corpus, and the output are the similarities to individual corpus documents.
Examples
>>> from gensim.similarities import MatrixSimilarity
>>> from gensim.test.utils import common_corpus
>>>
>>> index = MatrixSimilarity(common_corpus)
>>> similarities = index.get_similarities(common_corpus[1]) # get similarities between query and corpus
Notes
There is also a convenience wrapper, where iterating over self yields similarities of each document in the corpus against the whole corpus (i.e. the query is each corpus document in turn).
See also
gensim.similaritiesDifferent index implementations of this interface.
corpus (iterable of list of (int, number)) – Corpus in sparse Gensim bag-of-words format.
get_similarities(doc)¶Get similarities of the given document or corpus against this index.
doc ({list of (int, number), iterable of list of (int, number)}) – Document in the sparse Gensim bag-of-words format, or a streamed corpus of such documents.
load(fname, mmap=None)¶Load an object previously saved using save() from a file.
fname (str) – Path to file that contains needed object.
mmap (str, optional) – Memory-map option. If the object was saved with large arrays stored separately, you can load these arrays via mmap (shared memory) using mmap=’r’. If the file being loaded is compressed (either ‘.gz’ or ‘.bz2’), then `mmap=None must be set.
See also
save()Save object to file.
Object loaded from fname.
object
AttributeError – When called on an object instance instead of class (this is a class method).
save(fname_or_handle, separately=None, sep_limit=10485760, ignore=frozenset({}), pickle_protocol=2)¶Save the object to a file.
fname_or_handle (str or file-like) – Path to output file or already opened file-like object. If the object is a file handle, no special array handling will be performed, all attributes will be saved to the same file.
separately (list of str or None, optional) –
If None, automatically detect large numpy/scipy.sparse arrays in the object being stored, and store them into separate files. This prevent memory errors for large objects, and also allows memory-mapping the large arrays for efficient loading and sharing the large arrays in RAM between multiple processes.
If list of str: store these attributes into separate files. The automated size check is not performed in this case.
sep_limit (int, optional) – Don’t store arrays smaller than this separately. In bytes.
ignore (frozenset of str, optional) – Attributes that shouldn’t be stored at all.
pickle_protocol (int, optional) – Protocol number for pickle.
See also
load()Load object from file.
gensim.interfaces.TransformationABC¶Bases: gensim.utils.SaveLoad
Transformation interface.
A ‘transformation’ is any object which accepts document in BoW format via the __getitem__ (notation []) and returns another sparse document in its stead:
>>> from gensim.models import LsiModel
>>> from gensim.test.utils import common_dictionary, common_corpus
>>>
>>> model = LsiModel(common_corpus, id2word=common_dictionary)
>>> bow_vector = model[common_corpus[0]] # model applied through __getitem__ on one document from corpus.
>>> bow_corpus = model[common_corpus] # also, we can apply model on the full corpus
load(fname, mmap=None)¶Load an object previously saved using save() from a file.
fname (str) – Path to file that contains needed object.
mmap (str, optional) – Memory-map option. If the object was saved with large arrays stored separately, you can load these arrays via mmap (shared memory) using mmap=’r’. If the file being loaded is compressed (either ‘.gz’ or ‘.bz2’), then `mmap=None must be set.
See also
save()Save object to file.
Object loaded from fname.
object
AttributeError – When called on an object instance instead of class (this is a class method).
save(fname_or_handle, separately=None, sep_limit=10485760, ignore=frozenset({}), pickle_protocol=2)¶Save the object to a file.
fname_or_handle (str or file-like) – Path to output file or already opened file-like object. If the object is a file handle, no special array handling will be performed, all attributes will be saved to the same file.
separately (list of str or None, optional) –
If None, automatically detect large numpy/scipy.sparse arrays in the object being stored, and store them into separate files. This prevent memory errors for large objects, and also allows memory-mapping the large arrays for efficient loading and sharing the large arrays in RAM between multiple processes.
If list of str: store these attributes into separate files. The automated size check is not performed in this case.
sep_limit (int, optional) – Don’t store arrays smaller than this separately. In bytes.
ignore (frozenset of str, optional) – Attributes that shouldn’t be stored at all.
pickle_protocol (int, optional) – Protocol number for pickle.
See also
load()Load object from file.
gensim.interfaces.TransformedCorpus(obj, corpus, chunksize=None, **kwargs)¶Bases: gensim.interfaces.CorpusABC
Interface for corpora that are the result of an online (streamed) transformation.
obj (object) – A transformation TransformationABC object that will be applied
to each document from corpus during iteration.
corpus (iterable of list of (int, number)) – Corpus in bag-of-words format.
chunksize (int, optional) – If provided, a slightly more effective processing will be performed by grouping documents from corpus.
load(fname, mmap=None)¶Load an object previously saved using save() from a file.
fname (str) – Path to file that contains needed object.
mmap (str, optional) – Memory-map option. If the object was saved with large arrays stored separately, you can load these arrays via mmap (shared memory) using mmap=’r’. If the file being loaded is compressed (either ‘.gz’ or ‘.bz2’), then `mmap=None must be set.
See also
save()Save object to file.
Object loaded from fname.
object
AttributeError – When called on an object instance instead of class (this is a class method).
save(*args, **kwargs)¶Saves corpus in-memory state.
Warning
This save only the “state” of a corpus class, not the corpus data!
For saving data use the serialize method of the output format you’d like to use
(e.g. gensim.corpora.mmcorpus.MmCorpus.serialize()).
save_corpus(fname, corpus, id2word=None, metadata=False)¶Save corpus to disk.
Some formats support saving the dictionary (feature_id -> word mapping), which can be provided by the optional id2word parameter.
Notes
Some corpora also support random access via document indexing, so that the documents on disk
can be accessed in O(1) time (see the gensim.corpora.indexedcorpus.IndexedCorpus base class).
In this case, save_corpus() is automatically called internally by
serialize(), which does save_corpus() plus saves the index
at the same time.
Calling serialize() is preferred to calling :meth:`gensim.interfaces.CorpusABC.save_corpus().
fname (str) – Path to output file.
corpus (iterable of list of (int, number)) – Corpus in BoW format.
id2word (Dictionary, optional) – Dictionary of corpus.
metadata (bool, optional) – Write additional metadata to a separate too?
