gensim logo

gensim
gensim tagline

Get Expert Help From The Gensim Authors

Consulting in Machine Learning & NLP

• Commercial document similarity engine: ScaleText.ai

Corporate trainings in Python Data Science and Deep Learning

parsing.preprocessing – Functions to preprocess raw text

parsing.preprocessing – Functions to preprocess raw text

This module contains methods for parsing and preprocessing strings. Let’s consider the most noticeable:

Examples:

>>> from gensim.parsing.preprocessing import remove_stopwords
>>> remove_stopwords("Better late than never, but better never late.")
u'Better late never, better late.'
>>>
>>> preprocess_string("<i>Hel 9lo</i> <b>Wo9 rld</b>! Th3     weather_is really g00d today, isn't it?")
[u'hel', u'rld', u'weather', u'todai', u'isn']

Data:

STOPWORDS - Set of stopwords from Stone, Denis, Kwantes (2010).
RE_PUNCT - Regexp for search an punctuation.
RE_TAGS - Regexp for search an tags.
RE_NUMERIC - Regexp for search an numbers.
RE_NONALPHA - Regexp for search an non-alphabetic character.
RE_AL_NUM - Regexp for search a position between letters and digits.
RE_NUM_AL - Regexp for search a position between digits and letters .
RE_WHITESPACE - Regexp for search space characters.
DEFAULT_FILTERS - List of function for string preprocessing.
gensim.parsing.preprocessing.preprocess_documents(docs)

Apply DEFAULT_FILTERS to the documents strings.

Parameters:docs (list of str) –
Returns:Processed documents split by whitespace.
Return type:list of list of str

Examples

>>> from gensim.parsing.preprocessing import preprocess_documents
>>> preprocess_documents(["<i>Hel 9lo</i> <b>Wo9 rld</b>!", "Th3     weather_is really g00d today, isn't it?"])
[[u'hel', u'rld'], [u'weather', u'todai', u'isn']]
gensim.parsing.preprocessing.preprocess_string(s, filters=[<function <lambda>>, <function strip_tags>, <function strip_punctuation>, <function strip_multiple_whitespaces>, <function strip_numeric>, <function remove_stopwords>, <function strip_short>, <function stem_text>])

Apply list of chosen filters to s.

Default list of filters:

Parameters:
  • s (str) –
  • filters (list of functions, optional) –
Returns:

Processed strings (cleaned).

Return type:

list of str

Examples

>>> from gensim.parsing.preprocessing import preprocess_string
>>> preprocess_string("<i>Hel 9lo</i> <b>Wo9 rld</b>! Th3     weather_is really g00d today, isn't it?")
[u'hel', u'rld', u'weather', u'todai', u'isn']
>>>
>>> s = "<i>Hel 9lo</i> <b>Wo9 rld</b>! Th3     weather_is really g00d today, isn't it?"
>>> CUSTOM_FILTERS = [lambda x: x.lower(), strip_tags, strip_punctuation]
>>> preprocess_string(s, CUSTOM_FILTERS)
[u'hel', u'9lo', u'wo9', u'rld', u'th3', u'weather', u'is', u'really', u'g00d', u'today', u'isn', u't', u'it']
gensim.parsing.preprocessing.read_file(path)
gensim.parsing.preprocessing.read_files(pattern)
gensim.parsing.preprocessing.remove_stopwords(s)

Remove STOPWORDS from s.

Parameters:s (str) –
Returns:Unicode string without STOPWORDS.
Return type:str

Examples

>>> from gensim.parsing.preprocessing import remove_stopwords
>>> remove_stopwords("Better late than never, but better never late.")
u'Better late never, better late.'
gensim.parsing.preprocessing.split_alphanum(s)

Add spaces between digits & letters in s using RE_AL_NUM.

Parameters:s (str) –
Returns:Unicode string with spaces between digits & letters.
Return type:str

Examples

>>> from gensim.parsing.preprocessing import split_alphanum
>>> split_alphanum("24.0hours7 days365 a1b2c3")
u'24.0 hours 7 days 365 a 1 b 2 c 3'
gensim.parsing.preprocessing.stem(text)

Transform s into lowercase and stem it.

Parameters:text (str) –
Returns:Unicode lowercased and porter-stemmed version of string text.
Return type:str

Examples

>>> from gensim.parsing.preprocessing import stem_text
>>> stem_text("While it is quite useful to be able to search a large collection of documents almost instantly.")
u'while it is quit us to be abl to search a larg collect of document almost instantly.'
gensim.parsing.preprocessing.stem_text(text)

Transform s into lowercase and stem it.

Parameters:text (str) –
Returns:Unicode lowercased and porter-stemmed version of string text.
Return type:str

Examples

>>> from gensim.parsing.preprocessing import stem_text
>>> stem_text("While it is quite useful to be able to search a large collection of documents almost instantly.")
u'while it is quit us to be abl to search a larg collect of document almost instantly.'
gensim.parsing.preprocessing.strip_multiple_whitespaces(s)

Remove repeating whitespace characters (spaces, tabs, line breaks) from s and turns tabs & line breaks into spaces using RE_WHITESPACE.

Parameters:s (str) –
Returns:Unicode string without repeating in a row whitespace characters.
Return type:str

Examples

>>> from gensim.parsing.preprocessing import strip_multiple_whitespaces
>>> strip_multiple_whitespaces("salut" + '\r' + " les" + '\n' + "         loulous!")
u'salut les loulous!'
gensim.parsing.preprocessing.strip_non_alphanum(s)

Remove non-alphabetic characters from s using RE_NONALPHA.

Parameters:s (str) –
Returns:Unicode string with alphabetic characters only.
Return type:str

Notes

Word characters - alphanumeric & underscore.

Examples

>>> from gensim.parsing.preprocessing import strip_non_alphanum
>>> strip_non_alphanum("if-you#can%read$this&[email protected]#method^works")
u'if you can read this then this method works'
gensim.parsing.preprocessing.strip_numeric(s)

Remove digits from s using RE_NUMERIC.

Parameters:s (str) –
Returns:Unicode string without digits.
Return type:str

Examples

>>> from gensim.parsing.preprocessing import strip_numeric
>>> strip_numeric("0text24gensim365test")
u'textgensimtest'
gensim.parsing.preprocessing.strip_punctuation(s)

Replace punctuation characters with spaces in s using RE_PUNCT.

Parameters:s (str) –
Returns:Unicode string without punctuation characters.
Return type:str

Examples

>>> from gensim.parsing.preprocessing import strip_punctuation
>>> strip_punctuation("A semicolon is a stronger break than a comma, but not as much as a full stop!")
u'A semicolon is a stronger break than a comma  but not as much as a full stop '
gensim.parsing.preprocessing.strip_punctuation2(s)

Replace punctuation characters with spaces in s using RE_PUNCT.

Parameters:s (str) –
Returns:Unicode string without punctuation characters.
Return type:str

Examples

>>> from gensim.parsing.preprocessing import strip_punctuation
>>> strip_punctuation("A semicolon is a stronger break than a comma, but not as much as a full stop!")
u'A semicolon is a stronger break than a comma  but not as much as a full stop '
gensim.parsing.preprocessing.strip_short(s, minsize=3)

Remove words with length lesser than minsize from s.

Parameters:
  • s (str) –
  • minsize (int, optional) –
Returns:

Unicode string without short words.

Return type:

str

Examples

>>> from gensim.parsing.preprocessing import strip_short
>>> strip_short("salut les amis du 59")
u'salut les amis'
>>>
>>> strip_short("one two three four five six seven eight nine ten", minsize=5)
u'three seven eight'
gensim.parsing.preprocessing.strip_tags(s)

Remove tags from s using RE_TAGS.

Parameters:s (str) –
Returns:Unicode string without tags.
Return type:str

Examples

>>> from gensim.parsing.preprocessing import strip_tags
>>> strip_tags("<i>Hello</i> <b>World</b>!")
u'Hello World!'