Skip to content

fix: stemming #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
stephantul wants to merge 1 commit into
base: wandb
Choose a base branch
from
Open

fix: stemming #2

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions src/deep_impact/models/original.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ def get_query_document_token_mask(cls, query_terms: Set[str], term_to_token_inde
def process_query(cls, query: str) -> Set[str]:
query = cls.tokenizer.normalizer.normalize_str(query)
terms = map(lambda x: x[0], cls.tokenizer.pre_tokenizer.pre_tokenize_str(query))
filtered_terms = filter(lambda x: x not in cls.punctuation, terms)
filtered_terms = set(filter(lambda x: x not in cls.punctuation, terms))
stemmed_terms = set()
for term in filtered_terms:
if term not in cls.stemmer_cache:
Expand Down Expand Up @@ -141,13 +141,14 @@ def process_document(cls, document: str) -> Tuple[tokenizers.Encoding, Dict[str,

# filter out duplicate terms, punctuations, and terms whose tokens overflow the max length
for i, term in enumerate(document_terms):
if term not in cls.stemmer_cache:
cls.stemmer_cache[term] = cls.stemmer.stem(term)
term = cls.stemmer_cache[term]

if term not in filtered_term_to_token_index \
and term not in cls.punctuation \
and i in term_index_to_token_index:
# check if stemm is cached
if term not in cls.stemmer_cache:
cls.stemmer_cache[term] = cls.stemmer.stem(term)
term = cls.stemmer_cache[term]
filtered_term_to_token_index[term] = term_index_to_token_index[i]
return encoded, filtered_term_to_token_index

Expand Down