Batch text processing

[Definelymes]

Hi, I was wondering if there is a feature (or would be in future) to process multiple texts at a same time instead of processing them one by one in sequential manner.

[urchade]

You can do it like this:

def batch_predict_entities(self, texts, labels, flat_ner=True, threshold=0.5):
        """
        Predict entities for a batch of texts.
        texts:  List of texts | List[str]
        labels: List of labels | List[str]
        ...
        """

[Definelymes]

Thank you very much, I tried this approach and came accross this issue:
#63

Is there any plan to resolve it anytime soon on the new releases?

[urchade]

Fixed now. You can install v0.1.9 @Definelymes

[Hitish99]

Hi @urchade Urchade
Can you please share that while batch inference text are processed in parallel or concurrent?

[urchade]

Hi, The batch processing are processed in parallel @Hitish99

[batmech]

How does this batch processing really works?

cleaned_text_list = cleaned_text.split()

chunk_list = [' '.join(cleaned_text_list[i:i + 160]).strip() for i in range(0, len(cleaned_text_list), 160)]

for i in range(0, len(chunk_list), 8):

    chunk_batch = chunk_list[i:i + 8]

    batch_results = model.batch_predict_entities(chunk_batch, labels=labels)

Above is my implementation for self-study.
My question is: Does batch_predict_entities run on different CPU cores to facilitate batch processing (in parallel)?
@urchade

[amit-plutoflume]

Hi,

I’m trying to perform batch inference using GLiNER and noticed that the batch method seems slower than the sequential one.

Here’s what I tried:

Batch processing

model.batch_predict_entities(texts, labels, threshold=0.5)

Sequential processing

df["text"].apply(lambda x: model.predict_entities(x, labels, threshold=0.5))

Surprisingly, the second (sequential) approach is faster in my case.
Could you please suggest whether I might be using the batch processing method incorrectly, or if there are any configuration parameters (e.g., batch size, device setup) I should adjust?

Thank you

[Shrinivas23]

I agree with folks above, batch processing is slower than sequential. Making it cost prohibitive to use in production with high traffic sadly. @urchade

[vivekkalyanarangan30]

@Shrinivas23 sequence packing should solve this in v0.2.23, I pushed a PR for this and there is a ~2.5x speedup in workloads, benchmarks scripts contain different test distributions