Is GenAI All You Need to Classify Text? Some Learnings from the Trenches

Generative AI/LLMs perform poorly for text classification tasks compared to specialized models, showing a significant accuracy drop (~16%) and much higher computational costs

Smaller specialized models can be 1000x smaller than LLMs while being faster, more cost-effective, and more environmentally friendly

Using frozen pre-trained multilingual language models (sentence transformers) with a simple classifier layer provides good results across multiple languages due to language alignment in the latent space

Model optimization techniques like graph optimization and quantization can reduce response times by 2-3x and significantly decrease memory consumption

LLMs can still be useful for:

• Generating training data when labels are scarce
• Bootstrapping new categories
• Handling new languages without existing training data

For multilingual systems, using language-aligned embeddings allows training on one language while maintaining performance across others

Response time optimization is crucial for user experience - the new optimized model was 3x faster than legacy system and 100-1000x faster than using Palm2

Environmental and cost considerations strongly favor specialized models over LLMs for narrow classification tasks

Maintaining separate monolingual pipelines is complex and inefficient compared to a single multilingual model

Post-processing and manual curation of LLM outputs is often necessary due to hallucination issues