ModernBERT-base

ModernBERT-base is a modernized encoder-only transformer model from Answer.AI, designed as a drop-in replacement for BERT-style models with substantially improved efficiency, longer context (8192 tokens), and state-of-the-art performance on retrieval and classification tasks. It incorporates architectural advances such as rotary positional embeddings, alternating attention, and flash attention to outperform legacy encoders like BERT and RoBERTa. Source: supabase.