Gemini’s data-analyzing abilities aren’t as good as Google claims

Google's Gemini AI models are overhyped, struggling with large datasets and complex tasks, as recent studies reveal performance issues.

: Recent studies show Google’s Gemini AI models struggle with large datasets, contradicting the company’s claims. Tests reveal poor performance in understanding and reasoning tasks. Google's marketing of their context capabilities appears premature.

Recent research indicates Google's Gemini AI models, particularly Gemini 1.5 Pro and 1.5 Flash, struggle with processing and analyzing large datasets. The models performed poorly in tasks such as answering questions about lengthy fiction books and evaluating video content, often scoring worse than random chance.

One study by Marzena Karpinska and colleagues found the models answered true/false questions about books with less than 50% accuracy, revealing major comprehension issues. Another study at UC Santa Barbara showed Gemini 1.5 Flash’s struggles with simple reasoning tasks over images, with accuracy dropping significantly as task complexity increased.

Despite Google’s heavy marketing of Gemini's context capabilities, it appears the technology is not yet ready to meet those expectations. The studies suggest a need for better benchmarks and third-party critique to accurately assess AI capabilities. Google has not responded to these findings, but the sentiment within the research community suggests caution against overhyped claims.

GeminiAI ModelsData analysis