July 13, 2025

Large Language Models as Rankers, Judges, and Assistants: A Perspective on the Potential Over-Reliance on LLMs in IR

Abstract

Large language models (LLMs) are rapidly transforming information retrieval (IR), serving as core ranking components, synthetic data generators, and judges for automated system evaluation. Beyond these roles, LLMs are also empowering AI assistants that are changing how users create content across various domains. While the potential of LLMs is undeniable, this growing reliance raises concerns about potential over-dependence, particularly when multiple LLMs interact within an information service where users both create and access content. We present a comprehensive survey of LLMs in IR, examining the interconnected roles they play as rankers, data generators, judges, and as AI assistants. We focus on the novel challenge of understanding how interactions between LLMs used for different purposes influence outcomes across several use cases. Through experiments on public test collections, we analyze how specific LLM relationships influence retrieval effectiveness. This perspective paper provides a balanced view on LLM integration in IR, acknowledging both the benefits and potential risks of increased reliance on these powerful models. It includes experiment designs and illustrative findings to drive further investigations in this crucial area.

Authors

Krisztian Balog, Don Metzler, Zhen Qin

Venue

SIGIR 2025 <Perspective Papers>