Bridging Language Modeling & Divergence From Randomness Approaches: A Log-logistic Model for IR
Stéphane Clinchant, Eric Gaussier
We are interested in this paper in revisiting the Divergence from Randomness (DFR) approach to Information Retrieval (IR), so as to better understand the different contributions it relies on, and thus be able to simplify it. To do so, we first introduce an analytical characterization of heuristic retrieval constraints and review several DFR models wrt this characterization. This review shows that the first normalization principle of DFR is necessary to make the model compliant with retrieval constraints. We then show that the log-logistic distribution can be used to derive a simplified DFR model. Interestingly, this simplified model contains Language Models (LM) with Jelinek-Mercer smoothing. The relation we propose here is, to our knowledge, the first connection between the DFR and LM approaches. Lastly, we present experimental results obtained on several standard collections which validate the simplification and the model we propose.
ICTIR 2009 (International Conference in the Theory of Information Retrieval), Cambridge, UK, 10-12 September 2009.
Event organized by Microsoft Research and the Open University.
The proceedings will be published by Springer in the Lecture Notes in Computer Science series.