Skip to content

Metrics

Aliases


Aliases to use with ranx.evaluate and ranx.compare.

Metric Alias @k
Hits hits Yes
Hit Rate / Success hit_rate Yes
Precision precision Yes
Recall recall Yes
F1 f1 Yes
R-Precision r_precision No
Mean Reciprocal Rank mrr Yes
Mean Average Precision map Yes
NDCG ndcg Yes
NDCG Burges ndcg_burges Yes

Hits


Hits is the number of relevant documents retrieved.

Hit Rate / Success


Hit Rate is the fraction of queries for which at least one relevant document is retrieved. Note: it is equivalent to success from trec_eval.

Precision


Precision is the proportion of the retrieved documents that are relevant.

\[ \operatorname{Precision}=\frac{r}{n} \]

where,

  • \(r\) is the number of retrieved relevant documents;
  • \(n\) is the number of retrieved documents.

Recall


Recall is the ratio between the retrieved documents that are relevant and the total number of relevant documents.

\[ \operatorname{Recall}=\frac{r}{R} \]

where,

  • \(r\) is the number of retrieved relevant documents;
  • \(R\) is the total number of relevant documents.

F1


F1 is the harmonic mean of Precision and Recall.

\[ \operatorname{F1} = 2 \times \frac{\operatorname{Precision} \times \operatorname{Recall}}{\operatorname{Precision} + \operatorname{Recall}} \]

R-Precision


For a given query \(Q\), R-Precision is the precision at \(R\), where \(R\) is the number of relevant documents for \(Q\). In other words, if there are \(r\) relevant documents among the top-\(R\) retrieved documents, then R-precision is:

\[ \operatorname{R-Precision} = \frac{r}{R} \]

(Mean) Reciprocal Rank


Reciprocal Rank is the multiplicative inverse of the rank of the first retrieved relevant document: 1 for first place, 1/2 for second place, 1/3 for third place, and so on. When averaged over many queries, it is usually called Mean Reciprocal Rank (MRR).

\[ Reciprocal Rank = \frac{1}{rank} \]

where,

  • \(rank\) is the position of the first retrieved relevant document.

(Mean) Average Precision


Average Precision is the average of the Precision scores computed after each relevant document is retrieved. When averaged over many queries, it is usually called Mean Average Precision (MAP).

\[ \operatorname{Average Precision} = \frac{\sum_r \operatorname{Precision}@r}{R} \]

where,

  • \(r\) is the position of a relevant document;
  • \(R\) is the total number of relevant documents.

NDCG


Compute Normalized Discounted Cumulative Gain (NDCG) as proposed by Järvelin et al..

BibTeX
@article{DBLP:journals/tois/JarvelinK02,
    author    = {Kalervo J{\"{a}}rvelin and
                Jaana Kek{\"{a}}l{\"{a}}inen},
    title     = {Cumulated gain-based evaluation of {IR} techniques},
    journal   = {{ACM} Trans. Inf. Syst.},
    volume    = {20},
    number    = {4},
    pages     = {422--446},
    year      = {2002}
}
\[ \operatorname{nDCG} = \frac{\operatorname{DCG}}{\operatorname{IDCG}} \]

where,

  • \(\operatorname{DCG}\) is Discounted Cumulative Gain;
  • \(\operatorname{IDCG}\) is Ideal Discounted Cumulative Gain (max possibile DCG).

NDCG Burges


Compute Normalized Discounted Cumulative Gain (NDCG) at k as proposed by Burges et al..

BibTeX
@inproceedings{DBLP:conf/icml/BurgesSRLDHH05,
    author    = {Christopher J. C. Burges and
                Tal Shaked and
                Erin Renshaw and
                Ari Lazier and
                Matt Deeds and
                Nicole Hamilton and
                Gregory N. Hullender},
    title     = {Learning to rank using gradient descent},
    booktitle = {{ICML}},
    series    = {{ACM} International Conference Proceeding Series},
    volume    = {119},
    pages     = {89--96},
    publisher = {{ACM}},
    year      = {2005}
}
\[ \operatorname{nDCG} = \frac{\operatorname{DCG}}{\operatorname{IDCG}} \]

where,

  • \(\operatorname{DCG}\) is Discounted Cumulative Gain;
  • \(\operatorname{IDCG}\) is Ideal Discounted Cumulative Gain (max possibile DCG).