# Metrics

## Aliases

Aliases to use with ranx.evaluate and ranx.compare.

Metric Alias @k .p
Hits hits Yes No
Hit Rate / Success hit_rate Yes No
Precision precision Yes No
Recall recall Yes No
F1 f1 Yes No
R-Precision r_precision No No
Bpref bpref No No
Rank-biased Precision rbp No Yes
Mean Reciprocal Rank mrr Yes No
Mean Average Precision map Yes No
NDCG ndcg Yes No
NDCG Burges ndcg_burges Yes No

## Hits

Hits is the number of relevant documents retrieved.

## Hit Rate / Success

Hit Rate is the fraction of queries for which at least one relevant document is retrieved. Note: it is equivalent to success from trec_eval.

## Precision

Precision is the proportion of the retrieved documents that are relevant.

$\operatorname{Precision}=\frac{r}{n}$

where,

• $$r$$ is the number of retrieved relevant documents;
• $$n$$ is the number of retrieved documents.

## Recall

Recall is the ratio between the retrieved documents that are relevant and the total number of relevant documents.

$\operatorname{Recall}=\frac{r}{R}$

where,

• $$r$$ is the number of retrieved relevant documents;
• $$R$$ is the total number of relevant documents.

## F1

F1 is the harmonic mean of Precision and Recall.

$\operatorname{F1} = 2 \times \frac{\operatorname{Precision} \times \operatorname{Recall}}{\operatorname{Precision} + \operatorname{Recall}}$

## R-Precision

For a given query $$Q$$, R-Precision is the precision at $$R$$, where $$R$$ is the number of relevant documents for $$Q$$. In other words, if there are $$r$$ relevant documents among the top-$$R$$ retrieved documents, then R-precision is:

$\operatorname{R-Precision} = \frac{r}{R}$

## Bpref

Bpref is designed for situations where relevance judgments are known to be incomplete. It is defined as:

$\operatorname{bpref}=\frac{1}{R}\sum_r{1 - \frac{|\text{n ranked higher than r}|}{R}}$

where,

• $$r$$ is a relevant document;
• $$n$$ is a member of the first R judged nonrelevant documents as retrieved by the system;
• $$R$$ is the number of relevant documents.

## Rank-biased Precision

Compute Rank-biased Precision (RBP).

It is defined as:

$\operatorname{RBP} = (1 - p) \cdot \sum_{i=1}^{d}{r_i \cdot p^{i - 1}}$

where,

• $$p$$ is the persistence value;
• $$r_i$$ is either 0 or 1, whether the $$i$$-th ranked document is non-relevant or relevant, repsectively.

## (Mean) Reciprocal Rank

Reciprocal Rank is the multiplicative inverse of the rank of the first retrieved relevant document: 1 for first place, 1/2 for second place, 1/3 for third place, and so on. When averaged over many queries, it is usually called Mean Reciprocal Rank (MRR).

$Reciprocal Rank = \frac{1}{rank}$

where,

• $$rank$$ is the position of the first retrieved relevant document.

## (Mean) Average Precision

Average Precision is the average of the Precision scores computed after each relevant document is retrieved. When averaged over many queries, it is usually called Mean Average Precision (MAP).

$\operatorname{Average Precision} = \frac{\sum_r \operatorname{Precision}@r}{R}$

where,

• $$r$$ is the position of a relevant document;
• $$R$$ is the total number of relevant documents.

## NDCG

Compute Normalized Discounted Cumulative Gain (NDCG) as proposed by Järvelin et al..

BibTeX
@article{DBLP:journals/tois/JarvelinK02,
author    = {Kalervo J{\"{a}}rvelin and
Jaana Kek{\"{a}}l{\"{a}}inen},
title     = {Cumulated gain-based evaluation of {IR} techniques},
journal   = {{ACM} Trans. Inf. Syst.},
volume    = {20},
number    = {4},
pages     = {422--446},
year      = {2002}
}

$\operatorname{nDCG} = \frac{\operatorname{DCG}}{\operatorname{IDCG}}$

where,

• $$\operatorname{DCG}$$ is Discounted Cumulative Gain;
• $$\operatorname{IDCG}$$ is Ideal Discounted Cumulative Gain (max possibile DCG).

## NDCG Burges

Compute Normalized Discounted Cumulative Gain (NDCG) at k as proposed by Burges et al..

BibTeX
@inproceedings{DBLP:conf/icml/BurgesSRLDHH05,
author    = {Christopher J. C. Burges and
Tal Shaked and
Erin Renshaw and
Ari Lazier and
Matt Deeds and
Nicole Hamilton and
Gregory N. Hullender},
title     = {Learning to rank using gradient descent},
booktitle = {{ICML}},
series    = {{ACM} International Conference Proceeding Series},
volume    = {119},
pages     = {89--96},
publisher = {{ACM}},
year      = {2005}
}

$\operatorname{nDCG} = \frac{\operatorname{DCG}}{\operatorname{IDCG}}$

where,

• $$\operatorname{DCG}$$ is Discounted Cumulative Gain;
• $$\operatorname{IDCG}$$ is Ideal Discounted Cumulative Gain (max possibile DCG).