CoChat

Time-based calibration of effectiveness measures

Smucker, Mark D., Clarke, Charles L.A. · 2012 · Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval
Cited by 197

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Abstract

Many current effectiveness measures incorporate simplifying assumptions about user behavior. These assumptions prevent the measures from reflecting aspects of the search process that directly impact the quality of retrieval results as experienced by the user. In particular, these measures implicitly model users as working down a list of retrieval results, spending equal time assessing each document. In reality, even a careful user, intending to identify as much relevant material as possible, must spend longer on some documents than on others. Aspects such as document length, duplicates and summaries all influence the time required. In this paper, we introduce a time-biased gain measure, which explicitly accommodates such aspects of the search process. By conducting an appropriate user study, we calibrate and validate the measure against the TREC 2005 Robust Track test collection. We examine properties of the measure, contrasting it to traditional effectiveness measures, and exploring its extension to other aspects and environments. As its primary benefit, the measure allows us to evaluate system performance in human terms, while maintaining the simplicity and repeatability of system-oriented tests. Overall, we aim to achieve a clearer connection between user-oriented studies and system-oriented tests, allowing us to better transfer insights and outcomes from one to the other.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/2348283.2348300

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

  • Cumulated gain-based evaluation of IR techniques

    Järvelin, K., & Kekäläinen, J. (2002) · ACM Transactions on Information Systems · doi:10.1145/582415.582418 · Cited by 4,841 · cited 2×

    “Various researchers [7,10,43] suggest a generic framework that encapsulates many of the effectiveness measures in current use, including Normalized Discounted Cumulative Gain (nDCG) [17], Rank Biased Precision (RBP) [23], and Expected Reciprocal Rank (ERR) [9].”
    BackgroundMethod

Show all 1 references

Cited by

See all 197

Metadata

Authors
Smucker, M. D., & Clarke, C. L. A.
Venue
Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval
Publisher
ACM
Published
2012-08-12
Pages
95-104
OpenAlex
W2029009258
Cited by
197
References
1
FWCI
16.68
Type
conference paper
Language
en
Open access
closed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Time-based calibration of effectiveness measures. https://app.cochat.ai/papers/10.1145/2348283.2348300
The paper
Smucker, M. D., & Clarke, C. L. A. (2012). Time-based calibration of effectiveness measures. Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval. https://doi.org/10.1145/2348283.2348300

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.