Handbook on Constructing Composite Indicators: Methodology and User Guide

Using extreme values as thresholds could distort the results.

“A possible disadvantage of this choice is the risk of using extreme values as thresholds that could distort the results [11,18].”

Higher weighting assigned to indicators with broader coverage or more trustworthy sources could result in a biased selection in favour of proxies that cannot properly capture the desired information.

“However, an issue here is that this could result in a 'biased selection' in favour of proxies that are not able to identify and capture properly the information desired to measure (see e.g.Custance and Hillier 1998, pp. 284-285;OECD 2008, p. 32).”

The OECD ten-step checklist aims to establish a common guideline for developing composite indices and to enhance transparency and soundness of the process.

“the OECD (2008, p. 15) identifies a ten-step process, namely a 'checklist'.Its aim is to establish a common guideline as a basis for the development of composite indices and to enhance the transparency and the soundness of the process.”

Equal weighting is the most common scheme appearing in the development of composite indicators.

“Equal weighting is the most common scheme appearing in the development of composite indicators (Bandura 2008;OECD 2008).”

Even if statistical significance applies, correlation does not imply causality but shows similar or opposite co-movement between indicators.

“Moreover, even if statistical significance applies, it does not imply causality but rather shows a similar or opposite co-movement between indicators (Freudenberg 2003;OECD 2008).”

Correlation analysis is mostly used in the first steps of the construction process to examine the structure and dynamics of indicators in the data set.

“Correlation analysis is mostly used in the first steps of the construction process to examine the structure and the dynamics of the indicators in the data set (Booysen 2002;OECD 2008).”

If the participatory audience does not clearly understand a framework, these methods would lead to biased results.

“What is more, in the case in which the participatory audience does not clearly understand a framework (e.g. to evaluate the importance of an indicator/phenomenon or what it actually represents), these methods would lead to biased results (OECD 2008).”

Neutrality states that all alternatives, such as countries, must be treated equally.

“In brief, neutrality states that 'all alternatives (e.g.countries) must be treated equally' (OECD 2008, p. 105).”

Linear aggregation implies constant compensability between indicators and dimensions, so a unit could compensate for the loss in one dimension with a gain in another.

“Understandably, this trade-off implies constant compensability between indicators and dimensions; thus, a unit could compensate for the loss in one dimension with a gain in another (OECD 2008;Munda and Nardo 2009).”

The choice of the proper annotation of weights essentially boils down to the choice of the proper aggregation method.

“The choice of the proper annotation, though, essentially boils down to the choice of the proper aggregation method (Munda 2005a, p. 118;OECD 2008, p. 33).”

Aggregation methods may be divided into three distinctive categories: linear, geometric, and multi-criteria.

“According to the latest handbook on constructing composite indices, aggregation methods may be divided into three distinctive categories: linear, geometric, and multi-criteria (see OECD 2008, p. 31, Table 4).”

Among the compensatory aggregation approaches, linear aggregation is the most commonly used in composite indicators.

“Among the compensatory aggregation approaches, the linear one is the most commonly used in composite indicators (Saisana and Tarantola 2002;Freudenberg 2003;OECD 2008;Bandura 2008Bandura , 2011)).”

Linear aggregation assumes preferential independence among indicators.

“The first is that it assumes 'preferential independence' among indicators (OECD 2008, p. 103;Fusco 2015, p. 621), something that is conceptually considered as a very strong assumption to make (Ting 1971).”

Geometric aggregation is adopted when the developer of an index prefers only some degree of compensability.

“This approach is adopted when the developer of an index prefers only 'some' degree of compensability (OECD 2008, p. 32).”

Linear and geometric aggregation schemes lie within the compensatory aggregation scheme, while the non-compensatory aggregation scheme contains other multi-criteria approaches.

“Consequently, according to the latter categorisation of aggregation approaches (i.e. that of 'compensatory' and 'non-compensatory'), the linear 21 and geometric 22 aggregation schemes lie within the 'compensatory' aggregation scheme, while the 'non-compensatory' aggregation scheme contains other multi-criteria approaches, considering preferential relationships from the pairwise comparisons of the”

A misspecified aggregate measure may radically alter the results.

“Indeed, a misspecified aggregate measure may radically alter the results, and drawing conclusions from it is inadvisable in policy practices (Saltelli 2007;OECD 2008).”

Robustness analysis should not be misinterpreted as a guarantee of the sensibility of the composite index.

“However, it should not be misinterpreted as a guarantee of the sensibility of the composite index.”

The OECD Handbook cannot provide any form of aid to the developer regarding which theoretical framework fits best.

“Unfortunately, but no doubt reasonably, the Handbook cannot provide any form of aid to the developer regarding which theoretical framework fits best.”

One cannot interpret an assessment of robustness as the validation of a 'sensible' index.

“However, one of the first points stressed in the OECD's Handbook is that one cannot interpret an assessment of robustness as the validation of a 'sensible' index (OECD 2008, p. 35).”

The creation of a sound theoretical framework determines whether the index is actually sensible.

“Rather, it is the creation of a sound theoretical framework that determines whether the index is actually sensible.”

Robustness analysis is often completely missing for the vast majority of composite indices, while some only partially use it.

“Despite its importance, robustness analysis is often found to be completely missing for the vast majority of the composite indices (OECD 2008), while some only partially use it (Freudenberg 2003;Dobbie and Dail 2013).”

Robustness analysis might only help the developer answer questions related to the fit of the model and the meaning of its concept.

“Robustness might only help the developer to answer the questions related to the fit of the model and the meaning of its concept (OECD 2008).”