Does evidence support the high expectations placed in precision medicine? A bibliographic review

May combine two claims.

The heuristic analysis found 30 studies that compromised homoscedasticity: 11 (5.3%) with higher variance in the treated arm and 19 (9.1%) with lower variance.

“We found 30 studies that compromised homoscedasticity: 11 (5.3%) with higher variance in the treated arm and 19 (9.1%), with lower variance (see heuristic method in Table 1).”

Probably not a substantial claim: may not pass S3 (load-bearing).

More red points on the left indicate that changes in the average accompany reductions in the variance.

“We observe more red points on the left, indicating that changes in the average accompany reductions in the variance.”

May combine two claims.

In the over-time comparison, studies to the right have significantly higher variance at the end of the study in the treated group, while those on the left are significantly larger at the beginning.

“As for the over-time comparison, the studies to the right have a significantly higher variance at the end of the study in the treated group, while those on the left are significantly larger at the beginning of the study.”

May combine two claims.

Based on classical variance comparison tests, 41 studies (19.7%) had statistically significant differences between outcome variances; 15 (7.2%) favored greater variance in the treated arm; and 26 (12.5%) were in the opposite direction.

“Based on the classical variance comparison tests (sensitivity analysis), these figures were slightly higher: 41 studies (19.7%) had statistically significant differences between outcome variances; 15 (7.2%) favored greater variance in the treated arm; and 26 (12.5%) were in the opposite direction.”

May combine two claims.

In the random-effects model, among 208 studies comparing outcome between treatment arms, 14 (6.7%) had increased variability, 26 (12.5%) decreased, and 168 (80.8%) not changed.

“Outcome between treatment arms | 208 | Random-effects model | 14(6.7%) | 26 (12.5%) | 168(80.8%)”

May combine two claims.

In comparisons over time of 95 treated arms, 16.8% had significantly greater variability at the end of the study and 23.2% at the beginning.

“Larger proportions were obtained from the comparisons over time of 95 treated arms: 16.8% had significantly greater variability at the end of the study and 23.2% at the beginning.”

May combine two claims.

The review included 208 articles with enough information to conduct the analysis.

“The review provided 208 articles with enough information to conduct the analysis.”

• Also stated in: § Population · plain

  “208 (17.1%) contained enough information to enable us to conduct the analysis”

May combine two claims.

In the between-arm comparison, studies to the right of the triangle have variances significantly larger in the treatment arm than in the control arm, while those on the left are significantly larger in the control arm.

“In the between-arm comparison, the studies (represented by the circles) to the right of the triangle have variances that are significantly larger in the treatment arm than in the control arm, while those on the left are significantly larger in the control arm.”

May combine two claims.

The estimated baseline heterogeneity (τ2) was 0.31, which is a very high value.

“The second heuristic analysis was motivated by the fact that the estimated baseline heterogeneity (τ 2) was 0.31 ( Supplementary File 1 : Table S4, Model 1 with CDB), which is a very high value that could be explained by methodological flaws similar to those presented by Carlisle 23.”

May combine two claims.

Variability seems to decrease for treatments that perform significantly better than the reference; otherwise, it remains similar.

“Regarding the subgroup analyses, we found that variability seems to decrease for treatments that perform significantly better than the reference; otherwise, it remains similar.”

Probably not a substantial claim: may not pass S4 (based).

There are studies where the variability in the response is glaringly different, indicating the presence of a non-constant effect.

“First, there are studies where the variability in the response is glaringly different, indicating the presence of a non-constant effect.”

May combine two claims.

Without further specifications, it seems those trials were either hoping for the treatment effect to be the same for all patients or assuming that it was not useful to try and investigate this.

“Therefore, without further specifications, it seems that they were either hoping for the treatment effect to be the same for all patients or assuming that it was not useful to try and investigate this.”

Classified as probably not a substantial claim.

When studies reported more than one numerical endpoint and failed to clarify the primary endpoint, the primary endpoint was determined using hierarchical criteria.

“For studies that reported more than one numerical endpoint and failed to clarify which endpoint was the primary endpoint, the latter was determined using the following hierarchical criteria: (1) objective or hypothesis; (2) sample size determination; (3) main statistical method; (4) first numerical variable reported in results.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Data were collected by two researchers and later verified by two statisticians.

“Data were collected by two researchers (NM, MkV) in two phases: 2004/2007 and 2010/2013. Later, two statisticians (JC, MtV) verified the data and made them accessible to readers through a Shiny application and through the Figshare repository 18.”

Probably not a substantial claim: may not pass S4 (based).

The baseline variance ratio is expected to have heterogeneity equal to 0 due to randomization, provided no methodological impurities are present.

“we first modeled the baseline variance ratio as a response that is expected to have heterogeneity equal to 0 due to randomization – provided no methodological impurities are present”

Probably not a substantial claim: may not pass S4 (based).

When both arms have equal variances, the treatment effect is constant.

“When both arms have equal variances, then a simple and believable interpretation is that the treatment effect is constant, which, if correct, would render futile any search for predictors of differential response.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The existence of eligibility criteria at baseline may have limited the initial variance, leading to the variance increasing naturally over time.

“For example, the existence of eligibility criteria at baseline may have limited the initial variance (a hypertension trial might recruit patients with baseline SBP between 140 and 159 mm Hg), leading to the variance increasing naturally over time.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The constant effect assumption makes the average causal effect relevant to each patient.

“Following Holland 13, this assumption has the advantage of making the average causal effect relevant to each patient.”

• cites [13] R Wang, 2007, Statistics in medicine--reporting of subgroup analyses in clinical trials., N Engl J Med. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The simplest and strongest assumption about the treatment effect is that the effect is constant.

“The simplest and strongest one is that the effect is constant.”

May combine two claims.

The evidence of effect variation in only 1 out of 5 trials suggests a limited role for tailored interventions.

“That is, we have found evidence of effect variation in only 1 out of 5 trials (40/208), suggesting a limited role for tailored interventions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Three different methods were used to compare variances: a random-effects model, a heuristic procedure based on heterogeneity from the random-effects model, and a classical test for equality of variances.

“Three different methods were used to compare the variances: 1) a random-effects model; 2) a heuristic procedure based on the heterogeneity obtained from the previous random-effects model; and 3) a classical test for equality of variances.”

May combine two claims.

If both arms have equal variances, the average treatment effect can be seen as an individual treatment effect, supporting a unique clinical guideline for all patients within eligibility criteria.

“This means that the average treatment effect can be seen as an individual treatment effect (not directly observable), which supports the use of a unique clinical guideline for all patients within the eligibility criteria, thus in turn also supporting the use of parallel controlled trials to guide decision-making in these circumstances.”

Classified as probably not a substantial claim.

Heteroscedasticity may suggest a need to specify further the eligibility criteria or search for an additive scale.

“Otherwise, heteroscedasticity may suggest a need to specify further the eligibility criteria or search for an additive scale 25, 29.”

• cites [25] S Senn, 2004, Controversies concerning randomization and additivity in clinical trials., Stat Med. sourceNot checked yet

  Not checked in the free run.

• cites [29] K Schulz, 2010, CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials., BMJ. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Different patients and stakeholders may vary in their weighting of efficacy outcomes, harm, and cost of interventions, bridging the gap between common evidence and personalized decisions.

“More specifically, this is because different patients and stakeholders may vary in their weighting not only of efficacy outcomes, but also of the harm and cost of the interventions – thus bridging the gap between common evidence and personalized decisions.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Looking only at averages can cause us to miss out on important information.

“Looking only at averages can cause us to miss out on important information.”

Probably not a substantial claim: may not pass S2 (checkable).

There is a need to measure a greater range of features to determine the response to treatment.

“Our results uphold the assertion by Horwitz et al. that there is a “need to measure a greater range of features to determine [...] the response to treatment” 31.”

• cites [31] R Horwitz, 2013, Medicine. (De)personalized medicine., Science. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The treatment effect may not be additive on the scale used for analysis, suggesting that it would be suitable to explore other metrics and transformations.

“As mentioned above, another reason might be that the treatment effect is not additive on the scale used for analysis, suggesting that it would be suitable to explore other metrics and transformations.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

This study focuses on numerical endpoints; time-to-event or categorical outcomes are out of scope.

“And, third, this study focuses on numerical endpoints; thus, time-to-event or categorical outcomes are out of scope.”

May combine two claims.

Trials are usually powered to test constant effects and thus the presence of greater variability would lead to an underpowered design.

“Third, trials are usually powered to test constant effects and thus the presence of greater variability would lead to an underpowered design; that is, if the control group variance is used to plan the trial, increased treatment group variance would reduce power (perhaps leading to non-publication).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The outcomes of some types of interventions such as surgeries are greatly influenced by the skills and training of those administering the intervention; these situations could have some effect on increasing variability.

“Second, the outcomes of some type of interventions such as surgeries, for example, are greatly influenced by the skills and training of those administering the intervention; and these situations could have some effect on increasing variability.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

The heterogeneity could also be the result of relevant undetected factors interacting with the treatment, which would justify the suitability of precision medicine.

“On the other hand, this heterogeneity could also be the result of relevant undetected factors interacting with the treatment, which would indeed justify the suitability of precision medicine.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The heterogeneity observed in the random-effects model may be the result of methodological inaccuracies.

“Fourth, the heterogeneity observed in the random-effects model may be the result of methodological inaccuracies 23 arising from typographical errors in data translation, inadequate follow-up, insufficient reporting, or even data fabrication.”

• cites [23] J Carlisle, 2017, Data fabrication and other reasons for non-random sampling in 5087 randomised, controlled trials in anaesthetic and general medical journals., Anaesthesia. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A constant effect is the simplest explanation for homoscedasticity.

“Nevertheless, a constant effect is the simplest explanation for homoscedasticity (Section VIII of Supplementary File 1: Conditions for homoscedasticity to hold without a constant effect under an additive model).”

May combine two claims.

Although a constant effect always implies homoscedasticity on the chosen scale, the reverse is not true; homoscedasticity does not necessarily imply a constant effect.

“The main limitation of our study arises from the fact that, although a constant effect always implies homoscedasticity on the chosen scale, the reverse is not true; i.e., homoscedasticity does not necessarily imply a constant effect.”