A Sleeping, Recovering Bandit Algorithm for Optimizing Recurring Notifications
Open accessAbstract
Many online and mobile applications rely on daily emails and push notifications to increase and maintain user engagement. The multi-armed bandit approach provides a useful framework for optimizing the content of these notifications, but a number of complications (such as novelty effects and conditional eligibility) make conventional bandit algorithms unsuitable in practice. In this paper, we introduce the Recovering Difference Softmax Algorithm to address the particular challenges of this problem domain, and use it to successfully optimize millions of daily reminders for the online language-learning app Duolingo. This lead to a 0.5%. increase in total daily active users (DAUs) and a 2%, increase in new user retention over a strong baseline. We provide technical details of its design and deployment, and demonstrate its efficacy through both offline and online evaluation experiments.
AI snapshot
Written from the paper’s full text.
TL;DR
The paper introduces the Recovering Difference Softmax Algorithm, a multi-armed bandit variant for recurring notifications that handles conditional arm eligibility (“sleeping arms”) and novelty/recency effects (“recovering arms”). Deployed on millions of daily Duolingo reminders, it increased total daily active users by 0.5% and new-user retention by 2% over a strong baseline.
Key findings
- In an online A/B test, the bandit produced +0.5% Daily Active Users and +0.4% Total Lessons Completed (both p < 0.05); new-user D1 retention rose +2.2% and new-user D7 retention +2.0% (both p < 0.05), while existing-user D1 retention rose +0.5% (p < 0.05) and existing-user D7 retention +0.2%.
- Offline evaluation against a random baseline (average reward 0.1295) showed template-only bandit +1.2%, template+UI-language bandit +1.8%, and template+UI-language with recency penalty +1.9%; a policy that always reused the last template scored -0.5%.
- Five months after launch, the 95% of rounds using the bandit’s optimized policy had 2.5% higher reward than a 5% random holdout set.
- Relative-difference scoring controlled for eligibility confounding: for example, arm A had average reward 0.2672 when used vs 0.2682 when eligible (-0.37%), while arm G had 0.1372 vs 0.1352 (+1.48%).
Methods
- Design: a bandit variant that scores arms by the relative difference between weighted-importance-sampling estimates of reward when an arm is used versus when it is eligible but not chosen; it adds empirical Bayes shrinkage for small samples, an exponential recency penalty, and softmax arm selection.
- Data: offline experiments used 34 days of Duolingo legacy uniform-random notification data, split into 15 days/88M rows for training and 19 days/114M rows for testing; online evaluation was an A/B test on millions of daily practice reminders, with control using the legacy uniform-random policy and experimental using the bandit.
- Analysis: offline off-policy evaluation via weighted importance sampling, grid search for recency hyperparameters (γ = 0.017, half-life h = 15), and online tracking of DAUs, lessons completed, and retention; production later used a 5% random holdout for ongoing monitoring.
Limitations
- Stated: offline evaluation likely underestimates the recency penalty’s production gain because of an uncorrected feedback effect; the μ−a baseline depends on the behavior policy and tends to increase over time, causing arm scores to drift downward; batch daily score updates delay feedback; the parametric recovery function was chosen over Gaussian-process kernels due to data needs; and the model does not incorporate quantity of prior selections or other user properties, which the authors leave for future work.
- Evident: the evaluation is limited to Duolingo’s notification setting, template pools, and UI languages, with no external replication reported; the online A/B results are summarized after two weeks, with longer-term evidence coming from a later 5% holdout rather than a full long-term controlled experiment.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
1 of 5 cited sources checked · 1 supported · 0 flagged
- Findingown data
The algorithm led to a 0.5% increase in total daily active users (DAUs) over a strong baseline.
“This lead to a 0.5 % increase in total daily active users (DAUs) and a 2 % increase in new user retention over a strong baseline.”
Quote in the paperSubstantial claim- Rests on an assumption: A substantial percentage of completed lessons are organic.
- Rests on an assumption: It is not possible to reliably distinguish organic lessons from notification-driven lessons.
- Findingown data
The algorithm led to a 2% increase in new user retention over a strong baseline.
“This lead to a 0.5 % increase in total daily active users (DAUs) and a 2 % increase in new user retention over a strong baseline.”
Quote in the paperSubstantial claim- Rests on an assumption: A substantial percentage of completed lessons are organic.
- Rests on an assumption: It is not possible to reliably distinguish organic lessons from notification-driven lessons.
- Findingnone given
Arm scores have high variance for arms with small sample sizes.
“The estimated arm scores from the previous section will have high variance for arms with small sample sizes (i.e., small |H a |).”
Quote in the paperSubstantial claim- Rests on an assumption: A substantial percentage of completed lessons are organic.
- Rests on an assumption: It is not possible to reliably distinguish organic lessons from notification-driven lessons.
- Findingown data
Arms with small sample sizes tend to have higher magnitudes of scores.
“We can easily see that arms with small sample sizes tend to have higher magnitudes.”
Quote in the paperSubstantial claim- Rests on an assumption: A substantial percentage of completed lessons are organic.
- Rests on an assumption: It is not possible to reliably distinguish organic lessons from notification-driven lessons.
- Findingown data
The threshold of 0.5% excludes an average of 3% of historical rounds.
“In our case, we chose to set the threshold to 0.5 % 7 , which only excludes an average of 3 % of historical rounds.”
Quote in the paperNot a substantial claim
- Findingown data
Offline evaluations showed that relative and absolute difference metrics performed similarly.
“However, our offline evaluations showed that both metrics performed similarly”
Quote in the paperSubstantial claim- Rests on an assumption: A substantial percentage of completed lessons are organic.
- Rests on an assumption: It is not possible to reliably distinguish organic lessons from notification-driven lessons.
- Findingown data
The template+UI-language arm variation performs substantially better than the one-template-per-arm variation (1.8% vs 1.2%).
“However, the variation that treats each template+UI-language pair as a distinct arm performs substantially better than the one-template-per-arm variation (1.8 % vs 1.2 %)”
Quote in the paperSubstantial claim - Findingown data
It takes roughly 1-5 weeks of data to narrow the 95% confidence interval to ±0.15 percentage points for most arms.
“In fact, it takes roughly 1-5 weeks of data to narrow the 95 % confidence interval to ±0.15 percentage points for most arms (see Figure 4).”
Quote in the paperNot a substantial claim - Findingown data
Increasing γ even to as much as 3.4 did not significantly impact business metrics in subsequent production A/B tests.
“But, subsequent production A/B tests showed that increasing γ even to as much as 3.4 did not significantly impact business metrics.”
Quote in the paperSubstantial claim - Findingown data
New user recurring retention increased by 2% in the online experiment.
“More striking, however, is that new user recurring retention increased by 2 %”
Quote in the paperSubstantial claim - Findingown data
The maximum of the grid search for γ is around 0.017.
“As shown in Figure 5, the maximum is around 0.017.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The recency penalty should mostly decay within the span of a few weeks to a month.
- Findingown data
Computing arm scores per UI language reveals substantial differences between UI languages for some templates.
“When we compute the arm scores for each template on a per-UI language basis, we do find substantial differences between UI languages for some templates.”
Quote in the paperSubstantial claim - Findingown data
In offline experiments, adding empirical Bayes estimation after the fact did not significantly change the final policy's total reward.
“a followup experiment showed that adding the empirical Bayes estimation after the fact did not significantly change the final policy's total reward in the offline experiments.”
Quote in the paperNot a substantial claim - Findingown data
The policy with the recency penalty beat the template+UI language policy by 0.2 percentage points (p < 0.1).
“We see that the policy with the recency penalty beat the one from the last section by 0.2 percentage points (p < 0.1).”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The recency penalty should mostly decay within the span of a few weeks to a month.
- Findingown data
Reusing the same template hurts reward by 0.5%.
“reusing the same template (as the bandit algorithm would often do if the recency penalty were not applied) hurts reward by 0.5 %.”
Quote in the paperSubstantial claim - Findingown data
The effect size of reusing the last template is around -0.0006.
“Since the effect size of reusing recent templates is so small in absolute terms (e.g., around -0.0006 if we reuse the last template, per Table 6)”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Table 6
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Rests on an assumption: A substantial percentage of completed lessons are organic.
- Rests on an assumption: It is not possible to reliably distinguish organic lessons from notification-driven lessons.
- Study design and conductreasoning
Weighted importance sampling corrects for bias in estimating μ+ and μ- that arises because the probability of an arm being used varies from round to round depending on the policy in use.
“Since the probability of an arm being used varies from round to round depending on the policy in use at that time, collecting all historical instances where an arm was used will not produce a representative sample: it will be biased towards rounds where the arm was historically more likely to be used, such as rounds where there were fewer alternative arms.”
- Also stated in: § 2.1 Sleeping Arms · plain
“We use weighted importance sampling to correct for this.”
Quote in the paperSubstantial claim- Rests on an assumption: A substantial percentage of completed lessons are organic.
- Rests on an assumption: It is not possible to reliably distinguish organic lessons from notification-driven lessons.
- Also stated in: § 2.1 Sleeping Arms · plain
- Study design and conductreasoning
Retaining only enough data in each arm history to ensure the arm score's 95% credible interval is smaller than threshold φ balances variance and responsiveness.
“To avoid this, we measure this variance using the arm score's 95 % credible interval and retain only enough data in each arm history to ensure that interval is smaller than a chosen threshold, φ, beyond which additional reductions in variance are of little value.”
Quote in the paperSubstantial claim - Interpretationreasoning
Conventional bandit algorithms that choose the template with the highest average historical reward may yield poor results when eligibility criteria confound rewards.
“Hence, choosing the template with the highest average historical reward, such as is done by conventional bandit algorithms cited earlier (e.g., ϵ-greedy [16], softmax [17], and Upper Confidence Bounds (UCB) [3]) may yield poor results.”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [16]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [17]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [3]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Study design and conductreasoning
Estimating μ+ and μ- from historical rounds that meet the same eligibility criteria controls for the effects of eligibility criteria on expected reward.
“Since both expectations are estimated from a sample of historical rounds that meet the same eligibility criteria, we effectively control for the effects of the eligibility criteria on the expected reward.”
Quote in the paperSubstantial claim- Rests on an assumption: A substantial percentage of completed lessons are organic.
- Rests on an assumption: It is not possible to reliably distinguish organic lessons from notification-driven lessons.
- Interpretationnone given
The Recovering Difference Softmax Algorithm provides a practical solution that solves both the recovering bandits and sleeping bandits sub-problems simultaneously.
“In contrast, the Recovering Difference Softmax Algorithm provides a practical solution that solves both of these sub-problems simultaneously.”
Quote in the paperNot a substantial claim - Premisereasoning
Many notification templates have eligibility criteria that are not applicable to all users, so not all arms are available every round.
“Many templates are based on conditions that are not applicable to all users.”
- Also stated in: § 2.1 Sleeping Arms · plain
“As mentioned previously, not all arms are available every round.”
Quote in the paperSubstantial claim - Also stated in: § 2.1 Sleeping Arms · plain
- Study design and conductreasoning
Excluding rounds where the behavior policy probability b_t(a|t) is very small prevents large errors in estimates of μ+ and μ-.
“To prevent this, we exclude from H a any rounds where b t (a | t) < θ or 1b t (a | t) < θ , where θ is a hyperparameter set to a very small value.”
Quote in the paperSubstantial claim - Interpretationreasoning
The 5% holdout set provides valuable training data, especially for arms that perform poorly and are rarely used.
“It provides valuable training data, especially for arms that perform poorly and are thus rarely used by the bandit algorithm.”
Quote in the paperSubstantial claim - Interpretationown data
The optimized reminders significantly help in retaining new users.
“showing that the optimized reminders significantly help in retaining new users (a critical component to an app's growth).”
Quote in the paperSubstantial claim - Interpretationreasoning
The design of the bandit system allows new templates to be introduced without any special handling.
“The design of the bandit system allows new templates to be introduced without any special handling.”
Quote in the paperNot a substantial claim - Interpretationreasoning
The delay in feedback loop is not a significant drawback for this application because arm scores do not change much in a single day.
“This is not a significant drawback for our application, however, because the arm scores do not change much in a single day.”
Quote in the paperSubstantial claim - Interpretationown data
The algorithm can be practically scaled to millions of users to boost user engagement.
“We furthermore showed via our online evaluation that the algorithm can be practically scaled to millions of users to boost user engagement.”
Quote in the paperSubstantial claim - Interpretationreasoning
The 5% holdout set provides a way to monitor the bandit's ongoing performance and detect if it fails to beat the random baseline.
“It provides us a way to monitor the bandit's ongoing performance and detect if the bandit ever fails to beat the random baseline”
Quote in the paperSubstantial claim - Interpretationnone given
The Recovering Difference Softmax Algorithm solves a pair of key problems common in recurring notification optimization that were not adequately addressed in the literature previously.
“This algorithm solves a pair of key problems that are common in this setting that were not adequately addressed in the literature previously.”
Quote in the paperSubstantial claim - Interpretationreasoning
The use of Bayesian priors to regularize relative reward estimates for small sample sizes is very important.
“However, this does mean that the use of Bayesian priors to regularize relative reward estimates for small sample sizes, as described in Section 2.2, is very important.”
Quote in the paperSubstantial claim - Interpretationown data
It is worthwhile to take different translations of each template into account when scoring arms.
“confirming that it is worthwhile to take the different translations of each template into account when scoring arms.”
Quote in the paperSubstantial claim - Interpretationreasoning
It would likely take an enormous amount of data to estimate a more complex recovery function with Gaussian process kernels.
“it would likely take an enormous amount of data to estimate a more complex recovery function with Gaussian process kernels”
Quote in the paperSubstantial claim - Interpretationown data
The bandit continues to maintain business gains on par with what was predicted in the offline experiments.
“further demonstrating that the bandit continues to maintain business gains on par with what was predicted in our offline experiments.”
Quote in the paperSubstantial claim
- Classified as probably not a substantial claim. 6
- May combine two claims. 11
- The quote only partly states this claim. 1
Classified as probably not a substantial claim.
PremisecitationThe multi-armed bandit problem provides a well-studied theoretical framework for optimizing the selection of notification templates.
“The multi-armed bandit problem provides a well-studied theoretical framework for optimizing the selection of templates in this context.”
- Also stated in: § 1.1 Related Work · plain
“The multi-armed bandit problem [13] is an optimization problem in which an agent repeatedly chooses from among a fixed set of arms over some number of rounds.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites [13] source
- Also stated in: § 1.1 Related Work · plain
May combine two claims.
Findingown dataOnly 50% of arms with smaller effective sample sizes come within ±3.4%, and 38% have scores outside ±5%.
“However, only 50 % of arms with smaller effective sample sizes come within this range, and 38 % dubiously have scores outside ±5 %, which could cause highly degenerate behavior in the bandit, if used as-is.”
Quote in the paperSubstantial claim- Rests on an assumption: A substantial percentage of completed lessons are organic.
- Rests on an assumption: It is not possible to reliably distinguish organic lessons from notification-driven lessons.
Probably not a substantial claim: may not pass S4 (based).
Findingnone givenDeployment of the Recovering Difference Softmax Algorithm to select daily reminders for millions of users at Duolingo resulted in substantial lifts in user engagement metrics.
“It has been deployed to select daily reminders for millions of users at Duolingo, resulting in substantial lifts in user engagement metrics.”
Quote in the paperNot a substantial claim- Rests on an assumption: A substantial percentage of completed lessons are organic.
- Rests on an assumption: It is not possible to reliably distinguish organic lessons from notification-driven lessons.
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenThe 5% random holdout set has a very small impact on the gains of the algorithm.
“This small holdout set has a very small impact on the gains of the algorithm, while providing two advantages:”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataFor some templates, arm scores differ by 1 percentage point or more across UI languages, which is substantial given most scores range only ±3.4%.
“In some cases, the scores are different by 1 percentage point or more, which is substantial since the range of most of the scores is only ±3.4 %.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataIn the online experiment, daily active users increased by 0.5% and lessons completed increased by 0.4%.
“We do in fact see a gain in daily active users and lessons completed of 0.5 % and 0.4 %, respectively.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataRelative difference scoring and recency penalties each contribute to maximizing the reward.
“Via offline evaluations, we demonstrated how the relative difference scoring and recency penalties -the latter of which was inspired by cognitively-motivated theories about novelty effects -each contribute to maximizing the reward.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The recency penalty should mostly decay within the span of a few weeks to a month.
May combine two claims.
Findingown dataBoth variations of the bandit algorithm (one-template-per-arm and template+UI-language) beat the random baseline, gaining more than 1% lift in reward.
“As expected, both variations of the bandit algorithm beat the random baseline, gaining more than 1 % lift in reward.”
Quote in the paperSubstantial claimThe quote only partly states this claim.
Findingown dataFive months after launch, the 95% of rounds using the bandit's optimized policy had 2.5% higher reward than the 5% random holdout set.
“The 95 % of rounds that used the bandit's optimized policy had 2.5 % higher reward than the holdout set”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Premiseown dataFresh templates that have not been recently seen tend to have higher impact on user behavior.
“Prior A/B tests 5 at Duolingo that introduced new templates tended to boost notification conversion rates, even when the those new templates did not outperform existing templates in the long run.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningGaussian process kernels require many more than two parameters and would need much more data to fit accurately than the parametric approach.
“Gaussian process kernels require many more than two parameters, and hence would need much more data to fit accurately.”
Quote in the paperSubstantial claimMay combine two claims.
Study design and conductreasoningThe offline experiment only estimates the short-term cost of exploration; its results do not imply argmax would yield the best reward in production in the long-run.
“We should emphasize that, since the training dataset is fixed in this offline setup, this experiment only estimates the short-term cost of exploration.Hence, these results do not imply that argmax would yield the best the reward in production in the long-run.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
InterpretationreasoningThis downward drift applies to all arms similarly, so it does not significantly affect the overall performance of the algorithm.
“This downward drift applies to all arms similarly, so it does not significantly affect the overall performance of the algorithm.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationreasoningμ- tends to increase over time as the algorithm learns more optimal policies, causing template scores to drift downwards over time.
“The drawback of this is that µ - a tends to increase over time as the algorithm learns more optimal policies, which causes the template scores to drift downwards over time.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataBased on Table 7, τ=0.0025 was estimated to allow a reasonable amount of exploration while retaining more than 90% of the potential gains.
“Based on this, we selected a value of 0.0025, which we estimated would allow a reasonable amount of exploration while retaining more than 90 % of the potential gains.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
InterpretationreasoningThe batch processing greatly simplifies the arm score update process and makes it more scalable.
“The batch processing greatly simplifies the arm score update process and makes it more scalable.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
A substantial percentage of completed lessons are organic.
“However, a substantial percentage of our completed lessons are organic: many users will complete a lesson whether we send a notification or not.”
The paper relies on this without support.
It is not possible to reliably distinguish organic lessons from notification-driven lessons.
“It is not possible to reliably distinguish between organic lessons from notificationdriven lessons, which adds a degree of noise when evaluating the performance of different arms.”
The paper relies on this without support.
Arms with new eligibility rules are introduced frequently.
“Furthermore, arms with new eligibility rules are introduced frequently.”
The paper relies on this without support.
The function being estimated is not complex.
“there isn't much reason to suspect the function being estimated is complex, anyway.”
The authors state this as an assumption.
The recency penalty should mostly decay within the span of a few weeks to a month.
“Based upon previous A/B tests at Duolingo, we guessed that the recency penalty should mostly decay within the span of a few weeks to a month, and so we chose a half-life of 15 days.”
The paper relies on this without support.
There is a significant amount of organic activity not originating from push notifications that would dilute the gains.
“This is because there is a significant amount of organic activity not originating from push notifications that would dilute the gains.”
The paper relies on this without support.
The pools of templates and other aspects of the notification system had already been optimized through years of A/B tests, providing a strong baseline.
“Since the pools of templates and other aspects of the notification system had already been optimized through years of A/B tests, this provided a strong baseline to compare against.”
The paper relies on this without support.
Most bandit algorithms assume instantaneous feedback.
“Most bandit algorithms, in fact, assume instantaneous feedback.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Arm A is described as above average despite a negative lift
MinorTable 2 gives arm A a relative difference of −0.37%, indicating a lower reward when used than its average reward when eligible. The text says it performs better than average.
“arm A barely performs better than average.”
“A 0.2672 0.2682 -0.37 %”
Posterior formula conflicts with the stated beta prior
MinorThe stated beta prior has total prior weight 1/σ, but the posterior formula uses σ as that weight. The posterior does not follow from the stated prior.
“μ+ a = μ+ a n + a + µ a σ n + a + σ”
“µ + a , µ - a ∼ Beta(µ a /σ, (1 -µ a )/σ )”
Cites
Reinforcement Learning: An Introduction
Johnson, J. D., Li, J., & Chen, Z. (2000) · Neurocomputing · doi:10.1016/s0925-2312(00)00324-6 · Cited by 8,821 · see citing works
Finite-time Analysis of the Multiarmed Bandit Problem
Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002) · Machine Learning · doi:10.1023/a:1013689704352 · Cited by 6,033 · see citing works
A generalization of sampling without replacement from a finite universe
Horvitz, D. G., & Thompson, D. J. (1952) · Journal of the American Statistical Association · doi:10.1080/01621459.1952.10483446 · Cited by 4,293 · cited 1× · see citing works
“We used off-policy evaluation via weighted importance sampling [8, 17] to estimate the average reward of the policy being evaluated.”
Method(2002) · doi:10.1137/s0097539701398375 · Cited by 2,275 · see citing works
Some aspects of the sequential design of experiments
Robbins, H. (1952) · Bulletin of the American Mathematical Society · doi:10.1090/s0002-9904-1952-09620-8 · Cited by 2,184 · cited 1× · see citing works
“The multi-armed bandit problem [13] is an optimization problem in which an agent repeatedly chooses from among a fixed set of arms over some number of rounds.”
BackgroundMemory; a contribution to experimental psychology,
Ebbinghaus, H., Bussenius, C. E., & Ruger, H. A. (n.d.) · University of Michigan Library Repository · Cited by 1,281 · see citing works
Using confidence bounds for exploitation-exploration trade-offs
Auer, P. L. (2003) · Cited by 1,219 · see citing works
Introduction to Multi-Armed Bandits
Slivkins, A. (2019) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000068 · Cited by 820 · see citing works
(2014) · SIAM Journal on Computing · doi:10.1137/sicomp · Cited by 618 · see citing works
A Trainable Spaced Repetition Model for Language Learning
Settles, B., & Meeder, B. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1174 · Cited by 232 · cited 1× · see citing works
“Exponential decay functions have been shown to model human memory well [7, 14].”
BackgroundSelf-citationRegret bounds for sleeping experts and bandits
Kleinberg, R., Niculescu-Mizil, A., & Sharma, Y. (2010) · Machine Learning · doi:10.1007/s10994-010-5178-7 · Cited by 204 · see citing works
Stochastic Multi-Armed-Bandit Problem with Non-stationary Rewards
Besbes, O., Gur, Y., & Zeevi, A. J. (2014) · doi:10.13140/rg.2.1.2862.3844 · Cited by 193 · see citing works
Rotting BanditsPreprint
Levine, N., Crammer, K., & Mannor, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.07274 · Cited by 29 · see citing works
Discrepancy-Based Algorithms for Non-Stationary Rested BanditsPreprint
Cortes, C., DeSalvo, G., Kuznetsov, V., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.10657 · Cited by 6 · cited 2× · see citing works
“A number of other papers [6, 11, 15] explore a related problem, known as the “rested bandits problem,” where an arm’s expected reward depends on the number of times the arm has been selected previously (rather than the how recently it was last selected).”
BackgroundRecovering BanditsPreprint
Pike-Burke, C., & Grünewälder, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.14354 · Cited by 2 · cited 3× · see citing works
“The two variations that most directly relate to the recurring notifications setting studied in this paper are the “recovering bandits problem” [12] and the “sleeping bandits problem” [10].”
BackgroundMethodRotting bandits are not harder than stochastic onesPreprint
Seznec, J., Locatelli, A., Carpentier, A., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.11043 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Wei Chu , Lihong Li , Lev Reyzin , and Robert Schapire . 2011 . Contextual bandits with linear payoff functions . In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics. 208--214 . Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. 2011. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics. 208--214.
- Leslie Kish. 1965. Survey Sampling.] ohn Wiley. New York (1965). Leslie Kish. 1965. Survey Sampling.] ohn Wiley. New York (1965).
Cited by
Sorted by citations
Nonstationary Experimental Design Under Structured Trends
Simchi‐Levi, D., Wang, C., & Zheng, Z. (2026) · Management Science · doi:10.1287/mnsc.2023.03329 · Cited by 3 · see citing works
Smart Notifications – An ML-based Framework to Boost User Engagement
Pinto, V. M. M., Oliveira, L. M. M., Sousa, Y. P. D., et al. (2023) · Brazilian Symposium on Multimedia and the Web · doi:10.1145/3617023.3617053 · Cited by 3 · see citing works
“A very elegant approach was presented in [25], where a recovering”
BackgroundAssessing Notification Timing Strategies for Improved Micro-Learning Engagement
Tran, H. T. N., She, W. J., Kajimura, S., & Shibuya, Y. (2025) · IEEE Access · doi:10.1109/access.2025.3559521
“Especially, Duolingo is known for using ‘‘emotional blackmail’’ notifications have proven effective in getting users to go back and start learning again [12].”
InfluentialActive Recommendation for Email Outreach Dynamics
Žid, Č., Alves, R. A. D. S., & Kordík, P. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3760832
“Frequency control optimization has been studied primarily by large companies – e.g. Facebook [15], LinkedIn [40], Twitter (X) [20], and Duolingo [39], via extensive A/B tests.”
Sustained Impact of Agentic Personalisation in Marketing
Jeunen, O., Hanna, E., & Wheeler, S. (2026) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3774935.3807908 · Cited by 1 · see citing works
“The efficacy of this shift and widespread adoption among leading platforms has been well-documented: Duolingo [22], Kuaishou [3], LinkedIn [4, 7, 17, 23], Meta[13],Netflix[8],Pinterest[25],Tencent[24]andTwitter/X[15] haveallpursuedthisasanactiveresearchdirection.”
- Authors
- Yancey, K. P., & Settles, B.
- Affiliations
- Duolingo (United States)
- ORCID
- Yancey, Kevin P.
- Venue
- Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
- Publisher
- ACM
- Published
- 2020-08-20
- Pages
- 3008-3016
- OpenAlex
- W3081419666
- Semantic Scholar
- 2c71f6fd971ae7c2d1b6134a35a8c686d835ccac
- Cited by
- 5
- References
- 11
- Citations by year
- 2026: 2 · 2025: 2 · 2023: 1
- FWCI
- 0.35
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD) (publishedVersion, cc-by-nd)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- multi-armed bandits, push notifications, email notifications, novelty effect, user engagement, user retention, language learning apps, Duolingo
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-12.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-12.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
