SWISH: semantic analysis of window titles and switching history
Abstract
Information workers are often involved in multiple tasks and activities that they must perform in parallel or in rapid succession. In consequence, task management itself becomes yet another task that information workers need to perform in order to get the rest of their work done. Recognition of this problem has led to research on task management systems, which can help by allowing fast task switching, fast task resumption, and automatic task identification. In this paper we focus on the latter: we tackle the problem of automatically detecting the tasks that the user is involved in, by identifying which of the windows on the user's desktop are related to each other. The underlying assumption is that windows that belong to the same task share some common properties with one another that we can detect from data. We will refer to this problem as the task assignment problem.To address this problem, we have built a prototype named Swish that: (1) constantly monitors users' desktop activities using a stream of windows events; (2) logs and processes this raw event stream, and (3) implements two criteria of window "relatedness", namely the semantic similarity of their titles, and the temporal closeness in their access patterns.In addition to describing the Swish prototype in detail, we validate it with 4 hours of user data, obtaining task classification accuracies of about 70%. We also discuss our plans on including Swish in a number of intelligent user interfaces and future lines of research.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Oliver et al. examined a temporal feature of window switches within a 5 minute time window in addition to semantic features and using an unsupervised approach; an evaluation based on 4 hours of a single participant showed a precision of 0.49 and recall of 0.72.
“Oliver et al. [27] examined a temporal feature of window switches within a 5 minute time window in addition to semantic features and using an unsupervised approach. An evaluation based on 4 hours of a single participant, showed a precision of 0.49 and recall of 0.72.”
From Detecting Developers' Task Switches and Types (2022) · cites this paper as [27]
Cites
Maximum Likelihood from Incomplete Data Via the <i>EM</i> Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444 · see citing works
Indexing by latent semantic analysis
Deerwester, S. C., Dumais, S., Furnas, G. W., Landauer, T. K., & Harshman, R. A. (1990) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199009)41:6 · Cited by 12,758 · see citing works
An algorithm for suffix stripping
Porter, M. F. (1980) · Program · doi:10.1108/eb046814 · Cited by 8,096 · see citing works
Bayesian Networks and Decision Graphs
Jensen, F. V., & Nielsen, T. D. (2007) · Information science and statistics · doi:10.1007/978-0-387-68282-2 · Cited by 4,157 · see citing works
Probabilistic Latent Semantic Indexing
Hofmann, T. (2017) · ACM SIGIR Forum · doi:10.1145/3130348.3130370 · Cited by 3,988 · see citing works
Probabilistic latent semantic indexing
Hofmann, T. (1999) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/312624.312649 · Cited by 3,975 · see citing works
Algorithm 457: finding all cliques of an undirected graph
Bron, C., & Kerbosch, J. (1973) · Communications of the ACM · doi:10.1145/362342.362367 · Cited by 2,492 · see citing works
An information-theoretic perspective of tf–idf measures
Aizawa, A. (2003) · Information Processing & Management · doi:10.1016/s0306-4573(02)00021-3 · Cited by 1,419 · see citing works
Bederson, B. B., & Hollan, J. D. (1994) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/192426.192435 · Cited by 638 · see citing works
Henderson, D. A., & Card, S. (1986) · ACM Transactions on Graphics · doi:10.1145/24054.24056 · Cited by 522 · see citing works
Bellotti, V. M. E., Ducheneaut, N., Howard, M., & Smith, I. (2003) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642672 · Cited by 370 · see citing works
An Experimental Comparison of Model-Based Clustering Methods
Meilă, M., & Heckerman, D. E. (2001) · Machine Learning · doi:10.1023/a:1007648401407 · Cited by 284 · see citing works
Rekimoto, J. (1999) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/320719.322582 · Cited by 243 · see citing works
Robertson, G. G., van Dantzich, M., ROBBINS, D. W., Czerwinski, M. P., Hinckley, K., Risden, K., Thiel, D., & Gorokhovsky, V. (2000) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/332040.332482 · Cited by 204 · see citing works
Baudisch, P., Good, N. S., & Stewart, P. (2001) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/502348.502354 · Cited by 196 · see citing works
Support for multitasking and background awareness using interactive peripheral displays
MacIntyre, B., Mynatt, E. D., Voida, S., Hansen, K. M., Tullio, J., & Corso, G. M. (2001) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/502348.502355 · Cited by 174 · see citing works
Hutchings, D. R., Smith, G., Meyers, B. R., et al. (2004) · Working Conference on Advanced Visual Interfaces (AVI) · doi:10.1145/989863.989867 · Cited by 136 · see citing works
Self-citationEvaluation and analysis of users' activity organization
Bannon, L. J., Cypher, A., Greenspan, S. L., & Monty, M. L. (1983) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/800045.801580 · Cited by 115 · see citing works
Kandogan, E., & Shneiderman, B. (1997) · doi:10.1145/258549.258720 · Cited by 115 · see citing works
A comparison of tiled and overlapping windows
Bly, S. A., & Rosenberg, J. K. (1986) · ACM SIGCHI Bulletin · doi:10.1145/22339.22356 · Cited by 106 · see citing works
A comparison of tiled and overlapping windows
Bly, S. A., & Rosenberg, J. K. (1986) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/22627.22356 · Cited by 104 · see citing works
Smith, G., Baudisch, P., Robertson, G. G., Czerwinski, M. P., Meyers, B. R., ROBBINS, D. W., & Andrews, D. E. (2003) · Cited by 98 · see citing works
Self-citationScalable Fabric: flexible task management
Robertson, G., Horvitz, E., Czerwinski, M., Baudisch, P., Hutchings, D. R., Meyers, B., Robbins, D., & Smith, G. (2004) · Proceedings of the working conference on Advanced visual interfaces · doi:10.1145/989863.989874 · Cited by 89 · see citing works
Self-citationDynamic space management for user interfaces
Bell, B. A., & Feiner, S. (2000) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/354401.354790 · Cited by 85 · see citing works
Aizawa, A. (2000) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/345508.345556 · Cited by 63 · see citing works
Methodology of Window Management
Hopgood, F. R. A., Duce, D. A., Fielding, E. V. C., et al. (1986) · Springer eBooks · doi:10.1007/978-3-642-70919-7 · Cited by 43 · see citing works
A lightweight framework for cross-application user monitoring
Fenstermacher, K. D., & Ginsburg, M. (2002) · Computer · doi:10.1109/2.989930 · Cited by 33 · see citing works
Design of a Collaborative Multi User Desktop System for Augmented Reality
Wurnig, H. (1998) · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Detecting Developers' Task Switches and Types
Meyer, A. N., Satterfield, C., Zuger, M., Kevic, K., Murphy, G. C., Zimmermann, T., & Fritz, T. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.2984086 · Cited by 14 · see citing works
“[27] examined a temporal feature of window switches within a 5 minute”
BackgroundMethodInfluentialA Study on Automatic Classification of Users’ Desktop Interactions
Mirza, H. T., Chen, L., Hussain, I., Majid, A., & Chen, G. (2015) · Cybernetics and Systems · doi:10.1080/01969722.2015.1012372 · Cited by 8 · see citing works
Switch detector: an activity spotting system for desktop
Mirza, H. T., Chen, L., Chen, G., Hussain, I., & He, X. (2011) · Proceedings of the 20th ACM international conference on Information and knowledge management · doi:10.1145/2063576.2063947 · Cited by 5 · see citing works
BUILDING USER TASK SPACE BY MINING TEMPORALLY PROXIMATE DESKTOP ACTIONS
Mirza, H. T., Chen, L., Majid, A., & Chen, G. (2011) · Cybernetics and Systems · doi:10.1080/01969722.2011.634675 · Cited by 2 · see citing works
Activity Frames: Deterministic Screen-Activity Compilation for Agent Memory and Replay
Iyamu, N. (2026) · arXiv
“Deriving task structure from interaction logs is a two-decade ambition: TaskTracer tied window, file, and clip-board events to declared tasks [21], and SWISH clustered windows into tasks from titles and switching [22].”
- Authors
- Oliver, N., Smith, G., Thakkar, C., & Surendran, A. C.
- Affiliations
- Microsoft (United States) (US)
- ORCID
- Oliver, Nuria
- Venue
- Proceedings of the 11th international conference on Intelligent user interfaces
- Publisher
- ACM
- Published
- 2006-01-29
- Pages
- 194-201
- OpenAlex
- W2146133727
- Cited by
- 89
- References
- 28
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 3 · 2023: 2 · 2022: 1 · 2021: 3 · 2020: 5 · 2019: 2 · 2016: 3 · 2015: 6 · 2014: 2 · 2013: 3 · 2012: 6
- FWCI
- 12.39
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Personal Information Management and User Behavior
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Keywords
- task management, task classification, intelligent user interfaces, information professionals, task switching
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
