What does mHealth app engagement mean for real-world mental health outcomes?

Managing mental health and wellness takes time and effort. Tracking symptoms, connecting with support systems, and participating in therapies are all part of achieving good quality of life. Currently, there are literally thousands of mobile health (mHealth) apps available that claim to make this job easier. The creators of these apps measure their success by how engaging they are. They define engagement as how often people use the app (frequency), how long they use it for (duration), and how many people have downloaded it or rated it.  

Many people think that these measures are problematic. First, they don’t capture people’s experience of using the app, i.e., does the app make them feel more in control of their health, or more connected to healthcare professionals when they need support?  Second, they also don’t say anything about how using an app connects to real-world health outcomes, i.e., does using an app frequently mean you are also taking prescribed medication, getting enough rest, and managing your stress?

The problem with quantifying user engagement with mHealth apps

There are a handful of metrics commonly used to track people’s interactions with mHealth apps: time spent using an app, and number of downloads, logins, visits and specific interactions (such as clicking on links or watching videos) (Fleming et al., 2018).  The assumption is that the higher the numbers are, the better the level of engagement with the app. But in truth, interpreting these metrics is tricky. I might stop using an app because I feel frustrated with it, but I might also stop because I’m well enough that I don’t need to rely on it at the moment. How do we know which of these experiences a user is having? 

The answer is, we don’t.  

 Here’s another example: if I spend 2 minutes on an app, am I significantly less engaged than the person who spends 5 minutes? That’s hard to say. And is this difference actually meaningful?

The result of all this quantification is that we still can’t reliably compare apps, studies, or user groups (Ng et al., 2019), and we don’t have good benchmarks for meaningful “in-app” engagement (Torous et al., 2018).

Let’s think differently about user engagement

What if, instead of measuring only what people do with an app, we consider how people think and feel about these tools? Why do they use them? What needs do they fulfill? And how do they work with other aspects of mental health management and care?  

This would require us to think differently about how we measure in-app engagement, because we can’t get this information counting clicks and downloads.  Measuring meaningful engagement involves asking people about their mental health goals, what they need an app to do, and how apps fit into their overall mental health journey. It’s also about recognizing that more use isn’t necessarily better, and apps might be utilized at specific points in time, in relation to life’s ups and downs. High app use may actually be a sign of poor mental health.

I might stop using an app because I feel frustrated with it, but I might also stop because I’m well enough that I don’t need to rely on it at the moment. How do we know which of these experiences a user is having?

Studies have shown apps can affect people negatively if they have usability issues, are overwhelmed by information, or are reminded constantly that they are living with a mental illness (Allan et al., 2019; Nicholas et al., 2017, Morton et al., 2019).  Different conditions come with different symptoms and challenges (Ng et al., 2019). Factors such as age, gender identity and cultural background intersect to create diverse lived experiences (Nicholas et al., 2017). 

Building better apps for more meaningful engagement

In order to build better measures, we need to build apps that fit people’s goals, health conditions and lived experiences, and that align with effective clinical practice.  When we bring these elements into an app, we have a better chance of identifying what indicates meaningful engagement, and how to connect this to real-world outcomes. 

Let’s start by looking at app features. Apps can contain many different kinds of content, like educational articles, quizzes, symptom trackers, and social connection tools. Rather than looking at the time spent on these different kinds of content, let’s think about what purpose each serves. Do they provide education/information? Symptom tracking? Social support? 

Some features, such as a quiz, might be accessed once. Others, like sleep or exercise logs, might be used or multiple times. User behaviour might also differ depending on a person’s stage in their mental health journey. Some information might be more relevant for people newly diagnosed versus those who have been managing their illness for some time. Understanding why and how people interact with content at different points in their recovery or illness stage could help us tailor content to what is most needed, whether it’s routine maintenance, affirmation, social support, or reduce information overload (Morton et al., 2019).

We might also wish to look more carefully at patterns of app use, typically captured by machine learning algorithms. These patterns are currently based on duration (short- vs. long-term) and frequency of use (low vs. high). This tells us that different people interact with the app in different ways, but we already know this to be true. We need to explore these patterns with users and in relation to real-world outcomes.  

Toward a more comprehensive view of in-app engagement 

It is tempting to want a magic formula for measuring mHealth engagement. This is the appeal of using duration and frequency of use metrics that are relatively easy and unobtrusive to collect. But if we really want to develop meaningful indicators of engagement, we need to learn how to COPE! This means that we: 

• Use different measures from clinical practice, e.g. symptom severity assessment scales and we talk to users to CORROBORATE and make sense of clicks, downloads and other metrics
• Focus on OUTCOMES, rather than outputs. If mHealth apps are meant to improve health and well-being, then how they are used becomes more important than how often they are used.
• Recognize that mental health is a PROCESS with natural ebbs and flows. We use this to appreciate how different people interact with apps over time rather than labelling them as low or high engagers based on algorithmic patterns.
• Bring in the EXPERTS, meaning people with lived experience and clinicians, to identify how an app should look and function, and what goals it should support.

The COPE approach is not an easy road. It involves collaboration and relationship building between people with lived experience, healthcare providers, and app designers. If we want to do it right, we need greater transparency about how design decisions are made, what is being measured and why, and how the resulting app fits into the broader mental health landscape.

About Dr. Heather O’Brien

Heather O’Brien is an Associate Professor at the iSchool, UBC, where she teaches and researches in the area of human information interaction. Dr. O’Brien is best known for her work in the area of user engagement, where she has contributed numerous publications, including two books, Why Engagement Matters: Cross-Disciplinary Perspectives and Innovations on User Engagement with Digital Media (edited with Paul Cairns, 2016) and Measuring User Engagement (authored with Mounia Lalmas and Elad Yom-Tov, 2014), as well as the User Engagement Scale (UES), an experiential questionnaire that is used internationally to understand digitally mediated user experiences.

Her current research is supported by a Social Sciences and Humanities Council of Canada (SSHRC) Insight Grant, SSHRC Partnership Development Grant. and UBC Hampton Research Fund Award.
You can find Dr. O’Brien on Google Scholar and on Twitter at @HLOBrienUBC. You can learn more about her and her work at the UBC Designing for People website and her UBC profile page.

References

1. Fleming, T, Bavin, L, Lucassen, M, Stasiak, K, Hopkins, S, Merry, S. Beyond the trial: systematic review of real-world uptake and engagement with digital self-help interventions for depression, low mood, or anxiety. J Med Internet Res. 2018; 20(6): e199. 
2. Ng, M. M, Firth, J, Minen, M, Torous, J. User engagement in mental health apps: a review of measurement, reporting, and validity. Psychiatr Serv. 2019. 70(7): 538-544. DOI: https://doi.org/10.1176/appi.ps.201800519
3. Fletcher, K., Foley, F., Murray, G. Web-based self-management programs for bipolar disorder: insights from the online, recovery-oriented bipolar individualised tool project. J Med Internet Res. 2018; 20(10): e11160. 
4. Torous, J., Lipschitz, J., Ng, M., Firth, J. Dropout rates in clinical trials of smartphone apps for depressive symptoms: a systematic review and meta-analysis. J Affect Disord. 2019; 263: 413-419. DOI: https://doi.org/10.1016/j.jad.2019.11.167
5. Nicholas, J, Fogarty, AS, Boydell, K, Christensen, H. The reviews are in: a qualitative content analysis of consumer perspectives on apps for bipolar disorder. J Med Internet Res.  2017; 19(4): e105. DOI: 10.2196/jmir.7273
6. Allan S, Mcleod H, Bradstreet S, Beedie, S, Moir, B, Gleeson, J, Farhall, J, Morton, E, Gumley, A. Understanding implementation of a digital self-monitoring intervention for relapse prevention in psychosis: protocol for a mixed method process evaluation. JMIR Res Protoc. 2019; 8(12): e15634. DOI:10.2196/15634 
7. Morton, E, Hole, R, Murray, G, Buzwell, S, Michalak, E. Experiences of a web-based quality of life self-monitoring tool for individuals with bipolar disorder: a qualitative exploration. JMIR Ment Health. 2019; 6(12): e16121. 

Previous Posts about Bipolar Bridges

BB Blog 3: Our App Survey Was a Massive Success!
Participants from all around the world shared their app ideas on our survey.

BB Blog 2: Engaging People in the “Anti-Psychiatry” Movement?
We have tried over the years to position ourselves carefully as a network that focuses on a particular condition: bipolar disorder.

BB Blog 1: Our First Whiteboarding Session!
Our first group planning session — a “whiteboarding session” — for Bipolar Bridges was hosted on Friday, November 15th.

Bipolar Bridges Launched on World Mental Health Day
It’s the perfect day for CREST.BD to officially launch our flagship project: ‘Bipolar Bridges: A Digital Health Innovation Targeting Quality of Life in Bipolar Disorder.’

CREST.BD receives CIHR funding to develop ‘Bipolar Bridges’ app
The project will build on CREST.BD’s digital health work to develop a new mobile app – Bipolar Bridges – which will take advantage of advances in online technologies…