Context
Context
Fact-checking the Media Portrayals of Voice User Interfaces for Older Adults
Fact-checking the Media Portrayals of Voice User Interfaces for Older Adults
Mass media advertising of voice user Interfaces (VUIs) as beneficial for older adults (OAs) may influence adoption by this population (OAs 65+), but disparities between expectations and experiences can discourage use and widen digital divides. During THAI-RS 2021, we performed an analysis of 11 older adults’ perceptions and expectations of VUIs based on pre-and post-deployment interviews during a two-week home trial with Amazon Echo.
My Role
Researcher (planning, interviews, analysis, presentation)
Outcome
Academic Publication
Project
Toronto Human-AI Interaction Summer Research School
Methods
User Interviews, usability testing, and diary study
Team
3 UX Researchers (including myself); 2 research advisors
Results
Results
Key Findings
Key Findings
After receiving reviewed feedback, a revised comparative analysis was published in the journal of Human-Computer Interaction, titled “Avoiding Mixed Messages: Research-based fact-checking the Media Portrayals of Voice User Interfaces for Older Adults.”
After receiving reviewed feedback, a revised comparative analysis was published in the journal of Human-Computer Interaction, titled “Avoiding Mixed Messages: Research-based fact-checking the Media Portrayals of Voice User Interfaces for Older Adults.”
Read Report
The following key findings are based on our initial attitudinal study on older adult’s perceptions of smart speakers in comparison to the media’s portrayal of their projected experiences. From our interview data, we learned that older adults are open to using VUIs, but see them as novelties that are not sophisticated enough for serious use. More specifically, we learned:
The following key findings are based on our initial attitudinal study on older adult’s perceptions of smart speakers in comparison to the media’s portrayal of their projected experiences. From our interview data, we learned that older adults are open to using VUIs, but see them as novelties that are not sophisticated enough for serious use. More specifically, we learned:
1.
VUIs must be useful enough to replace established methods.
VUIs must be useful enough to replace established methods.
2.
Personal connection has a stronger influence than the media for VUI adoption.
Personal connection has a stronger influence than the media for VUI adoption.
3.
VUIs’ limitations erode trust for serious work and essential services.
VUIs’ limitations erode trust for serious work and essential services.
4.
Full use of VUIs (i.e., integration with devices) relies on digital literacy and is costly.
Full use of VUIs (i.e., integration with devices) relies on digital literacy and is costly.
5.
VUIs must be useful enough to replace established methods.
VUIs must be useful enough to replace established methods.
Getting Started
Getting Started
Research Objectives
Research Objectives
Investigate OAs’ attitudes and influences toward VUI adoption.
Investigate OAs’ attitudes and influences toward VUI adoption.
Determine if mass media content reflects OAs’ actual VUI use.
Determine if mass media content reflects OAs’ actual VUI use.
We Asked
What are OAs’ perceptions of VUIs compared to mass media messaging, before and after having hands-on experience?
We Asked
What are OAs’ perceptions of VUIs compared to mass media messaging, before and after having hands-on experience?
Process
Process
Our Approach
Our Approach
We conducted a literature review on the design of and adoption of VUIs by older adults. We agreed to expand on Sin et. al’s systematic review of the media’s influence on older adults’ adoption or lack of adoption of VUIs.
We conducted a literature review on the design of and adoption of VUIs by older adults. We agreed to expand on Sin et. al’s systematic review of the media’s influence on older adults’ adoption or lack of adoption of VUIs.
The following 4 high-level themes across media articles on OAs’ and VUIs informed our research questions and interview guide:
The following 4 high-level themes across media articles on OAs’ and VUIs informed our research questions and interview guide:
Research
Research
Study Design
Study Design
Our study procedure is separated into 4 parts (detailed below). We recruited participants through the Taglab mailing list, and then we expanded our search to Honebeehub, a UofT startup that connects researchers with study participants.
Our study procedure is separated into 4 parts (detailed below). We recruited participants through the Taglab mailing list, and then we expanded our search to Honebeehub, a UofT startup that connects researchers with study participants.
Our research comprised mostly of attitudinal data, as our project is centered around how OA’s perceive smart speaker devices. The aim for the first interview session was to find out:
What biases OAs have going into their initial usage of VUIs
What attitude do they hold in regards to it?
How do they expect the device to embed in their lifestyles?
We considered whether to give our participants media articles related to our 5 themes or to never give them any articles before our first interview session. We decided to give them the articles because they were representative of our research themes and informed our discussion during the interviews. Throughout the interviews, we would ask participants how closely their current attitudes and experience aligned with VUIs based on these articles.
Our research comprised mostly of attitudinal data, as our project is centered around how OA’s perceive smart speaker devices. The aim for the first interview session was to find out:
What biases OAs have going into their initial usage of VUIs
What attitude do they hold in regards to it?
How do they expect the device to embed in their lifestyles?
We considered whether to give our participants media articles related to our 5 themes or to never give them any articles before our first interview session. We decided to give them the articles because they were representative of our research themes and informed our discussion during the interviews. Throughout the interviews, we would ask participants how closely their current attitudes and experience aligned with VUIs based on these articles.
Research
Research
Interviews
Interviews
We scheduled 20 participants, but completed 11 in-depth interviews (6 moderated by me) with novice users that have never owned a smart speaker. Our research study was conducted remotely due to the pandemic. We recruited older adults ages 60+ with a conversational command of English to participate in interviews.
We scheduled 20 participants, but completed 11 in-depth interviews (6 moderated by me) with novice users that have never owned a smart speaker. Our research study was conducted remotely due to the pandemic. We recruited older adults ages 60+ with a conversational command of English to participate in interviews.
How we brainstormed our interview guide:
How we brainstormed our interview guide:
Our team individually brainstormed questions for our interview script and then converged to compile and edit the document with the research advisor.
Example Theme:
Theme 2: Embeddedness in OA Life
Example Question:
Do you perceive smart speakers as helping you accomplish daily tasks with more or less independence?
Our team individually brainstormed questions for our interview script and then converged to compile and edit the document with the research advisor.
Example Theme:
Theme 2: Embeddedness in OA Life
Example Question:
Do you perceive smart speakers as helping you accomplish daily tasks with more or less independence?
Data Analysis
Data Analysis
Learning how OAs attitudes towards smart speakers are shaped by their own experience.
Learning how OAs attitudes towards smart speakers are shaped by their own experience.
Our team decided to use an inductive approach, segmenting our data into short, informative phrases. We downloaded the transcripts from Zoom, anonymized the data, and went through it meticulously, correcting errors and adding in non-verbal cues.
We conducted our data analysis on a rolling basis since the interviewing portion of our study was separated in two parts. When I was the notetaker, I took notes directly into Excel or Miro during the session. Our full process was to upload our transcripts into MAXQDA and code our own interviews in addition to a separate researcher coding the same interview. After finishing each session, the two coders reconciled their codes in Miro.
Our team decided to use an inductive approach, segmenting our data into short, informative phrases. We downloaded the transcripts from Zoom, anonymized the data, and went through it meticulously, correcting errors and adding in non-verbal cues.
We conducted our data analysis on a rolling basis since the interviewing portion of our study was separated in two parts. When I was the notetaker, I took notes directly into Excel or Miro during the session. Our full process was to upload our transcripts into MAXQDA and code our own interviews in addition to a separate researcher coding the same interview. After finishing each session, the two coders reconciled their codes in Miro.
In our final consensual thematic analysis session, our team replicated this process by reviewing and refining the initial themes across all sessions in order to identify high-level, overarching themes.
In our final consensual thematic analysis session, our team replicated this process by reviewing and refining the initial themes across all sessions in order to identify high-level, overarching themes.
Why our research matters
Why our research matters
Why our research matters
Comparing media messaging and academic work on VUIs can offer a deeper understanding of factors driving adoption by older adults. While media does not directly influence perceptions, it can reflect societal and commercial interests. Our analysis found differences between media messaging and older adults' experiences with VUIs, revealing under-addressed barriers to adoption. These barriers include perceptions of close connections, trust in VUI systems, digital literacy, and access to infrastructure. Overall, understanding these factors can inform better VUI design for older adults.
Comparing media messaging and academic work on VUIs can offer a deeper understanding of factors driving adoption by older adults. While media does not directly influence perceptions, it can reflect societal and commercial interests. Our analysis found differences between media messaging and older adults' experiences with VUIs, revealing under-addressed barriers to adoption. These barriers include perceptions of close connections, trust in VUI systems, digital literacy, and access to infrastructure. Overall, understanding these factors can inform better VUI design for older adults.
Comparing media messaging and academic work on VUIs can offer a deeper understanding of factors driving adoption by older adults. While media does not directly influence perceptions, it can reflect societal and commercial interests. Our analysis found differences between media messaging and older adults' experiences with VUIs, revealing under-addressed barriers to adoption. Overall, understanding these factors can inform better VUI design for older adults.
Reflection
Reflection
Key Takeaways
Key Takeaways
As the first research paper I co-authored to submit to CHI 22, there were some challenges that I encountered during the study and preparing it for submission.
As the first research paper I co-authored to submit to CHI 22, there were some challenges that I encountered during the study and preparing it for submission.
Consider the audience. The design Implications should have sufficiently discussed the importance for the HCI community and considered guidelines for how to design VUIs for OAs.
Participants with low digital literacy. Some participants struggled with setting up or connecting their device to their home networks. Troubleshooting problems over Zoom was slow.
Condensed timeline. Slow recruitment made us adjust our timeline, and some participants missed sessions and had to be rescheduled. We had to put in extra time to meet the CHI submission deadline.
Consider the audience. The design Implications should have sufficiently discussed the importance for the HCI community and considered guidelines for how to design VUIs for OAs.
Participants with low digital literacy. Some participants struggled with setting up or connecting their device to their home networks. Troubleshooting problems over Zoom was slow.
Condensed timeline. Slow recruitment made us adjust our timeline, and some participants missed sessions and had to be rescheduled. We had to put in extra time to meet the Chi submission deadline.
Next Project