Member Home Screen Usability Study
Background
Team: 10 internal stakeholders
I worked closely with both third party vender as well as internal KP designers and product stakeholders across multiple experience teams.
role: lead researcher
I was the sole researcher who was in charge of the test plan, timeline, sessions, team syncs, data analysis, and share-out of the study.
Timeline: 3 weeks
Due to external factors, the timeline for this study was shortened to accommodate the launch of MVP at the end of Q1 2021. Key findings were delivered in time to change low-hanging fruit design components before official launch.
product: Kaiser permanente website
This study aims to evaluate the proposed MVP design of the logged-in home page of the Kaiser Permanente website. Website prototype was tested on both desktop web view as well as mobile web view.
Methodology: Remote moderated usability study
Stakeholder questions involved identifying pain points, uncovering potential opportunities, and understanding user priorities which resulted a usability study as the chosen methodology. Due to the pandemic, this study was changed to be a remote usability study.
Research Goals
Identify understandability, discoverability, and usability pain points with regard to the proposed MVP design
Understand user feedback and expectations of home page on both desktop web view and mobile web view
Understand how different users’ expectations and needs for the KP home screen change depending on how familiar they are with the current KP home screen
research questions
How does the KP home screen perform on a desktop web experience compared to a mobile web experience?
How does the proposed MVP design help guide users to the appropriate care, if at all?
How do users view COVID information displayed on the proposed MVP design?
Methodology
Participants
(11) Kaiser Permanente members*
(6) Familiar with current KP website
(5) Unfamiliar with current KP website
SCAL or NCAL region (top two most populated KP region)
Uses web/internet for at least 5 hours a week not including work-related purposes
Age ranged 28-65** years old; skewing slightly older
A mix of female, male, and/or non-binary gender
A mix of demographics
*One participant was a no-show (originally intended to be 12 participants)
**Modified for confidentiality reasons
Procedure
Introduction (5min): Provide instructions, build rapport, set-up remote prototype
Familiarity (10min): Ask questions to better understand users’ general healthcare experiences
Tasks (40min): A list of 6 common and frequently performed tasks are given to users
6 participants performed tasks on desktop prototype first, if time allowed the mobile prototype was shown afterward
5 participants performed tasks on mobile prototype first, if time allowed the desktop prototype was shown afterward
Wrap-up Questions (5min): Follow-up questions were asked to allow participant to reflect on their overall experience
Post Study Questionnaire: SUS question distributed to participants to fill out right after the study
Stimuli: Figma
Desktop prototype
Mobile prototype
Application: microsoft Teams
Asked participants to phone call into teams meeting to allow name and number to remain anonymous
Leveraged Figma mirroring capabilities to follow participant as they moved throughout the prototype to complete tasks
Evaluation
SUS score was analyzed to measure the perceived usability of products and systems, specifically how the proposed MVP design fits into the rest of the KP web experience
Task performance was measured to determine how successful users’ were in completing common & frequently performed tasks
Qualitative data was collected throughout the session via participant attitudes, behavior, and feedback
Results
sus scores & task performance
Hidden to protect company information, let me know if you have any questions!
general Key insights
Overall: Blocks of content made information salient and easy to find
Familiarity of KP Site: Regardless of how familiar users were with KP website, the most important information to users were understanding how to get care, how to contact their doctors, and looking at their top notifications**
Mobile vs Desktop View: Users were more accustomed to scrolling on a mobile web device compared to a desktop web device suggesting that the visual flow of a mobile device was easy and natural for users to scan down to find information. In comparison, the visual flow of the desktop prototype did not follow a users’ natural “F” shaped pattern to view information on the screen.**
**Modified for confidentiality reasons
“It's just easier to look for stuff...everything has been packaged into these easy to read, calming blocks. Whereas the [current site] is scattered.” -[example quote]
How I worked with stakeholders
test plan Alignment
With such a tight timeline, I facilitated team working sessions with relevant product and design stakeholders to prioritize goals and research questions. Together the team aligned on tasks, prototype flow, and timeline within a span of about 1 week.
test session debriefs
After each test session day, I organized a 30-45 minute working session to understand what key insights stood out to stakeholders. These working session debriefs were valuable in understanding the purpose and intent of the design, as well as align on what the most important key patterns and trends.
working session next steps
Follow-ups with team after share-out to ensure key findings were understood and implemented correctly in MVP launch and north star iterations.
Impact
reduced avoidable ER visits to reduce contact
This study impacts how users access COVID-19 information. After negative user feedback, bright COVID alerts were removed from MVP design.** COVID information is now better integrated within the design of the home screen.
This study changed get care messaging in MVP design to make users feel more confident in navigating and utilizing the right channel of care, encouraging virtual care encounters for low urgent situations.
increased member self efficacy
Impacted organization, layout, and messaging of healthcare notifications on MVP design. This change in the MVP will allow users to efficiently access their most urgent tasks/updates related to test results, upcoming appointments, prescription delivery, etc.
Key findings led to prioritization of healthcare notifications to be the first component on the home screen in north star iterations.**
increase member retention
This study works to increase member retention by better personalizing the user experience in a non-invasive and low effort way.
**Modified for confidentiality reasons
Next Steps
Launch of MVP scheduled for end of March. North star design iterations continues via working session with internal team. Looping in engineering stakeholders to align on MVP designs and North star designs in the next coming weeks.