Mobile App Beta Survey

Background

Team: 2 internal stakeholders

I worked closely with the product owner to align on survey goals, questions, and execution. The entire team consisted of about 30 internal and external stakeholders, all consisting of KP designers, external vendor designers, product owners, and engineers.

Timeline: 2 months

The timeline of this study was determined by the beta soft launch per region. Because the soft launch was different for every region, the survey was conducted in different phases depending on the launch schedule, all within a 2 month time period.

role: lead researcher

I was the sole researcher who was in charge of the planning, creation, data analysis, and share-out of the study.

product: soft launch of mobile app redesign

The mobile app was redesigned to improve the digital front door experience on the mobile device, a platform that is used more and more consistently. The mobile app was launched to KP employees in (3) regions (not KP members). This survey was sent to all 150 beta app users to collect data on initial responses and identify pain points before launching to official KP members.

Why survey?

This method was chosen because the team wanted to quickly capture what beta users were doing and their initial impressions. The team wanted a pulse check on how satisfied beta users were, what were they able to do and what were they not, would they recommend the redesigned app to KP members. Beta users were able to provide an explanation for their responses through a written free response, however that was not the focus. A number of usability tests had been done prior with prototypes to capture user feedback on the mobile app redesign. This survey was meant to validate the key insights from previous usability tests, as well as provide further insights from a fully functioning mobile app with beta users.

 Research Goals

  • Understand beta users initial impressions and satisfaction of the mobile app redesign

  • Identify tasks beta users are unable to complete with the redesigned mobile app redesign

  • Gauge redesigned mobile app responsiveness and navigation compared to previous design

research questions

  1. How do beta users perceive the KP mobile app before vs after redesign?

  2. What are beta users’ perception of the app’s responsiveness, navigation, accuracy of displayed information, etc?

  3. How satisfied are beta users with redesigned mobile app?

  4. What is the likelihood beta users recommend mobile app to KP members based on their expertise (e.g. clinicians, front line managers, etc.)?

Methodology

Participants

  • Beta users (KP employees: clinicians, front line managers, etc.)

  • Must have used the redesigned app for at least 3 weeks

  • Limited to no involvement in the creation of the redesigned app

Procedure

  • After KP employees agree to opt in to be the first to use the redesigned KP mobile app, their app automatically updates on the day of soft launch

  • Beta users use the redesigned app in their everyday life, and 3 weeks after soft launch day, the survey was emailed to beta users

  • 2-3 reminder emails were sent to beta users

  • beta users take about 10-15 minutes out of their day to provide feedback through the survey

Stimuli

  • Redesigned KP mobile app soft launch

  • No prototype used

Evaluation

  • Responses were quantitatively evaluated based on the survey responses

  • Qualitative feedback was collected via free responses to fill in the color of why they selected certain survey responses

Metrics

  • (40) beta participants responded to they survey which is about a 27% response rate

  • 90% confidence level (90% confident results are representative of the entire beta population, which in turn gives a glimpse into KP member results)

  • 10% margin of error (results could differ up to 10% from beta population)*

*The higher the sample size, the lower the margin of error

Results

Key insights:

  • About 50% of beta users reported a change in the app’s responsiveness and user expectation due to a shift in informational architecture

  • About 20% of beta users reported incorrect information, which upon further analysis was a misinterpretation of UI elements

  • Despite pain points, 60% of beta users were satisfied with the redesign, reporting high visual appeal

*percentages altered to protect company information private

Redesigned Mobile Beta User Satisfaction*

Impact

Overall

Impacted product roadmap releases for KP members to prioritize changes in pharmacy and appointments to address navigation, responsiveness, and perception of incorrect information Findings impact KP digital experience to ensure team hits 2021 KPIs including improving awareness, navigation to, and utilization of right channel of care.

pharmacy feature

Stakeholders reached out to discuss changes to implement in the same release due to beta users inability to interpret the information in their pharmacy centers within the redesigned mobile app.

Appointment feature

Stakeholders are looking to redo the entire appointment feature to increase usability and responsiveness.

Dashboard

Interactive components were removed to increase responsiveness. Further informational architecture usability studies will continue to understand users expectations, identify pain points, and improve the usability of the dashboard.

Next Steps

Next steps include modifying survey questions to send to KP members. Redesigned KP mobile app is officially launched and the survey will gather quick and inclusive feedback to understand what KP members are doing. Findings will help inform usability studies in the future to understand KP member impressions of redesigned app. Further informational architecture studies will continue to align user expectation with KP mobile app.