Mixed Method: Creator Rewards Program Beta Launch

Background

Team: Creator and Content monetization

role: lead researcher

Timeline: 7-8 weeks

product: Creator rewards program (CRP) Additional Rewards

Previously, the Creator Rewards Program consisted only of Standard Rewards, which were calculated based on the video views and Revenue Per Mille (RPM). The Creator and Content team launched a new monetization strategy that not only rewarded creators from Standard Rewards, but also included Additional Rewards, which rewarded high quality content (e.g., 1080p+, niche, etc). This study aimed to assess the new CRP monetization strategy with a quantitative and qualitative approach.

Methodology: large scale survey + interviews

I chose to start with a quantitative large scale survey first to identify broad patterns and quantify satisfaction/perceptions. From there, I conducted in depth interviews to dig deeper into the “why”, leveraging quantitative insights to structure interviews. Probing into the motivations, pain points, and contextual nuances provided more color to complement survey insights.

Objective

The Creator and Content team beta-launched a new monetization model called “Additional Rewards.” Previously, creator monetization relied solely on “Standard Rewards,” which paid creators based on video views. Additional Rewards represented a strategic shift—rewarding not only high engagement (from Standard Rewards), but also high quality content (from Additional Rewards). I led a two-phase mixed method research study to assess creator satisfaction, understand perceptions of the design changes, and identify opportunities to refine and strengthen the monetization strategy overall.

research questions

1. How satisfied are creators with the new CRP Additional Rewards monetization strategy? Why?

2. How well do creators understand Additional Rewards? How have they adapted to produce more high quality content, if at all?

3. How does the new CRP design compare to the previous design?

Methodology

1. large scale Survey

participants

• N=About 2,400 responses across US, UK, KR, JP, MX, and BR regions.

• I launched the survey to all creators who participated in the beta launch (a total of ~80,000). We had about a 3% response rate with support from operations (e.g., account managers, partners, etc).

• *Due to the smaller sample sizes from Korea, Japan, and Mexico, findings from these regions were treated as directional signals rather than definitive insights. While they highlight potential trends and areas of opportunity, they should be validated through additional research.

• **All numbers are approximations and are changed due to confidentiality.

procedure

• Respondent List Preparation: The product team provided a list of all the creator UIDs who participated in the beta launch

• Launch: I launched with an internal survey tool called ByteSurvey which sends the survey via an in-app notification directly on the TikTok app.

• Data Collection: The survey was in field for about 3-5 days

• Analysis: After I aligned with the product team on the total sample size as well as a regional breakdown, I downloaded the data set and analyzed the data via R. One limitation of the internal tool was that it did not allow us to distinguish responses from top-tier, mid-tier, and lower-tier creators.

application: Internal survey tool called Bytesurvey

I launched via an internal tool called ByteSurvey, which closely mimics Qualtrics. The internal tool allows creators to take the survey from their TikTok in-app notification.

method justification: confidently capture precise perceptions and satisfaction

By capturing insights from a larger sample size, the team can reliably identify users’ perceptions and satisfaction of the new Additional Rewards feature across regional markets, capturing feedback from top tier, mid tier, and low tier creators. Survey insights led to better discussion guide questions to understand why success metrics (e.g., satisfaction, awareness, etc) were potentially low and identify opportunities for the team to improve.

2. In-depth interviews

participants

• I recruited participants from survey responses and reached out to participants who were interested in a participating in a follow-up interview.

• Must be from the US or UK (To maximize efficiency within the project’s timeline, I focused on participants that can speak English and drawing from the largest respondent pool (US N=1,000; UK N=800))

• N=12 total US or UK creators

  • 6 were Middle Tier Creators (~50k-500k followers)

  • 6 were Low Tier Creators (~10k-50k followers)

Given the timeline constraints, we were unable to recruit Top-Tier creators who had a lower sample size and were a very difficult segment to bring in for interviews.

Procedure

• Introduction (5min): Provide instructions and built rapport

• Overall Satisfaction of CRP (15min): Understand their general satisfaction with the Creator Rewards Program overall

• Additional Rewards Awareness/Perceptions (20min): Understand their awareness and perceptions of the new Additional Rewards

• New Design Feedback (15min): I asked creators to walk through how they normally look at their their Creator Rewards to better understand perceptions and usability of the new designs

• Wrap-up Questions (5min): Follow-up questions were asked to allow participant to reflect on their overall experience and provide any other feedback

application: Zoom calls

I recruited participants by reaching out via email, which they provided within the survey if they were interested in participating in a follow-up interview. I used Calendly to schedule a time, and then I interviewed participants via Zoom.

method justification: nuanced contextual understanding of perception and satisfaction

Given the survey limitations, the in-depth interviews allowed the team to fully understand the context of a creators experience, which can be vastly different depending on what tier they are, whether they create part time or full time, and the type of content they post. Survey results from phase 1 were leveraged to shape the discussion guide, allowing me to probe more deeply into the pain points highlighted by the survey.

Key Findings

1.Mid-low tier creators experienced limited benefits, driving lower satisfaction compared to top creators.

• Quant:

  • Survey results show that most creators (77%*) felt neutral about Additional Rewards.

  • Satisfaction was higher among top-tier creators (55%* top-2) compared to mid-tier (22%*) and low-tier creators (12%*). Top-tier results should be interpreted directionally due to the smaller sample size (n=167).

• Qual:

  • Interviews explained creator sentiments, revealing that most mid-low tier creators experienced little to no changes in revenue. Without clear benefits, mid-low tier creators reported more neutral sentiments.

  • Many mid-low tier creators often didn’t have access to the resources or equipment (e.g, multiple cameras, editing software, etc) to easily produce high quality content to fit the Additional Reward criteria. This suggests mid-low tier creators face a steeper ramp-up before realizing any meaningful benefits.

  • A few mid-tier creators (e.g., 300+* followers) near the top tier reported slight revenue increases, likely because their high-quality content already met the criteria, suggesting that top tier creators can benefit from rewards with less effort.

• Key Recommendation:

  • Focus initial outreach on top tier and potential top tier creators who can integrate Additional Rewards into existing content creation with minimal friction, optimizing for immediate impact.

  • Support mid-low tier creators in ramping up over time (e.g., intermediate milestones) to encourage the creation of high quality content for the broader creator ecosystem.

2. Many creators did not understand Additional Rewards, because nav to detail pages felt hidden and unintuitive.

• Quant:

  • Survey results showed only a few (5%*) creators recalled noticing the Additional Rewards detail page, which contained key information to explain Additional Rewards purpose and eligibility criteria.

• Qual:

  • Interview insights validated that navigation to detail pages were critical in helping creators understand Additional Rewards and how to align their content to match its criteria.

  • Most mid-low tier creators were unfamiliar with Additional Rewards. While some briefly noticed it on their CRP home screens, they did not seek out more information to understand the changes. As a result, creators lacked the guidance needed to adjust their content to meet the criteria, limiting their ability to benefit from Additional Rewards.

  • This is especially important for mid-tier creators who have potential to be top tier, and may only need minor adjustments to see increased revenue from rewards.

  • When asked to find more information, the entry point to the detail page felt hidden and unintuitive. Navigating to the detail pages took many taps (~3-4). When directed to the information, many were pleasantly surprised at how interactive and engaging it was, but often struggled to return after exiting. Most wanted the information to be surfaced higher in the navigation, rather than buried behind so many taps.

• Key Recommendation:

  • Elevate entry point to Additional Reward details so creators can easily reference important information when its most relevant (e.g., when ramping up to the new monetization changes).

3. Most creators (86%*) preferred the new CRP designs because the updated graphs were clear and easy to use.

Key Recommendations

1. 🎬 Focus on current and potential top tier creators: Focus initial outreach on top tier and potential top tier creators who can integrate Additional Rewards into existing content creation with minimal friction, optimizing for immediate impact. Support mid-low tier creators in ramping up over time (e.g., intermediate milestones) to encourage the creation of high quality content for the broader creator ecosystem.

2. ℹ️ Make information on Additional Rewards easier to access: Elevate entry point to Additional Reward details so creators can easily reference important information when its most relevant (e.g., when ramping up to the new monetization changes). 

3. 📊 Launch updated designs: Move forward with launching new CRP designs because the graphs are more clear, easy to use, and visually appealing compared to the old designs.

Impact

team impact:

• Showcased Research Team Adaptability: Seamlessly absorbed a complex project during an unexpected absence, highlighting the team’s ability to consistently deliver high-quality, on-time insights and provide uninterrupted cross-functional support across all workloads.

product impact:

• Launched Updated CRP Designs: Due to the positive feedback from both the surveys and user interviews, the team launched the new CRP designs. 

• Expedited Full Launch in EU: Results from this study led to an expedited timeline for a full roll-out in the EU countries. This may have been influenced by the higher concentration of top-tier creators in EU, who generally expressed more positive sentiment toward Additional Rewards.

business impact:

• Target Audience Shift: Insights drove a leadership decision to pivot the primary target audience toward top-tier creators. This strategy shift was incorporated into product roadmaps and feature prioritization. The team planned to initiate follow-up workstreams to ensure long-term opportunities were available for all creator tiers.