Incrementality testing unlocks upper funnel branding investments for Study.com
Measurement Strategist
M-Squared
The Brand
Study.com provides high stakes online learning solutions to more than 34 million learners and educators a month across professional test preparation, college credit and K-12 education. Recognized as one of the world’s most innovative companies by Fast Company and the GSV150, Study.com has helped save students more than $475 million in tuition costs through its College Saver program and donated some $29 million across social impact programs committed to increasing educational equity
The Challenge
The company's new customer acquisition efforts had been primarily centered around SEO and PPC investments, and managed like a classic performance marketing program, governed to tight CAC guardrails agreed upon between marketing and finance. The strategy served the organization well for many years helping the brand drive predictable and profitable growth, but eventually these channels reached maturation and could not continue the accelerated growth rate desired. So like many other brands in the same predicament, Study.com has been seeking diversification and growth opportunities in upper funnel channels.
As Study.com explored other channels it also aspired to build out a stronger brand, investing in video creative with a branding narrative. The brand campaign-focused on inspiring students and educators to reach their academic and career goals through Study.com's online college courses, exam preparation and classroom resources.
But as always the key question is “Will branding drive new customer demand?”.
When the company rolled out smaller scale paid video campaigns on platforms like YouTube, it saw few last-click conversion although other positive indicators led to the desire to continue investment at scale and with more precise measurement. There were several back-and-forth discussions to determine if the organization should put more budget behind this and to launch in CTV, but would it be a risk worth taking? Would this bring in new students? How can you measure it since it serves more as an upper funnel tactic?
To find out, Study.com embraced geo-based incrementality testing with M-Squared to learn the efficacy of branding campaigns on CTV and YouTube.
“Really just taking the test-and-learn approach and leaning into creativity”
- Emily Johnson
Watch as Emily Johnson unpacks her experience with us!
Exploratory Data Analysis
As with any brand, the devil is in the details. Real businesses have real complexity in their business model, and in their data. Marketing measurement practitioners have to process these complexities and assess which of those nuances are meaningful to incorporate into the measurement solution architecture and which ones are irrelevant to the business questions being considered.
For Study.com, there were many such dimensions of complexity:
- Like many businesses, that service multiple audiences with several products, there is a mix of hero and long tail products.
- Different products drive unique LTV and hence different levels of contribution margin for the business over the long term.
- The tactics being tested are demand generation (vs demand harvesting) and the video medium, which intuitively means the measurement has to account for a longer time-to-conversion period, and low last click attribution. Should the measurement plan consider upper funnel outcomes like Engaged Sessions or Email/Phone collection which could serve as leading indicators of demand being generated. Would the data collected so far support that measurement plan?
The Approach:
Geo Match Market Test - Design
At M-Squared we take a structured approach to designing a geo test.
Catch the 11 minute snippet on geo testing from the M-Squared masterclass:
Study.com already had evergreen YouTube campaigns in flight, and there had been a significant push over the summer months to launch a new branding video. Post geo testing feasibility analysis, we determined that the current spend levels on YouTube might not clear minimum detectable lift (MDL) levels and hence may not yield statistically significant reads with a holdout test. As a result, we tested YouTube with a scale cell where spend was intentionally elevated to support readability. A holdout cell was also included to measure the lift at current spend levels, with the understanding that the reads may come back inconclusive, but provide valuable insight on incremental vs marginal contributions.
CTV was a brand-new channel and had no prior spend or performance history. It was not meaningful to select a holdout treatment for incrementality testing. Instead, a scale cell was designed as part of the feasibility analysis. Since it was a new channel, warming up the channel was recommended before starting measurement. So the first 2 weeks were slotted for campaign warmup and the next 4 weeks for the read, running a total of 6 weeks in test flight.
Market selection algorithms were run and DMA’s were identified for the three different testing cells. Test budgets and test flight period were determined as part of feasibility analysis.
|
YouTube Scale cell |
YouTube Holdout cell |
CTV Scale cell |
|
|
|
The test was flighted in Q4 2024, and the flight was monitored for execution aligning to the test design that was put in place.
The Results:
Lift Reads & Interpretations for Growth
As with the design, at M-Squared, we take a structured process for estimating the lift reads from the test.
Catch the 7 minute mini course on estimating lift from the M-Squared masterclass:
After carefully estimating the lift with multiple algorithmic approaches, Study.com learned:
- CTV drove a 6% incremental lift to new member acquisition at a 3.8 ROAS
-
YouTube showed good potential as well 3% incremental lift on new member acquisition at a 2.5 ROAS
Both of these reads provided meaningful insights on employing upper funnel tactics for driving customer acquisition growth for Study.com.
- With reasonable assumptions on scale and diminishing returns, and annualizing the estimates for seasonality, employing YouTube and CTV in their customer acquisition plan could drive an estimated 20% growth for the brand.
The Conclusion:
These insights now provided Study.com a clear path to diversify its media mix into upper funnel channels using a test-learn-grow approach for risk mitigated and fiscally responsible growth.
Disclaimer: The data presented is blinded to protect brand’s P&L confidentiality but preserve insights for educational purposes.
