RCT Methodology Overview
Swayable pre-tests messages and creative to identify what stories change minds and who they persuade, before campaigns are launched.
By pre-testing messaging, it is possible to measure the impact on metrics like brand lift, purchase intent, consideration, as well as custom questions related to your brand or product.
The platform implements a gold-standard randomized controlled trial protocol, coupling efficient respondent acquisition with post-stratification reweighting to maximize statistical power. This document outlines key details of the methodology.
Randomized Controlled Trials:
Measuring impact, not just feedback
Measuring impact on attitudes is not the same as asking for feedback. People often give positive feedback to messages that have no impact on their opinions. Equally, a message can have real impact on opinions even when people say they dislike it. The goal of brand communications is not to get people to like an ad; it is to change attitudes in favor of your product, service, or company. That is what Swayable measures.
The gold standard for measuring causation is randomized controlled trial (RCT) methodology. This is the protocol used whenever reliable results are crucial, such as medical trials. At its core, Swayable is a tool that runs highly efficient RCT survey experiments.
Each respondent in a Swayable test is randomly allocated either to a control group or test group. They are then shown one piece of creative, and asked a set of questions. All respondents have the exact same experience except for the creative shown. If they’re in test group 1, they see creative asset 1; test group 2 sees asset 2, etc. The control group is shown irrelevant ‘placebo’ content, often a blank image or a PSA video. This ensures the control group has as similar an experience to the test groups as possible.
The questions are a mix of segmentation questions, which the platform uses to categorize people into demographic and behavioral segments (generally asked before the creative), and metrics questions on outcomes like brand favorability and purchase consideration (which must be asked afterward). Wherever possible, high resolution 0–100 scales are used for answers.
Swayable computes the Swayable Impact Score as a difference between the average answers in the test group compared to the control group. This is because the average answers in the control group reflect the baseline of the public opinion. If a message is successful at increasing any metric, average responses in that test group will be higher than the baseline. If it does nothing, they will be the same.
This design is a “captive” or “forced exposure” RCT test design. Its benefit is that the overwhelming majority of people who see the creative also complete the survey. By contrast, in-channel tests from advertising platforms lose at least 95% of viewers post-exposure, creating inefficiencies and biases. While they can measure distribution effectiveness well, ad platforms do not have reliable information on creative impact, which is the more important metric — because you can’t make up for ineffective creative with better reach or engagement. The job of the creative is to change attitudes, not just drive views and clicks.
RCTs are also more reliable than pre/post tests, because those require respondents to self-report a change in attitude, which may be an easier test to execute, but gives less reliable data.
Data Integrity
Swayable invests heavily in data integrity. Underlying the reliability of our results are two structural advantages. First, we treat our respondents fairly and they opt in to all tests, so it is a demonstrable fact that we start with much higher quality data than market research industry standard practices.
Mobile-First Respondent Acquisition
Swayable respondents are recruited from a network of mobile apps with the largest possible user bases, including puzzles, games and social networking. Our agreements with app publishers and their agents allow us to recruit users to take part in our tests in return for non-cash incentives, generally in-app currencies.
A.I. Population Modeling
With millions of respondents having participated in Swayable tests, the platform’s A.I. population modeling adds significant statistical power to all test results. In typical deployments, this gain is the equivalent of at least doubling the sample size at no additional cost in time or money. For results on smaller segments, gains are typically many times greater than this.
Post-Stratification Reweighting
Swayable Impact Scores are calculated using sophisticated post-stratification reweighting to ensure that all population segments are correctly represented in the results. This means that the platform is constantly comparing the demographic composition of the sample in every test against the known population characteristics of each market, as determined by the latest census data.
Trusted by brands, agencies, and campaigns
