Experiments, fixed before they start
How Torque Lift picks an experiment, fixes its metric and end date first, runs the traffic split check, and reads the result once.
Torque Lift picks the next experiment from your numbers, ships it behind a flag as a pull request, and reports the result, win or not. What counts as a win is written down first.
From a pick to a pull request
Torque Lift drafts candidates
From your week's numbers, 3 to 5 ideas. For each, Torque Lift works out what your traffic could detect, and drops anything that can't be tested cleanly. During pilots, we review the candidates and pick one to put to you.
Your growth lead answers in writing
Accept or reject. The answer goes in the action log, so the record shows who decided and why. You can also pick an experiment yourself in your console.
Torque Lift writes the plan down
Before anything ships, the experiment's metric, goal, end date and stopping rule are recorded. See the next section.
A pull request puts it behind a flag
The new version is written as new files, with today's version kept as the default. The flag starts off, and with it off your app behaves exactly as it does now. The tests in the pull request cover both flag states and a flag service that can't be reached. A person on your team merges it. See Code in your repo.
Torque Lift starts the experiment
Once the pull request is in, Torque Lift starts the experiment on its recorded day. Starting on any other day means recording it again first.
What is fixed before it starts
Each experiment has a record, written before launch (pre-registered) and cited in every readout. It can't change while the experiment runs. The flags and the statistics run on open-source GrowthBook.
| Fixed | What it says |
|---|---|
| Hypothesis | What you expect the change to do, and why |
| Who is in it | The people the experiment applies to, and whether they are counted by user ID or by anonymous ID |
| Primary metric | What is measured, how it is defined, and where it stands today |
| Guardrails | The other metrics that must not get worse |
| The smallest lift that counts | The smallest change the experiment is sized to detect (the minimum detectable effect, MDE) |
| How many people | Who is planned, and how many come in each week |
| The dates | When it starts, how long people are enrolled, how long each is followed, and the one day it is read |
| What happens next | A written rule for each result: ship, stop, or iterate |
People are split 50/50 between today's version and the new one. Results are two-sided, at 95% confidence and 80% power.
The checks every experiment gets
- The traffic split check (sample ratio mismatch, or SRM). From 48 hours after launch, Torque Lift checks that people really were split 50/50. A split far from it, more than chance explains, marks the experiment invalid. Its result isn't shown, and your readout says why.
- One read, on the read day. The effect is read once, over people who came in during the enrollment weeks and whose follow-up has closed. Before then, readouts show the experiment's health, never the lift. No peeking.
- A fixed horizon. An experiment ends on its recorded day, not when the numbers look good. If the flag service is set to stop an experiment early, Torque Lift flags the mismatch with the record.
- Enough traffic first. Before the first experiment, Torque Lift checks that your traffic could show a lift of up to 30% on some metric within 4 weeks, at 80% power. If not, the check says how many more people it needs, and Torque Lift holds the experiment. Your readout still comes.
How a result reads
The readout gives the change in your metric with the range it likely falls in, set against the lift you fixed before the start. It names one of three decisions: ship it, stop it, or iterate. An experiment that missed its bar is reported as exactly that.
After a read, Torque Lift also breaks the result down by the day people came in and by how they first arrived. A group is shown once both versions have at least 100 people in it. These break-downs never change the read.
When there isn't enough traffic
Torque Lift says so plainly and holds the experiment rather than run one that can't give a clear answer. See Readouts for the leads Torque Lift still gives you each week.
How exposures are recorded
When someone is put in an experiment, Torque Lift's tracking sends an Experiment Viewed event with the experiment and version. Like every event, it waits for your app to record that the visitor agreed to analytics, and it lands in your event tables.
Your weekly readout
What your weekly Torque Lift readout holds, and how to read growth accounting: new, retained, resurrected and churned users, and the quick ratio.
Your event tables
Your events in your own bucket as Apache Iceberg tables, with every column, the turned-away table, and a query to start with.