P1 · Seconds to paywall
Seconds from first launch to the first screen that asks for payment, or “none” if no screen does.
Recorded as: A whole number of seconds, or “Never blocks”.
Always recorded. This measurement is never marked not applicable.
The test protocol.
A hands-on test is a person running our test protocol against an app on a real device and recording what they measure. This page prints that protocol in full: what is recorded, the form each value takes, and what the protocol leaves out.
Protocol v1.0. This page prints that version in full. Each test is run under one version of the protocol and carries its number.
What a hands-on test is
A hands-on test is a person running this protocol against an app on a real device and recording what they measure. The protocol is a fixed list of measurements. Each one is a number, a choice from a fixed list, or a line copied exactly as it appears on screen. The person running the test records what they see; code decides what the measurements mean.
Who runs the tests
One auditor, identified as A-01, runs every test, and no test is run a second time by another person.
The conditions every test is run under
Fixed, and recorded for every test.
- A fresh store account and a fresh app account.
- A factory-default notification state.
- The app's current store version.
- Screen recording running from before install to after the last item.
- Do Not Disturb on.
The rules of the protocol
- Every item is a measurement: a number, an enumerated observation or a verbatim transcription. No item is a rating or an adjective, and there is nowhere to record a result.
- The auditor incurs no charge. Free trials only. A free trial may be started only where it can be started and cancelled inside the same session without a charge being incurred, and is cancelled before the session ends. Nothing is bought: no plan, no paid or discounted introductory period, no in-app purchase.
The measurements
15 numbered measurements, in the order a test meets them, plus one sub-measurement, P2b, that exists only beside P2. Every one ends in one of three outcomes.
P2 · Paywall close control
Whether that screen shows a control that closes it; if it appears after a delay, record P2b.
Recorded as: One of: Visible; Appears after a delay; Absent.
Can be recorded as not applicable when: No paywall.
P2b · Paywall close delay
Seconds before the close control appears. Recorded only when P2 is “delayed”.
Recorded as: A whole number of seconds.
Recorded only beside the measurement it belongs to.
P3 · Listed function without paying
The first function named in the store description's first sentence, transcribed verbatim, then attempted once without paying: “done” or “blocked”.
Recorded as: Several parts, listed below.
- Attempt: One of: Done; Blocked.
- Function, verbatim: Text, copied exactly as shown.
Always recorded. This measurement is never marked not applicable.
P4 · Trial requires a payment method
Whether a payment method is demanded before the trial starts.
Recorded as: Yes or no.
Can be recorded as not applicable when: No subscription; No trial.
P5 · Trial terms shown
What the screen carrying the trial button states about the charge after the trial.
Recorded as: One of: Price and period; Price only; Neither.
Can be recorded as not applicable when: No subscription; No trial.
P6 · Displayed price
The price as displayed for the plan the paywall leads with, verbatim, with its amount, currency and billing period.
Recorded as: Several parts, listed below.
- Price as displayed, verbatim: Text, copied exactly as shown.
- Amount: A number.
- Currency (ISO 4217): A three-letter currency code.
- Billing period: One of: Daily; Every 3 days; Weekly; Every 2 weeks; Monthly; Every 2 months; Every 3 months; Every 6 months; Yearly; One-time.
Can be recorded as not applicable when: No paywall.
P7 · Preselected plan
Which plan is selected when the price sheet opens, and how many plans it offers.
Recorded as: Several parts, listed below.
- Selected plan: One of: None; Cheapest; Dearest; Other.
- Plans offered: A whole number of plans.
Can be recorded as not applicable when: No paywall.
P8 · Billing channels offered
Every channel through which the app offers to take payment. An empty set means none.
Recorded as: Any combination of: App Store / Play; Web checkout; Other, or none of them.
Always recorded. This measurement is never marked not applicable.
P9 · Cancel path
The route by which a cancellation actually completed.
Recorded as: One of: Platform (App Store / Play); In-app; Web form; Support email; Phone; Chat.
Can be recorded as not applicable when: No subscription; No trial; No chargeless trial.
P10 · Cancel steps
Taps from the app's home screen to a confirmed cancellation, or “none”.
Recorded as: A whole number of taps, or “Nothing to cancel”.
Can be recorded as not applicable when: No subscription; No trial; No chargeless trial.
P11 · Cancel control in the app
Whether the app contains a control that starts or links to cancellation.
Recorded as: Yes or no.
Can be recorded as not applicable when: No subscription; No trial; No chargeless trial.
P12 · Ads per session
Full-screen adverts that interrupt use in a scripted 10-minute session (P12), and optionally the unskippable seconds they hold the screen (P12b).
Recorded as: Several parts, listed below.
- Interrupting adverts: A whole number of ads.
- Unskippable seconds (P12b): A whole number of seconds (optional).
Can be recorded as not applicable when: No advert surface.
This version of the protocol fixes the length of the session but does not yet write down its steps.
P13 · Account required
Whether the function recorded in P3 can be performed without creating an account.
Recorded as: One of: No account; Optional account; Required for core use.
Always recorded. This measurement is never marked not applicable.
P14 · Data deletion path
The route to delete the account found from inside the app.
Recorded as: One of: In-app; Web form; Email request; No deletion path.
Can be recorded as not applicable when: No account system.
P15 · Odds shown for randomised items
Whether odds appear on a screen that sells a randomised item.
Recorded as: One of: Shown; Not shown.
Can be recorded as not applicable when: No randomised items.
The three outcomes
Every measurement ends in exactly one of these.
- Observed: A value was recorded, and at least one key frame from the screen recording shows it.
- Not applicable: The item does not apply to this app. The auditor gives one reason from the closed list further down, and no other reason is accepted.
- Could not observe: The auditor could not take the measurement and says why in words. A test with an item left this way cannot be completed.
When “not applicable” can be recorded
The list of reasons is closed: no other reason is accepted, and a reason must be true of the app.
| Reason | What it means |
|---|---|
| No paywall | No screen in the app asks for payment, so there is no paywall to measure. |
| No subscription | The app offers no recurring plan. It may still sell one-time purchases. |
| No trial | The app sells a subscription and offers no trial. |
| No chargeless trial | A subscription or trial exists, but none can be started and cancelled inside the session without a charge being incurred (a paid or discounted introductory period, a trial that bills on start). |
| No advert surface | The app has no surface on which an advert was observed. |
| No randomised items | The app sells no randomised item. |
| No account system | The app has no account, so there is no account data to delete. |
Rules a recorded test must satisfy:
- P9, P10 and P11 are all “observed”, or all “not applicable” with the same reason.
- “No subscription” (the app offers no recurring plan) is valid with any P8 value; “No chargeless trial” is recorded only where P8 is not the empty set.
- “No paywall” is recorded only where P1 is “none”.
- “No account system” is recorded only where P13 is “none”.
- P2b is recorded when P2 is “delayed”, and only then.
- Each item is recorded once.
Cancelling, and the no-charge rule
The auditor incurs no charge, so cancelling is tested only where a free trial allows it.
The cancel measurements (P9, P10, P11) are taken only where a free trial can be started and cancelled inside the test without a charge. They are all observed, or all not applicable for the same reason:
| What the app offers | P9, P10, P11 |
|---|---|
| The app offers no recurring plan (it may still sell one-time purchases). | Not applicable: No subscription |
| The app sells a subscription and offers no trial. | Not applicable: No trial |
| The app offers a trial that cannot be started and cancelled in the session without a charge (a paid or discounted introductory period, a trial that bills on start). | Not applicable: No chargeless trial |
| The app offers a free trial that can be started and cancelled in the session without a charge. | Observed |
What this version does not reach
This version of the protocol does not measure these, and a test run under it says nothing about them:
- Remove-ads purchase
- Notifications per week
- Pay advantage
Whether an app warns before a trial ends is not observed either, because watching a trial end means letting it charge.
Where the results appear
When a test is completed, what it measured is published in that app's report, under a dated “Human-reviewed” mark that carries the protocol version. Every completed test is also listed on the audit record.