Money and consent
Whether pricing is clear, the free tier is honest, leaving is reasonable, pressure tactics are manipulative, and advertising respects the core experience.
Fairness, not general app quality.
What we measure should have one meaning. Everything outside that meaning stays supporting context.
Polish or popularity
A beautiful, famous app can still treat people unfairly. A niche or imperfect app can still charge honestly and respect consent.
Preferences and context
Offline support, account requirements, technical quality and privacy characteristics are reported as context, without distorting the fairness measurements.
Evidence strength decides what we stand behind.
Independence is counted by source kind, not observation count. Fifty reviewers repeating one complaint are still one source.
REPRODUCIBLE FACTMeasured
A deterministic observation—such as a shipped SDK class or disclosed weekly-only price—publishes from one version-stamped source.
ON THE CATCH LISTONE SOURCE KINDOne source only
The finding is shown with its source so readers can inspect it, but it is not on the catch list — unless it is a charge only users can see and enough of them report it (below).
SHOWN, NOT CONFIRMEDTWO SOURCE KINDSCorroborated
Independent ways of looking at the app agree. The finding reaches the catch list, labelled “Corroborated”.
ON THE CATCH LISTTHREE OR MOREStrongly corroborated
Multiple independent evidence paths converge. The finding reaches the catch list, labelled “Strongly corroborated”.
ON THE CATCH LISTA completed hands-on test is a further kind of evidence, and the only one a person supplies. One recorded measurement stands on its own, labelled “Measured in our test”. Where a test's measurement contradicts a finding, that finding stops counting and stays on the page with both sides shown.
INPUT 1Enough public reviews
+
At least 50 reviews must be analyzed before the review side of the assessment gate is satisfied.
INPUT 2Observed app-side material
=
At least one package, resource, manifest, store-metadata, or privacy source must actually yield coverage.
RESULTA full assessment publishes
If either input is missing, findings can still appear — and the page says which input we did not have.
When reviews alone are enough
Some harms are only ever visible to users — no artefact can show that a charge appeared after cancelling. So each charge below may publish from reviews alone, as “Reported by N users”, once enough reviewers raise it. This is a recorded amendment to our rule that a single kind of evidence is never enough, not a loophole.
Each charge has its own bar. Our pattern-matching counts reviewers whose words match a charge; not all of them are actually asserting it. So we measured, per charge, how many matched reviews really assert the harm, and divided our base bar — 25 genuine reporters and 2% of at least 500 reviews read — by that share. A charge whose patterns over-match needs proportionally more reviewers. A published “Reported by N users” names the reviewers, their share and the sample on one store's reviews, and all three clear that charge's row. Each charge also has a lower floor, in the first column: below it, review evidence for the charge is not shown at all, and between the floor and the bar it is shown as “One source only”.
| Charge | Reviewers needed to be shown at all | Matched reviews that assert it | Reviewers needed | Share needed | Reviews scanned, at least |
|---|---|---|---|---|---|
| Upgrade prompts use guilt-trip wording | 5 | 32 of 32 | 25 | 2.00% | 500 |
| Hard to cancel | 5 | 31 of 32 | 26 | 2.07% | 500 |
| Paywall blocks the basics | 5 | 31 of 32 | 26 | 2.07% | 500 |
| Price keeps rising | 5 | 31 of 32 | 26 | 2.07% | 500 |
| Aggressive ad load | 5 | 122 of 129 | 27 | 2.12% | 500 |
| Pay to progress | 5 | 63 of 69 | 28 | 2.20% | 500 |
| Progress pauses until you wait or pay | 5 | 29 of 32 | 28 | 2.21% | 500 |
| Reports that the ads misrepresent the app | 5 | 29 of 32 | 28 | 2.21% | 500 |
| Paywall with no way past it | 5 | 27 of 32 | 30 | 2.38% | 500 |
| Trial converts to a charge | 5 | 27 of 32 | 30 | 2.38% | 500 |
| Reports of unexpected charges | 20 | 26 of 32 | 31 | 2.47% | 500 |
| Tracking and data collection concerns | 5 | 83 of 101 | 31 | 2.44% | 500 |
| Countdown-timer purchase pressure | 23 | 10 of 19 | 48 | 3.80% | 500 |
How analysis runs at catalog scale
The default path is deterministic and makes no model calls. Where a model is used, it may veto a detector's candidate, or report an observation with a quote we verify word-for-word against the app's own text. It never decides what that text means.
Public-review analysis
Recurring pain points are tallied across distinct authors. A published theme needs repeated support, and all reviews together still count as one source kind.
Package and resource inspection
We extract SDK signatures, permissions, billing components, trackers, and shipped copy from the app package. The binary is discarded after derived evidence is recorded.
Listing and disclosure checks
Store prices, subscription periods, privacy labels, and developer claims are compared with the app-side evidence without laundering reviews into a second source.
Corroborate, grade, and version
The fusion layer labels every finding by how much evidence is behind it and publishes it with that evidence, the app version and the rules version. The band row is computed from the same measurements each time the catalogue is exported.
A hands-on test is a separate tier, and the programme that would sell one is not yet open. A person runs our test protocol against the app on a real device and records measurements; code decides what they mean. A test confirms or contradicts findings and adds measurements that no artefact we hold can supply — whether cancelling actually works, what the paywall does on a real device. It does not change a band. What it measured is published in the app's report, under a dated “Human-reviewed” mark that has no end date; a contradicted finding stays on the page with both sides shown. Tests are run on free trials only and the auditor incurs no charge, so cancelling is tested only where a free trial allows it. A fee would buy the labour, never the outcome; no price is set, and nobody has paid us for a test. Every completed test is listed on the audit record, voided ones included, and we run tests at our own cost so the mark is never reachable only by paying. No human review has been completed yet: every measurement on this site today is automated.
The system is allowed to say “we don't know.”
Saying “Not assessed” is a feature. It is safer than turning missing coverage or model confidence into false absolution.
Models never judge
The mass path needs no model calls. When a model is used, it may veto a candidate, or report an observation whose quote we verify word-for-word against the app's own text and discard if it is not there. It never decides what the text means.
No clean result from a failed scan
A collector counts only when it observed the material it was meant to inspect. A failed package download never becomes “no trackers found.”
No paid path around the rubric
Corrections to objective facts are free. A hands-on test, paid for or not, enters as evidence under the same published rules, and it can go against the app as readily as for it.
A band is a measurement, not a recommendation
The bands measure fairness, not taste.
An app can be technically great and still treat you unfairly — and an app you'd never personally choose can still be scrupulously fair. They're not the same question. A band is a published measurement, not a recommendation or a quality rating.