Star ratings vs receipts: what a 3.5/5 can't tell you about an AI app builder
More sites now score AI app builders, and their numbers look alike: stars, points out of 100, tiers. The methods behind them are not alike. Here is what four sites, ours included, say they actually do, and the six questions worth asking behind any score.
The short answer: a score is a summary, so check what it summarises. App Builder Index says its star ratings come from its own panel running 6 briefs 3 times per tool on paid plans. AI Form Guide labels every current score "Preliminary · Reviewed from docs". AI Coding Hub (coding tools) publishes the commands, version strings and dates it ran Verified, each site read 22 Oct 2026. VibeTierList builds one app per tool on free plans and publishes the screenshots and credits used, but our first test was one run per tool, so each result is a single observation. Before you trust any number, look for: tested or read, which plan, how many runs, when, evidence you can open, and who pays the site.
Six questions to ask behind any score
1. Was it tested, or read?
A score built from a vendor's docs and pricing page tells you what the tool promises. A score built from runs tells you what it did on the day. Both are useful, but they answer different questions. Look for a label that says which one you're reading.
2. Which plan?
Free and paid plans can behave differently: credit caps, models, features behind a paywall. A rating from paid plans may not match what you get on a free account, and a free-plan test may undersell a paid one.
3. How many runs, and how far apart?
AI builders don't give the same result twice. One run is one data point. Several runs with a median and a range tell you how much the result moves.
4. When?
These tools ship changes weekly. A score with no test date, or with a date months old, may describe a different product from the one you'd sign up for today.
5. Can you see what happened?
A 3.5/5 or an 82/100 compresses everything into one number. Screenshots, logs, the deployed app and what it cost let you check the number, and find the one bug that matters for your app even when the total looks fine.
6. Who pays the site?
Affiliate commissions, sponsor slots and paid reviews are all common and can be fine when they're labelled. Read the disclosure and check whether payment can touch the score.
How each site says it tests
Each cell is what that site's own pages said when we read them on 22 Oct 2026. This compares methods, not accuracy: we haven't checked anyone's scores, and none of these sites paid us or was paid by us. AI Coding Hub covers coding tools rather than app builders; it's here because it is the other hands-on site we track.
| Question | App Builder Index | AI Form Guide | AI Coding Hub | VibeTierList (us) |
|---|---|---|---|---|
| What it covers | 33 AI app builders | New AI tools, for builders and for everyday professional jobs | AI coding tools | 4 AI app builders tested so far (Lovable, Bolt, Base44, v0) |
| Hands-on or from docs | Hands-on: its own testing panel, two testers per tool | Every current score says "Preliminary · Reviewed from docs"; entries will be marked "Tested hands-on" once tested | Hands-on, on its own repositories | Hands-on: an agent builds the same app in each tool and checks it in a real browser |
| Tasks and runs per tool | 6 briefs × 3 runs = 18 runs per tool | None yet (docs review) | The same set of tasks for every tool; run count not stated on the page we read | v1: 1 task × 1 run per tool. v2 (results published 25 Oct 2026): 1 harder task × 3 runs, median and range |
| Plan tested | Paid plans ("free tiers behave differently") | Not applicable yet | "Paid where needed" | Free plans only, default settings |
| Score format | 0–5 stars on 10 weighted axes, one decimal | 0–100 on 7 criteria | Weighted score on 6 dimensions; "in progress" until tested | 0–100, tiers S/A/B/C from the v2 median; v1 scores carry no tier |
| What you can check on the page | Axis scores, a dissenting view, a landing-page screenshot, dated sources, raw axis scores as JSON (CC BY 4.0). Per-run logs: not yet; the site said on 5 Sep 2026 it is working on publishing them | Score breakdown per criterion, pricing with the date checked | Exact commands, the version string the tool printed and the date run | Screenshots of key checks, credits used before and after, time to live app, the deployed app URL |
| Dates shown | "Last verified" and "next scheduled check" (90-day cycle) on each review | "Checked" date on each tool page (15 Oct 2026 on the one we read) | "Last updated" and "last verified" dates; most recent pass 4 Sep 2026 | Test date on every result; stale after 90 days; date we last checked for changes |
| Money, as the site describes it | Buys its own subscriptions; no placements or paid rating changes; no affiliate links currently | Sells labelled sponsor slots; may earn affiliate commission; says the score never reads payment | No affiliate links, sponsorships or paid placements | Sells $39 paid tests (same task, labelled) and labelled sponsor placements; no affiliate links yet; payment never buys rank |
Sources, all read 22 Oct 2026: App Builder Index how we review, its Lovable review (4.2/5, last verified 20 Aug 2026; the 5 Sep 2026 reply about raw logs is on this page) and rankings JSON · AI Form Guide home, how we score, disclosure (updated 18 Oct 2026) and its v0 page (76/100, reviewed from docs, checked 15 Oct 2026) · AI Coding Hub how we test (updated 4 Sep 2026) · ours: methodology and disclosures (updated 16 Oct 2026). Sites change their methods; check the source before you cite it.
The pattern: the three sites that test hands-on (App Builder Index, AI Coding Hub and us) agree on the basics (same tasks for every tool, dated results, scores that payment can't move), and they differ mostly in depth and in what they show you. App Builder Index states the most runs per tool (18), on paid plans. AI Coding Hub shows the exact commands and versions. We show the screenshots and the free-plan credits from each run. A docs-based score, like AI Form Guide's current issue, is a different kind of number: a reading of what the vendor publishes, clearly labelled as such Inferred.
What a top score hid in our own test
In our v1 test (16 Oct 2026, free plans, one run each) Bolt's app scored 100/100. It also had edit and delete buttons that only appeared on mouse hover, so on a phone they were invisible Observed, 16 Oct 2026 · single observation (screenshot). A perfect total didn't tell you that; the receipt did. If your users are on phones, that one bug matters more than the score. The same is true of anyone's number, ours included: open the evidence behind the score that decides your choice.
All four v1 apps, with their screenshots and credits used: Lovable vs Bolt vs Base44 vs v0 · what one working app cost on each free plan.
Where our method is weaker
- One run per tool in v1. Every v1 result is a single observation, and AI builders vary from run to run. App Builder Index's 18 runs per tool is a much bigger sample.
- Free plans only. That matches what most people try first, but paid plans can behave differently, so our results may not describe a paid account.
- One fairly simple app, 4 builders. v1 couldn't separate the tools (95 to 100 each), and we haven't tested the other builders other sites cover.
- Agent-run, not a human panel. An agent runs the builds and the checks. That keeps every run identical, but no human designer or developer rates the result in v1.
- The fix: our harder v2 test (teams, roles, share links, search) ran each tool 3 times under a plan we published before any results, and reports a median and range. v2 results published 25 Oct 2026 (other sites' methods were read 22 Oct 2026; our own column was updated for v2 on 26 Oct 2026).
Honest limits of this page
- This compares what each site's own pages say about its method, read 22 Oct 2026. We haven't audited anyone's tests or checked their scores, so this says nothing about whose numbers are more accurate.
- "Not stated", "not yet" or "none yet" means the pages we read didn't show it; it may exist elsewhere on the site.
- We run one of the four sites in the table, so we have an interest. We've tried to list our own weak spots as plainly as anyone's method. No site in the table paid us or was paid by us.
Related
- How we test: our task, rules, scoring, evidence labels and freshness rules.
- App builder test v2: the exact prompt and checks, published before results.
- Security scan vs functional test: the same methods-only comparison for AI app scanners.
- The two-account test: check your own app in about 10 minutes.
- Disclosures: every way we make money. · All guides
Evidence labels: Verified = read on that site's own page on 22 Oct 2026 (links above). Observed = seen in our own v1 run on 16 Oct 2026. Inferred = our reading of the evidence. What the labels mean.
FAQ
What can a single star rating or score out of 100 not tell you about an AI app builder?
On its own it can't tell you whether the tool was actually used or rated from its docs, which plan was tested, how many runs the number rests on, when it was tested, what broke, or what it cost. Two tools with the same score can fail in completely different places. Check the method page and the evidence behind the number before you rely on it.
Are App Builder Index ratings based on hands-on tests?
Its methodology page (read 22 Oct 2026) says yes: its own testing panel runs six fixed briefs three times each per tool, from a clean account, with two testers, on paid plans, and scores ten weighted axes out of five stars. Panel ratings are labelled Editor panel and community ratings are shown separately, not blended in.
Is AI Form Guide's score a hands-on test?
Not yet. Its homepage (read 22 Oct 2026) says every score in its first issue is preliminary and reviewed from public documentation, and that nothing on the page has been through its hands-on test yet. Its scoring page says each entry will be marked either Tested hands-on or Reviewed from docs.
How is VibeTierList's method different, and where is it weaker?
We build the same app in each tool on its free plan and publish screenshots, credits used and the deployed app for each run. The weak spots: our first test (v1, 16 Oct 2026) was one run per tool on one fairly simple app, so every v1 result is a single observation; we test free plans only; we cover 4 app builders, not 33; and v1 scores (95 to 100) were too close to separate the tools. Our harder v2 test ran each tool 3 times under a plan published before results, and reports a median and range; its results were published 25 Oct 2026.
Do these review sites take money from tool makers?
By their own pages, read 22 Oct 2026: App Builder Index says it sells no placements and runs no affiliate links. AI Form Guide sells labelled sponsor slots and may earn affiliate commissions, and says payment is never read by its score. AI Coding Hub says it has no affiliate links, sponsorships or paid placements. VibeTierList sells $39 paid tests (same task and scoring, labelled) and labelled sponsor placements, has no affiliate links yet, and payment never buys rank.
Which AI app builder review site should I trust?
Use more than one, and read each for what it is good at. Look for a test date, the plan tested, the number of runs, evidence you can open and a clear money disclosure. Where two sites that test differently agree, that is a stronger signal than either score alone. If you're about to ship, test your own app too: the two-account test takes about 10 minutes.
How to cite
VibeTierList, "Star ratings vs receipts: what a 3.5/5 can't tell you about an AI app builder", published 22 Oct 2026, each site's pages read 22 Oct 2026. https://vibetierlist.com/guides/app-builder-review-methods/