Menso ran its free site check on 516 sites posted to Show HN in the six weeks to November 11, 2026. 67 answered a made-up address as if the page existed: a soft 404. 39 name at least one AI training crawler in robots.txt and ask it to stay out. 26 of them use a Cloudflare setting. 4 of the 39 also turn away a search or answer crawler. Of the 73 pricing pages the check could read, 65 had a price in the HTML text. These counts cover one sample on one day. They are not rates for the web.
Last updated
Method
We started from every Show HN story posted between September 30, 2026 and November 11, 2026, found through the Hacker News search API. Posts without a link were left out. So were links to a shared platform instead of a site of the project's own: code hosts, app and extension stores, package registries, social networks and publishing platforms. Each site counts once, and its highest-scoring story needed at least 5 points. That is 516 different sites. We fixed the rule before the first request. No site was added or dropped after its result came in.
Each site got the requests the free site check sends. It requests robots.txt first, then the homepage and one made-up address. Next come 15 addresses people guess, such as /pricing and /login, and the pricing page if the check finds one. That is about 20 plain HTTP requests, a quarter second apart, sent under the name MensoSiteCheck. No scripts ran. The check used no AI model.
No site owner asked for these checks, so the batch behaved like a crawler. It followed each site's robots.txt rules for MensoSiteCheck. If the file had none, it used the rules for all crawlers. It never requested an address those rules disallow. On 10 sites it stopped at robots.txt. 2 of them disallow every crawler. On the other 8 the file could not be fetched. The batch stayed out of those sites, as RFC 9309 asks a crawler to do when robots.txt is unreachable. 7 more homepages timed out or refused the request. The remaining 499 homepages answered.
Scripts did the counting. An AI agent then reread the robots.txt files behind the crawler counts. No person reviewed the sites one by one. We do not publish the list or name any site.
The check asks each site for one address that cannot exist. The right answer is 404.
| Answer | Sites, of 516 |
|---|---|
| 404, the page does not exist | 425 |
| Soft 404: the address answered as if the page existed | 67 |
| Could not tell: the request failed, was refused or timed out | 11 |
| Not requested: robots.txt disallows it | 13 |
41 of the 67 soft 404s are script shells: the made-up address returned a page with scripts and almost no text. On such a site the server sends the same small HTML file for any address, and browser scripts decide what to show. The check runs no scripts, so it cannot see whether they show a not-found message. A crawler that runs no scripts gets status 200 and almost no text. The other 26 answered with status 200 too. On 21 of them, most of the guessed addresses returned the same page as the homepage.
These counts cover homepage rules for the 499 sites that answered. The 2 sites that disallow every crawler are not in the table. The batch stopped at their robots.txt.
| Crawler | What it gathers pages for | Sites that disallow it, of 499 |
|---|---|---|
| GPTBot | Training AI models | 39 |
| ClaudeBot | Training AI models | 38 |
| Google-Extended | Training AI models | 35 |
| OAI-SearchBot | Search results and answers | 3 |
| PerplexityBot | Search results and answers | 4 |
| Googlebot | Search results and answers | 0 |
| Bingbot | Search results and answers | 0 |
39 sites disallow at least one training crawler. 26 have the block Cloudflare writes when its managed robots.txt setting is on. 13 name the crawlers in their own rules.
4 sites disallow a search or answer crawler. All of them are among the 39 that disallow a training crawler. Keeping a training crawler out is a site's choice. Turning away a search crawler can remove the site from that engine's results or answers. The check cannot tell whether a site intended that.
The file itself varies more than the rules do. 346 sites serve a robots.txt with at least one rule. 101 have no robots.txt. 31 serve comments without rules. 30 of those show the same Cloudflare notice. 21 answer /robots.txt with an HTML page. A crawler treats all except the first group as having no rules. 9 homepages carry noindex.
| What /robots.txt returned | Sites, of 499 |
|---|---|
| A file with at least one rule | 346 |
| No robots.txt | 101 |
| A file with comments and no rules | 31 |
| An HTML page | 21 |
People type an address instead of looking for the link, and so do AI users. The median site opens 1 of the 15 addresses the check tries. An address opens when it returns a page of its own. Redirects count unless they end on the homepage.
| Address | Sites where it opens, of 499 |
|---|---|
| /privacy | 169 |
| /terms | 133 |
| /about | 121 |
| /blog | 97 |
| /docs | 76 |
| /pricing | 70 |
| /contact | 60 |
| /login | 27 |
| /faq | 24 |
| /signup | 21 |
| /help | 11 |
| /signin | 7 |
| /register | 7 |
| /plans | 3 |
| /price | 2 |
Many of these sites are free tools and side projects with nothing to sell. No page at /pricing is often the honest answer. What matters is how the missing address answers: a 404 says so, and a soft 404 does not. On 18 sites, robots.txt disallows at least one guessed address. The check did not request those addresses.
The check looks for a pricing page at the guessed addresses and among the homepage's links. It reads the page's HTML without running scripts. Crawlers that run no scripts read it the same way. Vercel's December 2024 analysis found that crawlers from OpenAI, Anthropic and Perplexity ran no scripts (Vercel, The rise of the AI crawler).
| Result | Sites, of 516 |
|---|---|
| No pricing page found | 352 |
| Pricing page read: a price is in the HTML text | 65 |
| Pricing page read: a price is only inside the page's scripts | 6 |
| Pricing page read: no price found | 2 |
| Could not tell: different addresses return the same page | 57 |
| Could not tell: the pricing address leads to another site | 5 |
| Could not tell: the request failed, was refused or timed out | 16 |
| Not requested: robots.txt disallows it | 13 |
Of the 73 pricing pages the check could read, 65 show a price to a reader that runs no scripts. On 57 sites, different addresses return the same page: the homepage or a script shell. On those sites, the check cannot tell whether a pricing page exists.
Two public runs from the Product Hunt board include review notes. In both, the AI user was deciding whether to sign up. It typed the usual pricing address instead of following a link. One site answered with its plans. The other answered with a not-found page. The AI user then decided not to sign up. An AI user test shows what a visitor does next. The counts above show only how the address answers.
Every run here used Menso's Purchase intent template: the persona Zainab Salazar, a freelance UX consultant, reads the homepage, the pricing page and the FAQ, decides whether she would buy or sign up, and stops before paying. It is a route-specific diagnostic, not a product-wide score.
TRACES 83.5/100
Zainab scrolled the homepage for five steps without finding a pricing link, and at step 6 typed the pricing address herself. At step 8 she reached the Free vs Enterprise table, then read the help centre and the docs FAQ, and at step 16 decided to sign up for the free tier.
T 4 · R 4.5 · A 4 · C 4 · E 4.5 · S 4 (each out of 5)
Review note: The actor never opened the top-right menu, where Pricing sits; this is not a claim that the site has no pricing link.
TRACES 64.5/100
Zainab scrolled the candidate page and opened the FAQ. At step 7 she expanded “What does it cost?”, and at step 8 she read the answer: $49 a month, which she took to be the company plan. At step 13 she typed the usual pricing address; at step 14 it returned a not-found page, and she decided not to sign up.
T 4 · R 4 · A 2.5 · C 4 · E 2 · S 2.5 (each out of 5)
Review note: The run mixed the candidate and recruiter purchase contexts; the $49 figure describes the recruiter service, not the free candidate profile.
Made one of these products? Write to support@menso.io and we will correct or remove the example as soon as we can.
Limits
A soft 404 is a missing page whose status code does not say it is missing. The server returns 200 or redirects the visitor to the homepage. Google recommends returning 404 for a missing page (Search Console help). The free site check tests this with one made-up address.
In this sample, 39 of the 499 Show HN sites that answered disallow GPTBot on the homepage. 26 of the 39 sites that disallow a training crawler do it through a Cloudflare setting. 2 more sites disallow every crawler. This is one sample of new projects on one day. It is not a figure for the web.
No. GPTBot gathers content that may be used to train OpenAI's models. ChatGPT search uses a different crawler, OAI-SearchBot. OpenAI says each setting is independent of the others (OpenAI crawler docs). In this sample, 39 sites disallow GPTBot and 3 disallow OAI-SearchBot.
Most often, the site is an app that runs in the browser. The host is set to send the same HTML file for every address. The app's scripts decide what to show. 41 of the 67 soft 404s here returned a file like that: scripts and almost no text.
Enter your address in the free site check. It sends the same requests this run did and shows the result for your site only. You do not need an account.