Can search engines and AI assistants read my site?
Menso sends about 20 plain HTTP requests to your site for three checks. Do common addresses like /pricing or /login open, redirect or return 404? Do missing pages wrongly return 200 or redirect home (a soft 404)? Are your pricing page's prices in raw HTML or added by scripts? Do robots.txt or noindex block search and answer crawlers on your homepage and pricing page? The check is free, needs no account and uses no AI.
03Search and AI crawlersOAI-SearchBotPerplexityBotGooglebotBingbot
Let an AI user try it
Menso Lab
Two runs where the AI user typed /pricing
tiun.
tiun.io
TRACES 83.5/100
6/
8/pricing
/help-center
16docs.tiun.io/faq
Zainab scrolled the homepage for five steps without finding a pricing link, and at step 6 typed the pricing address herself. At step 8 she reached the Free vs Enterprise table, then read the help centre and the docs FAQ, and at step 16 decided to sign up for the free tier.
Review note: The actor never opened the top-right menu, where Pricing sits; this is not a claim that the site has no pricing link.
Product Hunt week 38 (Sep 14 – 20, 2026), #2 on Product Hunt's board · tested Sep 21, 2026 · one AI user, one run (n=1) · Deep test · 16 of 16 steps · tiun.io
AINA
aina-tech.io
TRACES 64.5/100
/candidate
78/faq
13/
14/pricing
Zainab scrolled the candidate page and opened the FAQ. At step 7 she expanded “What does it cost?”, and at step 8 she read the answer: $49 a month, which she took to be the company plan. At step 13 she typed the usual pricing address; at step 14 it returned a not-found page, and she decided not to sign up.
Review note: The run mixed the candidate and recruiter purchase contexts; the $49 figure describes the recruiter service, not the free candidate profile.
Product Hunt week 38 (Sep 14 – 20, 2026), #7 on Product Hunt's board · tested Sep 21, 2026 · one AI user, one run (n=1) · Deep test · 14 of 16 steps · aina-tech.io
About these runs
Both public runs are from the Product Hunt board. Menso's AI user typed the usual pricing address to find prices. On one site, it opened plans and chose to sign up for the free tier. On the other, it got a not-found page and chose not to. These examples do not show a rate.
Every run here used Menso's Purchase intent template: the persona Zainab Salazar, a freelance UX consultant, reads the homepage, the pricing page and the FAQ, decides whether she would buy or sign up, and stops before paying. It is a route-specific diagnostic, not a product-wide score.
Made one of these products? Write to support@menso.io and we will correct or remove the example as soon as we can.
What the three checks look for
Guessed addresses. Visitors type /pricing, /login or /docs instead of finding a link. The check requests 15 such addresses and one random address. Missing pages should return 404. Returning 200 or redirecting home makes them look real to search engines.
Prices in the page. The check finds your pricing page and reads its HTML without running scripts, like many AI assistants. If prices need scripts, an assistant asked what you cost may find none.
Crawler access. The check reads robots.txt using standard crawler rules and looks for noindex on your homepage and pricing page. It checks OAI-SearchBot, PerplexityBot, Googlebot and Bingbot. Blocking a crawler can keep those pages out of its search results or answers. It also checks ChatGPT-User in robots.txt, though OpenAI says those rules may not apply to user-started visits. Blocking training crawlers like GPTBot is up to you; we show their access for reference.
What this check cannot see
Menso checks from outside as MensoSiteCheck. If your firewall or CDN blocks crawlers by IP, it may treat real crawlers differently. The check cannot see this, so check your CDN's bot settings.
The check does not run scripts. A page built entirely in the browser can look the same at every address. The result says this instead of guessing which pages exist.
Results are kept for 24 hours. Check the same site again within a day and you'll see the saved result and its time.
The check cannot tell whether visitors can sign up or buy. An AI user test can.
Keeping your site out of this check
Add a MensoSiteCheck group with Disallow: / to robots.txt. The check reads robots.txt first and stops at this rule.
Questions people ask
What is a soft 404?
A soft 404 is a page showing "not found" without returning HTTP 404. Google recommends 404 for missing pages (Search Console help). Google does not index URLs returning a 4xx code and removes those already indexed (Google's crawling documentation). Redirecting missing addresses to your homepage hides the same problem: visitors and crawlers never learn the page does not exist.
Does blocking GPTBot keep my site out of ChatGPT answers?
Blocking only GPTBot does not remove your site from ChatGPT search. GPTBot crawls content that may train models. ChatGPT search uses OAI-SearchBot. Opting out of OAI-SearchBot excludes your site from ChatGPT search answers, though navigational links can still appear. ChatGPT-User may visit pages when users ask ChatGPT questions, and robots.txt rules may not apply (OpenAI crawler docs).
Why does it matter if prices only appear after scripts run?
Crawlers that do not render JavaScript may miss prices that need it. Vercel's December 2024 analysis found OpenAI, Anthropic, Meta, ByteDance and Perplexity crawlers did not render JavaScript. It says Google's Gemini uses Googlebot's infrastructure, which does. Crawlers can change (Vercel, The rise of the AI crawler).
What did this check find on other sites?
On November 11, 2026, we ran this check once on 516 sites posted to Show HN. 67 answered a made-up address as if the page existed. Of the 499 sites whose homepage answered, 39 named at least one AI training crawler in robots.txt and asked it to stay out. 4 of the 499 turned away a search or answer crawler. Of the 73 pricing pages the check could read, 65 had a price in the HTML text. These counts cover one sample of new projects on one day. They are not rates for the web. See every count and the method.
Is the check free, and will it slow down my site?
The check is free and needs no account. It reads robots.txt first and sends about 20 plain HTTP requests a quarter second apart. Each IP address gets 5 checks an hour. Sites checked within 24 hours return the saved result instead of new requests.