On October 26, Google updated its user-generated spam documentation. It added a section called "Only index content meant for Search," and widened the document's opening to cover "tools that let users publish shared pages, generated content, or web apps."
Search Engine Roundtable and Search Engine Journal both covered it as guidance for platforms. That reading is correct. It is also why nobody who should act on this will.
Google's definition of a platform is not the one you are using.
TL;DR
- Google's new test: if users can create shareable pages, generated outputs, or interactive tools on your site, decide whether that content belongs in search at all.
- Quote calculators, booking confirmations, shareable reports, saved carts and review forms qualify. You are a platform.
- Google put this in the spam document, not the crawl budget document. That placement is the signal.
- Google also published the pattern it watches: a small number of users generating a large share of indexable URLs.
- For pages already in the index, noindex and robots.txt are not interchangeable. Wrong order leaves the pages stuck there.
What Google Actually Means By "Platform"
Here is the test, nearly verbatim from Google's doc: if your service lets users create shareable pages, generated outputs, or interactive tools, consider whether that content is meant to be found through search engines or only accessed directly by people with a link. For pages made for direct sharing, collaboration, or utility, Google says to apply noindex or block crawling in robots.txt by default.
Read it again with your own site in mind. A quote calculator that produces a results page with a URL. A booking confirmation. A shareable report your lead magnet generates. A saved-cart link. A review form. An AI assistant that writes a transcript to a URL.
If any of that lives on your domain, Google is talking to you. You never filed yourself as a platform, so you have never audited yourself as one.
The Placement Is the Whole Story
Google could have put this in the crawl budget documentation, or in the URL structure docs it also updated on October 23. It did not. It went in the document about preventing user-generated spam.
Crawl waste is an efficiency problem. You lose some discovery speed and nobody gets hurt. Spam is a classification problem. It is about whether a system decides your domain is a source of low-value machine-produced pages, the same systems behind what Google's spam updates actually punish.
Thin generated URLs used to be untidy. Google just told you where they sit in its own filing cabinet.
The Detection Pattern Google Published Is the Useful Part
Buried in the monitoring advice is the sharpest line in the update. Google tells platforms to watch for spikes in newly published public pages, or a small number of users generating a large share of indexable URLs.
Google rarely describes the shape of what it looks for. It just did.
A small number of sources producing most of the indexable URLs is not only a spammer on a free hosting tier. It is also a programmatic SEO build. Three hundred city pages from one template. We are not saying those stopped working. Done with real differentiation, they work. But the burden of looking different now sits with you.
Should You Noindex User-Generated Pages? Start With the Ones Nobody Chose
Pull up the Page Indexing report on a typical WordPress site built in the last five years. You will find indexed URLs no human ever decided to make public. Tag archives for tags used once. Author pages for a contributor who wrote one post in 2022. Internal search result URLs. Filtered product views that mint a new address for every combination. Attachment pages for every image in the library.
Nobody approved those. A theme default did. Then the site grew, and the generated URLs outgrew the written ones.
That ratio is the number to look at this week. Not your rankings. The ratio.
Noindex and robots.txt Are Not Interchangeable
Google's new guidance lists either one for pages that were never meant for search. For pages already sitting in the index, the order matters, and this is where cleanup efforts go wrong. Google's own documentation says so plainly.
Block a URL in robots.txt and Google cannot crawl it, which means Google cannot read the noindex tag on it. In Google's words, "the crawler will never see the noindex rule." The page can still appear in results if other pages link to it, and it shows up in Search Console's Page Indexing report as "Indexed, though blocked by robots.txt."
So the sequence is: apply noindex via the robots meta tag or an X-Robots-Tag header, leave the URL crawlable until the page drops out, then block it in robots.txt if you want the crawl budget back. Reverse those steps and you keep the pages you were trying to remove, minus the ability to remove them.
Indexability Is a Decision, Not a Default
For twenty years the instinct was more indexed pages, more surface area, more chances to rank. In an AI search environment that is expensive, because every thin generated URL is a page an answer engine can crawl and use to form a read on what kind of site you run.
For shared and generated pages, Google just wrote down the better posture. Keep them out by default. Index on purpose.
Your indexable layer should be the pages you would defend in a meeting.
What To Do This Week
- Export the indexed URL list from the Search Console Page Indexing report. Count how many are tag archives, author pages, internal search results, attachment pages, or query-string variants. Our rule of thumb: if that group is over 20 percent of your indexed pages, you have a curation problem, not a content problem.
- Inventory every feature that creates a URL from user input. Quote tools, calculators, report generators, shareable carts, confirmation pages, review forms, chat transcripts. For each, answer one question. Would we want a stranger finding this through a search? If no, add noindex via the robots meta tag or X-Robots-Tag.
- Filter Search Console for "Indexed, though blocked by robots.txt." Every URL there needs the robots.txt rule removed first, the noindex applied second, and the block restored only after the page clears the index.
FAQ
Should I noindex my blog tag pages?
Usually yes, unless a tag page is a curated topic hub with its own introduction and a reason to exist. A tag archive listing two posts is a thin duplicate of content you already have indexed. Noindex it, and keep it crawlable so internal links still pass through.
Does this mean Google will penalize my site for having too many pages?
No, and nothing in the update says that. This is guidance about keeping pages out of the index that were never meant for search. The risk is not a volume penalty. It is that a high ratio of thin generated URLs makes it harder for any system to read your site as a quality source.
My developer says robots.txt is the fast way to clean this up. Is that wrong?
For removing pages that are already indexed, yes, and Google's documentation says so directly. A robots.txt block stops crawling, which stops Google from ever seeing a noindex rule. The page can stay indexed indefinitely. Use noindex to remove, then robots.txt afterward.
The Point
The old job was getting found. The new job is deciding what is worth finding. One is a volume problem you solve by publishing. The other is a judgment problem you solve by choosing what your domain stands behind, which is most of what good SEO services should be doing now.
Google moved this guidance into the spam document. That is a hint about which job it thinks you should be doing.
If you want a straight read on how many of your indexed pages you actually chose to publish, book a 30-minute SEO strategy call. No pitch, no audit upsell, just a look at your Page Indexing report and an honest answer about what is in there. Get a free strategy call.