How Google May 'Understand' Unique Content
While machines will never truly understand content the way some of us do, Google's link information gain
Thanks to Rand’s excellent research and Barry’s expletive-laden ranting, we know that Google processes over five trillion searches each year. Trillion. Per day that’s 13.7 billion. Per second, 158,000.
There are some sizeable and growing caveats here:
Only 66.61% of the subsequent 32% go to the open web
This figure is rising each year
That still means Google processes 2.92 billion clicks to the open web every day. It’s still a figure worth fighting for - particularly for publishers whose business models heavily rely on a click.
So let’s not totally lose sight of what matters in the here and now. And unique content certainly fits that mould.
I have reviewed a few previous patents (Google’s in-depth article patent explained and how google ranks news sites) and it is not a thoroughly enjoyable experience. A granted patent protects an idea; it doesn’t prove deployment or real world use cases - and it’s certainly not unlike big tech to claim ownership of something just so it can’t be used elsewhere.
Generally if;
The patent is cited regularly and recently? This patent (Contextual estimation of link information gain) has been cited 24 times and as recently as last year.
Whether it has international filings? Yes, but with some caveats. US, China, ceased in Europe and worldwide, but extended in the US to 2039 very recently.
Whether Google has protected the ranking technology around the world? Yes, again with some caveats.
Does it broadly align with your understanding of the concept (in this case non-commodity content)? Very much so. As the rasping breaths of SEO-first, commodity content make even iron lungs work hard, it would be inconceivable for Google to not measure or evaluate uniqueness in some manner.
It is more likely to be used in some capacity.
TL:DR
Google has multiple public and leaked systems that appear to evaluate originality, effort and unique contribution - see OriginalContentScore and ContentEffort.
The patent describes an information gain score (potentially in a 0 - 1 framing) that is assigned to a document based on how much new information it adds beyond documents a user has already seen on the same topic.
In mine - and many other’s - opinion, Google’s systems reward originality in some way. Whether that’s directly through an information gain score and re-ranking system, a Bayesian predictive score or indirectly through positive engagement signals I couldn’t tell you
Originality doesn’t mean an entirely different document. As little as 10% difference could be the delineator between marketing success or failure
How does it work in practice?
This patent is not about the information gain applied to the current set of search results. It’s about the subsequent set of results - ranking the next set of search results based on wider user search behaviour, personalisation and added document value.
It highlights that documents:
may be reranked
may be excluded
may be significantly demoted
may no longer appear in results
Based on the amount of novel, relevant information provided when compared to other similar documents.
For any tech SEO geeks out there, you’ll be well aware of the concept of preloading. In nerd circles, preloading tells browsers which resources should be prioritised to improve the page load speed and above the fold rendering.
I think this patent works in a similar manner, but with bloody unreliable people instead of machines. Maybe bfcache is a more apt comparison but I haven’t really got stuck into technical SEO for a while, so forgive me for my appalling analogies.
Step-by-step
A user reads a document about a certain topic - let’s say growing an apple tree.
Google understands that the majority of users don’t stop at one page here. It’s a rich topic. When should I plant one? Where? What do I feed it?
With 13 months of click and engagement data to hand, Google knows - with I imagine an unerring level of accuracy - what piece of content each user should be shown and when based on goal fulfilment.
But new content is written everyday. Pages are updated. So this isn’t a static corpus to work with. And maybe someone has a novel way of growing apple trees?
So pages are compared. A user reads a document (d1). Google then compares a new or updated article (d2) to the original.
If d2 generates a favourable information gain score, it will likely be shown to the user as part of their journey. If it doesn’t it’s fucking doomed.
"An information gain score for a given document is indicative of additional information that is included in the given document beyond information contained in other documents that were already presented to the user."
Let’s say two documents are chosen based on a user’s search and search history. They’re represented as d1 or d2. D1 is an already-consumed document and d2 is brand spanking new. Well, to the user at least. These documents can be represented as a vector (or some other semantic representation) to help the model fake understanding of the document and its position against similar documents.
The system provides a quantitative score to assess whether the user should also view d2 after having viewed d1. If the machine learning model generates an information gain score of document d2 over document d1, then d2 is likely to be shown - for future use cases, possibly at the expense of d1.
There are some incredibly practical implications here.
If a topic has been done to death, you have a more limited chance to rank and generate value without providing something extra. In a scenario where your article scores 0, the system has assessed it provides nothing extra and a user who has seen d1 is less likely to see d2 - your article.
If nothing else, make sure you stand out above your closest competitors in some manner.
A lot of this describes the foundations of creating brilliant content. Being different and standing out.
As with so many of these Google-led ideas or initiatives there are flaws. You don’t have to follow it to the letter. But EEAT and ‘information gain’ are sound principles. You have to be memorable. There is no alternative.
How important is it?
I think uniqueness and standing out is more important than ever. Strip the patent out of the conversation. People or brands who publish content won’t survive if they aren’t memorable to people and - by proxy - search engines.
So you’ve got to do something differently.
In Google’s case, I think it’s more about efficiency than anything else. If they know the information gain scores of two documents are virtually identical, then a user isn’t going to be shown both versions of the document. The second document will be deprioritised in favour of richer, more unique content.
Google has enough engagement data to go along with these proxy scores to understand what document should be shown and when. They can get a user closer to their goal by removing overly similar pages from a user’s SERP or AI response.
Which may be exactly why they’re thinning their index - the removal of non-value add content. Well, that and all the AI slop you’re creating.
It is quite literally down to a) computational resources (money) and b) getting the user to the point of completion quicker. In the DOJ Antitrust trial, Pandu Nayak's sworn testimony called Navboost “one of the important signals that we have.”
“…a shorter query session or fewer dialogue turns can provide a corresponding reduction in the resource demands of the system e.g. with respect to memory and/or power usage of the system.”
And the Quality Rater Guidelines make numerous references to effort, originality and talent. Frameworks like EEAT and the product reviews update really highlight the importance of actually using products and showcasing the effort you have gone to. The amount of ‘effort’ you put in is quite literally quantified (highly recommend Sean’s breakdown here). It is part of the Helpful Content update (booooooo) and the more difficult your page is to replicate, the better chance it has of success all things being equal.
These are not stupid principles. They’re very good ones. The problem is, effort is expensive. The fewer clicks content produces, the less each article will generate.
In an attributable manner at least.
Google is building an audience loyalty ecosystem
Don’t take my word for it, take Barry’s. Google has wanted to get rid of click-chasing churnalism for years. Now it can. And it is - in most cases I think, a positive.
They are trying to build something around engaged users - like every publisher out there. Your most engaged users are your most valuable. Google’s quietly building a subscriber ecosystem that could one day rival their ad business. No reason to think that
Publishers that can demonstrate they have an audience outside of SEO are being ‘rewarded.’ Although I suspect you could replace rewarded with crushed a little more slowly.
You can follow your favourite publisher on via Preferred Sources and as a Search Profile via the Discover feed (US-only at the time of writing this) and badges like ‘highly cited’ have been in play for some time. It doesn’t work very well, but they are trying to promote unique reporting.
You can now see how content from social and video platforms performs on Google Search if you meet the requirements. Your digital footprint and impact within the industry you’re in really matters. Particularly when you consider how prevalent social and creator accounts are in Discover.
I worry that this is completely impossible to explain what is happening to users. What the fuck is Preferred Sources vs a Search Profile?
It’s tough to force people to follow you on platforms - maybe that’s the point. Which I kind of understand - but I think one of these would’ve sufficed.
If you want to know a little more about where Discover is heading, I made a short video about it:
Does information density matter?
Yes and no. Long articles are not necessarily more effective at satisfying the user.
Google has methods to normalise the length of an article, to prevent additional keywords and semantically relevant phrases from ranking the document too highly. Factors like TF-IDF normalisation prevents long documents with high word counts from artificially inflating their relevance scores just because they’re quote unquote richer.
More detail may be the wrong phrasing here. Detail and rigour are typically positives. But it’s less important than answering the question and getting the user closer to their end goal.
User satisfaction is quantified through goal completions and Navboost data - it trumps everything else.
How does it affect AI systems?
Well, traditional search ranking is still crucial in AI systems - whether that’s how effectively you rank for the primary search, your inclusion in the training data, RAG or suite of fan out searches run concurrently. And AI searches are extremely personalised - something that’s likely to only increase over time.
When Claude starts knowing what toilet paper I buy or selects a poorly chosen ‘Happy Mother’s Day’ card for my mum’s birthday that showcases my lack of effort and empathy, it’s time to call it a day.
According to Kevin Indig’s latest excellent research, First-party research is rare in AI citations, but it earns 3.3x more. And original data is the strongest single predictor of page originality. Good for traditional SEO, good for AI search. Who knew?
The ideas described in this patent map almost too neatly onto how modern AI search systems retrieve relevant information. Of the SGE. It helps anticipate the user’s next interest in an assistant-like context. Personalised, ‘helpful’ and with extreme memory.
As Roger Montti pointed out, this may give a clearer indication of how AIOs use pages that the user in question may be interested in. Their entire job is to synthesise answers from multiple sources and searches to provide the perfect jumping off point. I suspect this scoring system is an excellent way to avoid computationally expensive, unnecessary utilisation of documents.
contentEffort - described as a ‘Large Language Model (LLM)-based effort estimation for article pages’ - estimates the amount of effort invested in creating an article. As slop makes up more than 50% of the internet, this is seemingly one of Google’s way of dealing with it.
How can I use this effectively?
Make differentiated, non-commodity content. It’s really simple. Apply what we call information gain in this context to your own content - if you cannot add anything of value to the existing index, then don’t bother.
You can use do this with:
Original data
First-hand experience
Interviews
Real reporting
Being first on the scene and developing the story as it happens
Proprietary analysis
You don’t need a big budget. You can do amazing thing with a few free data sources, some creativity and a bit of rope. Just make sure the article has an element of uniqueness.
I think this really helps frame whether content is still worth creating. If you’re doing something just for SEO reasons and you can’t add anything extra to the existing suite of information, kill it. If a document contributes very little new information, the patent suggests it’s a strong candidate to be deprioritised when selecting subsequent documents.
Still costs time and money to make, but is less and less likely to drive any real value. Stay in your lane, but drive a nicer car.
I have a feeling your indexation report in GSC is invaluable here. Beige content has a shelf life so low it’s in the running for the new UK Prime Minister. So check for any pages dropping out of the index at scale for more serious issues.