Complete and utter shite if I’ve had a drink.

They all read like headlines you’d see in Discover and scroll past very fucking quickly.

Maybe in a few years that’ll be all that’s left and that’s what Googlers are prepping us for. Personally I think it’s far more likely their idea of quality, interesting content is a just a bit rubbish.

Marble vs grape juice - what a stupid title. Although interesting that they specify this is a video. Don’t hate the shoe one. No idea how that will make money for anyone however… Doesn’t matter to Google.

Anyway, here’s how I think you can create unique, interesting content that still drives actual value to your business. (Hint: it’s not about grape juice).

TL;DR

1. Commodity content is doomed for two reasons - it is easily summarised (because it has been done to death) and it doesn’t make (as much) money in a zero click world

2. If you are creating content just for SEO and have nothing unique to offer, stop. You are throwing money down the drain

3. Be more than an SEO. Help other teams structure their workflows to generate the maximum value from all channels with things like demand analysis

4. Google calculates the uniqueness of a document using a custom ‘information gain’ score at a query and document level

Subscribe now

Why commodity content is doomed

People are like water. We take the easiest possible route. One that really doesn’t include clicking to find an answer, even if said answer is riddled with bullshit.

Commodity content - content that has been the bedrock of evergreen search strategies for years - can be very effectively summarised and synthesised by answer engines. So effectively, that people will be satisfied with said clickless search.

Direct from the greedy horses’ mouth.

This means we have to focus our efforts elsewhere.

We have to focus our time and efforts on content more likely to drive legitimate value. Content that cannot easily be summarised by AI, adds something of real value to the user and hasn’t already been thrashed to death by savvy SEO teams.

You do not have the time, money or resource anymore to spend time on content that doesn’t drive value.

Does this mean things like search volume are useless?

At an individual keyword level, search volume has been declining in value for a long time. We just can’t generate the value we once could and it isn’t coming back.

But search volume just indicates demand. If you’re savvy and use monthly data, you can help content, social, paid marketing and editorial teams understand when users really care about a topic.

In this capacity, your job is to help teams understand when to create or update content, what that content should cover and crucially, why it’s spiking in search at this particular time.

If we take searches for family holidays in Google Trends as an example, there is clear and obvious consistency. Searches spike every January as people plan their family holidays for the year ahead in the bleak midwinter.

So you should still get your core family holiday content ready for January. But as we shouldn’t operate in a silo, you should share this with social and travel teams so they know what time of year this type of content will generate the most value.

Planning and structure take centre stage.

It is no longer about create x, get y. That click-based marketing is dead.

Commodity or not commodity

Loosely this header was a Shakespearean-based to be or not to be joke, which is a) clunky and b) outside of my wheelhouse.

Now I’ve had to explain it.

I wrote about this in how to do evergreen content in 2026 and beyond. Which is, ironically, quite a commodity topic. But it has evolved. There’s new shit to share. You can make commodity, non-commodity.

But you need to have a level of understanding and expertise that can really elevate a topic. That requires experience, a level of uniqueness and a platform. Your content needs to be found and what we have always done in search is unlikely to be anywhere near as valuable.

Subscribe now

The pillars of non-commodity content

• Uniqueness

• E-E-A-T

• Engagement

• Structure

Uniqueness

Uniqueness is the bedrock of everything when it comes to content that will continue to drive value. Without uniqueness, there’s no E-E-A-T. You won’t generate any shares, likes, comments or links. Certainly not any good ones.

You can make this as fancy as you like.

If you’re lucky enough to have access to high-quality data sources like Similarweb, you can create some truly brilliant proprietary metrics that elevate your content above and beyond.

Let me give you an example.

Similarweb give excellent engagement data at a site level. App-level too. If I was to combine these three metrics (pages per session, session duration and bounce rate) I have a composite engagement score.

Something no one else has.

If I took that engagement score and correlated it with third party traffic data or something like branded search/backlinks, I could correlate engagement data with traffic from search over time.

This is what stands out. This is what audiences will read, share and crucially, remember. It requires more effort.

And as we know from the Google Leak (this brilliant warehouse from Daniel Foley Carter is superb), effort is quite literally estimated and scored by Google. Things that are difficult to replicate are rewarded.

Unless they’re absolutely insane. Then probably the opposite.

You don’t get good at this overnight. But Google has been prepping us for this for some time. If you look at the declining youth engagement in the above graph, maybe people have too.

Not everyone is fortunate enough to have access to Similarweb data. But that doesn’t matter. Creativity and quality research is more important (and more readily available) than ever.

There are so many quality free data sources - Google Trends (combined with Glimpse), Keyword Planner, some free plans on tools like Ahrefs or Similarweb etc. You just need to identify metrics and combine them to make something bigger and better.

Google attempts to quantify information gain

Google has a patent (US20200349181A1) called Contextual estimation of link information gain that shows how the search giant may score the added value each document provides when compared to other similar documents.

Patents aren’t absolute. Just because a patent is present, it doesn’t mean it is always in use. If they’re frequently cited, recently updated and have worldwide applications, that’s usually a very good sign they have a level of importance.

But ‘ranking factors’ aren’t absolute either. SERPs and topics are vastly different. It’s why we have subtopics like local SEO, YMYL et al.

What matters for one term or topic may not matter as much, if at all, for another. It’s the nuance of the job and why trial and error is so important.

You don’t know until you know.

Consider the Four E’s

Your content needs a purpose.

Yes it needs to convert. That is a business purpose. But it needs a purpose for people. Is it designed to entertain? Educate? As audiences turn away from news (and probably more widely commodity content), this matters more than ever.

What we now term as commodity content was never designed to do any of the above. It was just designed to make money. Over the years, anything shit propped up by Google just to make money has died.

This is the next cab of the rank.

E-E-A-T

E-E-A-T has taken a bit of a kicking recently. Not without reason.

The premise is sound. Not unreasonable for readers to expect the author to be, you know, a real person, who knows something and has some kind of online presence. And Google absolutely does track authorship and entities. Plenty of evidence of that.

Google has built and maintained its Knowledge Graph for decades and entities have been the bedrock of news SEO for years. But E-E-A-T requires you to join the dots. To remove ambiguity - something we call disambiguation.

Doesn’t mean doing this is incredibly valuable, but it’s foundational. Particularly in this modern day iteration of the internet.

Remember, E-E-A-T projects have to add value

The problem with the whole - use experts, showcase expertise, prove you test everything, create video, make an effort in the industry etc - is now twofold;

1. It’s fucking expensive.

2. And less valuable than ever.

Having that person build some kind of profile in the industry. A platform that their content can be shared from and that reduces reliance on search can only be a good thing.

A moat if you will.

If they’re a legitimate expert on the topic, know how to structure great content and effectively showcase expertise, then you’re onto a bloody winner.

Which is why commodity content is doomed. Because people don’t care about it and now it doesn’t drive value.

Subscribe now

Engagement

I have always felt that links should be a happy byproduct of creating and sharing brilliant stuff.

I’ve never made an effort to build links. I have just made an effort to write stuff I think is interesting, made some semi-libellous jokes and got out there in the industry.

That is, more or less the Google definition of link building. In their world of sunshine and sweet smelling farts, links are just earnt by doing beautiful things. I am, in this scenario, the poster boy for white hat SEO.

The problem is, people need to make money and links still drive rankings. So there’s a market there. And if you’re a student of the scriptures like I am, you’ll know the buying and selling of links is the oldest recorded job.

Or was that prostitution?

Either way my inbox is full.

Anyway, your content has to fulfill a need. We’re moving away from straight laced content being able to do that for you as a publisher. Traditional ad revenue and the volume model sucks and you sure as hell aren’t going to drive any subscriptions with what time is x or how to tie your shoes.

I really hope this is a good thing for SEOs and publishers. I want us to focus on content that really makes a difference to people’s lives. Content that makes them smile or think.

Content that makes people angry has been a big hit when it comes to numbers for a long time. But I don’t think anger is the emotion you should shoot for.

Measurement

You need to measure quality engagement, on and off-site. That means;

On-site

No need to overcomplicate it for now.

• Session duration

• Bounce rate

• Link clicks

• Pages per session

• Comments

• Read time

Off-site

Very much depends on the platform and the purpose, but I would focus on;

• Links

• Shares

• Comments

• Saves

• Watch time

You need to track metrics that tell you clearly whether people truly care about what you are creating. Clicks are dying, so I’d rather be measured against something a) more valuable and b) less miserable.

Create a composite metric(s) that gives you and your creators something to clearly focus on. Make their job easy by guiding their content with simple, straightforward metrics. Metrics that don’t just chase page views.

Structure

Structure’s not sexy. Let’s be honest.

But it matters. If for some reason you think LLMs are the zenith of society and content consumption, then you should know that models are more likely to cite or reference content from the top or bottom of the page, thanks to their inability to properly follow an argument.

This is known as the lost in the middle effect.

Unless of course, the entity and topic is repeatedly referenced throughout.

I shouldn’t have to tell you that this is a shit idea and your content will become unreadable to living, breathing people.

But maybe you don’t care about that anymore.

Proper structure really matters. People have expectations (and accessibility needs). In more traditional commodity content they want their question answered immediately. If you satisfy that - and the intro to your article isn’t abysmal - you might generate a longer session, a click or hell maybe even a conversion.

Theoretically, non-commodity content accessed via search should still be intent-driven. Possibly more so if we’re to believe the more qualified users with longer tail queries theory Google espouses.

So you still need to follow a similar, highly coherent page structure;

• Answer the fucking question

• Some form of TL;DR article summary

• Argument

• Concluding thoughts

• Coherent FAQs (if applicable)

One that logically answers queries in the appropriate format - text, video, image, list etc - and is highly consumable.

The argument section is where LLMs tend to lose their ability to accurately and appropriately cite and reference content. Which is not at all dissimilar to people.

I am not saying you need to continually refresh and restate the entity in question. That may be construed as keyword stuffing. It needs to read well for people. But you need to be clear, concise and accurate to make consuming your content simple.

Don’t people consume content in different ways?

You’re absolutely right my pedantic friend, they do. Broadly, I think there are four types of consumption;

1. Scanners - The vast majority. Too lazy or illiterate to read the whole thing, but will be satisfied from a headline, bold text, bullet points, and headers. They treat a page like a map, not a story.

2. Answer seekers - They find what they want and leave. But still leave satisfied.

3. Visual/audio consumers - A cohort that either refuse to or cannot read, but will stare at a pretty picture for 60 seconds.

4. Deep readers - A small cohort, but a deeply engaged one, desperate for you to get something wrong.

I suspect these groups cover >90% of people. There are also fact-checkers - who skip the narrative and head straight for the citations, data points, or the “About Us” section before deciding if the content is worth their time.

And community-readers, who scroll to the bottom of the article to see the community reaction before deciding whether the content is worth their time. This is (obviously) more of a social trait. Particularly from younger audiences.

Your content can and should satisfy all of these people. It must;

• Answer the question

• Be highly scannable

• Broken up with clear, distinct headers

• Form a concise, easy to follow narrative

• Be highly scannable

• Easy to share

• Visually appealing (audio and video options available)

• Cite sources and clearly explain your methodology if appropriate

Anyway, that’s it. Thanks for reading as always!

Other good stuff

Some good stuff I’ve enjoyed recently;

• The Whole Point was the Mess - Pedro Dias

• It Works Until It Doesn’t: AI Content Strategies That Backfire - Lily Ray

• The New York Times is becoming subscription infrastructure for other publishers - The Media Stack

Leave a comment

Google’s market share has broadly held firm in the wake of everything AI. By held firm, I mean its share price has gone through the fucking roof and its AI offering is growing ever stronger.

But I don’t think all is as rosy as it seems.

Google’s search product isn’t addictive - as much as they’re trying to change that. Nobody hangs out there except saddos like us. And audiences - particularly younger ones - have options.

They’re turning away from more traditional methods of information retrieval and that’s a big fucking problem. Even for Google.

Even the search engine giant isn’t immune.

Older audiences - those already ingrained in the system - are taking up a larger percentage of their audience. The younger ones have more exciting and addictive options and best believe they’re using them to find shit.

Across every engagement metric, 18–24 year olds have deteriorated faster than 65+ users over the same period. Shorter visit duration, fewer pages per visit and a worse bounce rate. And it’s declining more rapidly with younger audiences.

Evolution for Google and the wider web is a necessity.

Although interesting to note that the 18-24 year old audience share has only suffered a small decline according to Similarweb data. The real losses were in the 25-34 cohort.

TL;DR

1. The publishing industry and Google have more in common than perhaps either of us care to admit.

2. The changes Google have made are a very deliberate effort to engage with - and retain - younger audiences. Audiences who behave differently.

3. Engagement data on news websites (pages per visit, bounce rate and time on site) declines with audience age. Exactly the same is true of Google.

4. AI Mode is Google’s attempt to create a ‘sticky’ product. One aimed at younger audiences.

Subscribe now

What’s changed?

Well, the obvious;

• Google’s started answering virtually everything in its search interface

• AI Mode is becoming something of a default response

• Personalisation is running amok

• It’s harder to generate clicks - reach and more traditional KPIs are less valuable

• Consumer decisions are made before someone reaches your website

• The agentic web is somewhere on the horizon

• Discover has become less publisher-friendly and more social and creator-led

• An increase in video and UGC in the SERPs (obviously all to YouTube and Reddit)

Just look at the SERP for almost any term, particularly middle of the funnel comparison ones.

What people apparently want is not very publisher, or legacy-search friendly. What they want, is video.

Particularly the bloody youth.

• 18-34 year olds spend 88 minutes a day on YouTube alone.

• Data from the UK market suggests children are averaging 2 hours and 14 minutes on TikTok per day.

The younger you are, the more time you spend watching, the less you spend reading. So the obvious counter (from a company who primarily organises written content) is to saturate the market with video content.

Obviously, it’s very fucking helpful if you own the market.

And this doesn’t just affect organic search. Adverts are more expensive to run because AIOs have destroyed the entire search ecosystem’s CTR. So for almost all businesses, customer acquisition is more expensive.

You could say that’s Google’s way of paying for AIOs - a far more expensive SERP to generate - due to the massive computational power and energy needed to run large language models (LLMs).

But I am not going to insinuate anything of the sort. It would be incomprehensible to me that the guys who earn the entire ad and search market would make the ad side of the business more expensive to run to pay for their search experiments.

Wait a minute…

Why now?

I think this is a direct response to two things;

1. The 2023 Code Red Google sent out in response to OpenAI

2. Younger audiences shifting information retrieval methods

One is obvious.

OpenAI forced Google to move quicker than they would’ve liked. Hence all the absolute shite in AI Overviews in the beginning. Well, and sort of now. It smacked of a product that hadn’t gone through the required amount of rigorous testing.

Two is more nuanced.

This data correlates almost perfectly with the Similarweb data I pulled. In isolation, this may not be a problem. Could be as simple as saying younger audiences will grow into it.

But I don’t think that argument works. We see it in news and publishing. We are living through it and we’re watching the decline in real time.

Younger audiences have the highest recorded screen time on record (globally, 7h 22m), but are spending less and less time reading. More on far more visually engaging, stimulating and addictive technologies.

Based on screen time alone, younger audiences should spend the most time on Google. But they don’t. I’m sure that is blatantly obvious to the Googlers.

Whilst content consumption is at an all-time high, the way a person consumes content is not conducive to more traditional publishing practices.

Just 4m a day on news websites for younger audiences vs 18m for the over 55s. A 350% increase.

The same principle is true of more traditional search.

At the risk of sounding a bit too AI-y, this is a really seismic shift. Ironically not one driven by AI. Not entirely. One driven by a combination of big tech’s insatiable appetite for money, a lack of trust in more traditional brands and the rise of the creator ecosystem.

And AI, obviously.

As someone in the comments said, Google is unc. Maybe a little like news websites. Their ability to attract younger audiences has diminished.

I think we can clearly correlate the changes Google has made to the reduction in younger audience share for publishers. A generation less inclined to click.

One could argue that the traffic losses so many seem to have suffered are almost exclusively from younger audiences. I certainly am.

Audiences more likely to adopt new technologies - particularly flashy ones.

Subscribe now

There are clear parallels between news and search

Google has got richer, as has the AI bubble. All that money has to come from somewhere.

It’s everyone else who struggles.

These changes are designed to counter a younger generations shift towards people and ultra engaging platforms that encourage passive or more incidental methods of information retrieval.

Since 2015, interest in news has declined - more significantly (43%) in 18-24 year olds than any other age group. And just 64% of 18–24s consume news on a daily basis, compared with 87% of people 55 and over.

Historically, news has been sought out.

Either you browsed a news website (a real paper if you felt fancy) or you searched for it. But the discovery layer changed and search - the engine that powered the volume-driven publishing model for two decades - is responding.

Responding to younger audiences’ shifting consumption habits. Just like publishers and websites will have to.

Passive consumption is just the norm now with younger audiences. This is why 44% of 18-24s see social media as main source of news, compared to just 15% of 55+.

They expect you to just appear. Algorithmic consumption has reduced the need, want and desire to actively seek something out. If what you serve isn’t delivered directly to their feed, you don’t exist.

Combine this with diminishing trust in more traditional brands, zero click searches and the rise of the creator and you can see why publishers and Google are having to change.

There have been alternatives to Google when it comes to accessing and retrieving information- Instagram, Amazon, YouTube et al - for years.

Really this is, or has been, Search Everywhere Optimisation. It has been around for a decade. It is also, IMO, why reframing SEO as GEO or some other bullshit because of LLMs is so moronic.

And now the individual has become the competition. The creator economy - soon to be worth $480 billion - has produced a new class of competitor: individuals with direct audience relationships, authentic voices, and none of the structural cost of a legacy newsroom.

51% of 18-24s pay attention to creators and personalities, compared to 39% who pay attention to traditional media and journalists - a 12 PP inversion.

And this is a problem for Google too. People used to use their organisational skills to satisfy all of their needs. Now it is so heavily navigational that it's hard to know how much ‘new’ stuff people really use it for.

Outside of news at least, ironically.

Will this work?

If it’s anything like news publishers, their primary concern is to continually generate new and engaged audiences with habitual products. AI Mode could absolutely be that product. Discover is their version of a social network. They are, in their own way, engaging products.

Although the low intent nature of Discover makes the advertising shit and Google not really care about it. Sad, but true.

Like Google, the engagement data for publishers tells a pretty bleak story.

If we isolate this to the youngest and oldest audience, it’s pretty clear what is going on.

Younger audiences;

• Are far less engaged with the traditional news offering than older audiences

• Use these (and any) websites differently

There’s no denying that younger audiences have more diverse and engaging options. This means they use websites like news publishers differently. To fact-check. To confirm something isn’t just spurious bullshit. To scan and skim.

The same is true of Google. Less of a discovery journey. More one of fact checking and navigational searching.

Subscribe now

So, what’s the answer?

To lean into what the new generation like. Adapt and evolve.

The same’s true for search (internally and externally) and publishers. If you work for Google, makes complete sense you would try to expand your video presence in the SERP and prioritise ‘quality’ UGC.

The quality part is lacking as most of the internet - as we’re finding out - is a stinking pile of shit.

But notoriously, the tide is tricky to swim against.

For publishers, it means working with creators, leveraging their audiences and ability to deliver things quickly. Differently. And creators can benefit from the trust associated with proper news organisations.

Is it that unreasonable to think Google should do the same?

Instead of abusing their position, they could start by giving people an idea of the impact of AIOs and AI Mode. I’m not a financial guru, but I reckon Google has enough money to build and foster creator and publisher programs that are not one-sided. That bring genuine value to people and the wider information retrieval ecosystem.

In this scenario, everyone benefits. When AI companies refuse to pay for publisher content, everyone loses.

• LLMs lose because they have less unique, human-created, quality content to train on

• Publishers lose because they are forced to suppress their visibility and don’t get any money

• Users lose because the end output isn’t as good

Model collapse is on the horizon. AI learning on AI falsehoods. A repetitive cycle of shit. Joyous.

These companies should invest in the ecosystems that built them. Particularly Google.

For publishers,

1. Build owned channels. Get the fuck away from relying on big tech

2. Create brilliant, unique journalism

3. Supplement it with habit forming products - puzzles being the obvious example

4. Build and sponsor audio and video programs that reach your intended audience

5. Implement channel-specific strategies

Even the NYT don’t rely solely on subscriptions from written content. Not by a long shot. It isn’t enough.

Final thoughts

Unfortunately I think the recent spate of job losses in the publishing industry is just the beginning. Bauer, the BBC, The Washington Post. It’s not UK or SEO-specific. 100k roles are becoming 70k ones. Teams are shrinking. And there are real world ramifications.

We are not in a good moment. Some of this can be attributed to AI. But I think more of it is due to longer-term economic difficulties, audiences switching off from traditional news and things like the Site Reputation Abuse update destroying much needed revenue lines overnight.

It is fucking hard to make these businesses profitable. Google doesn’t have that problem. But they’re not immune to changing behaviours and becoming yesterday’s news either.

Should you be enough of a psychopath, you can follow the job cuts via this updated Press Gazette article.

Leave a comment

AIOs have eroded clicks. Answer engines have given people options. And to be fair, people are fucking bored of the 2,000+ word article answering ‘what time does x start.’ Or recipes where the ingredient list is hidden below 1,500 words about why daddy didn’t like me.

In response to this, publishers say it will be important to focus on more original investigations and less on things like evergreen content (-32 percentage points).

So you’ve got to be smart. This has to be framed as a commercial decision. Content needs to drive real business value. You’ve got to be confident in it delivering.

That doesn’t mean every article, video or podcast has to drive a subscription or direct conversion. But it needs to play a clear part in the user’s journey. You need to be able to argue for its inclusion:

• Is it a jumping off point?

• Will it drive a registration?

• Or a free subscriber, save or follow on social

More commonly known as micro-conversions, these things really matter when it comes to cultivating and retaining an audience. People don’t want more bland, banal nonsense. They want something better.

The antithesis to AI slop will help your business be profitable.

TL;DR

1. If you’re creating evergreen content ‘just for SEO’ that is, IMO, a bad sign. It needs more strategic thought than search volume + clicks.

2. Prioritise micro-conversions. Evergreen content may not drive direct conversions, but it should help users become habitual somehow.

3. People are tired of low-quality, low-effort slop. Standing out is crucial. You have to add value with information gain - which will increase production costs.

4. You have to sweat your content. Make it shareable. Connect with people. Build owned properties and on rented land. Drive unattributable brand demand.

Subscribe now

What’s the problem with evergreen content?

Inherently, nothing. It’s a foundational part of the content pyramid.

In most cases it’s been done to death and AI is very effective at summarising a lot of this bread and butter content.

Over the last 10 years it’s been pretty easy to build a strategy around evergreen content - particularly if you go down the parasite SEO route. Remember Forbes’ Advisor and the great affiliate cull?

The epitome of quantity over quality - it worked and made a fortune.

But I digress.

An authoritative enough site has been able to drive clicks and follow up value with sub-par content for decades. That is, slowly diminishing. Rightly or wrongly.

And not because of the Helpful Content shite. Google nerfed all the small sites long before the goliaths. Now they’ve gone after the big fish.

We have to make commercial decisions that help businesses make the right choice. Concepts like EEAT have had an impact on the quality of content (a good thing). It’s also had an impact on the cost of creating quality content.

• Working with experts

• Unique imagery

• Video

• Product and development costs

• Data

This isn’t cheap. Once upon a time, we could generate value from authorless content full of stock images and no unique value. Unless you’re willing to bend the rules (which isn’t an option for most of us), you need an updated plan.

Is it still worth it?

It depends.

You need to establish how much your content now costs to produce and the value it brings. Not everything is going to drive a significant conversion. That doesn’t mean you shouldn’t do it. It means you need to have a very clear reason for what you’re creating and why.

If particular topics are essential to your audience, service and/or product, then it should at least be investigated.

One of the joys of creating evergreen content has always been that it adds value throughout the year(s). A couple of annual updates, even relatively light touch, could yield big results.

Commissioning something quality in this space is likely more expensive. It needs to be worth it - it has to form part of your multi-channel experience to make it so.

• Unique data and visuals that can be shared on socials

• Building campaigns around it (or it’s part of a campaign)

• You can even build authors and your brand around it

• And if it resonates, you can rinse and repeat year after year

And this type of content or campaign can increase demand for a topic. You can become a thought leader by shifting the tide of public opinion.

For publishers and content creators, that is foundational.

Reframing value - a zero click strategy

Two broadly rhetorical questions:

1. Do you think in a world of zero click searches, clicks and reach are sensible tier one goals?

2. Do you want to be targeted against a metric that is very likely to go down each year?

I don’t - on both counts. We should want to be targeted on driving real value for the business.

Something like;

1. Tier 1: Value - core, revenue and value driving conversions

2. Tier 2: Registrations (and things that help you build your owned properties), links, shares and comments

3. Tier 3: Page views, returning visits and engagement metrics

Micro-conversions over clicks. We’re focusing on registrations, free or lower value subscriptions. Whatever gets the user into the ecosystem and one step closer to a genuinely valuable conversion.

Now, could a click be a micro-conversion? If you know that someone who reads a secondary article (by clicking a follow up link) is 10x more likely to register, that follow-up click could be a sensible micro-conversion.

This type of conversion may not directly drive your bottom line. But it forces you and your team to focus on behaviours that are more likely to lead to a valuable conversion.

That is the point of a micro-conversion. It changes behaviours.

You can tweak the above tiers to better suit your content offering. Not all content is going to drive direct tier one or even two value. You just need to have a very clear idea of its purpose in the customer journey.

How can we make sure it adds value?

If what you’re creating already exists, you better make sure you add something extra. You’ve got to force your way into the conversation and unless you can offer something unique, you’re (almost certainly) wasting your time IMO.

I’ll break all of these down, but I think (in order of importance);

1. Writing content for people

2. Information gain

3. Getting it fucking found

4. Creating it at the right time

5. Structuring it for bots

Subscribe now

Writing content for people

Everyone is obsessed with getting cited or being visible in AI.

I think this is completely the wrong way of framing this new era. Getting cited there, or being visible, is a happy byproduct of building a quality brand with an efficient, joined-up approach to marketing.

The more you understand your audience, the more likely you will be to create high-quality, relevant content that gets cited.

If you know your audience really care about a topic, that’s step one taken care of. If you know where they spend time and how they’re influenced, that’s step two. And if you know how to cut through the noise, that’s step three.

Really this is an evolution in SEO and the internet at large.

• Invest in and create content that will resonate with your audience

• Create a cross-channel marketing strategy that will genuinely reach and influence them

• Share, share, share. Be impactful. Get out there.

• Make sure it’s fucking easy to read, share and consume

Your content still needs to reach and be remembered by the right people. Do that better than anybody else and wider visibility will come.

Information gain

In SEO, we have a different definition of information gain than more traditional information retrieval mechanics. I don’t know if that’s because we’re wrong (probably), or that we have a valid reason…

Maybe someone can enlighten me?

In more traditional machine learning, information gain measures how much uncertainty is reduced after observing new data. That uncertainty is captured by entropy, which is a way of quantifying how unpredictable a variable is based on its probability distribution.

Events with low probability are more surprising and therefore carry more information. High probability events are less surprising and novel. Therefore, entropy reflects the overall level of disorder and unpredictability across all possible outcomes.

Information gain, then, tells us how much that unpredictability drops when we split or segment the data. A higher information gain means the data has become more ordered and less uncertain - in other words, we’ve learned something useful.

To us in SEO, information gain means the addition of new, relevant information. Beyond what is already out there in the wider corpus.

Google wants to reduce uncertainty. Reduce ambiguity. Content with a higher level of information gain isn’t only different, it elevates a user’s understanding. It raises the bar by answering the question(s) and topic more effectively than anyone else.

So, try something different, novel even, and watch Google test your content higher up in the SERPs to see if it satisfies a user.

This is such an important concept for evergreen content because so many of these queries have well established answers. If you’re just parroting these answers because your competitors do it, you’re not forcing Google’s hand.

Particularly if you’re still just copying headers and FAQs from the top three results. Audiences are not arriving at publisher destinations through direct navigation at the same scale. They encounter journalism incidentally, through social feeds, not through habitual site visits.

You’ve got to meet them there and force their hand. Understand how Google ranks evergreen content and add real value.

According to this patent - Contextual estimation of link information gain - Google scores documents based on the additional information they offer to a user, considering what the user has already seen.

Structuring it for bots

Bots, like people, need structure to properly ‘understand’ content.

Elements like headings (h1 - h6), semantic HTML and linking effectively between articles help search engines (and other forms of information retrieval) understand what content you deem important.

Whilst the majority of semi-literates ‘understand’ content, bots don’t. They fake it. They use engagement signals, NLP and the vector model space to map your document against others.

They can only do this effectively if you understand how to structure a page.

• Frontloading key information

• Effectively targeting highly relevant queries

• Using structured data formats like lists and tables where appropriate (these are more cost effective forms of tokenisation)

• Internal and external links

• Increasing contextual knowledge gain with multimedia (yes, Google can interpret them)

The more clearly a page communicates its topic, subtopics, and relationships, the more likely it is to be consistently retrieved and reused across search and AI surfaces. This has a compounding effect.

Rank more effectively (great for RAG, obviously) - feature more heavily in versions of the internet - force your way into model training data.

If you need to get development work put through, frame it through the lens of assistive technology. Can people with specific needs fully access your pages?

As up to 20% need some kind of digital assistive technology, this becomes a ‘ranking factor’ of sorts.

Starting or steering the wave

I won’t go through this in much detail as I’ve written a really detailed post on it. Basically;

• Track and pay very close attention to spikes in demand (Google Trends API being a very obvious option here)

• Make sure you’re adding something of value to the wider corpus

• If quality content is already out there and you have nothing extra to add, consider whether it’s worth spending money on (SEO is not fucking free)

Whilst this is primarily for news, you can apply a similar logic to evergreen content if you zoom out and follow macro trends.

Creating it at the right time

Evergreen content still spikes at different times throughout the year. Take Spain as an example. There’s much more limited interest in going to Spain in the Winter months from the UK. But January (holiday planning or weekend breaks) and summer (more immediate holiday-ing with the kids) provide better opportunities to generate traffic.

You’re capturing the spike in demand by updating content at the right time. Particularly if you understand the difference in user needs when this spike in demand happens.

• In January, get your holiday planning content ready

• In the summer, get your family-friendly and last minute holiday content up and running

Demand for evergreen topics can be cyclical. In this example you would want to capture the spike(s) with carefully planned updates, so you have up-to-date content when a user is really searching for that product, service or information.

Subscribe now

How do I know what to create?

Well, what matters to your brand and your users? Have you asked them?

By the very nature of new and evolving topics and concepts, not everything ‘evergreen’ has been done.

New topics rise. Old ones fall. Some are cyclical.

My rule(s) of thumb would be to establish;

• Is the topic foundational to your product and service?

• Does your current (and potential) audience demand it?

• Do you have something new to add to the wider corpus of information?

If the answer to those three is a broad variation of yes, it’s almost certainly a good bet. Then I would consider topic search volume, cross-platform demand and whether the topic is trending up or down in popularity.

Is it ‘just for SEO?’

There are some things you should be doing ‘just for SEO.’ Content isn’t one of them. You can yell topical authority until you’re blue in the face. If you’re creating stuff just for SEO - kill it.

IMO these plays have been dead or dying for some time. The modern day version of the internet (in particular search) demands disambiguation. It demands accuracy. Verification that you are an expert. Otherwise you’re competing with those who have a level of legitimacy that you do not.

Social profiles, newsletters, real people sharing stories. You’re competing with people who aren’t polishing turds.

If all you’re thinking about is search volume or clicks I don’t think it’s worth it.

Video still sells

YouTube and TikTok are fucking flying. The young mind cannot escape big tech’s immeasurable evil.

They’re bored of reading the news, but they really, really like video. They will watch it.

The good news for you (and me), is that platforms like YouTube are still very viable opportunities to build something brilliant. Memorable even. They’re also far more AI-resilient - even if Google desperately tries to summarise fucking everything with AI.

Build on rented land

And this brings me nicely onto rented land. Platforms you don’t own.

We’ve spent years creating assets (your websites) to deliver value in search. Owning all of your assets and prioritising your site above all else. But that is changing. In many cases, people don’t reach your website until they’re already made a purchasing decision.

So you have to get your stuff out there. Create large, unique studies. Cut them into snippets and short-form videos. Use your individual platform to boost your profile and the content’s chances of soaring.

This is IMO, particularly prescient for publishers. You’ve got to get out there. You’ve got to share and reuse your content. To make the most of what you’ve created.

Sweat your assets. Even if senior figures aren’t comfortable with this - you need to make it happen.

People have been espousing how important it is to feature as part of the answer. And that may be true. But you’re going to have to be fucking good at selling your projects in if there’s no clear attribution or value.

Share. Share. Share.

It might not have the spikes of news, but evergreen interest still spikes at certain times in the year.

Get people - real people - to share it. To have their spin on it.

Outperform the expected early stage engagement and maximise your chance of appearing in platforms like Discover with wider platform engagement.

You have to work harder than before.

I shared an example of this around a year ago, but to revisit it - I now have 11 recommendations from other Substacks.

They have accounted for over 40% of my total subscribers. Admittedly mainly from Barry, Shelby and Jessie. But they are, if I may be so bold, superhumans.

And when our main driver of evergreen traffic to the site (Google) has really leant into the evil that surrounds big tech, we’ve got to be cannier. We have got to find ways to get people to share our content.

Even evergreen content.

It’s not great - but we needed a bit of a reset

If we’re being honest, a lot of SEO content has been rubbish. Churned out muck.

People are still churning out muck at an incredible rate. When what you’ve got is shit, more shit isn’t the answer. I think people are turned off. They’re tuning out of things at an alarming rate - especially young people.

It is all about getting the right people into the system. Evergreen content is still foundational here. You just have to make it work harder. Be more interesting. Be shareable.

Hopefully this makes decisions over what we should and shouldn’t create easier.

Leave a comment

It is, IMO, a very good way of framing how you can generate real value from trends.

The value of our traditional (if we’re being fair, fucking beige) utility content is on the way out. We had a good run, but the 2,000 word ‘what time is x’ posts are, thankfully, done with.

Or at least should be.

We’re entering an era of SEO where we have to be better at marketing. Where we have to think more critically about the value of our content and how it will be found.

TL;DR

1. If you’re the first on the scene, cover the story effectively and quickly. Start the wave.

2. Assess whether covering a trend with a straight news line is enough, or whether you should be ‘steering’ the demand.

3. Evergreen content isn’t dead by any means, but it requires more effort.

4. You should be looking to create demand for your content and for your brand (personal or otherwise) by using this method.

Value is not what it was

It’s fair to say the value we were able to generate a few years ago is dwindling. Particularly for more informational or comparison-type queries.

If you’re still targeting queries easily answered and synthesised by AI platforms and content with no unique value or offering, you’re probably doomed.

But, Google Zero is a load of old arse.

It is still the biggest and most valuable platform for a huge percentage of brands. Well, in terms of direct value. People may not discover you via search, but when they come for you, you better be ready.

You better be doing something really fucking good and different. Something with a bit of pizzazz.

Subscribe now

Is evergreen content doomed?

No. It just needs an overhaul. None of us need to see another ultimate guide unless the creator(s) go the extra mile. Then it’s a question of value and prioritisation.

Does the extra effort still generate value. Real revenue?

I’ve seen some brilliant pieces of work recently where publishers have invested time and effort into creating something stand-out. I don’t want to blow more smoke up the NYT’s arse, but there are a couple of really standout pieces of work that prioritise expertise, uniqueness and depth.

Things that add real value to a type of content lots of people will tell you is doomed.

• The 57 best sandwiches in NYC (The NYT)

You can’t beat a good sandwich from the world’s greatest city. Certainly for food. For something that prominent, that meaningful, you need standout content. Particularly if a huge part of your offering is the cooking section.

• The 100 best movies of all time (The NYT)

Real, legitimate experts. Video. User engagement. Creating something that gets people coming back. Using the opinions of bona fide experts. Evergreen lifestyle content taken to a level very few people are able to operate at.

However… Straight explainers are, broadly, a difficult sell. I’m not sure they’ve been a great idea for audiences for some time, but there you are. Unless you’re creating something better than what’s out there, I’d really consider it.

These FAQ-type pieces may have done numbers a couple of years ago, but we can and should do better. They can be massively upgraded with a bit of unique imagery. Data. Video. Something that can add value on and off-site. Something that humanises your brand and the people within it.

The majority of people still need a jumping off point into complex topics. Global conflicts (of which there are a fucking lot) still need some fundamental analysis to help people understand what’s really going on.

Capturing the wave of demand

This is all about understanding the current state of play. If you aren’t first on the scene with something breaking, you need to be cannier.

As far as I see it, you have three options;

1. You start the wave. Your competitors haven’t cottoned onto this emerging ‘trend’ and you can be the early bird.

2. You steer the wave. Stories have been published on the topic. In a news sense, the basics have been covered, but the story could still be spun. You can create a new angle.

3. The wave has crashed. Demand has begun to stall. Drop even. This is the point to figure out whether you can still add (and generate) any value

In all of these scenarios, it’s still better to be right than first.

In this scenario there are a few things you need;

1. Access to Google Trends

2. Glimpse (there’s a free version too)

3. Some vague level of critical thinking

For those of you concerned, I think two of the three are probably fine.

You can also add in either Keywords Everywhere’s new emerging trends tool (free) or Exploding Topics (I think there’s also a free version). These are great for understanding the long-term demand for a category or keyword as opposed to the near real-time view you get from Google Trends.

Glimpse integrates directly with Google trends and whilst it has its limitations, it’s really useful to see search volume in the platforms. Ditto Keywords Everywhere.

Not sponsored, just useful tools.

Subscribe now

A practical example

Alright, let’s see how I would actually do this. Let’s use everyone’s favourite rage baiting, cigar stubbing shithouse as an example.

Joey Barton is currently trending because he’s been arrested. Again. Over the last four hours the trend has skyrocketed and the straight story has been covered by multiple, very large outlets.

There’s a Top Stories box. The demand is high. But now the story has been covered, it’s not about starting the wave. It’s about steering it. Understanding whether there’s value in your version of the story.

Will your audience care?

In this scenario, I would think of stories like;

• Joey Barton’s history of arrests in images

• Analysis: Joey Barton’s criminal record makes for grim reading

• Why giving thugs like Joey Barton a voice is dangerous for our society

• An interview with someone who knows the real Barton (not the headline)

All of these stories take advantage of the trend. They could and should still do well in search and Discover. Crucially they are different. They position you as a leader of sorts. They are news-adjacent.

If you’re a paywalled publisher, this is critical to future success.

If your audience is interested and you should cover it, you can still create a straight news story. I’d still try to establish some level of information gain, but if you should cover it, then you should cover it.

If you think the story and interest has peaked, will this story positively matter to your bottom line?

Why this matters now

Google is a navigational engine. That means people need to search for you, your brand and/or your services. With informational content harder to generate a click from, we work smarter than years gone by.

So you’ve got to get out there and create the demand.

• Figure out what your moat is

• Put your stuff out there

• Fail in public (it’s not that bad)

• Try new things, experiment

• Build relationships with people in the industry (people will share your content, as long as it’s not shit)

• Stand out

This is, broadly, how tech companies work. Hook you in and shut up shop when you need them.

An excellent Ahrefs example

The Ahrefs team shared a very good example of this. They created demand.

Our ability to do well as marketers and SEOs will rely on this skillset more and more over the next few years.

They saw AI Overviews were trending. Big fucking news.

So Ahrefs created news-adjacent content around them. They used their incredible moat (big data) to create demand for the topic.

But they didn’t stop there. They pushed it out there. They have built personal brands that have a halo effect on the entire company. Tim Soulo, Ryan Law, Louise Linehan et al. I don’t pay attention to the main Ahrefs account(s), but I do follow and interact with these guys.

Then people started searching directly for their research. They have become a legitimate authority on the topic. In traditional search and AI platforms.

They steered a wave. They made it navigational. They became legitimate experts. Happy, halcyon days.

Leave a comment

Trump, Farage, Polanski, Mamdani. Hate them or love them, their ability to dominate the narrative and social media is winning out over traditional means.

Trump’s particular approach is, of course, the most egregious. By saturating the market to the nth degree, it’s hard to even remember what happened last week. Let alone care about it.

I think this is the problem facing more traditional media. Younger audiences have grown up with this as the norm. Fake news. A brutal job market. Housing so unaffordable that living in a shipping container is quite an attractive proposition.

The negativity of news is not a good escape. It is, in fact, quite the opposite.

TL;DR

1. Individuals overpower brands. Mistrust in legacy publishers and social media have helped the creator economy boom. This is the reverse halo effect in action.

2. For every visitor that reaches your website, 10 are getting to know your brand on platforms you don’t control.

3. 56% of young men and 60% of young women get the majority of their ‘news’ from social media.

4. Over the last decade, interest in news fell by 40% among 18-24 year olds and 38% for 25-34 year olds.

5. Since 2024, the average user has spent nearly 43% more time on TikTok.

Subscribe now

(Young) people don’t care as much about news

Interest in the news is plummeting.

Content consumption is at an all-time high, but between 2015 and 2024, interest in news fell by 40% among 18-24 year olds. 38% for 25-34 year olds. While everyone has lost interest over that time, the younger you are, the more significant this is.

It’s not a market problem. Not entirely. It’s a marketing problem. A content problem. An engagement problem. The fragmentation of the news and content industry means people have options. All pretty terrible in their own way.

Creators are soaring, but publishers are playing the same game they did 10 years ago. We’re competing with platforms so addictive the written word doesn’t cut the mustard anymore. This isn’t a gun fight anymore, it’s an all out nuclear war. And we are still fighting with a rolled up newspaper (albeit the digital edition).

News is seen as depressing. It isn’t ‘fun’ and according to a large proportion of younger audiences, doesn’t represent them. It hasn’t done for some time.

Really, it’s hard to care about anything

The average teen spends 7 hours and 22 minutes looking at screens each day. Nearly five hours a day on social media. More than three hours watching videos.

What doesn’t feature in these reports is how much time is spent reading content. The majority of influencers don’t write because it doesn’t hook people in the same way.

Or they’re illiterate.

When every app has been built to manipulate you psychologically, it’s hard to stop consuming. When you’re constantly on, breaking news becomes a background story. Unless the story is astronomical, it just fades into obscurity as the next disaster strikes 10 minutes later.

This is a structural and psychological shift

According to research from IPSOS and Joe, young people (16-34 years old) trust social media as a news source despite its obvious misinformation.

• 56% of young men and 60% of young women get the majority of their ‘news’ from social media

• This doesn’t even include YouTube (16% of young men and 9% of women)

• 72% of young men trust YouTube, compared to just 47% of women

• 79% of them are concerned about misinformation in the content they consume

20 years ago my dad read The Telegraph and The Times. Yours probably did too. We’re the last generation to have grown up with a broadsheet paper on the kitchen table. But this doesn’t exist anymore. Certainly not at the same scale.

This really impacts trust.

The kitchen table paper has been replaced by a personalised algorithm. One we all have a different experience of. As individuals and people who work for or with publishers, understanding this shift and combatting misinformation is fundamental to acquiring a younger audience.

Remember, we don’t see the same version of the world. Where you live, how old you are, what you engage with shapes what you see. You build your very own echo chamber. These algorithms have fundamentally reshaped the nature of free speech by determining who sees what.

Yours is all ‘stop the boats’ and mine’s cat videos and street fights.

Interestingly, trust in news in social media is rising among this age group, up from 45% in 2024 to 52% in 2025. How much of that is just because it’s easy and ‘agrees’ with your world view? Who knows.

Like it or not, the shift is happening with or without us.

Young people don’t care about brands

It’s why the creator economy is absolutely booming. The $250 billion market is predicted to nearly double to $480 billion by 2027, according to Goldman Sachs’ 2023 Creator Economy report.

It has completely reshaped modern marketing, attention and legacy brands ability to attract and retain audiences. People don’t trust brands. Not to the extent they did. Particularly young people.

Authenticity wins in the current environment. People value ‘realness.’ It’s why the spotty teenager can become a millionaire from his mum’s bedroom. There is very little connection with a publisher. To many they’re seen as the establishment.

The British Brainwashing Corporation you could say.

84% of teenagers express a negative sentiment when asked what word best describes news media. They think journalists are unethical and deceitful.

That is, by anyone’s reckoning, shit.

Subscribe now

Some see the solution as the problem

In an era where publishers are hemorrhaging organic traffic, UGC sites have bucked the trend. Outside of Reddit, has any single site done better than Substack?

Google owned products excluded of course.

And it’s not a mistake. It’s not luck. It’s not because Google has a deal with Substack (cough, Reddit, cough). It’s because people (and by proxy Google) trust people.

Whilst this isn’t limited just to younger audiences, it’s indicative of what is happening in society. People want to hear people’s real opinions. They don’t want watered down versions or over-optimised content that fits a mould.

So the solution is to build individuals. Individuals that will create a halo effect for your brand.

I can’t think of a better example than Dave Jorgenson - The Washington Post’s TikTok guy. Dave was a senior video producer at the Post between 2017 and 2025, and was responsible for launching the paper’s Tiktok channel in 2019. He built a huge TikTok following, saw an opportunity and bolted.

He also took the audience with him. But them’s the brakes. You’ve got to give people freedom to grow and build something brilliant. They’ll leave. Some to competitors, others to do their own thing.

But it’s the best option. You can’t escape the fact that individual news creators have greater weight than ever. Brands have to lean into it.

Future are doing some really interesting things in this space. By forging brand partnerships with individual creators with Future Collab they’re doing just this. Or at least trying to.

Build people’s profile. Absorb the halo effect to your brand. Rinse and repeat.

The discovery layer has changed

Google isn’t a homepage for younger people.

It’s not their first port of call. It may not even be their second or third. But that isn’t new. And news brands have spent a decade or more focusing on driving clicks to their site. Their entire business (outside of the subscription models) relies on traffic from external sites.

Why Google when you can endlessly swipe through 30 second, AI-made videos telling you how unsuccessful and stupid you are for paying tax? From platforms (and people) specifically designed to make you dependent and unhappy.

But referral traffic is drying up. And Google is certainly not the worst offender. It is still the most valuable platform by some distance for last click conversions.

A quote I stole from Rand I think sums this up perfectly:

There is no big tech company who wants to do you a favour. They identify their moat (usually customer data), open the gates and close for monetisation in brutal fashion.

This is the modern day version of the 1980s crack epidemic, but with worse people at the helm. Back then it was crack and flares, now its brightly coloured vapes and TikTok.

Google is trying to be sexy

Google’s latest and almost certainly most irritating shift is to attempt turn their search engine into a destination. Prioritising UGC sites, individual creators, videos and answering the question without the need for a click.

Facebook have quietly built an advertising business so evil that children are targeted with some pretty despicable practices. And who’d have though OpenAI and Google would be so evil that we’d all be cheering for Anthropic?

Well, actually I would back the first two to be fucking dreadful. I just though Anthropic would’ve followed suit. The bar is so low in Silicon Valley, that not using your company for mass surveillance and making unmanned WMDs at an incomprehensible scale would be cheered.

I suspect this is quite a clever ploy from Anthropic. Maybe they know they can’t beat Google, but they can become the go-to for people who value security and safety. Which I hope is fucking everyone.

Always worth remembering that you can’t beat the algorithm.

My friends father-in-law started using social media in the last two years. His views have changed. Significantly. He’s been targeted by a world that he didn’t know exists.

Subscribe now

Video dominates the young mind

Nothing grabs the young mind like short-form video. And no platform has mastered the art of short-form addiction like TikTok.

TikTok is almost completely at odds with how legacy publishers define themselves. Since 2024, the average user has spent nearly 43% more time on TikTok. When you compare that to all the other platforms, that is an extraordinary difference.

Here today gone tomorrow. No real quality control. Fact-checking. A human touch.

And TikTok’s not much better. (Nice little joke for you there).

Video really relies on individuals being comfortable in front of the camera. Being engaging and, crucially, being willing to do it. Which is a very, very different skill to writing. It’s crucial for a journalist and creator to master, but there are too few who can do it.

And if they can, they’ll probably be better off going it alone.

Links & References

I couldn’t find all of these numbers in the same report, so here is where I sourced the watch hours per user for each platform. Some inconsistencies here of course as the data doesn’t completely align. I accept I am flawed.

• Data Reportal 2024 report

• Digital 2026 Global Overview Report: A flagship joint study by We Are Social and Meltwater.

• TikTok Dominance Analysis (Birdeye)

• Whatsapp statistics

• Instagram users

Legacy media publications haven’t kept up

The older you are, the more likely you’re using social media to read news stories. The younger you are, the more likely you are to use it to watch videos.

Some publisher’s social strategies amount to sharing their articles. On platforms that actively devalue an external link.

Make it make sense.

Your content needs to be platform-specific. You have to add value on that platform. You can’t expect Instagram to drive direct conversions. It doesn’t work like that. You should be aiming for a positive interaction. A stop and smile. A save.

Google is pushing UGC and video content because people want to see more of it. Whilst they don’t give a shit about spam or stealing your traffic, they aren’t stupid. They see the shift and are maintaining their relevance at all costs.

You have to make your content work harder for you. Shares videos and photos on the proper channels and your website. Share quotes of articles. Get the writer of a big story to run an AMA. Answer the top comments. Add value to your audience wherever they are.

And, of course, create platform-specific content.

How the NYT have combatted this

But it’s not all doom and gloom.

The NYT now have over 12 million total subscribers. That is, by any measure, a lot. Their affiliate revenue, digital and print subs, advertising revenue and M&A (The Athletic) alongside smaller digital subscription offerings are all thriving.

• 11.3+ million, subscribe only to the company’s digital products

• Subscription revenue increased 9.6 percent year over year, to $481.4 million.

• Digital advertising revenue rose 18.7 percent, to $94.4 million, beating the company’s estimates, because of strong demand in areas like games and sports

But that’s not all.

This is a truly excellent (paid) look at how they have become so successful - they aren’t a news company anymore. The NYT now has only 1.5 million news-only subscribers, compared to 10.8 million on a bundle or other single-product plans.

They have managed the decline of interest in news and built a diverse, resilient company that can withstand the decline in its core product.

Worth noting that the decline in news-only subs doesn’t represent that level of decline in people who subscribe for news. Just that fewer people only subscribe to news.

From the horses’ mouth:

It’s not all doom and gloom

You have to accept you can’t compete with these platforms on scale. But that’s not a bad thing when it’s all shite. According to Matt Southern from Search Engine Journal, AI-generated slop accounts for 21% of YouTube Shorts shown to new users.

This Kapwing study of 15,000 trending channels identified;

• 278 channels producing nothing but AI slop.

• Amassing 63 billion views

• 221 million subscribers

• An estimated $117 million in annual ad revenue

But for you and I, this is short-term bullshit. It’s black hat guff. Drop domain abuse. It works until it doesn’t. It’s not something you or I should or could do.

But scale isn’t where this is won. Meaningful connections are where this is won.

What can we do?

Reporting on the news isn’t enough anymore. Just saying what happened isn’t enough to generate a click. Let alone loyalty. Fucking miles away from a subscription.

Legacy media outlets need to cut through the noise in their own way. Things like BBC Verify are a very good step. But we’ve got to get out there.

We’ve seen right and left wing publications get further away from the middle, I think, directly because of this. Pushing the boundaries further and further to resonate in some way with people.

The questions for all publishers are the same;

• How do we reach the right people?

• How do we cut through the noise?

• How do we become habitual?

The answer is very unlikely to be ‘write more.’

It’s about connecting with audiences at the right time and place in the right format. Follow the four E’s of content production. Entertain them when they want to be entertained. Educate them when they want to be educated. Empower and engage them in equal measure. Create great stuff.

Younger people have grown up in a pretty harsh environment when it comes to money, work and screens. It’s not easy. Good for us to remember that.

Leave a comment

Tomorrow myself, Jessie Willms, Shelby Blackley, and Barry Adams will share our insights and expertise on all things paywalls with those of you who've bought tickets for our online Paywall masterclass at 11am ET tomorrow (4pm UK).

(Yes, this was lifted almost word for words from Barry’s Linkedin post).

Buy Now

In this masterclass you'll learn:

➡️ The fundamentals of technical SEO for paywalls
➡️ Business and commercial strategies for paywalls
➡️ Editorial approaches that drive subscriptions

And you'll get the opportunity to ask us anything about paywalls (the most in vogue thing in publishing right now) or SEO for news in general. It’s a couple of hours, so settle in and enjoy. Should be fun!

Use code HARRY10 for 10% off the price at checkout.

Thanks for supporting!

Harry x

For the convenient price of £1500 an hour, people like Peter Attia and Andrew Huberman can teach how to do it. As long as they aren’t busy on a notorious paedophile’s Caribbean island I mean.

Allegedly.

But we’re boring. Grounding to us isn’t an ancestral tenet. We don’t eat pig testicles for breakfast. It’s all AI baby. When we’re talking about grounding, we mean fact-checking the hallucinations of planet destroying robots and tech bros.

If you want a non-stupid opening line, when models accept they don’t know something, they ground results in an attempt to fact check themselves.

Happy now?

TL;DR

1. LLMs don’t search or store sources or individual URLs, they generate answers from pre-supplied content.

2. RAG anchors LLMs in specific knowledge backed by factual, authoritative and current data. It reduces hallucinations.

3. Retraining a foundation model or fine-tuning it is computationally expensive and resource-intensive. Grounding results is far cheaper.

4. With RAG, enterprises can use internal, authoritative data sources and gain similar model performance increases without retraining. It solves the lack of up-to-date knowledge LLMs have (or rather don’t).

Subscribe now

What is RAG?

RAG - Retrieval Augmented Generation - is a form of grounding and a foundational step in answer engine accuracy. LLMs are trained on vast corpuses of data and every dataset has limitations. Particularly when it comes to things like newsy queries or changing intent.

When a model is asked a question it doesn’t have the appropriate confidence score to answer accurately, it reaches out to specific trusted sources to ground the response. Rather than relying solely on outputs from its training data.

By bringing in this relevant, external information, the retrieval system identifies relevant, similar pages/passages and includes the chunks as part of the answer.

This provides a really valuable look at why being in the training data is so important. You are more likely to be selected as a trusted source for RAG if you appear in the training data for relevant topics.

It’s one of the reasons why disambiguation and accuracy are more important than ever in today’s iteration of the internet.

Why do we need it?

Because LLMs are notoriously hallucinatory. They have been trained to provide you with an answer. Even if the answer is wrong.

Grounding results provides some relief from the flow of batshit information.

All models have a cutoff limit in their training data. They can be a year old or more. So anything that has happened in the last year would be un-answerable without the real time grounding of facts and information.

Once a model has ingested a sizeable amount of training data, it is far cheaper to rely on a RAG pipeline to answer new information rather than re-training the model.

Dawn Anderson has a great presentation called You Can’t Generate What You Can’t Retrieve. Well worth a read, even if you can’t be in the room.

Do grounding and RAG differ?

Yes. RAG is a form of grounding.

Grounding is a broad brush term applied used to apply to any type of anchoring AI responses in trusted, factual data. RAG achieves grounding by retrieving relevant documents or passages from external sources.

In almost every case you or I will work with, that source is live web search.

Think of it like this;

• Grounding is the final output - ‘please stop making things up.’

• RAG is the mechanism. When it doesn’t have the appropriate confidence to answer a query, ChatGPT’s internal monologue says ‘don’t just fucking lie about it, verify the information.’

• So grounding can be achieved through fine tuning, prompt engineering or RAG

• RAG either supports its claims when the threshold isn’t met or finds the source for a story that doesn’t appear in its training data

Imagine a fact you hear down the pub. Someone tells you that the scar they have on their chest was from a shark attack. A hell of a story. A quick bit of verifying would tell you that they choked on a peanut in said pub and had to have a nine hour operation to get a part of their lung removed.

True story - and one I believed until I was at university. It was my dad.

There is a lot of conflicting information out there as to what web search these models use. However, we have very solid information that ChatGPT is (still) scraping Google’s search results to form its responses when using web search.

Why can no-one solve AI’s hallucinatory problem?

A lot of hallucinations make sense when you frame it as a model filling the gaps. The fails seamlessly.

It is a plausible falsehood.

It’s like Elizabeth Holmes of Theranos infamy. You know it’s wrong, but you don’t want to believe it. The you here being some immoral old media mogul or some investment firm who cheaped out on the due diligence.

That is a direct quote from OpenAI. The hallucinatory horses’ mouth.

Models hallucinate for a few reasons. As argued in OpenAI’s most recent research paper, they hallucinate because training processes and evaluation reward an answer. Right or not.

If you think of it in a Pavlovian conditioning sense, the model gets a treat when it answers. But that doesn’t really answer why models get things wrong. Just that the models have been trained to answer your ramblings confidently and without recourse.

This is largely due to how the model has been trained.

Ingest enough structured or semi-structured data (with no right or wrong labelling) and they become incredibly proficient at predicting the next word. At sounding like a sentient being.

Not one you’d hang out with at a party. But a sentient sounding one.

If a fact is mentioned dozens or hundreds of times in the training data, models are far less-likely to get this wrong. Models value repetition. But seldom referenced facts act as a proxy for how many ‘novel’ outcomes you might encounter in further sampling.

Facts referenced this infrequently are grouped under the term the singleton rate. In a never before made comparison, a high singleton rate is a recipe for disaster for LLM training data, but brilliant for Essex hen parties.

Even when the training data is 100% error-free, the model will generate errors. They are built by people. People are flawed and we fucking love confidence.

Several post-training techniques - like reinforcement learning from human feedback or, in this case, forms of grounding - do reduce hallucinations.

Subscribe now

How does RAG work?

Technically, you could say that the RAG process is initiated long before a query is received. But I’m being a bit arsey there. And I’m not an expert.

Standard LLMs source information from their databases. This data is ingested to train the model in the form of parametric memory (more on that later). So, whoever is training the model is making explicit decisions about the type of content that will likely require a form of grounding.

RAG adds an information retrieval component to the AI layer. The system:

➡️ Retrieves data

➡️ Augments the prompt

➡️ Generates an improved response.

A more detailed explanation (should you want it) would look something like:

1. The user inputs a query and it’s converted into a vector

2. The LLM uses its parametric memory to attempt to predict the next likely sequence of tokens

3. The vector distance between the query and a set of documents is calculated using Cosine Similarity or Euclidean Distance

4. This determines whether the model’s stored (or parametric) memory is capable of fulfilling the user’s query without calling an external database

5. If a certain confidence threshold isn’t met, RAG (or a form of grounding) is called

6. A retrieval query is sent to the external database

7. The RAG architecture augments the existing answer. It clarifies factual accuracy or adds information to the incumbent response

8. A final, improved output is generated

If a model is using an external database like Google or Bing (which they all do), it doesn’t need to create one to be used for RAG.

This makes things a shit ton cheaper.

The problem the tech heads have is that they all hate each other. So when Google dropped the num=100 parameter in September 2025, ChatGPT citations fell off a cliff. They could no longer use their third party partners to scrape this information.

It’s worth noting that more modern RAG architectures apply a hybrid model of retrieval, where semantic searching is run alongside more basic keyword type matches. Like updates to BERT (DaBERTa) and RankBrain, this means the answer takes the entire document and contextual meaning into account when answering.

Hybridisation makes for a far superior model. In this agriculture case study, a base model hit 75 percent accuracy, fine-tuning bumped it to 81 percent, and fine-tuning + RAG jumped to 86 percent.

Parametric vs non-parametric memory

A models parametric memory is essentially the patterns it has learned from the training data it has greedily ingested.

During the pre-training phase, the models ingest an enormous amount of data - words, numbers, multi-modal content etc. Once this data has been turned into a vector space model, the LLM is able to identify patterns in its neural network.

When you ask it a question, it calculates the probability of the next possible token and calculates the possible sequences by order of probability. The temperature setting is what provides a level of randomness.

Non-parametric memory stores (or accesses) information in an external database. Any search index being an obvious one. Wikipedia, Reddit etc too. Any kind of ideally well structured database. This allows the model to retrieve specific information when required.

RAG methodologies are able to ride these two competing, highly complementary disciplines.

1. Models gain an ‘understanding’ of language and nuance through parametric memory.

2. Responses are then enriched and/or grounded to verify and validate the output via non-parametric memory.

Higher temperatures increase randomness. Or ‘creativity.’ Lower temperatures the opposite.

Ironically these models are incredibly uncreative. It’s a bad way of framing it, but mapping words and documents into tokens is about as statistical as you can get.

Subscribe now

Why does it matter for SEO?

If you care about AI search and it matters for your business, you need to rank well in search engines. You want to force your way into consideration when RAG searches apply.

You should know how RAG works and how to influence it.

If your brand features poorly in the training data of the model, you cannot immediately change that. Well, for future iterations you can. But the model’s knowledge base isn’t updated on the fly.

So you rely on featuring prominently in these external databases in order to be part of the answer. The better you rank, the more likely you are to feature in RAG-specific searches.

I highly recommend watching Mark Williams-Cook’s From Rags to Riches presentation. It’s excellent. Very reasonable and gives some clear guidance on how to find queries that require RAG and how you can influence them.

Basically, again, you need to do good SEO

1. Make sure you rank as high as possible for the relevant term in search engines

2. Make sure you understand how to maximise your chance of featuring in an LLMs grounded response

3. Over time, do some better marketing to get yourself in the training data

All things being equal, concisely answered queries that clearly match relevant entities that add something to the corpus will work. If you really want to follow chunking best practice for AI retrieval, somewhere around 200-500 characters seems to be the sweet spot.

Smaller chunks allow for more accurate, concise retrieval. Larger chunks have more context, but can create a more ‘lossy’ environment, where the model loses its mind in the middle.

Top tips (same old)

I find myself repeating these at the end of every training data article, but I do think it all remains broadly the same.

• Answer the relevant query high up the page (front-loaded information)

• Clearly and concisely match your entities

• Provide some level of information gain

• Avoid ambiguity, particularly in the middle of the document

• Have a clearly defined argument and page structure, with well structured headers

• Use lists and tables. Not because they’re less resource intensive token-wise, but because they tend to contain less information

• My god be interesting. Use unique data, images, video. Anything that will satisfy a user.

• Match their intent.

As always, very SEO. Much AI.

Leave a comment

From the training data to similarity scoring and machine’s ability to ‘understand’ our tired, sad bullshit - this is the third in a series I’ve titled, information retrieval for morons.

TL;DR

1. In the vector space model, the distance between vectors represents the relevance (similarity) between the documents or items

2. Vectorisation has allowed search engines to perform concept searching instead of word searching. It is the alignment of concepts, not letters or words

3. Longer documents contain more similar terms. To combat this, document length is normalised and relevance is prioritised

4. Google has been doing this for over a decade. Maybe for over a decade, you have too

Subscribe now

Things you should know before we start

Some concepts and systems you should be aware of before we dive in.

I don’t remember all of these and neither will you. Just try to enjoy yourself and hope that through osmosis and consistency you vaguely remember things over time.

• TF-IDF stands for term frequency-inverse document frequency. It is a numerical statistic used in NLP and information retrieval to measure a term's relevance within a document corpus.

• Cosine similarity measures the cosine of the angle between two vectors, ranging between -1 to 1. A smaller angle (closer to 1) implies higher similarity.

• The bag-of-words model is a way of representing text data when modelling text with machine learning algorithms.

• Feature extraction/encoding models are used to convert raw text into numerical representations that can be processed by machine learning models.

• Euclidean distance measures the straight-line distance between two points in vector space to calculate data similarity (or dissimilarity).

• Doc2Vec (an extension of Word2Vec), designed to represent the similarity (or lack of it) in documents as opposed to words.

What is the vector space model?

The Vector Space Model (VSM) is an algebraic model that represents text documents or items as ‘vectors.’ This representation allows systems to create a distance between each vector.

The distance calculates the similarity between terms or items.

Commonly used in information retrieval, document ranking, and keyword extraction, vector models create structure. This structured, high-dimensional numerical space, enables the calculation of relevance via similarity measures like cosine similarity.

Terms are assigned values. If a term appears in the document, its value is non-0. Worth noting that terms are not just individual keywords. They can be phrases, sentences and entire documents.

How does it work?

Once queries, phrases and sentences are assigned values, the document can be scored. It has a physical place in the vector space as chosen by the model.

Based on its score, documents can be compared to one another based on the inputted query. You generate similarity scores at scale. This is known as semantic similarity. Where a set of documents are scored and positioned in the index based on their meaning.

Not just their lexical similarity.

Why does it work so well?

Machines don’t just like structure. They bloody love it.

Fixed length (or styled) inputs and outputs create predictable, accurate results. The more informative and compact a dataset, the better quality classification, extraction and prediction you will get.

The problem with text is that it doesn’t have much structure. At least not in the eyes of a machine. It’s messy. This is why it has such an advantage over the classic Boolean Retrieval Model.

In Boolean Retrieval Models, documents are retrieved based on whether they satisfy the conditions of a query that uses Boolean logic. It treats each document as a set of words or terms and uses AND, OR and NOT operators to return all results that fit the bill.

Its simplicity has its uses, but cannot interpret meaning.

Think of it more like data retrieval than identifying and interpreting information. We fall into the term frequency (TF) trap too often with more nuanced searches. Easy, but lazy in today’s world.

Whereas the vector space model interprets actual relevance to the query and doesn’t require exact match terms. That’s the beauty of it.

It’s this structure that creates much more precise recall.

Subscribe now

The transformer revolution (not Michael Bay)

Unlike Michael Bay’s series aimed at pubescent boys, the real transformer architecture replaced older, static embedding methods (like Word2Vec) with contextual embeddings.

While static models assigned one vector to each word, Transformers generate dynamic representations that change based on the surrounding words in a sentence.

And yes, Google has been doing this for some time. It’s not new. It’s not GEO. It’s just modern information retrieval that ‘understands’ a page.

I mean, obviously not. But you, as a hopefully sentient, breathing being, understand what I mean. But transformers, well, they fake it;

1. Transformers weight input by data by significance

2. The model pays more attention to words that demand or provide extra context

Let me give you an example.

Bat is an ambiguous term. Ambiguity is bad in the age of AI.

But transformer architecture links bat with ‘teeth,’ ‘flew’ and ‘cave,’ signalling that bat is far more likely to be a bloodsucking rodent* than something a gentleman would use to caress the ball for a boundary in the world’s finest sport.

*No idea if a bat is a rodent, but it looks like a fucking rat with wings

BERT strikes back

BERT. Bidirectional Encoder Representations from Transformers. Shrugs.

This is how Google has worked for years. By applying this type of contextually aware understanding between the semantic relationships between words and documents. It’s a huge part of the reason why Google is so good at mapping and understanding intent and how it shifts over time.

BERT’s more recent updates (DeBERTa) allow words to be represented by two vectors - one for meaning and one for its position in the document. This is known as Disentangled Attention. It provides more accurate context.

Yep, sounds weird to me too.

Bert processes the entire sequence of words simultaneously. This means context is applied from the entirety of the page content (not just the few surrounding terms).

Synonyms baby

Launching in 2015, RankBrain was Google’s first deep learning system. Well, that I know of anyway. It was designed to help the search algorithm understand how words relate to concepts.

This was kind of the peak search era. Anyone could start a website about anything. get it up and ranking. Make a load of money. Not need any kind of rigour.

Halcyon days.

With hindsight, these days weren’t great for the wider public. Getting advice on funeral planning and commercial waste management from a spotty 23 year old’s bedroom in Halifax.

As new and evolving queries surged, Rankbrain and the subsequent neural matching was vital.

Then there was MUM. Google’s ability to ‘understand’ text, images and visual content across multiple languages simulatenously.

Combatting document length issues

Document length was an obvious problem 10 years ago. Maybe less. Longer articles, for better or worse, always did better. I remember writing 10,000 word articles on some nonsense about website builders and sticking it on a homepage.

Even then that was a shit idea…

In a world where queries and documents are mapped to numbers, you could be forgiven for thinking that longer documents will always be surfaced over shorter ones.

Remember 10-15 years ago when everyone was obsessed when every article being 2,000 words.

If you see another ‘what time is x’ 2,000 word article, you have my permission to fucking shoot me.

Longer documents will – as a result of containing more terms – have higher TF values. They also contain more distinct terms. These factors can conspire to raise the scores of longer documents

Hence why, for a while, they were the zenith of our shitty content production.

Longer documents can broadly be lumped into two categories:

1. Verbose documents that essentially repeat the same content (hello keyword stuffing my old friend)

2. Documents covering multiple topics, in which the search terms probably match small segments of the document, but not all of it

To combat this obvious issue, a form of compensation for document length is used, known as Pivoted Document Length Normalisation. This adjusts scores to counteract the natural bias longer documents have.

The cosine distance should be used because we do not want to favour longer (or shorter) documents, but to focus on relevance. Leveraging this normalisation prioritises relevance over term frequency.

It’s why cosine similarity is so valuable. It is robust to document length. A short and long answer can be seen as topically identical if they point in the same direction in the vector space.

Subscribe now

Well, so what?

Great question.

Well, no one’s expecting you to understand the intricacies of a vector database. You don’t really need to know that databases create specialised indices to find close neighbours without checking every single record.

This is just for companies like Google to strike the right balance between performance, cost and operational simplicity.

Kevin Indig’s latest, excellent research shows that 44.2% of all citations in ChatGPT originate from the first 30% of the text. The probability of citation drops significantly after this initial section, creating a ‘ski ramp’ effect.

Even more reason not to mindlessly create fucking massive documents because someone told you to.

In ‘AI search,’ a lot of this comes down to tokens. According to Dan Petrovic’s always excellent work, each query has a fixed grounding budget of approximately 2,000 words total, distributed across sources by relevance rank.

In Google at least. And your rank determines your score. So get SEO-ing.

Metehan’s study on what 200,000 Tokens Reveal About AEO/GEO really highlights how important this is. Or will be. Not just for our jobs, but biases and cultural implications.

As text is tokenised (compressed and converted into a sequence of integer IDs), this has cost and accuracy implications.

• Plain English prose is the most token-efficient format at 5.9 characters per token. Let’s call it 100% relative efficiency. A baseline.

• Turkish prose has just 3.6. This is 61% as efficient.

• Markdown tables 2.7. 46% as efficient.

Languages are not created equal. In an era where capex costs are soaring and AI firms have struck deals I’m not sure they can cash, this matters.

Top tips

Well, as Google has been doing this for some time, the same things should work across both interfaces.

1. Answer the fucking question. My god. Get to the point. I don’t care about anything other than what I want. Give it to me immediately (spoken as a human and a machine).

2. So frontload your important information. I have no attention span. neither do transformer models.

3. Disambiguate. Entity optimisation work. Connect the dots online. Claim your knowledge panel. Authors, social accounts, structured data, building brands and profiles.

4. Excellent E-E-A-T. Deliver trustworthy information in a manner that sets you apart from the competition.

5. Create keyword rich internal links that help define what the page and content is about. Part disambiguation. Part just good UX.

6. If you want something focused on LLMs, be more efficient with your words.

   1. Using structured lists can reduce token consumption by 20-40% because they remove fluff. Not because they’re more efficient*.

   2. Use commonly known abbreviations to also save tokens

*Interestingly they are less efficient than traditional prose

Almost all of this is about giving people what they want quickly and removing any ambiguity. In an internet full of shit, doing this really, really works.

Last bits

There is some discussion around whether markdown for agents can help strip out the fluff from HTML on your site. So agents could bypass the cluttered HTML and get straight to the good stuff.

How much of this could be solved by having a less fucked up approach to semantic HTML I don’t know. Anyway, one to watch.

Very SEO. Much AI.

Leave a comment

Every C-suite in the country is desperate to get this right. Decision makers need to feel confident that you and I are the right people to lead us into the new frontier.

We need to learn the fundamentals of information retrieval. Even if your business shouldn’t be doing anything differently.

Here, that starts with understanding the basics of model training data. What is it, how does it work and - crucially - how do I get in it.

TL;DR

1. AI is the product of its training data. The quality (and quantity) the model trains on is key to its success.

2. The web-sourced AI data commons is rapidly becoming more restricted. This will skew data representativity, freshness, and scaling laws.

3. The more consistent, accurate brand mentions you have that appear in training data, the less ambiguous you are.

4. Quality SEO, with better product and traditional marketing, will improve your appearance in the training data and eventually with real-time RAG/retrieval.

Subscribe now

What is training data?

Training data is the foundational dataset used in training LLMs to predict the most appropriate next word, sentence and answer. The data can be labelled, where models are taught the right answer, or unlabelled, where they have to figure it out for themselves.

Without high-quality training data, models are completely fucking useless.

From semi-libellous tweets, to videos of cats and great works of art and literature that stand the test of time, nothing is off limits. Nothing. It’s not just words either. Speech-to-text models need to be trained to respond to different speech patterns and accents. Emotions even.

How does it work?

The models don’t memorise, they compress. LLMs process billions of datapoints, adjusting internal weights through a mechanism known as backpropagation.

If the next word predicted in a string of training examples is correct, it moves on. If not, it gets the machine equivalent of Pavlovian conditioning.

Bopped on the head with a stick or a ‘good boy.’

The model is then able to vectorise. Creating a map of associations by term, phrase and sentence.

• Converting text into numerical vectors AKA Bag of Words

• Capturing semantic meaning of words and sentences, preserving wider context and meaning (word and sentence embeddings)

Rules and nuances are encoded as a set of semantic relationships - this is known as parametric memory. ‘Knowledge’ baked directly into the architecture. The more refined a model’s knowledge on a topic, the less it has to use a form of grounding to verify its twaddle.

Worth noting that models with a high parametric memory are faster at retrieving accurate information (if available), but have a static knowledge base and literally forget things.

RAG and live web search is an example of a model using non-parametric memory. Infinite scale, but slower. Much better for news and when results require grounding.

Crafting better quality algorithms

When it comes to the training data, drafting better quality algorithms relies on three elements;

1. Quality

2. Quantity

3. Removal of bias

Quality of data matters for obvious reasons. If you train a model on poorly labelled, solely synthetic data, the model performance cannot be expected to exactly mirror real problems or complexities.

Quantity of data is a problem too. Mainly because these companies have eaten everything in sight and done a runner on the bill.

Leveraging synthetic data to solve issues of scale isn’t necessarily a problem. The days of accessing high-quality, free-to-air content on the internet for these guys are, largely gone. For two main reasons;

1. Unless you want diabolical racism, mean comments, conspiracy theories and plagiarised bullshit, I’m not sure the internet is your guy anymore.

2. If they respect company’s robots.txt directives at least. Eight in ten of world’s biggest news websites now block AI training bots. I don’t know how effective their CDN-level blocking is, but this makes quality training data harder to come by.

Bias and diversity (or lack of it) is a huge problem too. People have their own inherent biases. Even the ones building these models.

Shocking I know…

If models are fed data unfairly weighted towards certain characteristics or brands, it can reinforce societal issues. It can further discrimination.

Remember, LLMs are neither intelligent nor databases of facts. They analyse patterns from ingested data. Billions or trillions of numerical weights that determine the next word (token) following another in any given context.

How is training data collected?

Like every good SEO, it depends.

1. If you’re an idiot and you built an AI model explicitly to identify pictures of dogs, you need pictures of dogs in every conceivable position. Every type of dog. Every emotion the pooch shows. You need to create or procure a dataset of millions, maybe billions, of canine images.

2. Then it must be cleaned. Think of it as structuring data into a consistent format. In said dog scenario, maybe a feline friend nefariously added pictures of cats dressed up as dogs to fuck you around. Those must be identified.

3. Then labelled (for supervised learning). Data labelling (with some human annotation) ensures we have a sentient being somewhere in the loop. Hopefully an expert to add relevant labels to a tiny portion data so that a model can learn. For example, A daschund sitting on a box looking melancholic.

4. Pre-processing. Responding to issues like cats masquerading as dogs. Ensuring you minimise potential biases in the dataset like specific dog breeds being mentioned far more frequently than others.

5. Partitioned. A portion of the data is kept back so the model can’t memorise the outputs. This is the final validation stage. Kind of like a placebo.

This is, obviously, expensive and time consuming. It’s not feasible to take up hundreds of thousands of hours of expertise from real people in fields that matter.

Think of this. You’ve just broken your arm and you’re waiting in A&E for six hours. You finally get seen, only to be told you had to wait because all the doctors have been processing images for OpenAI’s new model.

Data labeling is a time-consuming and tedious process. To combat this, many businesses hire large teams of human data annotators (AKA humans in the loop, you know, actual experts), assisted by automated weak labelling models. In supervised learning, they sort the initial labelling.

For perspective, one hour of video data can take humans up to 800 hours to annotate.

Subscribe now

Micro models

So companies build micro-models. Models that don’t require as much training or data to run. The humans in the loop (I’m sure they have names) can start training micro-models after annotating a few examples.

The models learn. They train themselves.

So over time, human input decreases and we’re only needed to validate the outputs. And to make sure the models aren’t trying to undress children, celebrities and your coworkers on the internet.

But who cares about that in the face of ‘progress.’

Types of training data

Training data is usually categorised by how much guidance is provided or required (supervision) and the role it plays in the model’s lifecycle (function).

Ideally a model is largely trained on real data.

Once a model is ready, it can be trained and fine-tuned on synthetic data. But synthetic data alone is unlikely to create high-quality models.

• Supervised (or labelled): where every input is annotated with the ‘right’ answer.

• Un-supervised (or unlabelled): work it out yourself robots, I’m off for a beer.

• Semi-supervised: where a small amount of the data is properly labelled and model ‘understands’ the rules. More I’ll have a beer in the office.

• RLHF (Reinforcement Learning from Human Feedback): humans are shown two options and asked to pick the ‘right’ one (preference data). Or a person demonstrates the task at hand for the mode to imitate (demonstration data).

• Pre-training and fine-tuning data: Massive datasets allow for broad information acquisition and fine-tuning is used to turn the model into a category expert.

• Multi-modal: images, videos, text etc.

Then some what’s known as edge case data. Data designed to ‘trick’ the model to make it more robust.

In light of the let’s call it ‘burgeoning’ market for AI training data, there are obvious issues of ‘fair use’ surrounding it.

“We find that 23% of supervised training datasets are published under research or non-commercial licenses.”

So fucking pay people.

The spectrum of supervision

In supervised learning, the AI algorithm is given labelled data. These labels define the outputs and are fundamental to the algorithm being able to improve over time on its own.

Let’s say you’re training a model to identify colours. There are dozens of shades of each colour. Hundreds even. So whilst this is an easy example, it requires accurate labelling. The problem with accurate labelling is its time-consuming and potentially costly.

In unsupervised learning, the AI model is given unlabeled data. You chuck millions of rows, images or videos at a machine, sit down for a coffee and then kick it when it hasn’t worked out what to do.

It allows for more exploratory ‘pattern recognition.’ Not learning.

While this approach has obvious drawbacks, it’s incredibly useful at identifying patterns a human might miss. The model can essential define its own labels and pathway.

Models can and do train themselves and they will find things a human never could. They’ll also miss things. It’s like a driverless car. Driverless cars may have less accidents than when a human is in the loop. But when they do, we find it far more unpalatable.

It’s the technology that scares us. And rightly so.

Combatting bias

Bias in training data is very real and potentially very damaging. There are three phases;

1. Origin bias

2. Development bias

3. Deployment bias

Origin bias references the validity and fairness of the dataset. Is the data all-encompassing? Is there any obvious systemic, implicit or confirmation bias present ?

Development bias includes the features or tenets of the data the model is being trained on. Does algorithmic bias occur because of the training data?

Then we have deployment bias. Where the evaluation and processing of the data leads to flawed outputs and automated/feedback loop bias.

You can really see why we need a human in the loop. And why AI models training on synthetic or inappropriately chosen data would be a fucking disaster.

In healthcare, data collection activities influenced by human bias can lead to the training of algorithms that replicate historical inequalities. Yikes.

Leading to a pretty bleak cycle of reinforcement.

The most frequently used training data sources

Training data sources are wide ranging in both quality and structure. You’ve got the open web, which is obviously a bit mental. X if you want to train something to be racist. Reddit if you’re looking for the Incel Bot 5000.

Or highly structured academic and literary repositories if you want to build something, you know, good… Obviously then you have to pay something.

Common Crawl

Common Crawl is a public web repository - a free, open-source storehouse of historical and current web crawl data available to pretty much anyone on the internet.

The full Common Crawl Web Graph currently contains around 607 million domain records across all datasets, with each monthly release covering 94 to 163 million domains.

In the Mozilla Foundation's 2024 report Training Data for the Price of a Sandwich, 64% of the 47 LLMs analysed used at least one filtered version of Common Crawl data.

If you aren’t in the training data, you’re very unlikely to be cited and referenced. The Common Crawl Index Server lets you search any URL pattern against their crawl archives and Metehan’s Web Graph helps you see how ‘centred you are.’.

Wikipedia (and Wikidata)

The default English Wikipedia dataset contains 19.88 GB of complete articles that help with language modeling tasks. And Wikidata is an enormous, incredibly comprehensive knowledge graph. Immensely structured data.

While representing only a small percentage of the total tokens, Wikipedia is perhaps the most influential source for entity resolution and factual consensus. it is one of the most factually accurate, up-to-date and well structured repositories of content in existence.

Some of the biggest, shittiest guys have just signed deals with Wikipedia.

Publishers

OpenAI, Gemini etc have multi million dollar licensing deals with a number of publishers.

• News Corp (WSJ, New York Post) signed a $250M+ deal in 2024

• The Atlantic signed a deal with OpenAI

• As did the FT

The list goes on, but only for a bit… And not recently. I’ve heard things have clammed shut. Which given the state of their finances may not be surprising.

Media & libraries

This is mainly for multi-modal content training. Shutterstock (images/video), Getty Images have one with Perplexity, and Disney (a 2026 partner for the Sora video platform) provide the visual grounding for multi-modal models.

As part of this three-year licensing agreement with Disney, Sora will be able to generate short, user-prompted social videos based on Disney characters.

As part of the agreement, Disney will make a $1 billion equity investment in OpenAI, and receive warrants to purchase additional equity.

Books

BookCorpus turned scraped data of 11,000 unpublished books into a 985 million-word dataset.

We cannot write books fast enough for models to continually learn on. It’s part of the soon to happen model collapse.

Code repositories

Coding has become one of the most influential and valuable features of LLMs. Specific LLMs like Cursor or Claude Code are incredible. GitHub and Stack Overflow data have built these models.

They’ve built the vibe-engineering revolution.

Public web data

Diverse (but relevant) web data results in faster convergence during training, which in turn reduces computational requirements. It’s dynamic. Ever-changing. But, unfortunately, a bit fucking nuts and messy.

But if you need vast swathes of data, maybe in real-time, then public web data is the way forward. Ditto for real opinions and reviews of products and services. Public web data, review platforms, UGC and social media sites are great.

Subscribe now

Why models aren’t getting (much) better

While there’s no shortage of data in the world, most of it is unlabelled and thus can’t actually be used in supervised machine learning models. Every incorrect label has a negative impact on a model’s performance.

According to most, we’re only a few years away from running out of quality data. Inevitably, this will lead to a time when those genAI tools start consuming their own shit.

This is a known problem that will cause model collapse. If I was eating my own shit, I’d probably want out too.

• They are being blocked by companies who do not want their data used pro bono to train the models.

• Robots.txt protocols (a directive, not something directly enforceable), CDN level blocking and terms of service pages have been updated to tell these guys to get to fuck.

• They consume data quicker than we can produce it

Frankly, as more publishers and websites are forced into paywalling (a smart business decision), the quality of these models only gets worse.

So, how do you get in the training data?

There are two obvious approaches I think of.

1. To identify the seed data sets of models that matter and find ways into them

2. To forgo the specifics and just do great SEO and wider marketing. Make a tangible impact in your industry

I can see pros and cons to both. Finding ways into specific models is probably highly unnecessary for most brands. To me this smells more like grey hat SEO. Most brands will be better off just doing some really fucking good marketing and getting shared, cited and you know, talked about.

These models are not trained on directly up-to-date data. This is important because you cannot retroactively get into a specific model’s training data. You have to plan ahead.

If you’re an individual, you should be;

• Creating and sharing content

• Going on podcasts

• Attending industry events

• Sharing other people’s content

• Doing webinars

• Getting yourself in front of relevant publishers, publications and people

There are some pretty obvious sources of highly structured data that models have paid for in recent times. I know, they’ve actually paid for it. I don’t know what the guys at Reddit and Wikipedia had to do to get money from these guys and maybe I don’t want to.

How can I tell what datasets models use?

Everyone has become a lot more closed off with what they do and don’t use for training data. I suspect this is is both legally and financially motivated. So you’ll need to do some digging.

And there are some massive ‘open source’ datasets I suspect they all use:

• Common Crawl

• Wikipedia

• Wikidata

• Coding repositories

Fortunately most deals are public and it’s safe to assume that models use data from these platforms.

Google has a partnership with Reddit and access to an insane amount of transcripts from YouTube. They almost certainly have more valuable, well structured data at their fingertips than any other company.

Grok trained almost exclusively on real-time data from X. Hence why it acts like a pre-pubescent school shooter and undresses everyone.

Worth noting that AI companies use third party vendors. Factories where data is scraped, cleaned and structured to create supervised datasets. Scale AI is the data engine that the big players use. Bright Data specialise in web data collection.

A checklist

OK, so we’re trying to feature in parametric memory. To appear in the LLMs training data so the model recognises you and you’re more likely to be used for RAG/retrieval. That means we need to:

1. Manage the multi bot ecosystem of training, indexing and browsing

2. Entity optimisation. Well structured, well connected content, consistent NAPs, sameAs schema properties and Knowledge Graph presence. In Google and Wikidata.

3. Make sure your content is rendered on the server side. Google has become very adept at rendering content on the client side. Bots like GPT-bot only see the HTML response. JS is still clunky.

4. Well structured, machine readable content in relevant formats. Tables, lists, properly structured semantic HTML.

5. Get. Yourself. Out. There. Share your stuff. Make noise.

6. Be ultra, ultra clear on your website about who you are. Answer the relevant questions. Own your entities.

You have to balance direct associations (what you say) with semantic association (what others say about you). Make your brand the obvious next word.

Modern SEO, with better marketing.

Leave a comment

1. Disambiguation is the process of resolving ambiguity and uncertainty in data. It’s crucial in modern day SEO and information retrieval

2. Search engines and LLMs reward content that is easy to ‘understand',’ not content that is necessarily best

3. The clearer and better structured your content, the harder it is to replace.

4. You have to reinforce how your brand and products are understood. When grounding is required, models favour sources they recognise from training data

The internet has changed. Channels have begin to homogenise. Google is trying to become something of a destination and the individual content creator is more powerful than ever.

Oh and we don’t need to click on anything.

But what makes for great content hasn’t changed. AI and LLMs haven’t changed what people want to consume. They’ve changed what we need to click on. Which I don’t necessarily hate.

As long as you’ve been creating well structured, engaging, educational/entertaining content for years. All this chat of chunking is a bit smoke and mirrors for me.

However, it is absolutely not all bullshit. Concepts like ambiguity are a more destructive force than ever. If you permit a quick double negative, you cannot not be clear.

The clearer you are. The more concise. The more structured on and off page. The better chance you stand. There’s no place for ambiguous phrases, paragraphs and definitions.

This is known as disambiguation.

Subscribe now

What is disambigation?

Disambiguation is the process of resolving ambiguity and uncertainty in data. Ambiguity is a problem in the modern day internet. The deeper down the rabbit hole we go, the less diligence is paid towards accuracy and truth. The more clarity your surrounding context provides, the better.

It is a critical component of modern day SEO, AI, Natural Language Processing (NLP), and information retrieval.

This is an obvious and overused example, but consider a term like apple. The intent and understanding behind it is vague. We don’t know whether people mean the company, the fruit, the daughter of a batshit, brain dead celebrity.

Years ago, this type of ambiguous search would’ve yielded a more diverse set of results. But thanks to personalisation and trillions of stored interactions, Google knows what we all want. Scaled user engagement signals and an improved understanding of intent, keywords, phrases and context are fundamental.

Yes, I could’ve thought of a better example, but I couldn’t be arsed.

You see my point.

Why should I care?

Modern day information retrieval requires clarity. The context you provide really matters when it comes to a confidence score systems require when pulling the ‘correct’ answer.

And this context is not just present in the content.

There is a significant debate about the value of structured data in modern day search and information retrieval. Using structured data like sameAs to signify exactly who this author is and tying all of your company social accounts and sub-brands together can only be a good thing.

The argument isn’t that this has no value. It makes sense.

• It’s whether Google needs it for accurate information parsing anymore

• And whether it has value to LLMs outside of well structured, pre-rendered HTML.

Ambiguity and information retrieval have become incredibly hot topics in data science. Vectorisation - representing documents and queries as vectors - helps machines understand the relationships between terms.

It allows models to effectively predict what words should be present in the surrounding context. It’s why answering the most relevant questions and predicting user intent and ‘what’s next’ has been so valuable for a long time in search.

See Google’s Word2Vec for more information.

Google has been doing this for a long time

Do you remember what Google’s early, and official, mission statement regarding information was?

Their former motto was “don’t be evil.” Which I think in more recent times they may have let slide somewhat. Or conveniently hidden it.

Organising the world’s information has become so much more effective thanks to advances in information retrieval. Originally Google thrived on straightforward keyword matching.

Then they moved to tokenisation.

Their ability to break sentences into words and match short-tail queries was revolutionary. But as queries advanced and intent became less obvious, they had to evolve.

The advent of Google’s Knowledge Graph was transformational. A database of entities that helped create consistency. It created stability and improved accuracy in an ever changing web.

Now queries are rewritten at scale. Ranking is probabilistic instead of deterministic and in some cases, fan out processes are applied to create an all-encompassing answer. It’s about matching the user’s intent at the time. It’s personalised. Contextual signals are applied to give the individual the best result for them.

Which means we lose predictability depending on temperature settings, context and inference path. There’s a lot more passage-level retrieval going on.

Thanks to Dan Petrovic, we know that Google doesn’t use your full page content when grounding its Gemini-powered AI systems. Each query has a fixed grounding budget of approximately 2,000 words total, distributed across sources by relevance rank.

The higher you rank in search, the more budget you are allotted. Think of this context window limit like crawl budget. Larger windows enable longer interactions, but cause performance degradation. So they have to strike a balance.

Subscribe now

Hummingbird, BERT, RankBrain - foundational semantic understanding

These older algorithm shifts were pivotal in making Google’s systems treat language and meaning differently.

• Hummingbird (2013) Helped Google identify entities and things quickly, with greater precision. This was an step toward semantic interpretation and entity recognition. Think of keywords at a page level. Not query level.

• RankBrain (2015) To combat the ever-increasing and never before seen queries, Google introduced machine learning to interpret unknown queries and relate them to known concepts and entities.

RankBrain was built on the success of Hummingbird’s semantic search. By mastering NLP systems, Google begin mapping words to mathematical patterns (vectorisation) to better serve new and ever-evolving queries.

These vectors help Google ‘guess’ the intent of queries it has never seen before by finding their nearest mathematical neighbors.

The Knowledge Graph Updates

In July 2023, Google rolled out a major Knowledge Graph update. I think people in SEO called it the Killer Whale Update, but I can’t remember who coined the phrase. Or why. Apologies. It was designed to accelerate the growth of the graph and reduce its dependence on third party sources like Wikipedia.

As somebody who has spent a long time messing around with entities, I can really understand why. It’s a giant, expensive time-suck.

It explicitly expanded and restructured how entities are recognised and classified in the Knowledge Graph. Particularly person entities with clear roles such as author or writer.

• The number of entities in the Knowledge Vault increased by 7.23% in one day to over 54 billion

• In July 2023, the number of Person entities tripled in just four days

All of this is an effort to combat AI slop, provide clarity and minimise misinformation. To reduce ambiguity and to serve content where a living, breathing expert is at the heart of it.

Worth checking whether you have a presence in the Knowledge Graph here. If you do and can claim a Knowledge Panel, do it. Cement your presence. If not, build your brand and connectedness on the internet.

What about LLMs & AI Search?

There are two main ways LLMs ‘retrieve’ information;

• By accessing their vast, static training data

• Using RAG (a type of grounding) to access external, up-to-date sources of information

RAG is why traditional Google Search is still so important. The latest models no longer train on real-time data and lag a little behind. Before the primary model dives in to respond to your desperate need for companionship, a classifier determines whether real time information retrieval is necessary.

They cannot know everything and have to employ RAG to make up for their lack of up to date information (or verifiable facts through their training data) when retrieving certain answers. Essentially trying to make sure they aren’t chatting shite.

Hallucinating if you’re feeling fancy.

So each model needs its own form of disambiguation. Primarily, this is achieved via;

• Context aware query matching. Seeing words as tokens and even reformatting queries into more structured formats to try and achieve the most accurate result. This type of query transformation leads to fan out and embeddings for more complex queries.

• RAG architectures. Accessing external knowledge when an accuracy threshold isn’t reached.

• Conversational agents. LLMs can be prompted to decide whether to directly answer a query or to ask the user for clarification if they don’t meet the same confidence threshold.

Remember, if your content isn’t accessible to search retrieval systems it can’t be used as part of a grounding response. There’s no separation here.

What should you do about it?

If you have wanted to do well in search over the last decade, this should’ve been a core part of your thinking. Helpful content rewards clarity.

Allegedly. It also rewards nerfing smaller sites out of existence.

Doesn’t mean you can’t be both. Great content entertains, educates, inspires and enhances.

Subscribe now

Use your words

You need to learn how to write. Short, snappy sentences. Help people and machines connect the dots. If you understand the topic, you should know what people want or need to read next almost better than they do.

• Use verifiable claims

• Cite your sources

• Showcase your expertise through your understanding

• Stand out. Be different. Add information to the corpus to force a mention and/or citation.

Structure the page effectively

Write in clear, straightforward paragraphs with a logical heading structure. You really don’t have to call it chunking if you don’t want to. Just make it easy for people and machines to consume your content.

• Answer the question. Answer it early.

• Use summaries or hooks.

• Tables of contents.

• Tables, lists and actual structured data. Not schema. But also schema.

Make it easy for users to see what they’re getting and whether this page is right for them.

Intent

Lots of intent is static. Commercial queries always demand some level of comparison. Transactional queries demand some kind of buying or sales process.

But intent changes and millions of new queries crop up every day.

So you need to monitor the intent of a term or phrase. News is probably a perfect example. Stories break. They develop. What was true yesterday may not be true today. The courts of public opinion damn and praise in equal measure.

Google monitors the consensus. Tracks changes to documents. Monitors authority and - crucially here - relevance.

You can use something like Also Asked to monitor intent changes over time.

The technical layer

For years, structured data has helped resolve ambiguity. But we don’t have real clarity over its impact on AI search. Cleaner, well structured pages are always easier to parse and entity recognition really matters.

• sameAs properties connect the dots with your brand and social accounts.

• It helps you explicitly state who your author is and crucially isn’t.

• Internal linking helps bots navigate across connected sections of your website and build some form of topical authority.

• Keep content up to date, with consistent date framing - on page, structured data and sitemaps

If you like messing around with the Knowledge Graph (who the hell doesn’t), you can find confidence scores for your brand.

According to Google’s very own guidelines, structured data provides explicit clues about a page’s content, helping search engines understand it better.

Yes yes, it displays rich results etc. But it removes ambiguity.

Entity matching

I think this ties everything together. Your brand, your products, your authors, your social accounts.

What you say about your brand matters now more than ever.

• The company you keep (the phrases on a page)

• The linked accounts

• The events you speak at

• Your about us page(s)

All of it helps machines build up a clear picture of who you are. If you have strong social profiles, you want to make sure you’re leveraging that trust.

At a page level, title consistency, using relevant entities in your opening paragraph, linking to relevant tag and articles page and using a rich, relevant author bio is a great start.

Really, just good, solid SEO. Don’t @ me.

PSA: Don’t be fucking boring. You won’t survive.

Leave a comment

1. To generate the most value from Discover, view it through an entity-focused lens. People, places, organisations, teams et al.

2. Your best chance of success in Discover with an individual article is to make sure it outperforms its expected performance early. So share, share, share

3. Then analyse the type of content you create. What makes it clickable? What resonates? What headline and image combination work?

4. High CTR is key for success, but ‘curiosity gap’ headlines that fail to deliver kill long-term credibility. User satisfaction trumps clickiness over time.

Discover isn’t a completely black box. We have a decent idea of how it works and can reverse engineer more value with some smart analysis.

Yes, there’s always going to be some surprises. It’s a bit mental at times. But we can make the most of the platform without destroying our credibility by publishing articles about vitamin B12 titled:

This kind of curiousity gap is killing your site’s credibility. With the algorithm and you know, real people. As John Shehata would say, be clicky, not baity.

Subscribe now

Key tenets of Discover

Before diving in headfirst, let’s check the depth of the pool.

• Search and Discover are intrinsically linked. Google has admitted this at their Zurich Search Central conference

• Discover feeds off fresh content. Whilst evergreen content pops up, it is very closely related to the news.

• More lifestyle-y, engaging content tends to thrive on the clickless platform.

• Just like news, Discover is very entity, click and early engagement driven.

• The personalised platform groups cohorts of people together. If you satiate one, more of that cohort will likely follow.

• If your content outperforms its predicted early stage performance, it is more likely to be boosted.

• Once the groups of potentially interested doomscrollers have been saturated, content performance naturally drops off.

• Google is empowering our ability to find individual creators and video content on the platform, because people trust people and like watching shit. Stunned.

Obviously loads of people know how to game the system and have become pretty rich by doing so. If you want to laugh and cry in equal measure, see the state of Google’s spam problems here.

Most algorithms follow the Golden Hour Rule. Not to be confused with the golden shower rule, it means the first 60 minutes after posting determine whether algorithms will amplify or bury your content.

If you want to go viral, your best bet is to drive early stage engagement.

What datapoints should you analyse?

This is focused more on how you as an SEO or analyst can get more value out of the platform. So let’s take conversions and click/impression data as read. We’re going deeper. This isn’t amateur hour.

I think you need to track the below and I’ll explain why.

• CTR

• Entities

• Subfolders

• Authorship

• Headlines and images

• Content type (just a simple breakdown of news, how-tos, interviews, evergreen guides etc)

• Publishing performance

You need to already get traffic from Discover to generate value form this analysis. If you don’t, revert back to creating high-quality, unique content in your niche(s) and push it out to the wider world.

Create great content and get the right people sharing it.

Worth noting you can’t accurately identify Discover traffic in analytics platforms. You have to accept some of it will be mis-attributed. Most companies make an educated guess of sorts, using a combination of Google and mobile/android to group it together.

CTR

CTR is one of the foundational metrics of news SEO, Top Stories, Discover and almost any form of real time SEO. It is far more prevalent in news than traditional SEO because the algorithm is making decisions about what content should be promoted in almost real time.

Evergreen results are altered continuously, based on much longer-term engagement.

This is weighted alongside some kind of traditional Navboost engagement data - clicks, on page interactions, session duration et al - to associate a clickable headline and image with content that serves the user effectively.

It’s also one of the reasons why clickbait shit has (broadly) started to die a death. Like rampant AI slop, even the mouth breathers will tire of it eventually.

To get the most out of CTR, you need to combine it with;

• Image type

• Headline type (content type too)

• And entity analysis

Subscribe now

Entity analysis

Entities are more important in news than any other part of SEO. Whilst entity SEO has been growing in popularity for years, news sites have been obsessed with entities (arguably without knowing it), for years.

Whist it isn’t as easy to just frontload headlines with relevant entities to get traffic anymore, there’s still an real value from analysing performance at an entity level.

Particularly in Discover.

You want to know what people, places and organisations (arguably these three make up 85%+ of all entities you need to care about) drive value for you and users in Discover.

To run proper entity analysis you cannot do this manually. At least not well or at scale.

My advice is to use a combination of your LLM of choice, an NER (Named Entity Recognition) tool and either Google’s Knowledge Graph or WikiData.

You can then extract the entity from the page in question (the title), disambiguate using the on page content (this helps you assess whether ‘apple’ is the computing company, the fruit or an idiotic celebrities daughter) and confirm it with WikiData or Google’s Knowledge Graph.

Subfolder

Relatively straightfoward, but you want to know which subfolders tend to generate more impressions and clicks on average in Discover. This is particularly valuable if you work on larger sites with a lot of subfolders and high content production.

You want to make sure that everything you do maximises value.

This becomes far more valuable when you combine this data with the type of headline and entities. If you begin to understand the type of headline (and content) that works for specific subfolders, you can help commissioners and writers make smarter decisions.

Subfolders that tend to perform better in Discover give individual articles a better chance of success.

Generate a list of all of your subfolders (or topics if your site isn’t setup particularly effectively) and tracking clicks, impressions and CTR over time. I’d use total clicks, impressions and CTR and an average per article as a starting point.

Authorship

Google tracks authorship in search. No ifs, no buts. The person who writes the content has significance when it comes to EEAT and good, reliable authorship makes a difference.

How much significance, I don’t know. And neither do you.

In breaking down all metrics from the leak that mention the word ‘author,’ the below is a how Google perceives and values authorship. As always, this is an imperfect science, but it’s interesting to note that of the 35 categories I reviewed, almost half are related just to identifying the author.

Disambiguation is one of the most important components of modern day search. Semantic SEO. Knowledge graphs. Structured data. EEAT. A huge amount of this is designed to counter false documents, AI slop and misinformation.

So it’s really important for search (and Discover), that you provide undeniable clarity.

For Discover specifically, you should see authors through the prism of:

• How many articles have they written that make it onto Discover (and that perform in Search)?

• What topic/entities do they perform best with?

• Ditto headline type

Headline type

This is a really good way of viewing the type of content that tends to perform for you. For example, you want to know whether curiousity gap headlines work well for you and whether headlines with numbers in have a higher or lower CTR on average.

• Do headlines with celebrities in the headline work well for you?

• Does this differ by subfolder?

• Do first person headlines have a higher CTR in Money than in News?

These are all questions and hypotheses that you should be asking. Although you can’t scrape Discover directly (trust me I’ve tried), you can hypothesise which h1, page title and OG title is the clickiest.

What’s interesting in this example is that ‘how-to’ headlines are not portrayed as very Discover-friendly. But it’s the concept that sells it. It’s different.

Start by defining all the types of headlines you use - curiousity gap, localised, numbered lists, questions, how-to or utility type, emotional trigger, first person et al - and analyse how effective each one is.

Use a machine learning model (you can absolutely use ChatGPT’s API) to categorise each headline.

• Train the model to identify place names, numbers, questions and first-person style patterns

• Verify the quality of the categorisation

• Break this down by subfolder, author, entity or anything else you choose

Worth noting that there are five different headlines you and Google can and should be using to determine how content is perceived. Discover is known to use the OG title more frequently than traditional search.

It’s an opportunity to create a ‘clickier’ headline than you would typically use in the h1 or page title.

Subscribe now

Images

Images fall into a similar category as headlines. They’re crucial. You can’t definitively prove which image gets pulled through into Discover. But as long as your featured image is 1200px wide, it’s safe-ish to assume this is the one that’s used.

CTR is arguably the single biggest factor in determining early success. Continued success I believe is more Navboost related - more traditional-ish engagement.

And CTR in Discover is determined by two things:

1. The headline

2. The image

Well, two things in your control. You could be pedantic and say ooo your brand is an important factor in CTR actually. Psychologically people always click on…

And I’d tell you to bore off. We’re talking about an individual article. We’ve done a significant amount of image testing and know that in straight news, people like seeing people looking sad. They like real-ness.

In money, they like people looking at the camera looking happy. It makes them feel safe in a financial decision.

Stupid, I know. But we’re not an intelligent race. Sure, there’s a few outliers. Galileo. Einstein. Noel Edmonds. But the rest of us are just trying not to throw shit at each other outside Yates’s on a Friday night.

It is actually why clickbait headlines have worked for years. It works until it doesn’t.

You’ll need to upload a set of images to help train the model and please for Christ’s sake don’t take it as gospel. Check the outputs. For the basics - whether people are present, where they’re looking, colour schemes etc - great. For more nuanced decisions like trustworthiness or emotional meaning, you’ll need to do that yourself.

Worth noting that lots of publishers trial badges and logos on images. And for good reason. Images with logos consistently click higher for larger brands (to the best of my knowledge) and if you’re a paywalled site, but have set live blogs to free, it’s worth telling people.

You should breakdown this image analysis into;

• Human presence and gaze

• Facial expression

• Emotional resonance

• Composition and framing

• Colour schemes

• Photo-type

Then you can use machine learning to bucket photos into groups to help determine CTR. For example, people directly looking at a camera + smiling could be one bucket. Not looking at a camera + scowling.

Publishing Performance

The more you publish, the more this matters.

Large newsrooms run analysis on publishing volumes, times and content freshness fairly consistently and at a desk-level. If you only have 50 or fewer articles per month making it into Discover, you probably don’t need to do this.

But if we’re talking about hundreds or thousands of articles, these insights can be really useful to commissioners.

I would focus on;

• Publishing days

• Publishing times

• Content freshness

• Republishing vs publishing

Your output should give really clear guidance to desks, commissioners and publishers around when is best to publish for peak Discover performance.

We never make direct recommendations solely for Discover for a number of reasons. Discover is a highly volatile platform and one that does reward nonsense. It can lead you down the garden path with all sorts of thin, curiousity gap style content if you just follow the numbers.

And it has limited direct impact on your bottom line.

How do you tie this all together?

You need a clear set of goals. Goals that help you deliver analysis that directly impacts the value of your content in Discover. When your set your analysis. focus on elements you have more control over.

For example, you might not be able to control what commisioners choose to publish, but you can change the headline (h1, title and/or OG) and image prior to publish.

1. Set a clear goal around conversions and traffic

2. Understand what you have more control over

3. Deliver insights at a desk or subfolder level

Understanding whether your role is more strategic or tactical is crucial. Strategic roles are more advisory in nature. You can offer some thoughts and advice on the type of headlines and entities to avoid or choose, but may not be able to change them.

Tactical roles mean you have more say in the implementation of change. Headlines, publish times, entity targeting etc.

Simple.

Some other goodies on this topic

That’s it, here’s some other cool stuff about Discover.

• Unlocking Google Discover by John Shehata

• Top 12 Ways to Optimize For Google Discover in 2025 by Metehan Yeşilyurt

• How To Succeed In Google Discover by Abby Hamilton

Leave a comment

Honestly I don’t think Google can handle this at all. The scale is unprecedented. They went after publishers manually with the site reputation abuse update. More expired domain abuse is reaching the top of the SERPs than any time I can remember in recent history. They’re fighting a losing battle and they’ve taken their eye off the ball.

A few years ago, search was getting on top of the various spam issues ‘creative’ SEOs were trialling. The prospect of being nerfed by a spam update and Google’s willingness to invest and care in the quality of search seemed to be winning the war. Trying to recover from these penalties is nothing short of disastrous. Just ask anybody hit by the Helpful Content update.

But things have shifted. AI is haphazardly rewriting the rules and big tech have bigger, more poisonous fish to fry. This is not a great time to be a white hat SEO.

TL;DR

1. Google is currently losing the war against spam, with unprecedented scale driven by AI-generated slop and expired domain and PBN abuse

2. Google’s spam detection monitors four key groups of signals - content, links, reputational and behavioural

3. Data from the Google Leak suggests its most capable detection focuses on link velocity and anchor text

4. AI ‘search’ is dozens of times more expensive that traditional search. This enormous cost and focus on new AI products is leading to underinvestment in core spam fighting

Subscribe now

How does Google’s Spam Detection System work?

Via SpamBrain. Previously the search giant rolled out Penguin, Panda and RankBrain to make better decisions based on links and keywords.

And right now, badly.

SpamBrain is designed to identify content and websites engaging in spammy activities with apparently ‘shocking’ accuracy. I don’t know whether shocking in this sense is meant in a positive or negative way right now, but I can only parrot what is said.

Over time, the algorithm learns what is and isn’t spam. Once it has clearly established signals associated with spammy sites, it’s able to create a neural network.

Much like the concept of seed sites, if you have the spammiest websites mapped out, you can accurately score everyone else against them. Then you can analyse signals at scale - content, links, behavioural and reputational signals - to group sites together.

• Inputs (content, linking reputational and behavioural signals)

• Hidden layer (clustering and comparing each site to known spam ones)

• Outputs (spam or not spam)

If your site is bucketed in the same group as obviously spammy sites when it comes to any of the above, that is not a good sign. The algorithm works on thresholds. I imagine you need to sail pretty close to the wind for long enough to get hit by a spam update.

But if your content is relatively thin and low value add, you’re probably halfway there. Add some dangerous links into the mix, some poor business decisions (parasite SEO being the most obvious example) and scaled content abuse and you’re doomed.

What type of spam are we talking about here?

Google notes the most egregious activities here. We’re talking;

• Cloaking

• Doorway abuse

• Expired domain abuse

• Hacked content

• Hidden text and content

• Keyword stuffing

• Link spam

• Scaled content abuse

• Site reputation abuse

• Thin affiliate content

• UGC spam

Lots of these are grossly intertwined. Expired domain abuse and PBNs. Keyword stuffing is a little old hat, but link spam is still very much alive and well. Scaled content abuse is at an all time high across the internet.

The more content you have spread across multiple, semantically similar websites, the more effective you can be. Using exact and partial match anchors to leverage your authority towards ‘money’ pages, the richer you will become.

Let’s dive into the big ones below.

Fake news

Google Discover - Google’s engagement baiting, social network-lite platform - has been hit by the unscrupulous spammers in recent times. There have been several instances of fake, AI-driven content reaching the masses. It’s become so prevalent, it has even reached legacy media sites (woohoo).

From changing the state pension age to free bus passes and TV licenses, the spammers know the market. They know how to incite emotions. Hell hath no fury like a pensioner scorned and whilst you can forgive the odd slip-up, nobody can be this generous.

The people who have been working by the book are being sidelined. But the opportunities in the black hat world are booming. Which is, in fairness, quite fun.

Scaled content abuse

At the time of writing, over 50% of the content on the internet is AI slop. Some say more. From nearly a million pages analysed this year, Ahrefs say 74% contain AI-content. What we see is just what slips through the mammoth-sized cracks.

According to award winning journalist Jean-Marc Manach’s research, he has found over 8,300 AI-generated news websites in French and over 300 in English (the tip of the iceberg, trust me).

He estimates two of these site owners have become millionaires.

By leveraging authoritative, expired domains and PBNs (more on that next) SEOs - the people still ruining the internet - know how to game the system. By faking clicks, manipulating engagement signals and utilising past link equity effectively.

Expired domain abuse

The big daddy. Black hat ground zero.

If you engage even a little bit with a black hat community, you’ll know how easy it is right now to leverage expired domains. In the example below, someone had bought the London Road Safety website (a once highly authoritative domain) and turned it into a single page ‘best betting sites not on GamStop’ site.

Betting and crypto are ground zero for all things black hat, just because there’s so much money involved.

I’m not an expert here, but I believe the process is as follows;

1. Purchase an expired, valuable domain with a strong, clean backlink history (no manual penalties). Ideally a few of them

2. Then you can begin to create your own PBN with unique hosting providers, nameservers and IP addresses, with a variety of authoritative, aged and newer domains

3. This domain(s) then becomes your equity/authority stronghold

4. Spin up multiple TLD variations of the domain i.e. instead of .com it becomes .org.uk

5. Add a mix of exact and partial match anchors from a PBN to the money site to signal its new focus

6. Either add a 301 redirect for a short period of time to the money variation of the domain or canonicalise to the variation

These scams are always short-term plays. But they can be worth tens of hundreds of thousands of pounds when done well. And they are back and I believe more valuable than ever.

Right now, I think it’s as simple as buying an old charity domain, adding a quick reskin and voila. A 301 or equity passing tactic and your single page site about ‘best casinos not on gamstop’ is printing money. Even in the English speaking market.

According to notorious black hat fella Charles Floate, some of these companies are laundering hundreds of thousands of pounds a month.

PBNs

A PBN (or Private Blog Network) is a network of websites that someone controls that link back to the money site. The variation of the site designed to generate typically advertising or affiliate revenue.

A private blog network has to be completely unique to each other. They cannot share breadcrumbs that Google can trace. Each site needs a standalone;

• Hosting provider

• IP address

• Nameserver

The reason PBNs are so valuable is you can build up an enormous amount of link equity and falsified topical authority to mitigate risk. Expired domains are risky because they’re expensive and once they get a penalty, they’re doomed. PBNs spread the risk. Like the head of a Hydra, one dies, another rises up.

Protecting the tier 1 asset (the purchased aged or expired domain) is paramount. Instead of pointing links directly to the money site, you can link to the sites that link to the money site.

This indirectly boosts the value of the money site, protecting it from Google’s prying eyes.

Subscribe now

What does the Google Leak show about spam?

As always this is an inexact science. Barely even pseudo science really. I’ve got the tinfoil hat on and a lot of string connecting wild snippets of information around the room to make this work. You should follow Shaun Anderson here.

If I take every mention of the word ‘spam’ in the module names and descriptions, there are around 115 once I’ve removed any nonsense. Then we can categorise those into content, links, reputational and behavioural signals.

Taking it one step further, these modules can be classified as relating to things like link building, anchor text, content quality et al. This gives us a rough sense of what matters in terms of scale.

A few examples:

• spambrainTotalDocSpamScore calculates a document’s overall spam score

• IndexingDocjoinerAnchorPhraseSpamInfo and IndexingDocjoinerAnchorSpamInfo modules identify spammy anchor phrases by looking at the number, velocity, the days the links were discovered and the time the spike ended

• GeostoreSourceTrustProto helps evaluate the trustworthiness of a source

Really the takeaway is how important links are from a spam sense. Particularly anchor text. The velocity at which you gain links matters. As does the text and surrounding content. Linking seems to be where Google’s algorithm is most capable of identifying red and amber flags.

If your link velocity graph spiked with exact match anchors to highly commercial pages, that’s a flag. Once a site is pinged for this type of content or link related abuse, the behavioural and reputational signals are analysed as part of SpamBrain.

If these corroborate and your site exceeds certain thresholds, you’re doomed. It’s why this has (until recently), been a relatively fine art.

Ultimately they’re just investing less in traditional search

As Martin McGarry pointed out, they just care a bit less… They have bigger, more hallucinogenic fish to fry.

In 2025, we have had four updates, with a duration of c. 70 days. In 2024, we had seven that lasted almost 130 days. Productivity levels we can all aspire to.

It’s not hard to guess why…

The bleeding edge search experience is changing. Google is rolling out preferred publisher sources globally and inline linking more effectively in its AI products. Much needed changes.

I think we’re seeing the real time moulding of the new search experience in the form of The Google Web Guide. A personalised mix of trusted sources, AI Mode, a more classic search interface and something inspirational. I suspect this might be a little like a Discover-lite feed. A place in the traditional search interface where content you will almost certainly like is fed to you to keep you engaged.

Unconfirmed, but apparently Google has added persona-driven recommendation signals and a private publisher entity layer among other things. Grouping users into cohorts is I believe a fundamental part of Discover. It’s what allows content to go viral.

Once you understand enough about a user to bucket them into specific groups, you can saturate a market over the course of a few days Discover. Less even. But the problem is the economics of it all. Ten blue links are cheap. AI is not. At any level.

According to Google, when someone chooses a preferred source, they click through to that site twice as often on average. So I suspect it’s worth taking seriously.

Why are AI searches so much more expensive?

Google is going to spend $10 billion more this year than expected due to the growing demand for cloud services. YoY, Google’s CAPEX spend is nearly double 2024’s $52.5 billion.

It’s not just Google. It’s a Silicon Valley race to the bottom.

Whilst Google hasn’t released public information on this, it’s no secret that AI searches are significantly more expensive than the classic ten blue links. Traditional search is largely static and retrieval-based. It relies on pre-indexed pages to serve a list of links and is fucking cheap to run.

An AI Overview is generative. Google has to run a large language model to summarise and generate a natural language answer. AI Mode is significantly worse. The multi-turn, conversational interface processes the entire dialogue in addition to the new query.

Given the query fan-out technique - where dozens of searches are run in parallel - this process demands significantly more computational power.

Custom chips, efficiencies and caching can reduce the cost of this. But this is one of Google’s biggest challenges. I suspect exactly why Barry believes AI Mode won’t be the default search experience. I’d be surprised if it isn’t just applied at a search/personalisation level too. There are plenty of branded and navigational searches where this would be an enormous waste of money.

And these guys fucking love money.

According to The IET, if the population of London (>9.7 million) asked ChatGPT to write a 100-word email this would require 4,874,000 litres of water to cool the servers – equivalent to filling over seven 25m swimming pools

Subscribe now

LLMs already have a spam problem

This is pretty well documented. LLMs seem to be driven at least in part by the shear volume of mentions in the training data. Everything is ingested and taken as read.

When you add a line in your footer describing something you or your business did, it’s taken as read. Spammy, low-quality tactics work more effectively than heavy lifting.

Ideally we wouldn’t live in a world where low lift shit outperforms proper marketing efforts. But here we are.

Like in 2012, ‘best’ lists are on the tip of everyone’s tongue. Basic bitch SEO is making a comeback because that’s what is currently working in LLMs. Paid placements, reciprocal link exchanges. You name it.

If it’s half-arsed, it’s making a comeback.

As these models rely on Google’s index for searches that the model cannot confidently answer (RAG), Google’s spam engine matters more than ever. In the same way that I think publishers need to take a stand against big tech and AI, Google needs to step up and take this seriously.

I’m not sure anyone is going to…

I’m not even sure they want to right now. OpenAI has signed some pretty extraordinary contracts and their revenue is light years away from where it needs to be. And Google’s CAPEX expenditure is through the fucking roof.

So things like quality and accuracy are not top of the list. Consumer and investor confidence is not that high. They need to make some money. And private companies can be a bit laissez-faire when it comes to reporting on revenue and profits.

According to HSBC, OpenAI needs to raise at least $207bn by 2030 so it can continue to lose money. Being described as ‘a money pit with a website on top’ isn’t a great look.

Let’s see them post-hoc rationalise their way out of this one. That’s it. Thank for you reading and subscribing to my last update of the year. Certainly been a year.

Further reading

• How Google turned authoritative domains into casino hotspots

• This post by Britney Muller on why LLMs make stuff up

• SEO myth busting quiz with Sarah Marshall

Leave a comment

To paywall.

To create a value exchange that reduces reliance on third party platforms. To become as self-sufficient as possible. Like an off-grid cabin or your mum’s basement, a paywall gives you a sense of security you just cannot put a price on.

As we’re all finding, any kind of reliance on these guys doesn’t put us in a good position. They do not want to send us traffic.

On the 4th March, 2026: Jessie and Shelby (WTF is SEO), Barry (SEO for Google News) and I will be running a paywall masterclass for publishers. We’ll run a near two hour long session on all things content, technical and commercial for paywall implementation and management.

Paywall Masterclass

TL;DR

1. Subscriber revenue is intrinsically more valuable to a business because it is predictable. Subscription and advertiser revenue are not created equal.

2. Don’t paywall everything. Use dynamic/metered paywalls and leave high-reach, generally lower-quality platforms like Google Discover free for email signups.

3. Subscription success relies on your USP - whether that’s exclusive data, deep, niche insights or a certain vibe - you have to stand out.

4. The customer experience and understanding of your audience matters. Create habit forming connections and products. Become an essential part of their life.

Subscribe now

But what about our traffic?

Your traffic will decline. But guess, what? You’re already hemorrhaging clicks and have been for some time. And traffic doesn’t pay the bills.

The only way to sustain rankings over time is with high quality engagement data. Navboost stores and uses 13 months of data to identify good vs bad clicks, click quality, the last longest click and on page interactions to establish the most relevant content. All at a query level.

Paywalls are not your friend when it comes to user engagement. Not for the masses. But for a small cohort of people who like you enough to pay, your engagement data will be excellent.

In an ultra personalised world, you will still do well to the people who really matter.

We have data that pretty perfectly highlights the impact of a paywall on rankings. Over the course of 3-4 months in traditional search, your rankings start to steadily drop before settling in severe mediocrity. You’ve got to fight for every click. With great content, marketing, savviness. Everything.

In Google Discover - a highly personalised, click and engagement driven platform - this is even more pronounced. Whilst Discover’s clickless traffic is lower quality, there will be a small cohort of highly engaged users that develop over time you can target with a paywall.

Unpaywall for the masses, build your owned channels and paywall for the highly engaged. The platform will take care of the personalisation for you.

So, maximise your value exchange with ads and email signups for most users, but don’t neglect those with a high return rate.

There’s some psychology involved in all of this. When a brand becomes widely known for paywalling, I suspect the likelihood of a click goes down as users know what to expect. Or maybe what not to expect.

This likely perpetuates over time, so you should clarify what articles are free to air.

Is our content good enough?

To nail SEO bingo, it depends. It depends what your value is in the market. There is a lot of free stuff out there already. But broadly shit. So as long as the bar keeps dropping, we’ll all be fine.

I am old ish. I like words. Writing great content isn’t easy and is usurped in many cases by richer, more visually striking content. Content that satisfies all types of users. Scanners, deep readers, listeners and get the answer and go-ers.

In some ways, you can satisfy all types of users more effectively than ever. I think you have to hit three of the four E’s of content creation. Make it resonate, be consistent and understand your audience. Whatever you create stands a chance.

But that doesn’t mean creating great stuff is any easier. If you work for a traditional publisher, the chances are you’ve brought a spoon to a gun fight. The war for attention is being fought on all fronts and straight words are losing.

Fortunately, not every subscription model relies on the quality of the prose. It might be that you have unique data, granular insights into a specific market or are just a bloody good laugh.

Subscriptions come in all shapes and sizes.

Ultimately, it comes down to your market, marketing, positioning and your USP. You have to know and speak to your audience and you have to stand out. As Barry would say, if you’re forgettable, you’re doomed.

Subscribe now

How do we know if people will pay?

When it comes to paying for news, some markets are far more ‘advanced’ than others. The Scandinavian market is light years ahead of almost everyone else when it comes to paying for news. You have to do your research to understand;

• How many people currently pay for news?

• What demographic of person pays?

• How saturated is the market already?

• What is your niche?

Whilst it doesn’t align perfectly, it’s not surprising that those most likely to pay for news have higher income levels. Higher disposable income tends to create an environment where people buy more stuff.

Shocking I know.

But that doesn’t tell the whole story. Norwegian news outlets have (apparently) a long history of trust with their audience and have never had access to free multi-day newspapers. Ditto other Scandinavian countries. In an age of shit and spin, trust and E-E-A-T are more important than ever.

And whilst the UK sits in a pretty shocking looking position, almost 24 million of us pay for a BBC license fee. That is, in essence, paying for news. Insert joke about BBC bias and woke cultural agendas here.

Cultural and societal factors really matter. As does your understanding of the market.

In an era of shit and slop, paying for something good is not a bad thing.

Macro and micro factors are influential

You can only control what you can control. But you shouldn’t dismiss the wider climate.

In the UK and arguably globally, there is a cost of living crisis. Globally there have been a number of very significant geopolitical issues that affect the wider economy. Money doesn’t go as far as it once did and most subscriptions are a luxury purchase.

Is a £20 or £30 monthly subscription more valuable than a £10 Netflix one? Or Spotify? These are questions you need to ask. Why would someone subscribe and stick around?

And we aren’t just competing with other publishers. Whilst screen time and content consumption is at an all-time high, video consumption and the creator economy are booming.

It is quite literally a near half a trillion dollar market.

Not strengths for traditional publishers. Whilst there have been some very good success stories in recent times (see Wired turning their journalists into individual subscription machines), legacy publishers need to adapt.

So your pricing strategy, customer service and overall experience is hugely important. You are almost certainly going to be a nice-to-have. So make sure your customer journey and path to conversion are premium and your audience feel listened to.

You need to speak to your audience. You don’t have to go into this blind. Forging real connections with people is not impossible and making them feel listened to will go a long way.

You can try to figure out what they really value, how much they’re willing to spend and what’s stopping them.

Should I paywall everything?

No. Content is designed to do different things and not everything is a premium product. Whatever journalists will tell you. If you shut down your site entirely, you become too closed off an ecosystem in my opinion.

• Commercial content: If you have affiliate-led content, paywalling that is a questionable decision. It' may not be wrong per se, but think about whether the pros outweigh the cons. Typically it’s a good gateway drug for the rest of your content. And makes some fucking money.

• Content you can get elsewhere: Evergreen content of a comparable quality to what already exists in the wider corpus is not a profitable opportunity. I’d argue that leaving this free-to-air has more pros than cons. You can always unpaywall the 100 best albums of all time, but gate the richer, individual album reviews.

• Lower-quality platforms: A user that comes from a platform like Discover is far less likely to convert than someone who comes from organic search. So think about the role each platform plays in your content access ecosystem.

• Paywall vs newsletter signup: It is far easier to convert people to a paying subscriber from a newsletter database than an on-page paywall. And the user journey is far less interrupted. Building an owned channel is never a bad thing, so think about how engaged users are and whether an email would be a more effective starting point.

The type of paywall matters (now more than ever)

LLMs do not respect paywalls. As it turns out, neither does Google.

I for one am stunned.

As of just a few months ago, the search giant asked that publishers with paywalls change the way they block content to help Google out. The lighter touch paywall solution (a Javascript-based one) includes the full content in the server response.

According to Google, they are struggling to determine the difference. So the problem is on us, not them. They (and I strongly suspect other LLMs) are ingesting this content and training their models on us whether we like it or not.

For those of you who haven’t heard of Common Crawl, it stores a corpus of open web data accessible to ‘researchers.’ By researchers we now mean tech bros who don’t want to pay for, surprisingly, anything.

It doesn’t stop there either. Even if you block all non-whitelisted bots from accessing your site at a CDN level, you may have syndication partnerships in place. If so, it’s likely your content is making it out into the wider world.

The internet is not exactly a leakproof vessel. If you’re setting one up now, try to implement a server-side option.

Subscribe now

What is the right paywall for me?

I have written about the types of paywall available to you and the pros and cons of each. Generally I think a metered or dynamic paywall is the best option for most publishers. At the very least a freemium model. Something that gives people enough to draw them in.

And you can’t exactly draw them in if you just hard paywall everything.

You have to think of this as a full blown marketing strategy. You need to know where people come from. How much of your content they have consumed. Whether it’s better to show them a newsletter signup as opposed to a paywall.

It is absolutely worth knowing that over time, a strong email database will convert far more effectively than a hard paywall.

So encouraging free signups and taking a longer-term view to conversions (you’ll need a good customer journey here) may be far more effective.

How can I set one up?

There are a number of paywall management options out there for publishers. Leaky Paywall, Zephr, Piano. There are plenty.

The best ones integrate with your existing tech stacks, have excellent personalisation and customisation options, deploy ad blocking strategies and run flexible gating strategies.

Larger publishers tend to go with enterprise level options with deep analytics and CRM integrations. Smaller publishers can work with lighter touch, cheaper operators. You really just need to scope out what will work best for you.

Particularly when it comes to monthly costs and revenue share options.

How can I map the impact?

You’ll need to establish a few key things;

• The average drop in traffic you expect to see

• The subsequent loss of existing revenue (probably ad related, but there may be some knock on wider commercial impact)

• The average value of a subscription (and the expected conversion rate)

• Your customer LTV

Focusing on Customer LTV shifts marketing from chasing traffic to profitable, loyal audience relationships. It makes businesses understand that not all audiences or subscriptions are created equal.

You generate more subs through paid media because the net is larger. But lots slip through the net. So you need a quality product (in both a product and marketing sense) alongside UX and customer service that reduces friction.

Search and owned channels are smaller, but far more likely to pay because they have taken an action to find you. In some cases, they actually want you in their inbox. The quality is higher, but the overall returns are lower.

So you just can’t treat everybody the same.

Closing thoughts

Subscriber revenue is so valuable because it’s predictable. Subscription business models have boomed for that very reason. A pound of subscriber revenue is far more valuable than almost anything else and it should be the focus of your business.

But that doesn’t mean you put all your eggs in one basket. You can have multiple subscription types on your website and that can help you become habitual with all types of users. But you need to add value to their lives everyday.

Puzzles, recipes, short and long-form videos et al.

Businesses make money in many ways. A diverse business is resilient. Resilient to macro and micro factors that will decimate some publishers over the next few years. So talk to your audience, trial new ways of adding value and commit when one works. Become habitual.

And, shock horror, people want to belong to something. So whilst the digital experience is crucial, making an effort to connect with people IRL matters.

Further reading

• Barry’s guide to paywalling

• Subscription and paywalled content markup (Google)

• Solute’s guide to paywalls

• Independent’s video and podcast Studio aims to become biggest revenue driver - Press Gazette

• What publishers need to do about zero click search - WTF is SEO

Leave a comment

But I’ve seen enough cult documentaries to know you should spit it back into the glass.

Do I love the opportunities it’s provided me in a work sense? Absolutely. Do I think it’s fundamentally shifted the marketing ecosystem? No. I think it’s expedited what’s been happening for some time.

• Reddit’s resurgence is search dominated

• The booming creator economy shows people trust people

• Word of mouth still travels

• Content still goes viral

• People don’t click unless they have to

LLMs provide a good proxy as to how you’re seen online. And they really lean into review platforms and strong brands. Associating your brand with your core topics, removing ambiguity and strengthening your product positioning is never a bad thing.

It’s not just about search anymore. In reality, it never should have been. It’s about connecting. Generating value from the different types of media.

Subscribe now

TL;DR

1. The search customer journey spans TikTok, YouTube, Instagram and everything in between.

2. Last-click attribution is outdated: BOFU platforms get the credit, but creators, communities and discovery platforms do the heavy lifting.

3. AI hasn’t broken anything, it’s just exposing how messy, multi-platform and people-driven it’s always been.

4. Brands win by understanding their audience, investing in creators, and building experiences that cut through an enshittified internet.

The customer journey has changed

True. But it’s been changing for a long time. Paid channels are becoming more expensive, owned channels like search send fewer clicks (mainly a Google driven mechanic) and earned channels are looking more like the golden ticket to corporate stooges.

The majority of brands use last click attribution (gross, get away from me). A method that overvalues search. For the last decade or more there have been discovery platforms that are more valuable than search - TikTok, YouTube, Instagram. Pick one.

We tend to use search for finding products, brands or stories we know exist. And for comparison related searches. But as AI Mode rolls ever closer and Google looks to greedily take on middle of the funnel queries, Google’s roll as a discovery platform will change. Theoretically at least.

Like every big tech company, enough is never enough and they don’t want to send you clicks. Unless you pay for them, of course.

Search is no longer a single platform journey

The Rise at Seven SEO and Data teams analysed 1.5 billion searches across 5 channels of the most-searched keywords on the internet and found that;

• A buying journey can take anywhere from 2 days to 10 weeks, with up to 97 interactions along the way.

• Google only accounts for just 34.5% of total search share

• YouTube (24%), TikTok (16.7%), and Instagram (20.9%) make up more than 60%

• The average consumer now uses 3.6 platforms before making a purchase.

But Google isn’t really a discovery platform. Maybe a bit. Google Shopping. Some comparison searches. But it’s not what anyone is there for.

Someone sees a product on Instagram or TikTok. They read a review of it on Reddit (probably through Google, albeit with a branded search) and watch videos of it on TikTok or YouTube.

They might even buy direct or via Amazon. At best, they perform a branded search in Google.

Now tell me last click attribution makes sense.

I think it’s worth noting here that so many of these other platforms are driven by a clickless algorithm. Google requires a click. A fundamental search. The others have homepages that stare directly into your soul.

I don’t think any of this is new. And I suspect it’s been a while since search was a single platform journey. But it depends what you define as search I suppose.

Google’s Messy Middle (lovingly renamed by me as the Clusterfuck Centre) is about right. We have been living through an era of marketing desperately tied to trying to track every penny. Something that has been a near impossible job for some time. At some point you just have to sit down, try to know your audience better than anyone else and have at it.

We need to influence clicks via search before that happens. Brands have to focus their time on the right channels for their audience. Not just search. That’s why knowing your audience and using an attribution model that doesn’t just value the BOFU click matters.

Subscribe now

But has AI been the catalyst?

Probably a bit. Behaviour has been changing long before LLMs hit the public arena. It’s changed because people have better options. More visually decisive. More authentic. The creator economy has boomed because people trust people.

• When I’m doomscrolling on the bog or on the tube (praise be to the 5G gods), I might get served a new product

• If I want real opinions or reviews about said product, I might go to Reddit (albeit through Google) to see what someone thinks. Well, I wouldn’t because I’m an adult with a wife and a mortgage, but you see my point

• I might subscribe to specific Substacks or creators who use and speak about the product

• My favourite LLM might give me product ideas (which I would check very fucking carefully)

• Hell, I might even see something IRL on the tube

A lot of this ends with a Google search. Maybe all of it. Google is a navigational engine. Hence the last click attribution issue. I suspect the last click isn’t the most important session in the majority of cases.

Unless you’re young, lazy or both, AI just won’t cut the mustard. Hell, Google’s kingpin tells you not to blindly trust AI. Even the guys fundamentally selling us this stuff are telling us it has some pretty serious flaws.

It’s one of the reasons user journeys are so much more complex and elongated.

1. We have more effective platforms and opinions than ever

2. We have more shitty platforms and opinions than ever

Cutting through the noise is everything. For people and brands. So you have to learn how to build brands and products that are bold and get the right people talking and sharing.

90% of marketers say creator content yields stronger engagement and 83% link it to more conversions. And 61% of consumers trust recommendations from creators more than they trust brand advertising.

The algorithms trust people because people do.

Channel-by-channel breakdown

Things don’t happen in a silo. Call me old-fashioned, but I think we’d all do well to work together as a marketing department. AIOs don’t just affect search. They have a knock on effect on the entire ecosystem and it’s important we understand the what and the why.

SEO

Christ where do we start. I’ll try and be brief. The most obvious and direct threat is zero click search, which has been on the rise for some time. Whilst AI hasn’t been the key driver of this, it has and will continue to reduce referral traffic.

• AIOs have significantly reduced CTR, particularly for informational, TOFU queries

• AI Mode is there to steal middle of the funnel clicks to ‘help users make the right decision’

• LLMs offer something of an alternative to search. Although based on what people really use them for I think they are complementary, rather than a replacement

I think AI has done some very interesting things in the SEO space. Vibe engineering platforms like Cursor and prototyping platforms like Lovable have opened up new worlds.

If you can wade through the shit, you can do some brilliant things.

Then you have Profound’s Zero Click conference where one of the speakers said he felt sorry for anyone working in SEO. According to this turdy savant, there’s very little crossover between SEO and insert favourite acronym before proceeding to discuss lots of SEO ideas from 2012.

People who just do not understand marketing, SEO, the internet or people. These are the guys driving the enshittification of our day to day lives. (More on this later).

PPC

PPC and SEO are ugly cousins really. We operate in the same space, we target the same traffic. So it stands to reason that AIOs and AI Mode significantly impact paid search.

If you can believe it, it’s broadly a negative.

I know. I too am stunned.

Thanks to Seer Interactive, we have near conclusive data that proves how serious this impact has been. When an AIO is present and you are not cited, clicks are down over 78%.

Even when there’s no AIO present, paid clicks are down 20%. This is disastrous. Customer acquisition becomes more expensive and the blended CPAs are significantly more expensive.

This may show a real and significant shift in user behaviour. Users are becoming so used to getting what they want from a TOFU search, they don’t even follow up when an AIO isn’t present.

Attention is slipping everywhere.

Social

We’ve seen the rapid rise of disinformation in search. Google’s been promoting fake content to millions of people on Discover and has been struggling to block them for some time. Gaming the system isn’t new. PBNs, expired domain abuse, link schemes. You name it, it’s working.

Thankfully the Vote Leave Take Control team have put their talents to good use and can now tell me what casino site should I choose.

The scale is unprecedented. Bullshit flies everywhere.

And that’s where social comes in. Globally the average person spends 2 hours and 24 minutes on social media every day. That’s a lot of time to be hit by fake news. Personalised fake news too. So maybe it’s not surprise that social use has been on the decline for the last couple of years.

According to this study by the FT, social media use has decreased by 10% and has been driven by (*shakes fist*) the youth. I think these platforms are a shell of what they once were. The connections they provided has been replaced by absolute bullshit.

They will do literally anything to get and hold your attention. Except help you stay in touch with people or watch something that isn’t AI generated. The content quality bell curve we see in search is mirrored by the enshittification of social channels.

• First, the platform attracts users with some bait, such as free access

• Then the activity is monetised, bringing in the business customers with no thought for the user experience

• Once everyone is ‘trapped,’ the value is transferred to their executives and shareholders

People with no understanding of marketing or people thinking that auto-generated comments will boost their profile on Linkedin. Businesses using AI to cut corners to generate meaningless bullshit and throwing it at me. See Coca Cola advert for reference.

The lights are on, the wheels are turning but nobody is home. Or gives a shite. The Mark Zuckerburgs of the world are, hopefully, turning people off hyper addictive brain rot.

Impressive, I know. Thank god for Ryan Air.

Subscribe now

Email marketing

As email is an owned channel, there’s not an obvious issue with generative AI. However, the Litmus State of Email Report shows the top roadblocks and operational challenges encountered by teams.

AI makes all of these roadblocks worse. Crummy, personality-devoid content churned out at scale will lower engagement. And it doesn’t take a genius to figure out that execs would love to save on personnel.

Operationally, you’d think AI will help. But if producing high-quality content at scale and improving your core benchmarks are fundamental issues, I’m not sure AI is the answer.

Personalisation, research and distribution. Absolutely. Creating content that draws real people in and engages with them. Colour me sceptical.

Paid vs earned vs owned

This is all about the funnel. If it becomes more expensive to acquire customers in their unaware/aware phase with paid campaigns, your owned and earned channels need to work harder. They need to work harder to increase your conversion rate.

• Paid campaigns or projects are designed to do two things: reach a newer potential audience and retarget an existing, highly qualified one. But they’re becoming more expensive. Especially in a PPC sense.

• Most sensible companies are trying to build their email databases off the back of search and organic social. Owned media is simultaneously under threat and incredibly valuable.

• Earned media - public exposure through word of mouth and shared content - is arguably more important than ever. People really trust people’s opinions.

  

What should you do?

As an SEO and a marketer, you should focus on creating real connections with people. Understanding your audience. Leveraging people that have influence over your audience. Build, work with and promote brilliant creators and own your audience data internally.

Squeeze every last drop out of your content. Cut and share it in appropriate formats across multiple channels.

Email is almost certainly the most applicable channel for most brands. You actually own it. Then figure out the role your brand plays in that journey. Create a great user experience on and off site. Make sure it’s well documented and you own everything in your control. Speak to your PPC and social teams to understand the challenges they’re having.

• Help Centre

• FAQ and product pages

• ToV consistency and brand guidelines

• Reviews and complaints (On and off site)

• Technical site quality

• Content quality

• Large scale, TOFU campaigns

This isn’t just about marketing. Or LLMs. They are just a good proxy for how you are seen on the internet.

It’s about working together as a marketing department with a shared goal of creating and amplifying brilliant experiences to the right people. Maximising the value of your owned channels, to reduce the reliance on paid and doing things that create brand advocates and cause your earned media to soar.

There’s an opportunity here to do great things!

But whatever you do, don’t forget about good quality SEO. It’s the primary purpose of our job and it still works.

Leave a comment

1. Search Console has some pretty severe limitations when it comes to storage, anonymised and incomplete data and API limits

2. You can bypass a lot of these limitations and make GSC work much harder for you but setting up far more properties at a subfolder level

3. You can have up to 1,000 properties in your Search Console account. Don’t stop with one domain level property

4. All of this allows for far richer indexation, query and page level analysis. All for free. Particularly if you make use of the 2k per property API URL indexing cap

Now, this is mainly applicable to enterprise sites. Sites with a deep subfolder structure and a rich history of publishing a lot of content. Technically this isn’t publisher-specific. If you work for an ecommerce brand this should be incredibly useful too.

I and it loves all big and clunky sites equally.

Subscribe now

What is a Search Console Property?

A Search Console Property is a domain, subfolder or subdomain variation of a website you can prove that you own.

If you just set up a domain level property you still get access to all the good stuff GSC offers. Click and impression data, indexation analysis and the crawl stats report (only available in domain level properties) to name a few. But you’re hampered by some pretty severe limitations;

• 1,000 rows of query and page level data

• 2,000 URL API limit for indexation level analysis each day

• Sampled keyword data (and privacy masking)

• Missing data (in some cases 70% or more)

• 16 months of data

Whilst the 16 month limit and sampled keyword data require you to export your data to BigQuery (or use one of the tools below), you can massively improve your GSC experience by making better use of properties.

There are a number of verification methods available - DNS verification, HTML tag or file upload, GA tracking code. Once you have setup and verified a domain level property, you’re free to add any child-level property. Subdomains or subfolders alike.

The crawl stats report can be extremely useful for debugging issues like spikes in parameter URLs or from naughty subdomains. Particularly on large sites where departments do things you and I don’t find out about until it’s too late.

But by breaking down changes at a host, file type and response code level, you can stop things at the source. Easily identify issues affecting your crawl budget before you want to hit someone over the head with their approach to internal linking and parameter URLs.

Usually, anyway. Sometimes people just need a good clump. Metaphorically speaking of course.

Subdomains are usually seen as separate entities with their own crawl budget. However, this isn’t always the case. According to John Mueller, it is possible that Google may group your subdomains together for crawl budget purposes.

According to Gary Illyes, crawl budget is typically set by host name. So subdomains should have their own crawl budget if the host name is separate from the main domain.

How can I identify the right properties?

As an SEO, it’s your job to know the website better than anybody else. In most cases that isn’t too hard because you work with digital ignoramuses. Usually you can just find this data in GSC. But larger sites need a little more love.

Crawl your site using Screaming Frog, Sitebulb, the artist formerly known as Deepcrawl and build out a picture of your site structure if you don’t already know. Add the most valuable properties first (revenue first, traffic second) and work from there.

Some alternatives to GSC

Before going any further it would be remiss of me not to mention some excellent alternatives to GSC. Alternatives that completely remove these limitations for you.

SEO Stack

SEO Stack is a fantastic tool that removes all query limits, has an in built MCP style setup where you can really talk to your data. For example, show me content that has always performed well in September or identify pages with a health query counting profile.

Daniel has been very vocal about query counting and it’s a fantastic way to understand the direction of travel your site or content is taking in search. Going up in the top 3 or 10 positions - good. Going down there and up further down - bad.

SEO Gets

SEO Gets is a more budget-friendly alternative to SEO Stack (which in itself isn’t that expensive), SEO Gets also removes the standard row limitations associated with Search Console and makes content analysis much more efficient.

Create keyword and page groups for query counting and click and impression analysis at a content cluster level. SEO Gets has arguably the best free version of any tool on the market.

Indexing Insight

Indexing Insight - Adam Gent’s ultra-detailed indexation analysis tool - is a life saver for large, sprawling websites. 2k URLs per day just doesn’t cut the mustard for enterprise sites. But by cleverly taking the multi-property approach you can leverage 2k URLs per property.

With some excellent visualisations and datapoints (did you know if a URL hasn’t been crawled for 130 days it drops out of the index?), you need a solution like this. Particularly on legacy and enterprise sites.

All of these tools instantly improve your Search Console experience.

Subscribe now

Benefits of a multi property approach

Arguably the most effective way of getting around some of the aforementioned issues is to scale the number of properties you own. For two main reasons - it’s free and it gets around core API limitations.

Everyone likes free stuff. I once walked past a newsagent doing an opening day promotion where they were giving away tins of chopped tomatoes. Which was bizarre. What was more bizarre was that there was a queue. A queue I ended up joining.

Spaghetti bolognaise has never tasted so sweet.

Granular indexation tracking

Arguably one of Search Console’s best but most limiting features is its indexation analysis. Understanding the differences between Crawled - Currently Not Indexed and Discovered - Currently Not Indexed can help you make smart decisions that improve the efficiency of your site. Significantly improving your crawl budget and internal linking strategies.

Pages that sit in the Crawled - Currently Not Indexed pipeline may not require any immediate action. The page has been crawled, but hasn’t been deemed fit for Google’s index. This could signify page quality issues, so worth ensuring your content is adding value and your internal linking prioritises important pages.

Discovered - Currently Not Indexed is slightly different. It means that Google has found the URL, but hasn’t yet crawled it. It could be that your content output isn’t quite on par with Google’s perceived value of your site. Or that your internal linking structure needs to better prioritise important content. Or some kind of server of technical issue.

All of this requires at least a rudimentary understanding of how Google’s indexation pipeline works. It is not a binary approach. Gary Illyes said Google has a tiered system of indexation. Content that needs to be served more frequently is stored in a better quality, more expensive system. Less valuable content is stored in a less expensive system.

Less monkey see monkey do; more monkey see, monkey make decision based on the site’s value, crawl budget, efficiency, server load and use of Javascript.

The tiered approach to indexation prioritises the perceived value and raw HTML of a page. JavaScript is queued because it is so much more resource intensive. Hence why SEOs bang on about having your content rendered on the server side.

Adam has a very good guide to the types of not indexed pages in GSC and what they mean here.

Worth noting the page indexation tool isn’t completely up to date. I believe it’s updated a couple of times a week. But I can’t remember where I got that information, so don’t hold me to that…

If you’re a big news publisher you’ll see lots of your newsier content in the Crawled - Currently Not Indexed category. But when you inspect the URL (which you absolutely should do) it might be indexed. There is a delay.

Indexing API scalability

When you start working on larger websites - and I am talking about websites where subfolders have well over 500,000 pages - the API’s 2k URL limitation becomes apparent. You just cannot effectively identify pages that drop in and out of the ‘Why Pages Aren’t Indexed?’ section.

But when you set up multiple properties, you can scale effectively.

The 2k limit only applies at a property level. So if you set up a domain level property alongside 20 other properties (at the subfolder level), you can leverage up to 42k URLs per day. The more you do the better.

And the API does have some other benefits;

• You can request indexation or removal of specific URLs (2k per day limit again)

• You can send real-time notifications when pages are launched or removed. Think of it like Bing’s IndexNow

• Speed up the indexation of time-sensitive content

But it doesn’t guarantee indexing. It is a request, not a command.

To set it up, you need to enable the API in Google Cloud Console. You can follow this semi helpful quickstart guide. It is not fun. It is a pain in the arse. But it is worth it. Then you’ll need a Python script to send API requests and to monitor API quotas and responses (2xx, 3xx, 4xx etc).

If you want to get fancy, you can combine it with your publishing data to figure out exactly how long pages in specific sections take to get indexed. And you should always want to get fancy.

This is a really good signal as to what your most important subfolders are in Google’s eyes too. Performant vs under-performing categories.

Granular click and impression data

An essential for large sites. Not only does the default Search Console only store 1k rows or query and URL data, but it only stores it for 16 months. While that sounds like a long time, fast forward a year or two and you will wish you had started fucking storing the data in BigQuery.

Particularly when it comes to looking at YoY click behaviour and event planning. The teeth grinding alone will pay for your dentist’s annual trip to Aruba.

But by far and away the easiest way to see search data at a more granular level is to create more GSC properties. Whilst you still have the same query and URL limits, because you have multiple properties instead of one, the data limits become far less limiting.

Subscribe now

What about sitemaps?

Not directly related to GSC indexation, but a point of note. Sitemaps are not a particularly strong tool in your arsenal when it comes to encouraging indexing of content. The indexation of content is driven by how ‘helpful’ it is to users.

Now, it would be remiss of me not to highlight that news sitemaps are slightly different. When speed to publish and indexation are so important, you want to highlight your freshest articles in a ratified place.

Ultimately it comes down to Navboost. Good vs bad clicks and the last longest click. Or in more of a news sense, Glue - a huge table of user interactions, designed to rank fresh content in real-time and keep the index dynamic. Indexation is driven by your content being valuable enough to users for Google to continue to store in its index.

Thanks to decades of experience (and confirmation via the DoJ trial and the Google Leak), we know that your site’s authority (Q*), impact over time and internal linking structure all play a key role. But once its indexed, it’s all about user engagement. Sitemap or no sitemap, you can’t force people to love your beige, miserable content.

And sitemap indexes?

Most larger sites use sitemap indexes. Essentially a sitemap of sitemaps to manage larger websites that exceed the 50k row limit. When you upload the sitemap index to Search Console, don’t stop there. Upload every individual sitemap in your sitemap index.

This gives you access to indexation at a sitemap level in the page indexing or sitemaps report. Something that is much harder to manage when you have millions of pages in a sitemap index.

Take the same approach with sitemaps as we have discussed with properties. More is generally better.

Worth knowing that each document is also given a DocID. The DocID stores signals to score the page’s popularity: user clicks, its quality and authoritativeness, crawl data, and a spam score among others.

Anything classified as crucial to ranking a page is stored and used for indexation and ranking purposes.

What should I do next?

1. Assess your current GSC setup - is it working hard enough for you?

2. Do you have access to a domain level property and a crawl stats report?

3. Have you already broken your site down into ‘properties’ in GSC?

4. If not, crawl your site and establish the subfolders you want to add

5. Review your sitemap setup. Do you just have a sitemap index? Have you added the individual sitemaps into GSC too?

6. Consider connecting your data to BigQuery and storing more than 16 months of it

7. Consider connecting to the API via Google Cloud Console

8. Review the above tools and see if they’d add value

Ultimately Search Console is very useful. But it has significant limitations and to be fair, it is free. Other tools have surpassed it in many ways. But if nothing else you should make it work as hard as possible.

Other good stuff

• Reddit tips for publishers - Press Gazette

• The Company Quietly Funneling Paywalled Articles to AI Developers - The Atlantic

• Why you should verify your image cloud provider (subfolder or subdomain) in Search Console - Mark Williams-Cook/John Mueller

Leave a comment

Like a disgraced BBC employee, thoughts are my own. Not a perfect science.

TL;DR

1. Your site needs to be seen as a ‘trusted source’ with low SPAM, evaluated by a variety of proxies like site, author and trust scores in order to be eligible

2. I think Discover is driven by a six-part pipeline, using good vs. bad clicks (long dwell time vs. pogo-sticking) and repeat visits to continuously score and re-score content quality

3. Fresh content gets an initial boost. Success hinges on a strong CTR and positive early-stage engagement (good clicks/shares from all channels count, not just Discover).

4. Content that aligns with a user’s interests is prioritised. To optimise, focus on your areas of topical authority, use a compelling headline(s), be entity-driven and use large (1200px+) images.

Updated: I had used the wrong names for Google Leak proxies thanks to a touch of carelessness and a malfunctioning database and custom GPT. These have now been updated. Thank you to Johan Hülsen for pointing this out.

There are dozens of potential proxies that Google likely uses to guide satiate the doomscrollers desperate need for quality content in the Discover feed. It’s not that different to how traditional Google search works.

But traditional search (a high-quality pull channel) is worlds apart from Discover. Audiences killing time on trains. At their in-laws. The toilet. Given they’re part of the same ecosystem, they’re bundled together into one monolithic entity.

And here’s how it works.

Subscribe now

Google’s Discover guidelines

This section is boring and Google’s guidelines around eligibility are exceptionally vague:

• Content is automatically eligible to appear in Discover if it is indexed by Google and meets Discover’s content policies

• Any kind of dangerous, spammy, deceptive or violent/vulgar content gets filtered out

Then they give some solid, albeit beige advice around quality titles - clicky, not baity as John Shehata would say. Ensuring your featured image is at least 1200px wide and creating timely, value-added content.

But we can do better.

Discover’s six part content pipeline

From cradle to grave, let’s review exactly how your content does or in most cases doesn’t appear in Discover. As always remembering I have made these clusters up, albeit based on real Google proxies from the Google Leak.

But they seem to tally very well from my (and others) experience of the platform.

1. Eligibility check and baseline filtering

2. Initial exposure and testing

3. User quality assessment

4. Engagement and feedback loop

5. Personalisation layer

6. Decay and renewal cycles

Eligibility and baseline filtering

For starters, your site has to be eligible for Google Discover. This means you are seen as a ‘trusted source’ on the topic and you have a low enough SPAM score that the threshold isn’t triggered.

There are a few primary proxy scores I suspect account for eligibility and baseline filtering. Before your site can feature on the platform, it has to hit a few basic criteria.

• siteAuthority, trust or trustedScore: scores that evaluates a site’s reliability and reputation and form the basis of eligibility filtering at a platform and topic level.

• topicalityScore: a higher topicality score that helps Discover identify trusted sources at the topic level

• isAuthor and authorObfuscatedGaiaStr help Google identify authoritative sources alongside site level metrics.

The site’s reputation and topical authority are ranked for the topic at hand. These three metrics help evaluate whether your site is eligible to appear in Discover. We know authorship is scored and matters too. We just don’t know to what extent.

You can find out more about SPAM proxies like SpamRank or GibberishScore here.

Subscribe now

Initial exposure and testing

This is very much the freshness stage, where fresh content is given a temporary boost (because contemporary content is more likely to satiate a dopamine addicted mind).

• browsyTopic: Based on a lot of prior data, a Discover-worthy topic can be identified using an experimental topic score

• isHotDoc: Extremely new articles around trending topics get a short term ranking boost.

• freshnessboxArticleScores: helps Google prioritise more recent articles for time sensitive queries. Ditto contentfreshness.

• goldmineTrustFactor: Google uses this to prioritise articles and titles with a good reputation and history of reliability. See Shaun Anderson’s excellent work on this.

I would hypothesise that using a Bayesian style predictive model, Google applies learnings at a site and subfolder level to predict likely CTR. The more quality content you have published over time (presumably at a site, subfolder and author level), the more likely you are to feature.

Because there is less ambiguity. A key feature of SEO now.

User quality assessment

An article is ultimately judged by the quality of user engagement. Google has used the good and bad click style model from Navboost to establish what is and isn’t working for users for years.

Low CTR and/or pogosticking style behaviour downgrades an article’s chance of featuring. No reason to think any different in Discover.

• unsquashedClicks and impressions: Data that suggests a high CTR would work well for a person and cohort of users. See also dailyClicks.

• clicksTotalInterval: Total number of clicks received within a specific time period. For Discover, I suspect this is compressed.

• imageClickRank: Interesting to note Navboost stores image data. As these models have some pretty advanced reasoning, not unreasonable to suspect they know what type of headline and image combo will perform well.

The quality of any article is measured by its user engagement. Cohorts of users can be grouped together to form effective personalisation testing. People with the same general interests are served the content if they should be interested.

Speed and scale can be achieved on Discover far more rigorously.

But if the overly clicky or misleading title delivers poor engagement (dwell time and on page interactions), then the article may be downgraded. Over time this kind of practice can compound and nerf your site completely.

Important to note that this click data doesn’t have to come from Discover. Once an article is out in the ether - it’s been published, shared on social etc - Chrome click data is stored and is applied to the algorithm.

So the more quality click data and shares you can generate early in an article’s lifecycle (accounting for the importance of freshness), the better your chance of success on Discover.

Treat it like a viral platform. Make noise. Do marketing.

Engagement and feedback loop

Once the article enters the proverbial fray, a scoring and rescoring loop begins. Continuous CTR, impressions, interactions and click quality feed models like Navboost to refine what gets shown.

Valuable content is decided by the good vs bad click ratio. Quality visits are used to measure lasting satisfaction and re-rank top performing content.

• lastLongestClick: likely measures user satisfaction at a topic and cohort level in Discover

• clicksBad and clicksGood: once it has aggregated enough data, it can make a distinction at an article level for good vs bad click data.

• Google uses re-ranking functions called Twiddlers including a FreshnessTwiddler to boost newer content in traditional search.

In Discover this is all quite simple. Apply impression and click data at a topic and cohort level over a compressed period of time. Voila.

These behavioural signals define an article’s success. It lives or dies on relatively simple metrics. And the more you use it, the better it gets. Because it knows what you and your cohort are more likely to click and enjoy.

I imagine headline and image data are stored so that the algorithm can apply some rigorous standards to statistical modelling. Once it knows what types of headlines, images and articles perform best for specific cohorts, personalisation becomes effective faster.

Personalisation layer

Now I don’t want to panic you, but Google knows a lot about us.

It collects a lot of non-anonymised data (credit card details, passwords, contact details etc) alongside every conceivable interaction you have with web pages. It knows what we care about in the moment and how we’ve evolved over time.

Discover takes personalisation to the next level. I think it may offer an insight into how part of the SERP could look like in the future. A personalised cluster of articles, videos and social posts designed to hook you in embedded somewhere alongside search results and AI Mode.

All of this is designed to keep you on Google’s owned properties for longer. Because they make more money that way. And they have the click data in Discover to do it,

It starts getting a bit messy with specific peronalisation modules from the leak. Fair to say that the below help the algorithm improve user experience by leveraging past interactions and experiences.

• personalizationGameWebEventsConsolidatedEvents

• copleySourceTypeList

• RepositoryWebRefPersonalizationContextOutputs

• socialgraphNodeNameFp

If you search for things from or about specific people, places or organisations (entities!) more frequently, that will be taken into account. Think of everything you do as a fragment of your digital fingerprint. One Google almost monopolistically owns.

In search at least!

Content that matches well with your personal and cohort’s interest will be boosted into your feed.

You can see the site’s you engage with frequently using the site engagement page in Chrome (from your toolbar: chrome://site-engagement/) and every stored interaction with histograms. This histogram data indirectly shows key interaction points you have with web pages, by measuring the browser’s response and performance around those interactions.

It doesn’t explicitly say user A clicked x, but logs the technical impact. I.e. how long did the browser spending processing said click or scroll.

Decay and renewal cycles

Discover boosts freshness because people are thirsty for it. By boosting fresh content, older or saturated stories naturally decay as the news cycle moves on and article engagement declines.

For successful stories in Discover, this is probably through market saturation.

• Google stores substantial date data in its PerDocData (Google’s leaked core document model). This could be a applied at a topic and entity level in Discover.

• timeSensitivity is also stored to help identify content where recency and freshness is more important.

• And the NLPSemanticParsingModel stores the publication time in a direct effort to prioritise fresher content on topics specifically from news sites.

None of us spend time in Google. It’s not fun. It isn’t designed to hook us in and ruin our attention spans with constant spiking of dopamine.

But Google Discover is clearly on the way to that. They want to make it a destination. A social network. hence all the recent changes where you can ‘catch up’ with creators and publishers you care about across multiple platforms. Videos, social posts, articles… The whole nine yards.

I wish they’d fuck off with summarising literally everything with AI however.

Subscribe now

My 11 step workflow to get the most out of Google Discover

Follow basic principles and you will put yourself in good stead. Understand where your site is topically strong and focus your time on content that will drive value. Multiple ways you can do this.

If you don’t feature much in Discover, you can use your Search Console click and impressions data to identify areas where you generate the highest value. Where you are topically authoritative. I would do this at a subfolder and entity level (e.g. politics and Rachel Reeves or the Labour Party).

Also worth breaking this down in total and by article. or you can use something like Ahrefs’ Traffic Share report to determine your share of voice via third party data.

Then really focus your time on a) areas where you’re already authoritative and b) areas that drive value for your audience.

Assuming you’re not focusing on NSFW content and you’re vaguely eligible, here’s what I would do:

1. Make sure you’re meeting basic image requirements. 1200 pixels wide as a minimum.

2. Identify your areas of topical authority. Where do you already rank effectively at a subfolder level? Is there a specific author who performs best? Try to build on your valuable content hubs with content that should drive extra value in this area.

3. Invest in content that will drive real value (links and engagement) in these areas. Do not chase clicks via Discover. It’s a one way ticket to clickbait city. It’s shit there.

4. Make sure you’re plugged into the news cycle. Being first has a huge impact on your news visibility in search. If you’re not first on the scene, make sure you’re adding something additional to the conversation. Be bold. Add value. Understand how news SEO really works.

5. Be entity driven. In your headlines, first paragraph, subheadings, structured data and image ALT text. Your page should remove ambiguity. You need to make it incredibly clear who this page is about. A lack of clarity is partly why Google rewrites headlines.

6. Use the Open Graph title. The OG title is a headline that doesn’t show on your page. Primarily designed for social media use it is one of the most commonly picked up headlines in Discover. It can be jazzy. Curiousity led. Rich. Interesting. But still entity-focused.

7. Make sure you share content likely to do well on Discover across relevant push channels early in its lifecycle. It needs to outperform its predicted early stage performance.*

8. Create a good page experience. Your page (and site) should be fast, secure, ad-lite and memorable for the right reasons.

9. Try to drive quality onward journeys. If you can treat users from Discover differently to your main site, think about how you would link effectively for them. Maybe you use a pop-up ‘we think you’ll like this next’ section based on a user’s scroll depth of dwell time.

10. Get the traffic to convert. Whilst Discover is a personalised feed, the standard scroller is not very engaged. So focus on easier conversions like registrations (if you’re a subscriber first company) or advertising revenue et al.

11. Keep a record of your best performers. Evergreen content can be refreshed and repubbed year after year. It can still drive value.

Anyway that’s it, let me know what you think in the comments!

Some other Discover goodies

• Lily Ray’s guide to driving traffic in Discover

• Abby Hamilton’s guide to success in Discover

• SEJ interview with John Shehata about Discover

Leave a comment

1. Google can and does rewrite headlines and titles frequently. Almost anything on your page could be used.

2. The title is not all that matters. The entirety of your page - from the title to the on page content - should remove ambiguity

3. The title tag is the most important term. Stick to 12 words and 600 pixels to avoid truncation and maximise value from each word

4. Google uses three rough concepts - Semantic title and content alignment, satisfactory click behaviour and searcher intent alignment - for this

This is based on the Google Leak documentation and Shaun Anderson’s excellent article on title tag rewriting. I’ve jazzed it to make it more news and publisher-specific.

No idea if that’s true or not.

I’m sure it’s some old age advertising bullshit. But alongside the featured image, it is our shop window. Headlines are the gatekeepers. They need to be clickable, work for humans and machines and prioritise clarity.

So when you’ve spent a long-time crafting a headline for your own fucking story, why-oh-why does Google fuck you around?

I’m sure you get a ton of questions from other people in the SEO team and the wider newsroom (or the legal team) about this.

Something like;

You could rinse and repeat this conversation and theory for almost anything. Meta descriptions are the most obvious situation, where some research shows they’re rewritten 70% of the time. The answer will, unfortunately, always be that, because Google can and does do what it wants.

But it helps to know the what and the why when having these conversations.

Mark Williams-Cook and team did some research to show that up to 80% of meta descriptions were being rewritten and the rewriting increased traffic. Maybe the machine knows best after all.

Subscribe now

Why does Google rewrite title tags?

The search giant uses document understanding, query matching, content rewriting, and user engagement data to determine when a title or H1 should be changed in SERPs.

It rewrites them because it knows what is best satisfying users in real time. An area of search where we as publishers are at the bleeding edge. When you have access to that much data and you take a share of ad revenue, it would be a little obtuse not to optimise for clicks in real-time.

Does length matter?

No innuendos please, this is a professional newsletter.

Google’s official documentation doesn’t define a limit for title tags. I think it’s just based on the title becoming truncated. Given Google now rewrites so much, longer, more keyword-rich and descriptive titles, longer titles could help with ranking in Top Stories and traditional search results.

According to Gary Illyes there is real value in having longer title tags:

Sara Taher ran some interesting analysis (albeit on evergreen content only) that showed the average title length falls between 42-46 characters. If titles are too long, Google will probably cut them off or rewrrite them. Precision matters for evergreen search.

What are the key determinants?

Based on the Google Leak and Shaun’s analysis, I’d say there are three concepts at play Google uses to determine whether a title should be rewritten. I have made this up by the way, so feel free to use your own.

• Semantic title and content alignment

• Satisfactory click behaviour

• Searcher intent alignment

Subscribe now

Semantic title and content alignment

This is undoubtedly the most prominent section. Your on-page content and title/headline have to align.

This is why clickbait content and content written directly for Google Discover is so risky. Because you’re writing a cheque that your ass can’t cash. Create content specifically for a platform like Discover and you will erode your quality signals over time.

You need to understand how to analyse the platform properly.

The titlematchScore, h1ContentScore and spammyTitleDetection review the base quality of a headline based on the page’s content and query intent. Mismatched titles, headlines and keyword-stuffed versions are, at best, rewritten.

At worst, they downgrade the quality of your site algorithmically.

The titleMatchAnchorText ensures our title tags and header(s) are compared to internal and external anchors and evaluated in comparison to the hierarchy of the page (the headingHierarchyScore).

Finally, the ‘best’ title is chosen from on-page elements via the snippetTitleExtraction. Whilst Google primarily uses the title or h1 tag, any visible element can be used if it ‘best represents the page.’

Satisfactory click behaviour

Much more straightforward. Exactly how Google uses user engagement signals (think of Navboost’s good vs bad click signals) to best cultivate a SERP for a particular term and cohort of people.

The titleClickSatisfaction metric combines click data at a query level with on-page engagement data (think scroll depth, time on page, on-page interactions, pogo-sticking).

So ranking adjustments are made if Google believes the title used in the SERP is underperforming against your prior performance and the competition. So the title you see could be one of many tests happening simultaneously, I suspect.

For those unfamiliar with Navboost, it is one of Google’s primary ranking engines. It’s based on user interaction signals, like clicks, hovers, scrolls, and swipes, over 13 months to refine rankings.

For news publishers, Glue helps rank content in real time for fresh, real-time events. Source and page level authority. It’s a fundamental part of how news SEO really works.

Searcher intent alignment

Searcher intent really matters when it comes to page titles. And Google knows this far better than we do. So if the content on your page (headings, paragraphs, images et al) and search intent isn’t reflected by your page title, it’s gone.

Once a page title has been identified as not fit for purpose, the pageTitleRewriter metric is designed to rewrite ‘unhelpful or misleading page titles.’

And page titles are rewritten at a query level. The queryIntentTitleAlignment measures how the page title aligns with searcher intent. Once this is established, the page alignment and query intent is reviewed to ensure the title best reflects the page at a query level.

Then the queryDependentTitleSelection adjusts the title based on the specifics of the search and searcher. Primarily at the query and location-level. The best contextual match is picked.

Subscribe now

Suggestions for publishers

I’ll try to do this (in a vague order of precedence):

1. Make your title stand out. Be clickable. Front load entities. Use power words, numbers or punctuation where applicable.

2. Stick to 12 words and 600 pixels to avoid truncation and maximise value from each word.

3. Your title tag better represent the content on your page effectively for people and machines.

4. Avoid keyword stuffing. Entities in headlines = good. Search revolves around entities. People, places and organisations are the bedrock of search and news in particular. Just don’t overdo it.

5. Do not lean to heavily into clickbait headlines. There’s a temptation to do more for Discover at the minute. The headlines on that platform tend to sail a little too close to the clickbait wind.

6. Make sure your title best reflects the user intent and keep things simple. The benefit of search is that people are directly looking for an answer. Titles don’t always have to be wildly clicky, especially with evergreen content. Simple, direct language helps pass titleLanguageClarity checks and reduces truncation

7. Utilise secondary (h2s) and tertiary (h3s) headings on your page. This has multiple benefits. A well broken up page encourages quality user engagement. It increases the chances of your article ranking for longer-tail queries. And, bloody and, it helps provide the relevant context to your page for Google.

8. Monitor CTR and run headline testing on-site. If you have the capacity to run headline testing in real-time, fantastic. If not, I suggest taking headline and CTR data at scale and building a model that helps you understand what makes a headline clickable at a subfolder or topic level. Do emotional, first-person headlines with a front-loaded entity perform best in /politics, for example?

9. Control your internal anchor text. Particularly important for evergreen content. But even with news, there are five headlines to pay attention to. And internal links (and their anchors) are a pivotal one. The matching anchor text reinforces trust in the topic.

If you are looking into developing your Discover profile, I would recommend testing the OG title if you want to test ‘clickier’ headlines that aren’t visible on page.

Final thoughts

So the goal isn’t just to have a well crafted headline. The goal is to have a brilliant set of titles - clickable, entity and keyword rich, highly relevant. As Shaun says, it’s to create a constellation of signals - the <title>, the <h1>, the URL, the intro paragraph - that remove all ambiguity.

As ever, clicks are an immensely powerful signal. Google has more datapoints than I’ve had hot dinners, so had a pretty good ideas what will do well. But real clicks can override this. The goldmineNavboostFactor is proof that click behaviour influences which title is displayed.

The title tag is the most important headline on the page when it comes to search. More so than the <h1>. But they have to work together. To draw people in and engage them instantly.

But it all matters. Removing ambiguity is always a good thing. Particlarly in a world of AI slop.

Other good stuff

• Definitive Guide to On-Page SEO (Shaun again)

• Reddit Reputation Attack (Lars Lofgren)

• How Publishers Can Survive (Barry Adams)

Leave a comment

1. When it comes to LLM visibility, not all brands are created equal. For some it matters far more than others

2. LLMs give different answers to the same question. Trackers combat this by simulating prompts repeatedly to get an average visibility/citation score

3. Whilst simulating the same prompts isn’t perfect, secondary benefits like sentiment analysis are not SEO-specific issues. Which right now is a good thing

4. Unless a visibility tracker offers enough scale at a reasonable price, I would be wary. But if the traffic converts well and you need to know more, get tracking

A small caveat to start. This really depends on how your business makes money and whether LLMs are a fundamental part of your audience journey. You need to understand how people use LLMs and what it means for your business.

Brands that sell physical products have a different journey from publishers that sell opinion or SAAS companies that rely more deeply on comparison queries than anyone else.

Or a coding company destroyed by one snidey Reddit moderator with a bone to pick…

For example, Ahrefs made public some of their conversion rate data from LLMs. 12.1% of their signups came from LLMs from just 0.5% of their total traffic. Which is huge.

But for us, LLM traffic converts significantly worse. It is a fraction of a fraction.

Honestly, I think LLM visibility trackers at this scale are a bit here today and gone tomorrow. If you can afford one, great. If not, don’t sweat it. Take it all with a pinch of salt. AI search is just a part of most journeys and tracking the same prompts day in, day out has obvious flaws.

They’re just aggregating what someone said about you on Reddit while they’re taking a shit in 2016.

Subscribe now

What do they do?

Trackers like Profound and Brand Radar are designed to show you how your brand is framed and recommended in AI answers. Over time you can measure yours and your competitors visibility in the platforms.

But LLM visibility is smoke and mirrors.

Ask a question, get an answer. Ask the same question, to the same machine, from the same computer and get a different answer. A different answer with different citations and businesses.

It has to be like this, or else we’d never use the boring bastards.

To combat the inherent variance determined by their temperature setting, LLM trackers simulate prompts repeatedly throughout the day. In doing so, you get an average visibility and citation score alongside some other genuinely useful add-ons like your sentiment score and some competitor benchmarking.

“Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic.”

OpenAI Documentation

Simulate a prompt 100 times. If your content was used in 70 of the responses and you were cited seven times, you would have a 70% visibility score and a 7% citation score.

Trust me, that’s much better than it sounds… These engines do not want to send you traffic.

In Brian Balfour’s excellent words, they have identified the moat and the gates are open. They will soon shut. As they shut, monetisation will be hard and fast. The likelihood of any referral traffic, unless it’s monetised, is low.

Like every tech company ever.

How do they work?

As far as I can tell, there are two primary models.

1. Pay for a tool that tracks specific synthetic prompts that you add yourself

2. Purchase an enterprise-like tool that tracks more of the market at scale

Some tools, like Profound, offer both. The cheaper model (the pricepoint is not for most businesses) lets you track synthetic prompts under topics and/or tags. The enterprise model gives you a significantly larger scale.

Whereas tools like Ahrefs Brand Radar provide a broader view of the entire market. As the prompts are all synthetic, there are some fairly large holes. But I prefer broad visibility.

I have not used it yet, but I believe Similarweb have launched their own LLM visibility tracker, which includes real user prompts from Clickstream data.

This makes for a far more useful version of these tools IMO and goes some way to answering the synthetic elephant in the room. And it helps you understand the role LLMs play in the user journey. Which is far more valuable.

Subscribe now

The problem

Does doing good SEO improve your chances of improving your LLM visibility?

Certainly looks like it…

GPT-5 no longer needs to train on more information. It is as well-versed as its overlords now want to pay for. It’s bored of ravaging the internet’s detritus and reaches out to a search index using RAG to verify a response. A response, it does not quite have the appropriate level of confidence to answer effectively.

But I’m sure we will need to modify it somewhat if your primary goal is to increase LLM visibility. Increase expenditure on TOFU and digital PR campaigns being a notable point.

Right now LLMs have an obvious spam problem. One I don’t expect they’ll be willing to invest in solving anytime soon. The AI bubble and gross valuation of these companies will dictate how they drive revenue. And quickly.

It sure as hell won’t be sorting out their spam problem. When you have a $300 billion contract to pay and revenues of $12 billion, you need some more money. Quickly.

So anyone who pays for best page link inclusions or adds hidden and footer text to their websites will benefit in the short-term. But most of us should still build things actual, breathing, snoring people.

With the new iterations of LLM trackers calling search instead of formulating an answer for prompts based on learned ‘knowledge’, it becomes even harder to create an ‘LLM optimisation strategy.’

As a news site, I know that most prompts we would vaguely show up in would trigger the web index. So I just don’t quite see the value. It’s very SEO-led.

How you can add value with sentiment analysis

I found almost zero value to be had from tracking prompts in LLMs at a purely answer level. So let’s fuck all that off for a second and use them for something else. Let’s start with some sentiment analysis.

These trackers give us access to;

• A wider online sentiment score

• Review sources LLMs called upon (at a prompt level)

• Sentiment scores by topics

• Prompts and links to on and off-site information sources

You can identify where some of these issues start. Which, to be fair, is basically Trustpilot and Reddit.

I won’t go through everything, but a couple of quick examples;

1. LLMs may be referencing some not-so-recently defunct podcasts and newsletters as ‘reasons to subscribe.’

2. Your cancellation process may be cited as the most serious issues for most customers

Unless you have explicitly stated that these podcasts and newsletters have finished it’s all fair game. You need to tighten up your product marketing and communications strategy.

For people first. Then for LLMs.

These are not SEO specific projects. We’re moving into an era where solely SEO projects will be difficult to get pushed through. A fantastic way of getting buy-in is to highlight projects with benefits outside of search.

Highlighting serious business issues - poor reviews, inaccurate, out of date information at al - can help get C-Suite attention and support for some key brand reputation projects.

To me, this has nothing to do with LLMs. Or what our audience might ask an ill-informed answer engine. They are just the vessel.

It is about solving problems. Problems that drive real value to your business. In your case, this could be about increasing the LTV of a customer. Increasing their retention rate, reducing churn and increasing the chance of a conversion by providing an improved experience.

If you’ve worked in SEO for long enough, someone will have floated the idea of improving your online sentiment and reviews past you.

Fuck knows, Jeff. It really depends on what is holding you back compared to your competition. And like it or not, search is not very investible right now.

But that doesn’t matter in this instance. This isn’t a search-first project. It’s an audience-first project. It encompasses everyone. From customer service to SEO and editorial. It’s just the right thing to do for the business.

A quick hark back to the Google Leak shows you just how many review and sentiment-focused metrics may affect how you rank.

For a long time, search has been about brands and trust. Branded search volume, outperforming expected CTR (a Bayesian type predictive model), direct traffic and general user engagement and satisfaction.

This isn’t because Google knows better than people. It’s because they have stored how we feel about pages and brands in relation to queries and used that as a feedback loop. Google trusts brands because we do.

Most of us have never had to worry about reviews and sentiment. But this is a great time to fix any issues you may have under the guise of AEO, GEO, SEO or whatever you want to call it.

Lars Lofgren’s article titled How a Competitor Crippled a $23.5M Bootcamp By Becoming a Reddit Moderator is an incredible look at how Codesmith was nobbled by negative PR. Negative PR started and maintained by one Reddit Mod. One.

So keeping tabs on your reputation and identifying potentially serious issues is never a bad thing.

Subscribe now

Could I just build my own?

Yep. For starters, you’d need an estimation of monthly LLM API costs based on the number of monthly tokens required. Let’s use Profound’s lower end pricing tier as an estimate and our old friend Gemini to figure out some estimated costs.

• 200 prompts × 10 runs × 12 days (approx.) × 3 models = 24,000 monthly runs

• 24,000 runs × 1,000 tokens/query (conservative est.) = 24,000,000 tokens

Based on this, here’s a (hopefully) accurate cost estimate per model from our robot pal.

Right then. You now need some back-end functionality, data storage and some front-end visualisation. I’ll tot up as we go.

Back-end

• A Scheduler/Runner like Render VPS to execute 800 API calls per day

• A data orchestrater. Essentially, some Python code to parse raw JSON and extract relevant citation and visibility data

Data storage

• A database, like Supabase (which you can integrate directly through Lovable) to store raw responses and structured metrics

• Data storage (which should be included as part of your database)

Front end visualisation

• A web dashboard to create interactive, shareable dashboards. I unironically love Lovable. It’s easy to connect directly to databases. I have also used Streamlit previously. Lovable looks far sleeker but has its own challenges

• You may also need a visualisation library to help generate time series charts and graphs. Some dashboards have this built in

$96 all in. I think the likelihood is it’s closer to $50 than $100. No scrimping. At the higher end of budgets for tools I use (Lovable) and some estimates from Gemini, we’re talking about a tool that will cost under $100 a month to run and function very well.

This isn’t a complicated project or setup. It is, IMO, an excellent project to learn the vibe coding ropes. Which I will say is not all sunshine and fucking rainbows.

So, should I buy one?

If you can afford it, I would get one. For at least a month or two. Review your online sentiment. See what people really say about you online. Identify some low lift wins around product marketing and review/reputation management and review how your competitors fare.

This might be the most important part of LLM visibility. Set up a tracking dashboard via Google Analytics (or whatever dreadful analytics provider you use) and see a) how much traffic you get and b) whether it’s valuable.

The more valuable it is, the more value there will be in tracking your LLM visibility.

You could also make one. The joy of making one is a) you can learn a new skill and b) you can make other things for the same cost.

Frustrating, yes. Fun? Absolutely.

Other great stuff

• AI Survival Strategies for Publishers - Barry Adams

• AI Mode User Behaviour Study - Kevin Indig

• E-E-A-T Decoded: Google’s Experience, Expertise, Authoritativeness, and Trust - Shaun Anderson

• ChatGPT for Competitive Analysis - WTF is SEO

Leave a comment

TL;DR

1. LLMs are not replacing search. But they are shifting how people access and consume information

2. Asking (49%) and Doing (40%) queries dominate the market and are increasing in quality

3. The top three use cases - Practical Guidance, Seeking Information, and Writing - account for 80% of all conversations

4. Publishers need to build linkable assets that add value. It can’t just be about chasing traffic from articles anymore

Subscribe now

Chatbot 101

A chatbot is a statistical model trained to generate a text response given some text input. Monkey see, monkey do.

The more advanced chatbots have a two or more-stage training process. In stage one (less colloquially known as ‘pre-training’), LLMs are trained to predict the next word in a string.

Like the world’s best accountant, they are both predictable and boring. And that’s not necessarily a bad thing. I want my chefs fat, my pilots sober and my money men so boring they’re next in line to lead the Green Party.

Stage two is where things get a little fancier. In the ‘post-training’ phase, models are trained to generate ‘quality’ responses to a prompt. They are fine-tuned on different strategies, like reinforcement learning, to help grade responses.

Over time, the LLMs, like Pavlov’s dog, are either rewarded or reprimanded based on the quality of their responses.

In phase one, the model ‘understands’ (definitely in inverted commas) a latent representation of the world. In phase two, its knowledge is honed to generate the best quality response.

A quick word on temperature

Without temperature settings, LLMs will generate exactly the same response time after time, as long as the training process is the same.

Higher temperatures (closer to 1.0) increase randomness and creativity. Lower temperatures (closer to 0) make the model(s) far more predictive and precise.

So, your use case determines the appropriate temperature settings. Coding should be set closer to zero. Creative, more content-focused tasks should be closer to one.

I have already talked about this in my article on how to build a brand post AI. But I highly recommend reading this very good guide on how temperature scales work with LLMs and how they impact the user base.

What does the data tell us?

That LLMs are not a direct replacement for search. Not even that close IMO. This Semrush study highlighted that LLM super users increased the amount of traditional searches they were doing. The expansion theory seems to hold true.

But they have brought on a fundamental shift in how people access and interact with information. Conversational interfaces have incredible value. Particularly in a workplace format.

Who knew we were so fucking lazy?

Subscribe now

1. Guidance, Seeking Information and Writing dominate

These top three use cases account for 80% of all human-robot conversations. Practical guidance, seeking information and please help me write something bland and lacking any kind of passion or insight, wondrous robot.

I will concede that the majority of Writing queries are for editing existing work. Still. If I read something written by AI, I will feel duped. And deception is not an attractive quality.

2. Non-work related usage is increasing

• Non-work-related messages grew from 53% of all usage to more than 70% by July 2025.

• LLMs have become habitual. Particularly when it comes to helping us make the right decisions. Both in and out of work.

3. Writing is the most common workplace application

• Writing is the most common work use case, accounting for 40% of work-related messages on average in June 2025.

• About two-thirds of all Writing messages are requests to modify existing user text rather than create new text from scratch.

I know enough people that just use LLMs to help them write better emails. I almost feel sorry for the tech bros that the primary use cases for these tools are so lacking in creativity.

4. Less so coding

• Computer coding queries are a relatively small share, at only 4.2% of all messages*

• This feels very counter-intuitive, but specialist bots like Claude or tools like Lovable are better alternatives

• This is a point of note. Specialist LLM usage will grow and will likely dominate specific industries because they will be able to develop better quality outputs. The specialised stage two style training makes for a far superior product.

*Compared to 33% of work-related Claude conversations

It’s important to not that other studies have some very different takes on what people use LLMs for. So this isn’t as cut and dry as we think. I’m sure things will continue to change.

5. Men no longer dominate

• Early adopters were disproportionately male (around 80% with typically masculine names)

• That number declined to 48% by June 2025, with active users now slightly more likely to have typically feminine names.

Sure, us men have our flaws. Throughout history maybe we’ve been a tad quick to battle and a little dominating. But good to see parity.

6. Asking and Doing queries dominate

• 89% of all queries are Asking and Doing related

• 49% Asking and 40% Doing, with just 11% for Expressing

• Asking messages have grown faster than Doing messages over the last year and are rated higher quality

Subscribe now

7. Relationships and personal reflection are not prominent

• There have been a number of studies that state that LLMs have become personal therapists for people (see above)

• However, relationships and personal reflection only account for 1.9% of total messages according to OpenAI

8. The bloody youth (*shakes fist*)

• Nearly half of all messages sent by adults were from users under the age of 26

• This coincides perfectly with Kevin Indig’s research on how different demographic types consume AI Overviews. Young people are a) more likely to trust it and b) more likely to adopt it

Takeaways

I don’t think LLMs are a disaster for publishers. Sure, they don’t send any fucking referral traffic and have started to remove citations outside of paid users (classic). But none of these tech-heads are going to give us anything.

It’s a race to the moon and we’re the dog they sent on the test flight.

But if you’re a publisher with an opinion, an audience and - hopefully - some brand depth and assets to hand, you’ll be ok. Although their crawling behaviour is getting out of hand.

One of the most practical outcomes we as publishers can take from this data is the apparent change in intents. For eons we’ve been lumbered with navigational, informational, commercial and transactional.

Now we have Doing. Or Generating. And it’s huge.

SEO isn’t dead for publishers. But we do need to do more than just keep publishing content. There’s a lot to be said for espousing the values of AI, whilst keeping it at arm’s length.

Think BBC Verify. Content that can’t be synthesised by machines because it adds so much value. Tools and linkable assets. Real opinions from experts pushed to the fore.

But it’s hard to scale that quality. Programmatic SEO can drive amazing value. As can tools. Tools that answer users’ ‘Doing’ queries time after time. We have to build things that add value outside of the existing corpus.

And if your audience is generally younger and more trusting, you’re going to have to lean into this more.

Leave a comment

Google has a patent called Surfacing In-Depth Articles in Search Results. It’s a gold mine. Really. Full of indexes I didn’t know existed. Author, relevancy, article and site-pattern level scoring.

The whole nine bloody yards.

Sidenote, but I love how these patent naming conventions - US20150379140A1 - make them sound like US naval destroyers. Big, bold and brash they are not.

I first saw this patent from a post from Koray Tugberk (Holistic SEO). Well worth a follow.

TL;DR

1. The system creates a separate index for quality, long-form content called the In-Depth Article Index

2. There’s a two-tiered scoring system for in-depth articles: a pre-calculated In-Depth Article Score and a real-time Topicality Score

3. Authorship and subfolder performance really matter. Authors and subfolders with strong evergreen-like metrics will (theoretically) boost performance

4. Consistent link acquisition over time is far more important than a one-off burst. Truly evergreen content should naturally accrue the right signals over time

Subscribe now

What does this patent do?

It describes how Google identifies and surfaces the best in-depth articles in search results. It’s not just about traditional relevance signals. High-quality, long-form content demands more. Content that stands head and shoulders above the rest because it needs to.

The user’s query is nuanced.

Much like bloody Starmer’s Britain, there is a two-tiered scoring system. One that identifies the true quality of an article and then ranks it based on its relevance to a query. An entity, even.

This scoring system helps Google create a dedicated index for queries and topics. Each article is then ranked for a given query using a new metric - the Document Score.

Instead of treating all content equally, the system creates a standalone index of these ultra detailed articles. When a user searches for a query that demands depth, it is used.

What constitutes an (quality) in-depth article?

It’s not length, I’ll tell you that much.

I wrote in-depth (ironically) about how to write great content. I don’t think there’s much separation between the two.

Great content requires an appropriate level of depth. Proper exploration. It must;

• Answer the question(s) (and the appropriate follow-ups)

• Demonstrate obvious expertise and trust

• Hit three of the four E’s to truly resonate with the audience

• Be unique and original (in some way). Incredibly well researched. Owned data, analysis, visuals etc.

• Be remembered for the right reasons

• Be well structured and have the approprioate level of depth

For all of this to work. For in-depth content to fly, it has to engage. It has to be shared. SEO-wise, it needs to accrue links at a reasonable velocity over time. That’s one of the fundamental concepts of this patent.

You can’t do that with slop. The pig’s in the trough but he wants a change. Your regurgitated fermented grain mash isn’t cutting the proverbial mustard.

It is all about adding value.

Unless you can buy aged or expired domains and spin up casino content on them. Then who cares? You don’t need me.

Subscribe now

Key frameworks and scores

The patent revolves around three key frameworks and scores.

The In-Depth Article Score

A primarily pre-calculated, document-specific score that measures an article's perceived quality. Perceived by a robot, but perceived nonetheless. Independent of any specific search query.

It’s designed to calculate whether your pitiful article is worthy of inclusion in the In-Depth Article Index.

Think of it like Epstein Island or a Diddy Party. Only the select few make it. Then claim they were never really friends and become President of the US or a disgraced member of the Royal Family.

This composite score is derived from five others.

• Article Score: The system looks for long-form content with a high word count and longer paragraphs. Something that resembles a narrative via coverage, structure and length rather than a straight answer.

• Commercial Score: If the commercial score is too high, the article is filtered out of the index. Commercial phrases are a dead giveaway (offer, sale, or discount prices). As are interactive elements like credit card fields.

• Evergreen Score: Is the article adding value months or years after publication? A steady stream of links over a long period suggests sustained interest. For newer articles, Google use a Bayesian-type predictive model, using factors like word count (lol) and early link growth to establish its potential.

• Site Pattern Score: This is a probability score designed to estimate the long-term value of an article based on previous subfolder performance. For example, articles published under a /travel/ directory have a higher probability of being evergreen than those under a /news/ directory. Evergreen content that sits under /news/ will be suppressed compared to /travel/. Theoretically.

• Author Score: Authors hate being judged. Especially by machines. The system tracks the author’s history across documents and can assign weight if they have a track record of producing in-depth content. Authorship really does matter, particularly for evergreen content.

If you’re ever considering where evergreen content should sit for maximum effect, remember that the system learns statistical associations between URL paths and evergreen content. So evergreen content migrations can be clearly explained. As can author performance.

The Topicality Score

The Topicality Score is a relevance score that is (mostly) calculated in real-time, for each query. It measures how relevant each in-depth article is to that particular term.

Topics and entities associated with the user's query are compared to those found within the in-depth article. A higher degree of topical overlap results in a higher score.

Nothing new there.

It acts as a bridge between the user's immediate information need and the higher-quality content that exists in the specialised index.

Think of topical authority at an article level. Topical coverage. Query fanning. Semantic SEO. Whatever drivelling buzzword you want to use.

• Have you answered all of the relevant queries your audience has to offer here?

• Do you understand them well enough to know what they want to know on each page?

• Does it fit with with the wider search narrative?

The Document Score

The final boss. The Document Score.

You’ve created a supposedly in-depth article. At least by your own appalling standards. Now your document gets scored. Judged by a predetermined set of predictive analytics.

It is the final ranking used to select which in-depth articles to display. It is the definitive metric for a given search query.

It is calculated by combining the article's pre-existing IDA Score with its real-time Topicality Score for the given query. The Document Score can be a sum or product of the two scores.

The Document Score ensures that the in-depth articles displayed are both high-quality (high IDA Score) and highly relevant to the user's query (high Topicality Score).

This two-part approach prevents low-quality but relevant content from being shown, while also preventing high-quality but irrelevant content from being ranked.

Subscribe now

How the IDA Search Index is Created

Like a hangover recovery post 30, it is a multi-step process. One with slightly less groaning and electrolytes.

1. Stellar sources are identified: A set of seed websites - publishers known for high-quality, in-depth content - are identified. I suspect at a topic/query level. Probably driven by link acquisition, mentions and user engagement over time. Possibly also accreditations and authorship.

2. Based on the seed sites, target sites are then identified to expand the pool of trusted sources. Websites with similar anchor text profiles are considered topically similar. Low-quality seed websites, known for their commercial or low-quality content, are removed. This creates the final set of target websites to be analysed.

3. Process and score content: Once the target websites are identified, their content is analysed to determine an IDA Score for each article. Only content that passes a certain threshold is added to the dedicated index.

I don’t know exactly what the threshold is, but the concept is sound.

• Identify a quality target website

• Remove spam

• Score content

• Create index

We have a pretty good idea of what the seed websites are based on link acquisition and traffic/share of voice data per category (see below traffic share report). We also have this searchable database, which is a bit shit unfortunately.

And for publishers, I suspect making sure your audience selects you as a preferred source may have some positive ramifications. Never one to sniff at real user data.

How In-Depth Content is Ranked

When something is queried, the algorithm determines if in-depth article search results are required to satisfy the request. The majority of transactional or purely navigational queries don’t require an IDA.

If something can be zero-clicked, it will be. For people like us, the IDA index is where it’s at. If Google can answer the query itself, it will.

If one of the top-ranking general search results is from a stellar source, the system can use this as a trigger condition. But it’s not the only condition. In some implementations, in-depth articles may be triggered based on query type (intent-based, etc).

Triggered harder than Liz Truss seeing a cabbage, the system then ranks the articles from the IDA index using the Document Score. The higher the score, the higher the ranking.

Simple.

Worth noting that the IDA index is periodically updated. That means publishers can enter/exit the index as new content is published or as older content drops below thresholds. Not permanent inclusion. That would be stupid. And I can only count about 40 stupid things Google have done over the last year.

What it means for us

I don’t know if this changes anyone’s approach to evergreen content. For those of us who work for large publishers or on multiple products, your authorship and site structure really matter.

Understanding intent is more important than ever. And links. Links are still the foundation of ranking. The strongest trust signal available.

• In-depth, evergreen content is so important that it has its own index

• Quality is a quantifiable metric. Google actively quantifies and scores in-depth content for quality signals beyond the usual. It's not just about what you say, but how you say it (long-form, narrative style) and who you are (author credibility, publisher reputation).

• Site architecture impacts evergreen performance. The Site Pattern Score proves that your site structure affects how Google perceives the quality and evergreen nature of that content.

• Author performance is a ranking factor. The Author Score explicitly confirms that the system looks at the author's history and reputation. Because author and site history feed into IDA scores, a consistent body of evergreen, narrative work matters more than a single epic article.

• Anchor text and link velocity matter. The Evergreen Score's reliance on anchor distribution and link velocity over time means a consistent stream of quality links is foundational.

• Document Score vs Site Score. The final IDA Score includes the Site Pattern Score. The reputation of the site and its authors contribute directly to the IDA score, which in turn feeds into the Document Score. A dirty little nexus of sites, articles and authorship.

• Commercialisation of informational evergreen content: Articles that veer into sales pitches will be excluded. Keep in-depth and commercial separate where possible.

While not explicit, user preferences/personalisation may influence which in-depth articles get surfaced. For example, if a user has a history of engaging with a source, that could weigh into the decision.

That’s it. Let me know what you think. You know where to find me.

Leave a comment