We’re excited to announce the beta release of our new feature that allows you to track keyword positions on Large Language Models (LLMs) such as ChatGPT 4o-mini and Claude 3.5 Haiku.
This functionality integrates seamlessly into Nightwatch, making it as easy to track these models as it is with traditional search engines like Google, Bing, YouTube, DuckDuckGo, Google Places, and Yahoo.
This document will guide you through how to use this new feature and explain why it’s essential for your SEO strategy.
How to Use LLM Tracking in Nightwatch
Adding Keywords for LLM Tracking:
Navigate to the Add Keywords section of your Nightwatch dashboard.
When adding new keywords, you’ll now see ChatGPT 4o-mini and Claude 3.5 Haiku as options in the Search Engine dropdown.
Select either of these options based on the LLM you want to track.
Adjust the precise location if you are interested in a specific location that will be included in the query.
✅ Available Metrics:
LLM Tracking is available for desktop devices.
The available metrics differ from traditional search engines, like Google, as standard search visibility, search volume, and click potential data are not applicable.
Currently, the key metrics provided for LLM tracking are:
Average Rank – The average ranking position of a keyword across tracked queries.
Rank Distribution – A breakdown of how your keywords are ranking within different position ranges.
Keywords Up / Down – Indicates how many of your tracked keywords have moved up or down in ranking over a given period.
These metrics focus on ranking trends and keyword movement.
We are continuously working on enhancing LLM tracking capabilities and will update available data as new insights become available.
Segmenting LLM Data:
After adding your keywords, you can create custom views with a filter applied to track LLM-specific rankings.
To do this, go to the Filters section in your dashboard, and set Search Engine = LLM. This will isolate and display only the rankings from ChatGPT 4o-mini and Claude 3.5 Haiku.
You can save these views for easy access later and clearer view of the LLM performance.
Beta Access: This feature is currently available in beta. Any account that requests access by reaching out to [email protected] or our chat will receive 50 keywords to track on LLMs for free.
Watch the Tutorial Video
Insights into LLM Ranking and Why It Matters
Large Language Models like ChatGPT 4o-mini and Claude 3.5 Haiku are now integral to how users interact with content on the web. These models are driven by underlying search engines (like Bing for OpenAI and Google for others), but the way they rank and prioritize content is shifting SEO dynamics. Here's a deeper look:
Brand Mentions: LLMs place significant weight on brand mentions, especially those with specific keyword sequences around your brand name. The context in which your brand is mentioned plays a critical role in ranking on these platforms. This is why tracking your brand mentions is crucial.
Content Relevance Over Backlinks: Unlike traditional search engines where backlinks are pivotal, LLMs focus more on the relevance and context of the content. Monitoring LLM rankings allows you to adapt your content strategy accordingly, optimizing for both search engines and models like ChatGPT and Claude.
Search Engine Dependencies: LLMs such as ChatGPT rely on Bing for their search results, while Claude (developed by Anthropic) often leans on Google. By tracking these models, you can gain insights into how your content is ranked based on these search engine data sources.
AI Search vs. Traditional LLMs
The key difference between AI search and traditional LLMs is that AI search integrates real-time internet access, executing searches on external search engines.
It then processes the results using an LLM, depending on the specific configuration or service.
For OpenAI, AI search typically uses the latest, most cost-effective model based on the account’s pricing tier. If the goal is to understand how a website is ranked within an LLM, it’s important to analyze search results without LLM interference. This helps determine:
How rankings are embedded within the LLM.
How rankings appear in AI-powered search results.
Since OpenAI relies on Bing for searches, tracking relevant keywords on Bing can provide insights into how OpenAI’s AI search ranks websites. This distinction between AI searches (which use live web data) and LLM searches (which rely solely on the model’s pre-existing knowledge) is crucial.
Why LLMs Require External Search Tools
LLMs operate on a fixed dataset, meaning their knowledge is cut off at a specific point in time and does not update dynamically. Because of this, they require external tools to fetch real-time information and overcome their own limitations.
Tracking Rankings in AI Search
To effectively track rankings in AI search, the best approach is:
Simulating LLM searches to analyze how results appear within the model.
Tracking Bing and Google rankings independently, as these search engines significantly influence AI-powered search results across major LLM platforms.
Since OpenAI’s exact search mechanisms are still evolving, this dual approach currently offers the best strategy for understanding rankings and visibility for a specific brand or website.
How LLM Tracking Benefits Your SEO Strategy
Competitive Insights:
Tracking positions on LLMs provides unique insights into how your competitors are being ranked in these models.
For example, you can analyze Momondo and Skyscanner to see how they are positioned differently on ChatGPT 4o-mini compared to Google.
Identifying Ranking Shifts:
LLMs are constantly evolving. Tracking rankings in real-time lets you quickly identify changes in how your brand is perceived. For instance, if a competitor's rankings improve in ChatGPT 4o-mini, you’ll be able to identify the phrases or brand mentions that are giving them an edge.
Focus on LLM-Specific Keywords:
With custom views dedicated to LLM tracking, you can focus solely on how your selected keywords perform in these models, independent of traditional search engines.
Important Considerations for LLM Tracking
Beta Limitations: As the feature is in beta, each account has access to 50 keywords. If you need more keywords to track, please contact support.
Frequency of Updates: LLM rankings are updated daily as keywords tracked on other search engines. Be sure to monitor your rankings regularly to capture any significant fluctuations or trends.
Tracking LLM positions is now an essential part of your SEO toolkit. With the introduction of Nightwatch’s LLM tracking, you can stay on top of how your brand and competitors are performing in ChatGPT 4o-mini and Claude 3.5 Haiku.
As LLMs continue to shape the future of search, now is the time to get ahead and ensure your brand remains visible in this evolving landscape.