AI/LLM Tracking in Nightwatch

More and more people are skipping traditional search and going straight to AI tools to find recommendations, compare products, and make decisions. LLM Tracking lets you monitor exactly how your brand shows up in those conversations.

Instead of just knowing where you rank on Google, you can now see:

This is a new visibility layer that sits alongside your traditional rank tracking, and right now, very few businesses are paying attention to it.

Before adding prompts manually, use the Prompt Research feature to generate high-quality prompts based on real user questions. This helps ensure you're tracking what people actually ask, not just what you assume they ask.

You can use it in two ways:

After clicking Execute, you'll be taken to the SEO Agent workspace where the research runs, and a list of suggested prompts is generated. You can then review and add them directly to LLM Tracking.

Prompts are the real-world questions users ask AI tools, for example: "What's the best CRM for small businesses?"

Nightwatch automatically gathers and analyzes AI responses on a regular refresh schedule, so results update without any manual work on your end.

For each prompt, you'll see:

Click the 👁️ icon next to any prompt to preview the full AI response and see exactly how your brand is being talked about.

💡 Tip: Good prompts sound like real questions, not search queries. Think: "What's the best tool for tracking local SEO rankings?" rather than "SEO rank tracker."

The Overview dashboard is your high-level summary of how your brand is performing across all tracked prompts and AI models.

Average Visibility — The percentage of prompts where your brand appears, averaged across all models and the selected time period.

Visibility — Your current visibility score at this point in time.

Share of Voice — How much of the total AI conversation in your space belongs to your brand, compared to everyone else being tracked.

Sentiment — A score that reflects whether AI responses mentioning your brand are positive, neutral, or negative.

Entity Visibility graph — shows how your visibility changes over time. Switch between day, week, and month views to spot trends.

Citation Appearance chart — shows how often your brand is being cited across AI responses, broken down by source.

Top Performing Entities — a quick look at which brands are showing up most across your prompts.

Top Citation Sources — the websites being cited most frequently in AI answers related to your tracked prompts.

💡 Tip: Use the date range selector in the top right to compare different time periods and catch shifts early.

Prompts are the questions you track across AI models. Think of them the same way you think of keywords in traditional SEO — they represent what your potential customers are actually asking.

For each prompt, Nightwatch shows visibility, mention count, position, number of entities detected, and when it was last refreshed.

Write prompts the way a real user would ask an AI. Focus on intent, not keywords.

Good examples:

💡 Tip: Use the Prompt Research feature to get suggestions based on your industry. It's a fast way to discover angles you might not have thought of.

Brand Entities is where you set up the brands you want to track — your own and your competitors. Once added, Nightwatch detects when these brands appear in AI responses across all your prompts.

This makes it easy to spot if a competitor is gaining ground in AI responses, or if your own visibility is shifting, before it becomes a problem.

💡 Tip: Add your top competitors as entities from the start. The comparison data is often where the most useful insights come from.

Citations shows which websites AI models are referencing when they answer questions in your space. This is one of the most strategically valuable views in LLM Tracking.

When an AI answers a question, it draws on specific sources. The sites it cites most frequently have an outsized influence on what it says, and who it recommends.

If a highly cited publication is writing content that favors a competitor, that directly shapes AI recommendations. If you can get your brand or content featured on those same sources, you improve your chances of being recommended.

This creates a clear strategy: find the publications with the most influence in your space, and focus your content and outreach efforts there.

💡 Tip: Combine citation data with SEO Agent to turn what you're seeing into concrete outreach opportunities.

Source Matrix gives you a cross-view of which publications are mentioning which brands, across all your tracked prompts and AI models. It's displayed as a grid, with citation sources on one side and brand entities across the top.

The numbers in each cell show how many times a given source has mentioned a specific brand. Higher numbers mean a stronger association between that source and that brand in AI responses.

Source Matrix helps you answer questions like:

This view works particularly well for agencies — it gives a clear, visual competitive picture that translates easily into client conversations and reporting.

🔬 Note: Source Matrix is currently in beta and is actively being developed. New capabilities will be added over time.

Classifications let you categorize AI responses based on how your brand appears — whether as a direct recommendation, a comparison, a mention in passing, and so on.

This helps you understand not just whether you were mentioned, but in what context, so you can focus on the types of mentions that matter most to your goals.

LLM Tracking tells you what's happening. SEO Agent helps you address it.

Once you've identified which sources are influencing AI responses in your space, SEO Agent can help you act on that — from spotting content gaps and topic opportunities to supporting outreach to the publications that carry the most weight.

If you'd like SEO Agent credits enabled on your account, reach out to our support team, and we'll get it set up.

Which AI models can I track?

You can track across ChatGPT, Perplexity, Google AI Mode, and AI Overview. Gemini is supported for plans with 300 prompts +.

More models are being added over time.

How often are prompts refreshed?

Regular refresh frequency is daily. But it can be adjusted to weekly for custom enterprise plans. You can see the last refresh time next to each prompt in the Prompts view.

How is this different from traditional rank tracking?

Traditional rank tracking shows where you appear on search engine results pages. LLM Tracking shows how AI models talk about you — a different channel with distinct dynamics and influencing factors.

Can I use this for client reporting?

Yes. The data in LLM Tracking can be included in Nightwatch reports, making it easy to show clients their AI visibility alongside traditional ranking data.

I don't see my brand appearing at all. Is something wrong?

Not necessarily. It may mean your brand isn't being mentioned in AI responses for those prompts yet — which is exactly the kind of insight LLM Tracking is designed to surface. Start by reviewing which sources are being cited and work from there.

You can test and add different prompts so you see how it affects visibility.

Have questions or need help setting up LLM Tracking? Reach out to our support team — we're happy to help you get the most out of it.