How to change an AI monitoring question set while keeping results comparable

This procedure changes an existing monitoring question set. Monitoring size and general metric definitions are covered elsewhere. The outcome is a change register and a comparison that does not confuse a larger sample with improved visibility.

1. Preserve the previous configuration

Keep the text and ID of each question, set version, country, language, system and available settings. Record brand-recognition rules, observation cadence and collection-failure handling. Do not overwrite the old wording without history.

Register field	Value
Question ID and version	[identifiers]
Previous and new wording	[exact text]
Reason	[new offer or correction]
Effective date	[date and time zone]
Group	[unchanged; new; changed; retired]
Setting changes	[market; language; system; classification]
Owner	[approver]

2. Separate the common set

The control set contains unchanged questions observed under comparable conditions. Report new questions separately. When a question changes meaning, start a new series for it rather than treating it as the earlier version.

Even a wording correction may affect the answer; retain its history. No universal question count guarantees stability.

3. Fictional example

Version A has 10 questions. Version B keeps 8 unchanged, retires 2 and adds 4. The new set has 12 questions; the common set has 8.

Compare periods using those 8 questions under consistent conditions. Describe the 4 new questions as an expansion. Do not label the difference between the complete ten-question set and the complete twelve-question set as the effect of a content update. These numbers are illustrative, not client findings.

4. Check units and completeness

If a question is run repeatedly, the number of questions is not the number of answers. Define aggregation and record valid observations separately from failures. Check that both periods use comparable cadence and branded or unbranded groups.

Changes to alias rules, systems or markets can also reduce comparability. Show separate series and explain the limitation. Do not silently recalculate history.

5. Accept the change

Another reviewer should be able to identify additions, retirements, the result for the common set and the configuration in force. Retain the register, available data export and a report note. Do not assume every platform stores previous versions; check the tool you actually use.

Related guides and sources

Sources

How to Measure Brand Visibility in AI?
Semly blog explaining that questions and configuration should stay stable, that a scope change should be marked as a new series rather than attributed to better content, and that metric names across tools are not comparable.
AI visibility monitoring tools: how to compare coverage, evidence and execution
Semly knowledge base article describing the stable historical baseline criterion, the need to record additions to questions, models and markets, and the warning that a higher total after expanding the sample may reflect the new configuration rather than improved content.
How to Measure AI Visibility? A Complete Guide to Metrics and Tools
Semly knowledge base guide covering the 24-hour measurement cycle, Response Stability, non-determinism of AI answers, repeated runs across fresh sessions, and prompt-set design with a monthly audit.
How AI monitoring works in Semly
Semly help center article explaining why results change over time, including model updates, new sources, site changes, competitor content, seasonality and model differences, and noting that the trend over time matters most.
E-commerce Visibility in AI: Semly Report 2026
Semly report based on a 90-day panel of 100 stores, 10,000 prompts and 4.5 million AI responses, showing non-uniform change across stores and models and stating that results are not a guarantee of specific growth.
LLM stability study (Atil et al.)
Academic study of six deterministically configured models across eight tasks and five identical runs, showing accuracy spreads of 5 to 10 percent and non-normal variation even at temperature 0.
Sampling temperature effect study (Renze & Guven)
Academic study across nine LLMs and multiple temperatures showing no statistically significant accuracy change between 0.0 and 1.0, while text similarity decreases as temperature rises.
Question repetition study (Shaier et al.)
Academic study of 90,000 questions showing that repeating a question had no statistically significant effect on accuracy, indicating models are robust to redundancy.
Dice Roll Method (Zatuchin)
Academic study of roughly 190,000 observations across 270+ brands showing reliability rising with iteration count: about 0.58 at five iterations, 0.74 at ten and 0.81 at fifteen.
OpenAI API documentation on advanced usage
Provider documentation describing non-determinism, the seed parameter and the system fingerprint field used to detect backend configuration changes that may alter outputs.
Anthropic Claude model deprecations
Provider documentation describing Active, Legacy, Deprecated and Retired model states, retirement dates and advance notice, and warning that researchers lose access to models for ongoing and comparative studies.

Check if your company is visible in AI