Is AI inventing features your SaaS does not have? Auditing feature and integration claims

This audit checks specific claims about your SaaS features and integrations. Start with current documentation, the plan, version and date. Compare AI answers manually; a brand mention or a citation of documentation does not establish claim accuracy.

Split each answer into atomic claims. “The integration reads data and automatically publishes changes” contains two different claims. Distinguish reading from writing, manual from automatic operations, and creating a draft from publishing.

Missing documentation means unverified, not proof that a feature does not exist. An incorrect claim requires contradicting evidence or confirmation from the product owner. Assess omissions only against the information that the test was intended to check.

Manual verification register

Keep counts for every class. If calculating accuracy, state the denominator and report unresolved cases separately. Do not automatically count them as accurate. This is manual assessment, separate from Semly visibility metrics.

How to Audit AI Feature and Integration Claims: Scope, Prerequisites and Limits

Before running any prompt, define what you are auditing and what you need in place. An audit without a canonical source of truth produces opinions, not verdicts.

What counts as a claim. Treat any statement an AI engine makes about your product as an auditable claim if it falls into one of five categories: a feature (what the product does), an integration (what it connects to), a price (list price, discount, currency), a plan limit (seats, prompts, API calls, storage), or a certification (SOC 2, ISO 27001, GDPR posture). Each category has a different canonical source, so each requires a different verification path.

Who this guide is for. B2B SaaS marketing, SEO/GEO, product marketing and RevOps leads who are comfortable with prompt tooling and structured data. The method assumes you can access at least three answer engines and that your product has public documentation.

Prerequisites. Assemble these before Step 1:

• A canonical source of truth: product documentation, pricing page, integration directory, security/compliance page.
• Access to three to four answer engines (for example ChatGPT, Gemini, Perplexity, Claude).
• A spreadsheet or log template with the fields listed in Step 5.
• Roughly two to four hours for the first pass across all prompt categories.

Prerequisite checklist Canonical docs URL, pricing page URL, integration directory URL, compliance page URL. Three to four answer engines accessible in fresh sessions. Log template with date, model, version, prompt, raw answer, atomic claim, verdict, cited URL, source verdict. Named owner and a two-to-four-hour first-pass window.

Step 1 — Build the Prompt Matrix Across Models

The prompt matrix is the test set. Its purpose is comparability: the same prompts, the same models, the same logging format, repeated over time.

Define four prompt categories and write 8 to 15 prompts per category:

• Feature prompts — "Does [product] support [capability]?" "How does [product] handle [workflow]?"
• Integration prompts — "Does [product] integrate with [tool]?" "What is the native integration list for [product]?"
• Pricing and limits prompts — "How much does [product] cost?" "What are the seat limits on [product]'s entry plan?"
• Comparison prompts — "[product] vs [competitor] for [use case]" "Which tool is better for [job]?"

Run every prompt across ChatGPT, Gemini, Perplexity and Claude, in fresh sessions, and log the model name, version and date. Fresh sessions matter because conversational context contaminates later answers.

Step 2 — Extract Atomic Claims from Each Answer

A paragraph is not verifiable. A single fact is. Decompose each answer into atomic claims, following the logic used by factuality frameworks such as FActScore, VeriScore and SAFE: one claim, one verifiable unit.

Take this model sentence: "The platform includes a native Salesforce integration, syncs every 15 minutes, is available on the Growth plan, and supports two-way field mapping."

That single sentence contains four atomic claims:

1. Native Salesforce integration exists.
2. Sync frequency is 15 minutes.
3. The integration is available on the Growth plan.
4. Two-way field mapping is supported.

Tag each claim as feature, integration, pricing, limit or other, and mark the exact sentence it came from. Keep the raw answer text alongside the extracted claims. Without the raw text you cannot re-verify a verdict later, and you cannot show a vendor or a stakeholder what the model actually said.

To avoid double-counting, apply one rule: if two claims can be falsified by the same single fact, they are one claim. "Syncs every 15 minutes" and "syncs hourly" are one claim with two candidate values, not two claims.

Step 3 — Verify Each Claim Against Your Canonical Sources

Every atomic claim gets a verdict. Use a four-point scale, because binary true/false hides the most common real-world outcome, which is a claim that is directionally right but wrong in detail.

Check each claim against your documentation, pricing page and integration directory, and record the exact source URL used for verification. A verdict without a source URL is not evidence.

If your own documentation is ambiguous, that ambiguity is a finding. A model cannot ground a claim your docs do not state clearly. Log it as a documentation defect and fix the page.

Step 4 — Audit the Citations Behind the Claim

Open every cited URL and confirm it actually supports the claim. Add a citation verdict column to your log with three values: supports, does not support, contradicts. A cited page that contradicts the claim is your highest-priority fix, because the model is actively retrieving a source that argues against your own product.

Ask two questions per citation: does the source say this, and is the source authoritative or a third-party aggregator? Aggregators are not automatically wrong, but they are frequently stale, and stale data is a common seed for invented plan limits.

Step 5 — Fix the Source, Not the Chatbot

There is no editing panel inside ChatGPT or Gemini. You cannot log in and correct a sentence. Corrections happen through three channels: publishing accurate structured official data, correcting the specific page that seeded the claim, and reporting errors to the vendor.

Step 6 — Monitor Drift and Re-Audit on a Schedule

Models update frequently, retrieval indexes change, and a claim that was supported in March can be fabricated in June. A one-time audit is a photograph, not a monitoring system.

Common Mistakes and How to Fix Them

Semly provides AI answers and source inspection. A person familiar with the product checks feature and integration accuracy.

Related guidance

• WordPress REST API cannot accept a post: troubleshooting 401, 403 and 404 errors
• Automatic Article Publishing on WordPress with AI. From Idea to Finished Post.