An AI answer can recommend a SaaS platform, an agency and a WordPress plugin in one list. Interpreting it requires labels for solution type and customer task. This guide covers coding recommendations for a benchmark, not selecting a supplier.
1. Define the comparison unit and task
Decide whether you are analysing companies, products or specific offers. A company may sell software and consulting; one product is not the same object as the organisation. Keep separate fields for name, domain, product and operator. Do not merge similar names without checking identity.
Customer tasks might include monitoring AI answers, commissioning an audit or improving website content. A monitoring answer may list solutions addressing only part of the need. Being recommended does not confirm a solution’s capabilities.
2. Apply a type codebook
| Type | Object classified | Source to check |
|---|---|---|
| SaaS platform | Software delivered as a service | Product description and documentation |
| Agency service | Work commissioned from a team | Scope of the specific offer |
| Plugin | Extension installed in a website environment | Extension documentation and publisher |
| Directory or information source | Collection of offers or reference material | Source purpose and operator |
Type describes delivery form, not effectiveness. A plugin may provide monitoring, and a platform may have a WordPress extension. Record task, features and type in separate columns. Allow multiple labels when verified offers support them.
3. Code a mixed-answer example
The names and facts below are fictional. AI lists Vista Monitor, Studio Aurora and Schema Note. Assume verified descriptions of these example offers establish the following scope.
| Entry | Type | Confirmed task | Comparison group |
|---|---|---|---|
| Vista Monitor | SaaS platform | Monitoring AI answers | Monitoring |
| Studio Aurora | Agency service | Delivering an audit | Audit services |
| Schema Note | Plugin | Organising website markup | Website extensions |
These entries form one raw recommendation list, but are not three interchangeable answers to every task. If Studio Aurora also offers its own platform, record a second product or label according to the defined unit. Do not count the same entry twice merely because it has two labels.
4. Record evidence and uncertainty
Keep answer date, model, prompt, entry name, product, domain, type, task, source URL and reasoning. If official material does not settle the category, mark it unverified. The AI answer is being measured, so its description alone is not independent evidence.
Ask a second reviewer to assess disputed entries against the same codebook. Record the resolution and rule version. Do not relabel entries after seeing the result merely to improve your brand’s position.
5. Show raw results beside grouped results
Preserve the original list and visibility result for the complete measured scope. Show groups by task and type alongside it. If calculating an additional within-group share, name it separately and specify the denominator; do not present it as the original Semly result.
Semly provides AI answers and source inspection. Type labels and benchmark grouping here are analyst work. Keep scope consistent for the next measurement or explicitly document the change.
Quality checks
- The comparison unit is defined.
- Type, task and features are separate.
- Hybrids have evidence and do not create double counting.
- Unverified categories remain visible.
- Raw results and denominators remain available.
