Make the Next
Click Easier
How useful website chat turns questions into progress—and how to prove it.
A Conversation Is
Not a Conversion
A buyer arrives with a question that is too specific for a headline and too important for a guess: “Will this connect to the approval workflow we already use?” The page has a search field, a product tour, three pricing cards, and a form asking for a phone number. None answers the question. A chat window might. It might also produce a confident paragraph that is wrong, hide the actual documentation, or ask for a meeting before the buyer knows whether the product fits.
That is the distinction this paper starts with: engagement is visitor progress, not conversation count. A useful interaction helps someone understand fit, find the right material, complete a considered form, or reach a person with the right context. Sometimes the best engagement is a click to documentation and no chat at all. A bot should complement navigation, searchable content, clear forms, and human access. It is an additional path through a site, not a replacement for the site’s information architecture or the people behind it.
These numbers are deliberately side by side, but they are not a blended benchmark. The first is an average productivity result among support agents using an assistive system. The second is a vendor-reported count in a particular sales operation. The third is a stated preference among surveyed customers. They describe different populations, interventions, and outcomes. None tells a team what its own website chat will do.
The goal is not to make more conversations happen. It is to make the next confident action easier.
SWITCHCASE STUDIOS · OPERATING THESISThat makes the standard for a chatbot usefully demanding. It must be accurate enough to earn trust, restrained enough to admit uncertainty, and connected enough to the rest of the experience that a visitor can keep moving. It also needs a route to a human when the question is consequential, personal, or simply outside the bot’s knowledge. Staffing replacement is not the outcome here. Better access to useful context is.
Count progress signals, not greetings
A greeting is an event. A completed comparison, relevant document view, qualified next step, or completed human route is evidence that the visitor moved forward.
What the Evidence
Actually Says
The evidence is promising in places and narrower than the marketing around it. The right reading is not “chatbots convert.” The right reading is that assistive systems can help people handle work, a well-configured sales agent may create useful after-hours capacity, and an operationally improved knowledge system may correlate with better commercial outcomes. Those are reasons to test a specific job on a specific site—not reasons to skip measurement.
CASE 01 · An assistive tool changed agent productivity
Brynjolfsson, Li, and Raymond studied 5,172 customer-support agents and reported a 15% average increase in issues resolved per hour when agents used an AI assistance tool.[1] The gains were not evenly distributed: less-experienced workers benefited more, which is a useful clue about access to effective practice.
This is evidence about a tool helping humans do support work. The study follows a staggered rollout rather than a randomized website experiment. The measure is agent productivity in a support setting, not visitor progress on a public website. The practical implication is to look for places where a companion can make a human route clearer or better prepared.
CASE 02 · An overnight sales agent reported meetings
In an April 22, 2026 vendor report, Intercom described Fellow’s use of Fin for Sales and reported that its overnight bot booked 18 meetings in January while the human team’s booking rate stayed the same.[2] That may be a useful operating signal: an automated layer can cover a time window a human team does not.
But the report provides no holdout, denominator, or counterfactual. We cannot know from it how many eligible visitors saw the bot, how many would have booked without it, or whether the result generalizes. Without a denominator, a meeting count cannot tell us a conversion rate.
CASE 03 · Conversion grew alongside an operating system
Jukebox’s vendor case reports 40% conversion growth alongside improvements to its helpdesk, knowledge base, and backend operations.[3] That is not 40 percentage points, and it does not isolate the chatbot’s contribution. The honest lesson is that useful chat often sits inside a larger investment in content, routing, and service operations.
Balance those cases with Gartner’s December 2023 survey of 5,728 customers, published in July 2024: 64% said they would prefer companies did not use AI for customer service, with reaching a human among the top concerns.[4] That is stated preference, not observed abandonment. It is still a design warning. If people cannot tell what the bot knows or how to reach a person, trust falls before the conversation begins.
The evidence therefore supports a modest thesis: useful automation can extend access and reduce friction, especially when it is grounded in good information and paired with people. It does not support a universal conversion promise. Website chat earns its place by making a defined journey clearer, then proving that claim with a fair test.
Design for the Question
Before the Click
Start with the question visitors are trying to answer, not the widget you want to install. A visitor should be able to browse, search, read, compare, submit a form, or talk to a person without being forced into chat. The bot earns attention by being useful at the moment a question blocks progress.
| Experience | What the visitor gets | Where it fails |
|---|---|---|
| Browse-only | Indexable pages, clear labels, search, forms, and direct contact. | Questions cross page boundaries; the visitor must assemble the answer alone. |
| Useful chat | A labeled, optional path to grounded answers, relevant links, and a human route. | Knowledge is stale, uncertainty is hidden, or the route stops at a transcript. |
| Intrusive chat | An interruption before intent is known, often paired with a lead form. | It competes with the page, obscures content, and makes help feel like capture. |
Four high-value intents
Fit and plan clarity
Explain the kinds of teams, workflows, or constraints a product suits, with links to the evidence behind the answer.
Product and docs finding
Point to the right product page, guide, comparison, or setup document instead of recreating a long page inside chat.
Minimum-needed qualification
Ask only what is necessary for the next step, explain why, and get consent before collecting personal information.
After-hours context
Capture a useful question and a reliable route for human follow-up without implying a live handoff that is only an email queue.
Risk boundaries should be explicit. The bot must not invent price, availability, policy, or implementation facts. It should say when it does not know and give the actual contact route. If a person will reply tomorrow, say that; do not call it live support. Keep critical content in public HTML so it remains indexable, linkable, accessible, and usable without chat.
Respect the user’s control
Make chat optional. Do not autoplay audio or open an interruption. Support keyboard Escape and focus order, screen readers and live updates, reduced motion, and mobile CTAs that remain visible. Label AI plainly.
Accessibility is part of engagement, not a post-launch polish item. A visitor who cannot close a panel, read an update, or reach the form has not been helped. The same is true for a person on a small screen whose primary action disappeared below a chat composer.
Measure Progress,
Not Noise
Attribution asks, “Did this conversation appear before the outcome?” Incrementality asks, “Did chat cause more outcomes than would otherwise have happened?” The difference is selection: visitors who choose chat may already be more motivated. Do not judge the bot by comparing chatters with non-chatters.
Intercom’s July 7, 2026 revenue rule counts the full order value—even if only one product was discussed—when the product was discussed and purchased within 14 days.[5] That can be a useful attribution diagnostic. It is not causal proof. In practice, use attribution to understand paths and incrementality to decide whether the intervention created value.
A practical scorecard
| Measure | Why it matters |
|---|---|
| Qualified next steps / eligible visitors | Measures progress across the population that could receive the experience. |
| Answer accuracy | Independent review of sampled answers; a busy bot that misleads is a liability. |
| Human-route completion | Shows whether escalation reaches a real channel with usable context. |
| Repeat contact, unresolved, and abandonment | Separates productive assistance from loops and dead ends. |
| P95 answer latency and page performance | Protects the surrounding experience from slow or heavy chat. |
| Cost per incremental qualified outcome | Connects operating cost to outcomes that would not otherwise have occurred. |
Randomize eligible visitors before chat appears, analyze assigned groups, use consistent definitions and a pre-specified window, and include guardrails. Sample size should support a decision; an arbitrary 14-day window cannot create certainty. Microsoft’s experimentation guidance emphasizes data-quality checks and sample-ratio-mismatch guardrails before trusting a result.[6]
Illustrative worked example—not a reported result
Suppose 10,000 eligible visitors are assigned to each arm. The control produces 200 qualified actions (2.0%); chat produces 230 (2.3%). That is +0.3 percentage points, +15% relative, and 30 observed extra actions. It is not proof of statistical significance; confidence intervals and sample uncertainty still matter. If incremental operating cost is $300, then $300 ÷ 30 = $10 observed cost per extra action—only if the lift is real and incremental.
Write the decision rule before looking at the result: what lift would justify keeping the experience, what accuracy floor is non-negotiable, and what accessibility or contact failures stop the test immediately? Measurement is a way to keep judgment honest, not a way to make every visitor produce a number.
Run a Narrow Pilot.
Keep an Exit.
A pilot is a learning boundary, not a soft launch of an ambition. Choose one audience, one question class, one source set, one human owner, and one outcome. A narrow scope makes wrong answers visible while they are still inexpensive to fix.
Map the current questions
Review search terms, forms, support themes, page exits, and top unanswered questions. Define eligible visitors and a progress event before adding chat.
Approve sources and ownership
Limit answers to current product, documentation, and policy sources. Name a human owner for content, escalation, privacy, and weekly review.
Expose a small, measurable slice
Keep the bot optional and label it. Randomize where possible, watch performance and route completion, and sample answers before celebrating volume.
Improve content before expanding scope
Review wrong answers, unresolved questions, exits, and human handoffs. Update the source or narrow the job; do not simply tune the tone.
Stop when the system fabricates a policy, exposes personal data, blocks contact, or creates an accessibility regression. Course-correct low use by checking placement and relevance, not by adding interruption. Course-correct high use with poor outcomes by tightening content and scope. Scale only when value and guardrails pass together.
Document a privacy, minimization, and retention policy before collecting conversation data. Keep only what the operating purpose needs, define who can access it, and set a deletion horizon. Those are responsible design practices, not a claim of legal compliance; counsel and applicable requirements still matter.
Every pilot needs an exit that is as real as its launch.
SWITCHCASE STUDIOS · PILOT PRINCIPLEBuild the
Helpful Layer
Website chat is not the website. It is a layer that can help a visitor cross a gap between a question and a decision. The layer is strongest when the public page remains useful without it: content is clear, routes are visible, forms are understandable, and a human is reachable. Chat then acts as a guide across the system instead of becoming a second, less reliable system.
Where Companion fits
Companion is SwitchCase Studios’ website AI companion product. Teams select website and document knowledge; Companion retrieves relevant passages with source citations and can run as an inline or floating configurable widget with optional support routes. Teams can review helpfulness and knowledge gaps, while answer checks still need human factual review. Human response times and channels depend on the business’s configured support process. Learn more at switchcasestudios.com/work/companion or visit companion.switchcasestudios.com.[7]
Companion is our product; we are not presenting it as independent performance evidence.
The product question is therefore not “Can we add a bot?” It is “Which question, for which visitor, is currently expensive to answer—and what would a trustworthy next step look like?” Sometimes the answer is a better page. Sometimes it is a search improvement, a shorter form, a clearer price explanation, or a person. Sometimes it is a carefully bounded companion with evidence behind every answer.
The best chat does not make the site feel more automated. It makes the site feel more legible. It knows when to link, when to ask, when to admit uncertainty, and when to hand over. It shortens the distance between question and confident next step.
Make the answer findable. Make the route honest. Measure the progress. Keep the human door open.
Bibliography
- Brynjolfsson, E., Li, D., & Raymond, L. R. Generative AI at Work. arXiv:2304.11771v2, revised November 6, 2024. Read the study. Study of 5,172 support agents; assistive-tool evidence, not autonomous website conversion evidence.
- Intercom. Announcing Fin for Sales. April 22, 2026. Read the vendor report. Fellow case report; no holdout or denominator is supplied.
- Fin by Intercom. Jukebox customer story. Undated, accessed September 19, 2026. Read the vendor case. Reported growth accompanied broader helpdesk, knowledge, and backend changes.
- Gartner. Gartner Survey Finds 64% of Customers Would Prefer That Companies Didn’t Use AI for Customer Service. July 9, 2024. Read the survey release. Survey of 5,728 customers; stated preference, not observed abandonment.
- Intercom. Analyze the revenue Fin for Ecommerce generates for your store. July 7, 2026. Read the methodology. Attribution rule; not causal proof.
- Microsoft Research. Patterns of Trustworthy Experimentation: Post-Experiment Stage. December 24, 2021. Read the guidance. Covers data-quality checks and sample-ratio-mismatch guardrails.
- SwitchCase Studios. Companion. Owned product source, accessed September 19, 2026. Visit Companion. Product description only; no independent validation implied.