Case study · Independent consulting, early 2025
Scoping an In-House AI Support Chatbot
Replacing a vendor chatbot, starting with a four-week pilot
The Head of Product at a venture-backed company approached me about its customer support chatbot, which an outside vendor ran for them. I worked out why it was falling short and designed a four-week pilot to test an in-house replacement before any larger investment.
At a glance
| Engagement | Details |
|---|---|
| Client | Venture-backed South American professional services company serving more than ten thousand small-business customers |
| When | Early 2025 |
| Turnaround | Diagnosis and pilot proposal delivered within three days of first contact |
| Delivery | Fully remote, with the client on another continent |
| Challenge | A vendor-managed support chatbot that escalated too many conversations and gave the team no control to fix it |
| My role | Independent consultant: diagnostic analysis, solution design and pilot scoping |
| Deliverables | Diagnostic report with a target solution; four-week pilot proposal with success metrics |
| Scope | Discovery and pilot design |
Completed under a non-disclosure agreement as an independent consultant, before I began working under the Aevoxis name. Client details have been withheld.
The situation
The company had grown fast, and its customer support had to grow with it. A third-party chatbot handled first-line questions, integrated with the team's helpdesk.
The bot did its job, but too many conversations still ended up with a human agent, and the team had a clear internal target for how many the bot should resolve on its own. They were also willing to spend more to get there.
So I was asked two things: should the company build its own chatbot, and how could it test that without putting its whole support operation at risk?
Diagnosis: the team had no control over the bot
Most of the bot's problems came from one cause. The team could see what was going wrong but had no way to fix it themselves. I grouped the issues into three areas.
| Area | What the team experienced | Why it mattered |
|---|---|---|
| Control | No access to prompts, slow knowledge base updates, no say over infrastructure | Every fix depended on the vendor's roadmap and timeline |
| Capability | No handling of email attachments, limited integration with internal systems | Common customer requests could never be resolved by the bot |
| Quality | Hallucinated and repeated answers; only part of conversations were reviewed | Customers lost trust, and problems surfaced late |
Tuning the existing setup would not have helped much, because the parts that needed tuning were out of the team's reach.
Looked at as a governance problem, the gaps were serious. The company could not see or change how its AI answered customers, and most of those answers were never checked. Customers acted on what the bot told them, so this was a business risk as well as a service problem.
Recommendation: an in-house chatbot
I recommended replacing the vendor with an in-house platform built on Rasa, an open-source conversation framework, with Anthropic's Claude as the large language model. With its own setup, the team could change prompts, knowledge and integrations directly.
The target solution had six parts:
- Retrieval-augmented generation (RAG) over a restructured, expanded knowledge base, so answers come from approved content.
- Integrations for customer-specific data and document processing (OCR for email attachments).
- Prompt engineering designed to reduce hallucinations, refined continuously from escalations and feedback.
- A tiered escalation flow that hands complex cases to people without breaking the customer's trust.
- Full audit coverage using automated review, with tracking of response time, resolution and satisfaction.
- Multi-channel reach across messaging, website chat and email, with analytics dashboards.
Customer channels
Messaging, website chat, email
Rasa
Conversation flows and business rules
Claude with RAG
Answers only from the approved knowledge base
Answer
When confident
Human agent
Anything uncertain
Business rules live in Rasa, Claude answers only from approved content, anything uncertain goes to a person, and every conversation is logged.
For each part I listed the main drawback and how to reduce it, so the team could see the trade-offs up front.
Why Rasa, and not LangChain, LangGraph or LlamaIndex
I considered these frameworks, and they are strong choices for LLM applications, retrieval pipelines and agent workflows. This client's main problem, though, was a support bot that made things up and that the team could not control.
| What the client needed | Why Rasa fit | Where the usual options fall short |
|---|---|---|
| Predictable answers in a regulated domain | Business logic lives in defined conversation flows, with the LLM handling language | Agent frameworks let the LLM decide more of the next step, the behaviour the client wanted less of |
| Reliable multi-turn conversations | Dialogue management is built in | LangChain and LlamaIndex supply the building blocks; dialogue state has to be built and maintained on top |
| Messaging channels from day one | Ready-made channel connectors, including WhatsApp | Channel handling must be written and maintained separately |
| Full ownership after leaving a vendor | Open source and self-hosted, so prompts, data and infrastructure stay in-house | Comparable, but more of the surrounding platform must be assembled by the team |
| Costs that fit the client's budget | No licence fees, and less custom code to build and maintain | Also free to use, but the extra build work for dialogue and channels adds cost |
So I chose a platform built for customer conversations, with Claude handling the language understanding.
Designing the pilot
Moving everything at once would have been risky with an untested system, so I scoped a four-week pilot on a single channel. Its goal was to beat the current bot's resolution rate by a measurable margin before the company paid for the full build.
What stayed in and what came out
| Kept for the pilot | Deferred to the full build |
|---|---|
| Multi-turn conversations | Knowledge graph and automated data ingestion |
| RAG over a curated knowledge base | Legacy system integrations and attachment processing |
| One channel, chosen by customer usage | Additional channels |
| Prompt engineering and a feedback loop | Tiered escalation model (direct handover instead) |
| Clear fallbacks and smooth human handover | Real-time analytics dashboards |
| Basic analytics and a 20% manual audit | Full automated audit coverage |
I kept whatever directly affected the resolution rate and postponed whatever only mattered at a larger scale.
Starting with the right data
The pilot opened with a two-to-three-day audit of the existing knowledge base. If it was clean, it would be used whole. If not, I would apply the Pareto principle and select the smallest set of Q&A pairs covering about 80% of incoming questions, prioritized by:
- How often a question is asked
- How often it currently leads to escalation
- How strongly it affects customer satisfaction
The selected pairs would then be grouped by theme, such as account setup, billing and troubleshooting.
Four-week timeline
- Week 1, setup: build and configure the bot on the curated knowledge base and connect one channel.
- Week 2, controlled launch: release to internal testers or a small group of customers and log every gap.
- Week 3, optimization: refine prompts, answers and Q&A pairs from what the logs show.
- Week 4, pilot launch: open to the wider customer base and track resolution, escalations and feedback.
Measuring success
Success came down to one number the team could check themselves, the share of conversations the bot resolves without a human.
The current bot's rate served as the baseline. Early trends would appear in weeks 2 to 3, stable figures in weeks 3 to 4, and a clear go or no-go answer by the end of week 4.
Risks and how the plan handled them
| Risk | Mitigation |
|---|---|
| Disruption while moving away from the vendor | Phased approach: the pilot runs on one channel before any migration |
| Knowledge base goes stale | Scheduled audits now; automated ingestion in the full build |
| Slow integrations with older systems | Modular design, with critical features first and the rest added in stages |
| Prompts need skilled upkeep | Team training and a feedback loop from escalations |
| Bot-to-human handover feels abrupt | A/B testing and real-time feedback on handover messages |
| Monitoring adds overhead | Automated review in place of manual checks over time |
Outcome and takeaways
The company received a diagnosis, a costed target solution and a ready-to-run pilot plan with clear success criteria. This case study covers the analysis and planning rather than implementation results.
A few principles guided the work:
- I looked for the cause behind the symptoms. Hallucinations and escalations were symptoms, the lack of control was the cause, and that shaped the recommendation.
- I chose tools for the client's needs and budget. The stack was picked for predictable answers, ready-made channels and low running costs, and the pilot let the client test the idea for far less than the full build.
- I built oversight into the design from the start. The pilot audited a sample of conversations, the full build audited all of them, and the team owned the prompts and knowledge base.
- I listed a drawback and a mitigation for every component, so the client could judge the trade-offs for themselves.
- I kept the pilot narrow, with one channel, a curated knowledge base and one success metric.
- I defined success before any building started, using a formula the client could check without me.
Related work
- Designing a Customer Support Chatbot Using Flowise, a Coursera guided project I created and teach, taken by more than 2,500 learners
- Spec-Drift Chronometer, my open-source governance tool for EU AI Act oversight, a Top 300 finalist in the AWS 10,000 AIdeas Competition 2025
Work with me
I help growing companies decide whether and how to use AI in their operations, and how to keep it accountable once it is running. I usually start with a diagnosis and a small pilot that shows early on whether the idea works. If your team is facing a similar decision, I'd be glad to talk.
Vinita Silaparasetty, AI Governance Engineer