Case study · Independent consulting, early 2025

    Scoping an In-House AI Support Chatbot

    Replacing a vendor chatbot, starting with a four-week pilot

    The Head of Product at a venture-backed company approached me about its customer support chatbot, which an outside vendor ran for them. I worked out why it was falling short and designed a four-week pilot to test an in-house replacement before any larger investment.

    At a glance

    EngagementDetails
    ClientVenture-backed South American professional services company serving more than ten thousand small-business customers
    WhenEarly 2025
    TurnaroundDiagnosis and pilot proposal delivered within three days of first contact
    DeliveryFully remote, with the client on another continent
    ChallengeA vendor-managed support chatbot that escalated too many conversations and gave the team no control to fix it
    My roleIndependent consultant: diagnostic analysis, solution design and pilot scoping
    DeliverablesDiagnostic report with a target solution; four-week pilot proposal with success metrics
    ScopeDiscovery and pilot design

    Completed under a non-disclosure agreement as an independent consultant, before I began working under the Aevoxis name. Client details have been withheld.

    The situation

    The company had grown fast, and its customer support had to grow with it. A third-party chatbot handled first-line questions, integrated with the team's helpdesk.

    The bot did its job, but too many conversations still ended up with a human agent, and the team had a clear internal target for how many the bot should resolve on its own. They were also willing to spend more to get there.

    So I was asked two things: should the company build its own chatbot, and how could it test that without putting its whole support operation at risk?

    Diagnosis: the team had no control over the bot

    Most of the bot's problems came from one cause. The team could see what was going wrong but had no way to fix it themselves. I grouped the issues into three areas.

    AreaWhat the team experiencedWhy it mattered
    ControlNo access to prompts, slow knowledge base updates, no say over infrastructureEvery fix depended on the vendor's roadmap and timeline
    CapabilityNo handling of email attachments, limited integration with internal systemsCommon customer requests could never be resolved by the bot
    QualityHallucinated and repeated answers; only part of conversations were reviewedCustomers lost trust, and problems surfaced late

    Tuning the existing setup would not have helped much, because the parts that needed tuning were out of the team's reach.

    Looked at as a governance problem, the gaps were serious. The company could not see or change how its AI answered customers, and most of those answers were never checked. Customers acted on what the bot told them, so this was a business risk as well as a service problem.

    Recommendation: an in-house chatbot

    I recommended replacing the vendor with an in-house platform built on Rasa, an open-source conversation framework, with Anthropic's Claude as the large language model. With its own setup, the team could change prompts, knowledge and integrations directly.

    The target solution had six parts:

    1. Retrieval-augmented generation (RAG) over a restructured, expanded knowledge base, so answers come from approved content.
    2. Integrations for customer-specific data and document processing (OCR for email attachments).
    3. Prompt engineering designed to reduce hallucinations, refined continuously from escalations and feedback.
    4. A tiered escalation flow that hands complex cases to people without breaking the customer's trust.
    5. Full audit coverage using automated review, with tracking of response time, resolution and satisfaction.
    6. Multi-channel reach across messaging, website chat and email, with analytics dashboards.

    Customer channels

    Messaging, website chat, email

    Rasa

    Conversation flows and business rules

    Claude with RAG

    Answers only from the approved knowledge base

    Answer

    When confident

    Human agent

    Anything uncertain

    Audit layer: every conversation is logged and reviewed
    Recommended architecture: four stages, human handover and an audit layer.

    Business rules live in Rasa, Claude answers only from approved content, anything uncertain goes to a person, and every conversation is logged.

    For each part I listed the main drawback and how to reduce it, so the team could see the trade-offs up front.

    Why Rasa, and not LangChain, LangGraph or LlamaIndex

    I considered these frameworks, and they are strong choices for LLM applications, retrieval pipelines and agent workflows. This client's main problem, though, was a support bot that made things up and that the team could not control.

    What the client neededWhy Rasa fitWhere the usual options fall short
    Predictable answers in a regulated domainBusiness logic lives in defined conversation flows, with the LLM handling languageAgent frameworks let the LLM decide more of the next step, the behaviour the client wanted less of
    Reliable multi-turn conversationsDialogue management is built inLangChain and LlamaIndex supply the building blocks; dialogue state has to be built and maintained on top
    Messaging channels from day oneReady-made channel connectors, including WhatsAppChannel handling must be written and maintained separately
    Full ownership after leaving a vendorOpen source and self-hosted, so prompts, data and infrastructure stay in-houseComparable, but more of the surrounding platform must be assembled by the team
    Costs that fit the client's budgetNo licence fees, and less custom code to build and maintainAlso free to use, but the extra build work for dialogue and channels adds cost

    So I chose a platform built for customer conversations, with Claude handling the language understanding.

    Designing the pilot

    Moving everything at once would have been risky with an untested system, so I scoped a four-week pilot on a single channel. Its goal was to beat the current bot's resolution rate by a measurable margin before the company paid for the full build.

    What stayed in and what came out

    Kept for the pilotDeferred to the full build
    Multi-turn conversationsKnowledge graph and automated data ingestion
    RAG over a curated knowledge baseLegacy system integrations and attachment processing
    One channel, chosen by customer usageAdditional channels
    Prompt engineering and a feedback loopTiered escalation model (direct handover instead)
    Clear fallbacks and smooth human handoverReal-time analytics dashboards
    Basic analytics and a 20% manual auditFull automated audit coverage

    I kept whatever directly affected the resolution rate and postponed whatever only mattered at a larger scale.

    Starting with the right data

    The pilot opened with a two-to-three-day audit of the existing knowledge base. If it was clean, it would be used whole. If not, I would apply the Pareto principle and select the smallest set of Q&A pairs covering about 80% of incoming questions, prioritized by:

    • How often a question is asked
    • How often it currently leads to escalation
    • How strongly it affects customer satisfaction

    The selected pairs would then be grouped by theme, such as account setup, billing and troubleshooting.

    Four-week timeline

    1. Week 1, setup: build and configure the bot on the curated knowledge base and connect one channel.
    2. Week 2, controlled launch: release to internal testers or a small group of customers and log every gap.
    3. Week 3, optimization: refine prompts, answers and Q&A pairs from what the logs show.
    4. Week 4, pilot launch: open to the wider customer base and track resolution, escalations and feedback.

    Measuring success

    Success came down to one number the team could check themselves, the share of conversations the bot resolves without a human.

    Resolution rate = (queries resolved by the chatbot ÷ total queries received) × 100

    The current bot's rate served as the baseline. Early trends would appear in weeks 2 to 3, stable figures in weeks 3 to 4, and a clear go or no-go answer by the end of week 4.

    Risks and how the plan handled them

    RiskMitigation
    Disruption while moving away from the vendorPhased approach: the pilot runs on one channel before any migration
    Knowledge base goes staleScheduled audits now; automated ingestion in the full build
    Slow integrations with older systemsModular design, with critical features first and the rest added in stages
    Prompts need skilled upkeepTeam training and a feedback loop from escalations
    Bot-to-human handover feels abruptA/B testing and real-time feedback on handover messages
    Monitoring adds overheadAutomated review in place of manual checks over time

    Outcome and takeaways

    The company received a diagnosis, a costed target solution and a ready-to-run pilot plan with clear success criteria. This case study covers the analysis and planning rather than implementation results.

    A few principles guided the work:

    • I looked for the cause behind the symptoms. Hallucinations and escalations were symptoms, the lack of control was the cause, and that shaped the recommendation.
    • I chose tools for the client's needs and budget. The stack was picked for predictable answers, ready-made channels and low running costs, and the pilot let the client test the idea for far less than the full build.
    • I built oversight into the design from the start. The pilot audited a sample of conversations, the full build audited all of them, and the team owned the prompts and knowledge base.
    • I listed a drawback and a mitigation for every component, so the client could judge the trade-offs for themselves.
    • I kept the pilot narrow, with one channel, a curated knowledge base and one success metric.
    • I defined success before any building started, using a formula the client could check without me.

    Related work

    Work with me

    I help growing companies decide whether and how to use AI in their operations, and how to keep it accountable once it is running. I usually start with a diagnosis and a small pilot that shows early on whether the idea works. If your team is facing a similar decision, I'd be glad to talk.

    Vinita Silaparasetty, AI Governance Engineer