An AI chatbot playbook for government

How to build, protect, and govern AI-powered assistants

When I meet with technology leaders from government agencies, they often tell me the first generative AI (GenAI) tool they’ve implemented is an AI chatbot. In fact, as of January 2025, 80% of all documented GenAI use cases in the federal government involved chatbots and AI-powered virtual assistants. Most agencies have started with internal, employee-facing chatbots to make sure they’re comfortable with the technology and happy with the outcomes before creating external, customer-facing ones.

State and local government agencies are now similarly eager to implement their first AI-powered chatbots. Meanwhile, federal agencies that have completed internal deployments are planning external chatbots.

These AI chatbots are different from the traditional chatbots that have been used for years. Traditional (rule-based) chatbots follow fixed scripts and are excellent for simple tasks, but they fail with “off-script” questions. AI (conversational) chatbots use natural language processing (NLP), machine learning (ML), and large language models (LLMs) to understand intent, handle context, learn, improve, and integrate data for comprehensive answers to complex queries.

AI chatbots can provide fast, consistent, and compliant responses — without requiring humans. But they can be difficult to build, secure, and govern. To build a chatbot, they first need to find a use case that can provide a quick win and needed experience, then optimize design, choose the right LLM, and successfully integrate that model with properly curated data sources.

They also must secure the chatbot from attacks and malicious behavior. At the same time, they have to establish effective governance, ensuring the model isn’t providing inaccurate or inappropriate responses, which can lead to flawed decision-making, facilitate fraud, jeopardize regulatory compliance, and ultimately erode user trust.

Despite the challenges, there’s no question that federal, state, and local government agencies are moving forward with AI chatbot implementations. Following four best practices can help your organization maximize the value of these AI-powered services while minimizing the potentially serious risks.

1. Find the right use case

Building AI chatbots starts by adhering to the same principles as other applications: Understand the problem, design for users, and iterate based on feedback. For your first AI chatbot deployment, focus on solving a specific, clearly defined, high-volume problem, like answering frequently asked questions. As you plan that first project, examine both external and internal uses. And consider using a chatbot for triage — a first line of user interaction that determines how to best solve an issue.

Over time, you can incorporate additional capabilities to support more complex problems. For example, you could integrate translation features, support both voice and text, and connect to multiple systems.

External use cases

External, customer- or citizen-facing chatbots can span a wide range of functions. Here are just a few examples:

• US Citizenship and Immigration Services implemented “EMMA,” a chatbot which helps with public requests related to immigration services, green cards, passports, etc.

• The City of Atlanta improved 311 services by providing round-the-clock access to non-emergency assistance. Residents can report potholes or ask about waste pickup schedules at any time.

• The state of South Carolina is launching a new AI-powered “resident assistant,” nicknamed “Bradley,” which will enable users to find answers to frequently asked questions and get assistance with paying taxes, navigating water bills, and more.

External chatbots can be designed to support anonymous inquiries or provide personalized information. A city website’s chatbot might provide general information about home construction permits or upcoming events anonymously. But a labor, tax, or motor vehicles agency website could provide personalized responses with information about an individual’s specific claims, tax filings, or vehicles.

Internal use cases

Most public sector IT leaders I talk to are focusing on internal, employee-facing chatbots. Internal chatbots use retrieval-augmented generation (RAG) to leverage internal data. They can help individual employees find information and navigate potentially complex processes rapidly, without relying on colleagues.

As with external chatbots, these internal tools can be constructed to provide personalized responses or more generic information. For example:

• An HR chatbot could guide employees through onboarding, like the “Mrs. Landingham” Slackbot launched in 2015 by the General Services Administration (GSA). An HR chatbot could also provide information about benefits, compensation, and time off tailored to each individual.

• An IT chatbot could provide tech support, helping employees resolve common support tickets, such as password reset requests or installing new software.

• A case management chatbot could help case workers and benefits adjudicators quickly look up rules, summarize case histories, and pull specific information from internal systems.

Chatbots excel at triage — handling frequently asked questions and determining whether problems should be escalated to humans. They can add productivity to any customer service center if they are properly designed, and they can significantly cut response times for users, which is particularly important if the chatbot is supporting an IT service desk or another critical function. Human agents can then concentrate on more complex, nuanced, and non-routine issues.

2. Assemble foundational technologies

Whether your organization is planning to build an external or internal chatbot, you will need the same foundational elements: a model, a data set, and a means to retrieve data.

LLM

Most public sector organizations choose an existing LLM instead of going through the expensive, time-consuming process of building a model themselves. You might select a commercial LLM, like OpenAI’s ChatGPT, or an open-source LLM, like Meta’s Llama.

When you are evaluating different models, consider functionality, security, and cost. For example, you might want a chatbot that can handle multimodal input and output (like text, images, and audio) rather than just text. You could select an open-source model so you can host the model in your own controlled environment to maximize security. An open-source model also allows you to avoid subscription fees and the potentially expensive per-token pricing of commercial providers.

Data set

If you are trying to ground the chatbot in specific knowledge, you need a data set for the LLM to draw from. Depending on the use case, that data could include vehicle registration information for everyone in the state, your agency’s HR policies, or a document store that contains a county’s vital records. To support a chatbot, the data should be kept in a vector database, which is structured to enable LLMs to remember previous inputs.

RAG

Finally, you need a way to integrate the LLM with that data source so your chatbot can deliver relevant information to users. RAG enhances user prompts with information from that data source and then queries the LLM. The LLM then synthesizes an answer using the new context, instead of relying on its general training data.

3. Don’t rush design

Carefully designing the chatbot’s interface, platform support, and relationship to other content will be key to ensuring its successful adoption. Start with user research, then define a limited scope for the chatbot, like specific topics it will cover or a certain type of question-and-answer model. Gather data and feedback to continuously improve the bot’s performance and knowledge base.

To maximize usefulness and inclusivity, consider deploying chatbots across multiple channels, including your website, your mobile application, and a text platform. Support multiple languages (English and Spanish are common priorities) to increase accessibility.

Remember also that chatbot content must work together with other website / FAQ content. The chatbot is a tool for providing information in a way that approximates human interactions; it’s not a replacement for presenting information in other ways.

4. Strengthen security and governance

Designing and building a chatbot are just parts of the journey. You also need effective ways to secure and govern it — and that will involve controlling the data flowing into and out of the chatbot.

Control the input

It’s important to place guardrails on user prompts to prevent manipulation of the AI system. In particular, teams need to guard the model and monitor the prompts for key threats, such as:

• Prompt injection and jailbreaking: Users might attempt to override the chatbot’s rules or trick the model into bypassing safety settings by inputting malicious code.

• Inappropriate prompts: Some users might purposefully input hate speech or explicit content, trying to get the model to engage with inappropriate material. Doing so wastes computing resources and could ultimately damage the reputation of the organization if the chatbot responds with similarly inappropriate content.

Monitor the output

You also need to prepare for problematic output, including inaccurate responses. For example, New York City’s MyCity chatbot for small business owners was asked if it was OK for a restaurant to serve cheese nibbled on by a rodent. The response — obviously incorrect — was that a restaurant could still serve the cheese, as long as they assessed the extent of the damage caused by the rodent and informed customers.

Chatbots can also provide inappropriate and biased responses that damage the reputation of the agency and reduce public trust. In a well-publicized example from 2025, xAI’s Grok posted antisemitic rhetoric.

An AI firewall can help safeguard inputs and outputs, blocking prompt injection, model poisoning, excessive usage, and other threats that traditional security solutions can’t address. Sitting at the network edge, between the user and the model, the firewall can prevent malicious input from reaching the model and inappropriate or non-compliant output from reaching the user. The firewall can be fully controlled by the government agency, employing agency-set policies.

Start building, securing, and governing

Cloudflare offers several tools that can help streamline the process of deploying AI chatbots — all from a single, unified platform. For example, Cloudflare enables your teams to build the backend for chatbots with a vector database, zero-egress-fee global object storage, and service for enabling RAG patterns without having to manage infrastructure.

You can then build and deploy AI-powered chatbots on Cloudflare’s global network. AI Gateway services enable developers to incorporate multiple LLMs into their chatbots while gaining a single point of control and observability for AI app traffic. Implementing in-line Firewall for AI services, integrated into Cloudflare’s global network, can then help protect both prompts and outputs.

There’s no doubt that chatbots can help meet user expectations for immediate service while also adhering to today’s increased drive toward government efficiency. Still, taking the time to carefully design your chatbot and to implement sufficient security and governance will be critical for achieving your goals.

This article is part of a series on the latest trends and topics impacting today’s technology decision-makers

작성자

Dan Kent — @danielkent1
Cloudflare 공공 부문 필드 CTO

핵심 사항

이 글을 읽고 나면 다음을 이해할 수 있습니다.

• 조직에 가장 적합한 챗봇 활용 사례를 식별하는 방법

• 선택하고 통합해야 하는 핵심 기술

• 챗봇 보안을 강화하고 관리하는 전략

관련 자료

• 안전한 AI 관행 보장: 확장 가능한 AI 전략 수립 방법에 대한 CISO 가이드

• 주 정부와 지방 정부를 위한 안전한 AI 도입

• AI가 생성한 잘못된 정보 해결하기