Before you continue reading this, note that I assume you are going to self-host the model. Whether that means off-prem or on-prem does not matter, but you are not just calling some AI provider’s API.
When I decide to build a chatbot, an agent, or some extravagant multi-agent gizmo just to learn something new, I dive in and have fun. I start building and I do not stop until I am happy with it. I do not think about sleep, food, water, deadlines, budget or the users.
Client work is different. When someone is paying you to solve a problem, you cannot just accept “We need a chatbot for our FAQ page” as a spec. Nodding along to “We need a multi-agent system for our workflow” will result in lots of unnecessary pain and suffering for both parties.
“Chatbot” describes an interface, “Multi-agent system” describes an architecture. You need requirements.
A perfect spec is a pipe dream. But asking the right questions helps your client define what they actually need, and helps you with choosing the right trade-offs to get the best ROI. Almost anyone can take a general-purpose SOTA model and get the job done in a mediocre way. To get the best bang for the buck you need to understand what your system does NOT need to do.
Here are some questions to consider. I will leave it to your judgment, depending on the context and the client, which of them you will ask directly and which you will infer from the conversation.
Input and output
Think about what input you are going to get. How large will it be? Will it be structured or free-form? What type? For a basic chatbot, the input might be text only. For a voice assistant, it is audio. For an image-manipulation application, it might be text, images or both. Do you accept files and what types?
“Supports PDF files” sounds simple until someone uploads a 1000+ page scanned document full of multi-gigabyte-sized x-ray images. You did not expect that one, did you? Neither did I when it happened to me. But that’s life: it hurts, and then you die.
And if you support files think about how they will be used - should their contents be added directly to the model’s context, or should they become part of a searchable knowledge base? It will affect your KV cache size, the context degradation, and also the ability to answer some type of questions (local vs global questions). There is a case for each.
What language is used? Do you support multiple languages? If so, which ones? If not, how does your system respond to input in an unsupported language?
Will the system support long-running conversations or agents whose context grows over time? What context limit will you impose? When the system reaches it, will you summarize the conversation or discard everything?
How about the output? Does it need to be structured? If you intend to tool-call with the output, you need to know this. If the output does not parse, or if it is cut mid-JSON you get broken pipelines and problems. You need to choose models with reliable structured-output support, do validations, retries etc.
Perhaps you are creating images or audio? You need to have additional models or use multi-modal ones. How large will the output be? Is the output language different from the input one? Are you translating text, is it specific for some industry? This is an example where a general-purpose SOTA model might get you far worse results than a small specialized model.
Capabilities
Do you expect the system to have the ability to search the internet? Can it read from a local or network filesystem? Can it create, edit or delete files? Which tools and external services can it call and how often? What if they don’t respond or return gibberish?
Concurrency
How many users will the system have? Can one user run multiple conversations or tasks simultaneously? Keep in mind that this is important not only for the processing capabilities part but for the KV cache requirements. Will usage be steady, or will requests arrive in bursts?
Latency
How quickly must the system begin responding? How quickly must it generate the response? How long may the complete task take?
Deployment
What hardware and serving software are already available? If none are available, what budget is there for them? Must it work with an approved runtime such as vLLM or llama.cpp? Which is it exactly? Must the system run on-premises, in a particular country or region, or in an air-gapped environment?
Availability
When must the system be available? Is it working 9to5 or do you want five nines? Is occasional downtime acceptable, or is it part of a business-critical process? What happens to work already in progress when a model server fails?
Privacy and security
Does input contain PII? Do tool calls produce PII? Will it contain financial information, source code or business secrets?
Will you store the history of the work being performed? If so, for how long?
May prompts and outputs be logged? How long may those logs be retained, and who may access them?
What user roles and permissions exist, and how will the system enforce them?
Failure tolerance
Which mistakes are acceptable? And don’t say “none.”
Which mistakes are deal-breakers? Be precise. And don’t say “all of them.”
Will a person review every output? In some cases, that might be not only acceptable but expected. In others, it would be ridiculous.
May the system send messages, modify data or call external services without human approval? If so, which actions and services?
When you get a response from the system, does it need to cite its knowledge sources? This is especially important in high-risk contexts, such as medical software.
Shared context
Is the accumulated context, memory or knowledge shared among a group of users with different access rights? Be careful to note this one. As you cannot rely on the model to enforce access control by “cleaning” shared context for each user or role, you will need to develop a deterministic mechanism to protect the data which is a non-trivial requirement.
Conclusion
Now, that is a big list. I mentioned it will pay off and it will.
You will not ask the client all of these. Values like TTFT, tok/s follow from the described use case, KV cache sizing follows from concurrency and context size etc. Also, don’t expect that the client will give you failure tolerance on a platter. Ask “what if” questions until you can understand what is critical.
In some cases POCs can be built and demoed on available hardware, or using pay-by-the-hour GPU rental. You can skip the detailed conversations and deliver the demo. But for the real thing, you better be sure what is actually necessary before someone orders an expensive AI system. You don’t want to find out that a couple of questions could have saved a couple of mil. Ouch.
I will cover the reasoning for all of these in the following articles.