"Add AI" is now a line item on most product roadmaps, usually with no more detail than that. Sales wants something to demo, a competitor has announced a copilot, and engineering has three ideas with three very different price tags. Pick badly and you're locked into a vendor, leaking data across customer accounts, or a year into a model nobody uses.
The real choice is between buying an off-the-shelf capability, integrating a foundation model with your own data, or building something custom. The right answer for the AI integration existing software can actually support depends more on your data, your architecture and your obligations than on which model is top of the leaderboard this month.
So start there. Below is the decision tree we use with CTOs and product managers, with the cost, risk and data questions to answer at each branch.
The model is the easy part
Foundation models are a cloud API call away, so a first prototype takes days. The hard work is everywhere else: getting the right data to the model, making sure user A never sees user B's records, keeping costs predictable and meeting privacy obligations.
Two things raise the stakes for Australian teams. From 10 December 2026, privacy policies must describe certain automated decisions, and Allens' summary of the OAIC guidance reports that the disclosure obligation sits with the user of a product, not the vendor. And the deployment option you pick decides where data is processed. Microsoft's data, privacy and security documentation for Foundry models explains that standard deployments process prompts within the customer-specified geography, while "Global" deployments may process them in any geography where the model is deployed. Pick a Global deployment and your prompts may be processed outside Australia.
The three paths compared
| Path | What it means | Typical cost profile | Main risks | Data considerations |
|---|---|---|---|---|
| Buy | Turn on AI features in SaaS you already use, or add a specialist AI product | Licence or per-seat fees, low build cost | Vendor lock-in, limited control, features that don't fit your workflow | Data leaves your boundary under the vendor's terms; check residency and training use |
| Integrate | Call a foundation model API from your product, often with retrieval-augmented generation (RAG) over your data | Moderate build cost, usage-based run cost | Prompt injection, access control mistakes, cost growth with usage | Your data stays in your stores; retrieval must respect existing permissions |
| Build custom | Fine-tune or train a model, or build a specialised pipeline around one | Highest build and maintenance cost, needs ML skills | Model drift, evaluation burden, long time to value | Needs large, clean, labelled data and a governance process for it |
How to choose the AI integration existing software can support
Work down the list. Stop at the first branch that fits.
- Is the capability generic, not a differentiator? Meeting summaries, email drafting, document search inside a tool your team already uses. If so, buy. Read the vendor's data terms, residency options and whether your data is used for training. Only move on if the bought feature can't meet a specific requirement.
- Does the feature need your own content or records to be useful? Product documentation, policies, case notes, contracts, catalogue data. If so, integrate with RAG. AWS describes retrieval-augmented generation as having the model reference an authoritative knowledge base outside its training data without retraining, and calls it a more cost-effective way to introduce new data than retraining a foundation model.
- Does the feature need to act, not just answer? Creating records, updating orders or kicking off workflows moves you into agent territory. Integrate with tool calls, scope permissions tightly and keep human approval for high-impact actions. Our guide to starting small with AI agents covers the guardrails.
- Does a general model with good retrieval and prompting still fall short on a task you can measure? A specialised classification, a domain vocabulary, a strict output format at high volume. Only then look at building custom, and start by fine-tuning an existing model rather than training one.
- Will the output significantly affect people? Credit, pricing, eligibility, hiring and triage decisions carry extra obligations whichever path you take. Plan disclosure, logging and human oversight now, and read our piece on the automated decision-making transparency rules.
RAG for business: where it actually goes wrong
For most products, RAG for business content is the sensible middle ground. Your data stays in your own stores, answers can cite their sources and you skip the cost of training. AWS notes that RAG lets the model present information with source attribution, and that the underlying documents and their embeddings need updating so answers don't go stale.
The failures are rarely the model's fault. They're usually one of these:
- Access control. Microsoft's overview of RAG in Azure AI Search states plainly that users and agents must only retrieve content they're authorised to see. If one vector index holds every tenant's documents and the query has no tenant or permission filter, your new feature is a cross-customer data leak.
- Content quality. Out-of-date, duplicated or contradictory documents produce confident, wrong answers. Curate before you index. Three versions of the refund policy means the model will quote whichever one ranks highest.
- Chunking and metadata. How you split documents, and which metadata you keep (owner, date, product, customer), decides whether retrieval finds the right passage. Splitting a table across two chunks is a classic source of nonsense answers.
- Evaluation. Build a set of real questions with known good answers and score every change against it, including changes to chunk size and prompts.
- Cost per request. Work out tokens per request at production volume before launch. Stuffing ten long chunks into every prompt adds up fast.
Custom AI vs off-the-shelf: the costs that don't show up early
Off-the-shelf looks cheap until per-seat pricing scales across the whole workforce, or a key workflow needs something the vendor won't build. Custom looks attractive until you count the ongoing work: retraining, watching for drift, maintaining evaluation sets and keeping ML skills on the team.
Ownership is a useful test. If the AI capability is part of what customers pay you for, owning more of the stack is often worth it. If it's there for internal productivity, buying or a thin integration usually wins. Either way, agree the success metric first, as we argue in why most AI pilots never reach production.
Working through the decision with us
Our AI-driven software solutions team starts by reading what you already have: architecture, data stores, how permissions are modelled, and the AI clauses in your current vendor contracts. Then we map each candidate feature to buy, integrate or build. You get a short recommendation per feature covering indicative effort, run-cost drivers, data readiness and the privacy and security controls it needs.
When the answer is integrate, we build it into your product with your engineers: retrieval that enforces existing permissions, model calls routed through deployments that match your residency needs, evaluation sets, logging and cost tracking. We'll usually advise against fine-tuning as a first step, and against any shared index that can't filter by tenant.
Weighing up several AI options for an existing product? Request an AI integration review and we'll help you choose the path that fits your data, risk and budget.
Frequently asked questions
What is RAG and why is it popular for business AI?
Retrieval-augmented generation (RAG) is a pattern where your application searches your own content for relevant passages and passes them to a language model along with the user's question. The model answers from that content and can cite it. It's popular because it uses current company data without retraining, keeps the data in your own stores and usually costs less than fine-tuning.
Should we fine-tune a model or use RAG?
Start with RAG and good prompting for most business use cases, because it's cheaper and easier to keep current. Consider fine-tuning only when a measurable task still falls short, such as a specialised classification, a strict output format or domain language the model handles poorly. Fine-tuning brings ongoing work: managing training data, re-evaluating and redeploying when the base model changes.
Is it safe to send customer data to an AI model API?
It can be, with the right deployment and contract terms. Check where prompts are processed and stored, whether your data is used to train models, how long it's retained and who can access it. Pick deployment options that match your residency needs, send as little personal information as the feature needs, and make sure retrieval respects existing access permissions.
How much does it cost to add AI to an existing application?
It depends on the path. Buying AI features usually means licence or per-seat fees and little build work. Integrating a model adds engineering effort plus usage-based run costs that grow with traffic. Custom models add training, evaluation and specialist staff. Estimate cost per request at production volume, including monitoring and human review, before you commit.
AI for Business
Request an AI integration review
Tell us where you're stuck and we'll come back with practical next steps.
Get in touchSources
- Allens: Mandatory automated decision-making disclosure requirements are coming: key takeaways from the OAIC guidance
- Microsoft Learn: Data, privacy, and security for Foundry Models sold by Azure
- AWS: What is RAG (Retrieval-Augmented Generation)?
- Microsoft Learn: RAG and Generative AI, Foundry IQ (Azure AI Search)