Frame
Name the decision, the user, the failure modes and what success would look like.
Done when: A written definition of a wrong answer exists.
AI Delivery Framework (DELIVER)
AI Delivery Framework
A gated path from idea to live service, with evidence at every step.
The business problem
The demo worked. Production did not.
Models move from notebook to production by heroics, with no gates, no monitoring and no one on call.
What you hear inside the organisation
It worked on the data scientist's laptop.
Accuracy dropped and nobody noticed.
Who supports this model at night?
Every team builds AI its own way.
Who it serves and what changes
Your teams, before and after the AI Delivery Framework
Client teams and what they need
AI and data science
A repeatable path to production
IT operations
Monitoring and clear support
Security
Controls built into the release
Business owners
Users who adopt the result
The outcome you get
- Stage gates from idea to live service
- Monitoring for drift, cost and quality
- A run book and named support owner
- One delivery method across teams
A worked example
A telecom operator with a churn model
The problem we saw
The model worked in a notebook. In production it drifted for months before anyone noticed, and nobody owned the fix.
How the framework solved it
We added gates from idea to live service, automated testing and release, drift monitoring and a named support owner.
The business result
Drift now raises an alert, retraining is routine, and new models reach production through the same path.
Illustrative example based on common client situations.
In short
What is the AI Delivery Framework?
AI Delivery Framework (DELIVER): A gated path from idea to live service, with evidence at every step.
The gap between a demo and a production AI system is evaluation, hardening and operations.
- How it moves
- Each step closes with a check before the next begins
- Owned by
- Run by the delivery lead; each gate is signed by the business owner and by risk where the tier requires it.
- You keep
- Golden dataset and evals, Gate evidence pack, Operations runbook
How it works
How the AI Delivery Framework works
Evaluate is the stage most programmes skip, and the reason most pilots never earn a production decision.
Ground
Connect to certified data, define retrieval scope, entitlements and refresh.
Done when: Every answer can name its source; entitlements are enforced at query time.
Prototype
Build the thin slice end to end, including the human hand-off.
Done when: The slice runs on real data inside your controls.
Evaluate
Score against a golden dataset: groundedness, accuracy, refusal behaviour, Arabic quality.
Done when: The workload beats the agreed threshold, and regressions block release.
Harden
Red-team, bound the tools, add rate and cost limits, plan the rollback.
Done when: Known prompt-injection and data-exfiltration paths are closed and tested.
Launch
Release to a bounded population with monitoring and a human escalation path.
Done when: Attestation is current and on-call ownership is named.
Operate
Run evaluations continuously, watch cost and drift, and feed fixes back to the data layer.
Done when: Weekly evaluation and cost readings exist for every live workload.
Is it right for you?
Is the AI Delivery Framework right for you?
A strong fit if
- You have pilots that need to go live
- Live models fail without warning
- Several teams build AI differently
Probably not yet if
- You are still deciding whether to use AI
- You only consume AI inside packaged software
Free assessment
Discuss the AI Delivery Framework with our team
A working session with our senior team. We review what fits your situation, what does not, and where to start.
A focused conversation
We listen to your situation and goals.
Fit review
Which parts of this framework apply to you, and which do not.
A written view
Clear next steps, whether or not you work with us.
Go deeper
Applying the AI Delivery Framework in practice
Open only the part you need: how it starts, what your team keeps, maturity levels, sector rules and the evidence a reviewer will ask for.
The gap between a demo and a production AI system is evaluation, hardening and operations. DELIVER makes those explicit stages with exit evidence, which is why workloads reach production instead of a second pilot.
Version
v2.0
Last reviewed
July 2026
Family
AI Operating System
How this was developed: Built from delivery practice on Saudi and international engagements, then reconciled clause by clause against the published standards below. Revised when a standard is updated or a gate proves unworkable in the field.
One-pager
Download the DELIVER one-pager
Stages, exit gates, artefacts, standards crosswalk and FAQs for AI Delivery Framework on one printable page, share it with the committee.
- Week 1
Frame the decision, the user, and the failure modes you will accept and refuse.
- Week 2
Ground on a certified data domain and build the golden dataset before the model.
- Week 3
Prototype against real inputs, then evaluate with thresholds fixed in advance.
- Week 4
Harden, write the runbook and rehearse the rollback before the first user sees it.
- End of the first thirty days
Golden dataset and evals
Your questions, your expected answers, re-runnable on every change.
Gate evidence pack
Signed exit evidence for each of the seven stages.
Operations runbook
Monitoring, escalation, rollback and change procedure.
L1
Demo-driven
Prototypes impress and never leave the laptop.
L2
Piloted
A workload runs with a small group, unevaluated.
L3
Evaluated
Releases pass fixed thresholds on a golden dataset.
L4
Operated
Owners, monitoring, rollback and cost control are in place.
Banking & finance
Evaluate adds adverse-action explainability and a four-eyes review on customer impact.
- Who reviews it
- SAMA, plus PDPL and NCA controls
- Systems in scope
- Core banking, cards, KYC and AML systems, credit bureau feeds, the general ledger
- Where it lands first
- KYC and AML review, credit decisioning support, complaint and dispute handling
1
Frame
- Workload brief
- Failure-mode list
2
Ground
- Grounding design
- Entitlement rules
3
Prototype
- Working slice
- Prompt and tool spec
4
Evaluate
- Golden dataset
- Evaluation report
5
Harden
- Red-team report
- Rollback runbook
6
Launch
- Launch checklist
- Escalation path
7
Operate
- Operations dashboard
- Change log
Evaluate is the stage most programmes skip, and the reason most pilots never earn a production decision.
Stage
Exit gate
Artefacts
1
Frame
Name the decision, the user, the failure modes and what success would look like.
- Signed off by
- Business owner of the decision
- Typical duration
- Scoped with you
Exit gate
A written definition of a wrong answer exists.
Evidence: Problem statement with the accepted failure modes named
Artefacts
- Workload brief
- Failure-mode list
2
Ground
Connect to certified data, define retrieval scope, entitlements and refresh.
- Signed off by
- Data owner of the domain
- Typical duration
- Scoped with you
Exit gate
Every answer can name its source; entitlements are enforced at query time.
Evidence: AI-ready certificate for the domain the workload reads
Artefacts
- Grounding design
- Entitlement rules
3
Prototype
Build the thin slice end to end, including the human hand-off.
- Signed off by
- Delivery lead
- Typical duration
- Scoped with you
Exit gate
The slice runs on real data inside your controls.
Evidence: Working prototype plus the golden dataset it is judged on
Artefacts
- Working slice
- Prompt and tool spec
4
Evaluate
Score against a golden dataset: groundedness, accuracy, refusal behaviour, Arabic quality.
- Signed off by
- Evaluation owner, independent of the builder
- Typical duration
- Scoped with you
Exit gate
The workload beats the agreed threshold, and regressions block release.
Evidence: Evaluation report with pass thresholds set before the run
Artefacts
- Golden dataset
- Evaluation report
5
Harden
Red-team, bound the tools, add rate and cost limits, plan the rollback.
- Signed off by
- Security and platform leads
- Typical duration
- Scoped with you
Exit gate
Known prompt-injection and data-exfiltration paths are closed and tested.
Evidence: Red-team results and the fixes closed against them
Artefacts
- Red-team report
- Rollback runbook
6
Launch
Release to a bounded population with monitoring and a human escalation path.
- Signed off by
- Service owner
- Typical duration
- Scoped with you
Exit gate
Attestation is current and on-call ownership is named.
Evidence: Runbook, rollback path and named on-call ownership
Artefacts
- Launch checklist
- Escalation path
7
Operate
Run evaluations continuously, watch cost and drift, and feed fixes back to the data layer.
- Signed off by
- Service owner with the business owner
- Typical duration
- Ongoing
Exit gate
Weekly evaluation and cost readings exist for every live workload.
Evidence: Live monitoring with drift and cost alerts wired to a human
Artefacts
- Operations dashboard
- Change log
Standard
What this framework satisfies
CRISP-DM / TDSPBusiness understanding → deployment, extended with evaluation
The classic analytics lifecycle, extended with evaluation and hardening for generative systems.
MLOps / LLMOpsCI/CD/CT, model registry, continuous evaluation and rollback
Continuous evaluation, versioning, monitoring and rollback in Operate.
OWASP LLM Top 10LLM01 injection · LLM02 insecure output · LLM06 disclosure
Harden stage test list for injection, exfiltration and insecure tool use.
NIST AI RMFMEASURE 2 · MANAGE 2 (test, evaluation and response)
Measure and Manage functions realised as gates rather than documents.
Demo-driven delivery
A convincing demo with no golden dataset cannot be defended when it is wrong.
Skipping the hand-off
If nobody owns the escalation, the first bad answer stops the programme.
Prompt tuning as strategy
Quality gains that are not measured cannot be kept through the next change.
Built on
- 1
NIST AI Risk Management Framework 1.0 and Generative AI Profile
National Institute of Standards and Technology, Govern, Map, Measure, Manage.
- 2
ISO/IEC 42001: AI management systems
International Organization for Standardization, management system requirements for AI.
- 3
OWASP Top 10 for LLM Applications and MITRE ATLAS
The AI attack surface and adversarial technique catalogue.
- 4
MLOps and LLMOps maturity practice
Published maturity levels for automation of the model lifecycle.
Citations describe the parts of each standard the framework answers. Bilytica is not an accreditation or certification body.
In Saudi Arabia
What this family has to satisfy before a release in Saudi Arabia
The AI operating system frameworks carry a workload from a readiness score to a governed release. These are the Saudi checks each stage is built against.
AuthorityWhat it requiresWhat we buildWhat you keep
SDAIA, PDPL
Purpose, lawful basis and disclosure recorded before the first personal-data field is used.
Diagnosis and assurance stages block a build that has no recorded basis.
Processing register entry attached to the release record.
NCA ECC
Authorisation and containment for anything the system can do on its own.
Agent operations defines action classes, dual control and a tested rollback path.
Containment test result and an authority matrix signed by the business owner.
NDMO data management controls
No workload builds on an uncertified data domain.
The data foundation gate certifies ownership, lineage and quality thresholds first.
Signed domain certificate referenced by the release.
Where it runs
Training, evaluation and serving run inside Saudi Arabia on your infrastructure, with any hosted component documented as a boundary.
Arabic, as it is spoken here
Arabic quality is a release condition: evaluation sets include Saudi Arabic content, not translated English.
FAQ
Frequently Asked Questions
What is a golden dataset?
A fixed set of real questions with agreed correct answers and sources, used to score every change before release.
Can stages be run in parallel?
Frame and Ground often overlap. Evaluate and Harden cannot be merged into launch, that is where the risk lives.
How is Arabic quality measured?
Through the Arabic Quality Standard: terminology, register, RTL rendering and answer equivalence against the English evaluation set.
How fast is the first governed release?
Timelines depend on your data and scope, and we agree them with you once the data domain the use case depends on is certified. The gates are what keep delivery safe.
What stops it becoming another pilot?
Evaluate and Harden are exit gates, not optional steps: an evaluation set, security testing against the OWASP LLM Top 10, and an operate plan exist before launch.
How is quality measured before launch?
Against a task-specific evaluation set built with the business, covering accuracy, grounding, refusal behaviour and Arabic performance where the workload is bilingual.
What happens after launch?
The Operate stage owns monitoring, evaluation reruns, cost tracking and a change path, so the release stays measured instead of drifting quietly.
Where this sits on the spine
Where this connects
Rest of the library
DFFData Foundation FrameworkAIRMAI Readiness ModelVALUEAI Value FrameworkTRUSTAI Assurance FrameworkAGENTAgent Operations FrameworkSOVEREIGNSovereign AI FrameworkDMMData Management Maturity ModelDPAData Platform Architecture FrameworkDQFData Quality & Observability FrameworkDECIDEAnalytics & Decision FrameworkDGOVData Governance Operating ModelSDCFSaudi Data Compliance FrameworkAQSArabic Quality StandardKSA-READYIn-country Deployment Framework
Topic cluster
Explore the topic cluster
Curated links across our Saudi hubs, sector pages and long-form blog research.