FORGE: the ten-layer stack for building an accountable AI workflow

A pre-integrated, open-source blueprint — and how to stand it up in an afternoon

The evaluation tax

Two earlier articles covered the design questions to answer before you build, and the disciplines that keep a system right after it goes live. This one is about the middle: what you actually build it with.

Most teams pay a tax before they write a line of code. Someone is asked to "evaluate the options," and four weeks disappear into comparison tables — six gateways, four orchestration frameworks, three vector databases, a dozen observability tools. The tables get built, a choice gets made, and nobody can quite explain why. Six months later a different team in the same company runs the same exercise and reaches a different answer.

The tax is worth paying once. It is not worth paying every time.

FORGE is our answer to it: ten layers, one component per layer, each chosen for a specific job, wired together so it runs. We publish it because a good stack is not a secret — the hard part was never which tools to pick, it is what you do with them. If you take this blueprint and build on it without ever speaking to us, the article has done its job.

Two rules shape every choice in it, and they explain the choices better than any comparison table would.

Rule one: component count is a constraint, not an afterthought. Every extra component is another entry in a security review, another licence to clear, another vulnerability feed to watch, another thing that breaks at 2am. So the stack uses one database engine for relational data, vector search and graph queries, rather than three specialist databases. It carries ten components, not twenty-five.

Rule two: every component must implement a design decision as a working control. Not as a policy document, not as a good intention — as something in the system that will refuse. "A human must approve refunds over $5,000" is a slide. A workflow that genuinely stops executing and waits for a named person is a control. The second is what this stack is arranged to produce.

The ten layers

Here is the whole thing on one page. Read the "what breaks without it" column first — it is the honest reason each layer exists.

And here is what sits in each, as of August 2026. Every one is open-source under a licence that permits commercial use.

1 — Model gateway: LiteLLM. One endpoint in front of every model you use, whoever makes it. Your code asks for "the cheap classification model," not "claude-something" or "gpt-something," and the gateway resolves that to a real model from configuration. This is worth more than it sounds: changing provider becomes a config edit rather than a code change, every call gets costed against a workflow, and rules like "customer data must not leave this region" live here as routing policy rather than as something a developer has to remember. The gateway category has become crowded — Portkey, Bifrost, Kong and others are all credible. Pick one; the discipline of having exactly one call surface matters far more than which.

2 — Orchestration: LangGraph. Your workflow becomes an explicit graph of steps with typed state flowing between them. Two features do the heavy lifting. Checkpointing means the workflow survives a restart mid-case — no half-processed invoices lost to a deploy. Interrupts mean execution can genuinely stop and wait for a person. That second one is how an approval requirement stops being a promise. LangGraph reached its 1.0 release with a stable API, which removed the main reason enterprises hesitated. If your organisation is committed to a single model provider, that provider's own agent SDK is a reasonable alternative; we default to the vendor-neutral option because our workflows have to run anywhere.

3 — Local inference (optional): Ollama for pilots, vLLM in production. Carry this layer for one of exactly two reasons. Either a class of content is not permitted to leave your estate or your country — a regulatory reason — or you have measured that a high-volume, simple step is cheaper on a small self-hosted model than on a hosted API. Below a few hundred thousand calls a month, self-hosting is usually a false economy once you price the time to run it. If neither reason applies, leave this layer out entirely.

4 — Document retrieval: LlamaIndex + pgvector. Your documents get chunked, embedded and searched so the model answers from your material rather than its memory. The choice worth noticing is pgvector: vector search runs inside PostgreSQL, so there is no separate vector database to deploy, secure, back up and explain. At workflow scale — thousands to low millions of chunks — a dedicated vector database buys you very little and costs you a whole extra system.

5 — Graph retrieval (optional): GraphRAG + Apache AGE. For the small number of workflows that must traverse relationships — who ultimately owns this counterparty, which suppliers connect to this sanctioned entity. Also inside PostgreSQL. Most workflows never need it; carry it only when the questions genuinely require walking a network.

6 — Output validation: structured outputs, with Instructor and Pydantic. Every model output that feeds a decision is forced into a defined shape and checked — the refund decision comes back as {eligible: bool, amount: decimal, evidence: [...], unresolved: [...]}, never as a paragraph someone has to parse. The major providers now support structured outputs natively, so use that where you can; Instructor sits over the top as the provider-neutral layer and, importantly, retries when validation fails — feeding the error back to the model rather than passing broken data downstream. This is the highest-yield reliability control in the entire stack and the cheapest to adopt.

7 — Action layer: FastAPI + MCP. Every connection to a real system — your ERP, your document store, your ticketing tool — is written as an MCP server. Every single one, even when a direct API call would be three lines. That discipline is what makes the SAP connector you write for the first workflow usable in the fourth, and it means integration work survives a change of orchestration framework. Since MCP settled into a formal specification it has become the standard here, which makes this the least controversial choice in the stack. FastAPI exposes the workflow itself over HTTP for the review interface and callbacks.

8 — Telemetry and evaluation: OpenTelemetry → Langfuse, plus Prometheus and Grafana. Your application emits standard OpenTelemetry traces to a collector; the collector forwards them to Langfuse for inspection, prompt management and evaluation runs, and to Prometheus for infrastructure metrics. Do not skip the collector and instrument straight into your observability tool — that decision is what lets a client who mandates Datadog cost you one endpoint change instead of a rewrite. This layer is also where the golden-set evaluation runs live: the mechanism that tells you the system is still right.

9 — Review interface: Streamlit for pilots, Next.js for production. Streamlit gets a working review screen up in hours, which is enough to prove the oversight gate is real. The production version calls the same API. Whatever you build it in, one requirement is non-negotiable: the reviewer must see the evidence, not just the conclusion. A screen that shows a recommendation and an Approve button manufactures rubber-stamping.

10 — Data and state: PostgreSQL 17+ with pgvector, plus Redis or Valkey. One database holds workflow checkpoints, decision records, configuration, and document embeddings — and the entity graph if you carried layer 5. One engine, one backup story, one access review. Redis (or Valkey, if your policy prefers a BSD licence) handles session state and queues.

Standing it up

The stack comes up fast because it is ordinary infrastructure: containers, one database, a few services. The minimum that runs is seven containers — PostgreSQL with pgvector, Redis or Valkey, the model gateway, an OpenTelemetry collector, Langfuse, your own workflow application, and a review screen. Everything ships as a published image except the last two, which are yours.

On a laptop or a small cloud VM, that is an afternoon — most of it spent waiting for images to pull. The stack coming up is not the hard part; your workflow is. What follows is the sequence that keeps the hard part honest:

1. Bring up the infrastructure and prove the gateway. Point LiteLLM at two different providers and make the same call through both. You have just proved provider independence, which is the claim most stacks make and few test.
2. Write your workflow as a graph with stub steps. No real logic yet — just the shape, the conditional branches, and the interrupt where a human will approve. Run it end to end. Getting the shape right before the substance is what prevents the expensive rewrite.
3. Build from the deterministic end inward. Connectors first, then validation, then the model calls. Most of a workflow is not model calls, and doing the plumbing first stops you from using a model where a database lookup would do.
4. Turn on the evidence before the intelligence. Make every step write what it retrieved and what it concluded, as it happens. Evidence reconstructed afterwards is not evidence.
5. Add twenty-five graded cases and wire them to your pipeline. Ten ordinary, ten hard, five that the system should refuse. Nothing ships if the score drops.
6. Deploy the identical image to a second cloud and serve traffic on both. One afternoon's work that converts "it's portable" from an intention into a demonstrated fact — and it is the cheapest credibility you will ever buy.

Where the design decisions become controls

This is the part that connects the three articles. Each design question has a place in the stack where it stops being a decision and starts being a mechanism that will refuse.

That fourth row is the one worth pausing on. In this stack, the configuration file is checked at boot: if a step is marked as requiring a human and no named human is attached, the deployment refuses to start. Not a warning, not a lint rule — it will not run. That is the difference between governance written down and governance implemented, and it is something you can show a risk committee rather than describe to them.

And the last row is where a decision record gets its teeth. Reconstructing a decision months later needs more than what the agent did — it needs what the agent was permitted to do at that moment. That is why every record carries the hash of the configuration in force when it ran.

What we deliberately leave out

Restraint is a design decision too. Each of these is a good thing that costs more than it returns until a specific trigger fires.

The temptation is always to add the thing you will "eventually need." Eventually is doing a lot of work in that sentence. Every component you add today is one you maintain for the next two years.

Keeping it current — and what stays true

An honest word about the shelf life of anything like this. The component names above are correct as of August 2026 and some of them will be wrong within a year. We re-verify the whole list every quarter and before any client's infrastructure review, and we replace a component only when the job it does is better done elsewhere — never because something newer exists.

What does not go out of date is the shape: one call surface for models, one place workflows are orchestrated with real pauses, one database, one integration standard, telemetry that is portable by construction, outputs that must satisfy a schema, and a review screen that shows evidence. If you keep those seven disciplines and swap every brand name in this article, you still have FORGE. If you keep the brand names and lose the disciplines, you have a demo.

The stack on one page

Everything above, in the form worth keeping. The middle column is what we run today; the right-hand column is what else is credible, so you can substitute deliberately rather than by accident — and so you have something to re-check in six months.

Every default above is open-source under a licence permitting commercial use. Two to flag before someone else finds them in your infrastructure review: Grafana is AGPL 3.0 — fine as a deployed service, which is the ordinary pattern, but say so upfront — and Terraform is under the Business Source Licence, with OpenTofu as the drop-in substitute if your policy prohibits BUSL.

One reading tip for the right-hand column: swapping a default is cheap, swapping a discipline is not. Replacing Langfuse with Datadog costs you an endpoint change, because telemetry goes through OpenTelemetry either way. Replacing "every integration is an MCP server" with "each team calls APIs directly" costs you every integration you have written.

Start here

If you are about to build an agentic workflow, three concrete moves:

1. Stand up the seven containers this week and run one trivial workflow end to end through the gateway, with a trace you can open and a decision record you can read. A day's work that tells you more than a month of evaluation tables.
2. Write your configuration file before your code. Which steps exist, which run alone, which pause, who approves, what the thresholds are. If you cannot fill it in, you are not ready to build — you have a design question outstanding.
3. Add the graded cases at the start, not the end. Twenty-five cases with known-right answers, wired to your build pipeline. It is the difference between shipping with confidence and shipping with hope.

We are packaging this as ready-to-run images — a Docker bundle and a prepared VM with the layers pre-wired and a sample workflow already running — so that step one becomes an hour instead of an afternoon. If that would be useful to you, tell us and we will point you at it when it lands.

Everything in this stack is open-source. Take it, change it, disagree with a layer and replace it. The reason to publish a blueprint rather than guard it is straightforward: the scarce thing was never the component list. It is knowing which workflow deserves this treatment, where the human belongs in it, and what evidence you will need eighteen months from now — and that judgement is the same whether you build on our stack or your own.

By Sai Krupa Sagar. To go deeper on FORGE — or to have the stack and the workflow design worked out for a system you are about to build — reach us at aiforgrowth.io/contact.