There is a pattern I have seen in nearly every portfolio governance conversation over the past year. The operating partner wants to get AI governance right. They know it matters, they can feel regulatory pressure building, and they have seen what happens when a portco runs AI without any oversight structure. So they do what makes sense: they open a vendor evaluation. They pull together a shortlist of AI governance platforms. They issue an RFP.
This is the wrong move. Not because the platforms are bad. The problem is that they have skipped the only question that matters before procurement: where does governance live in your architecture?
Get that question wrong and no platform saves you. Get it right and the vendor decision almost answers itself.
The Architecture Question Comes Before the Vendor Question
AI governance is not a product you install. It is an architectural decision about where control sits in your AI stack. Every portco deploying AI is implicitly making this decision right now, usually without realizing it. When they add guardrails or logging to a specific model integration, they are making a distributed decision. When they funnel all model calls through a single routing layer, they are making a centralized decision.
Most portcos end up distributed by default, not by design. They added governance incrementally: a logging call here, a rate-limit there, a content filter bolted onto the most sensitive use case. The result is several different governance implementations that do not talk to each other, do not produce a unified audit trail, and require separate updates every time a policy changes.
The question that should precede any vendor evaluation is: do you want governance to live at the routing and gateway layer, or do you want it to live per model, per deployment, per team? That is the centralized-vs-distributed decision. It is an architecture decision. And it has to be made before the RFP goes out.
What Centralized Governance Actually Means
A gateway-layer governance architecture puts a control plane between every application and every model. All inference requests pass through it. The control plane enforces routing rules, cost caps, rate limits, content policies, and model selection. It logs every request and response into a single, unified audit trail. No model call gets made that the control plane has not seen and approved.
The architectural advantage is significant. One policy change propagates everywhere. One audit query covers all models, all portcos, and all use cases. One set of token budgets applies across every application. When you add a new model or a new portco to the stack, it inherits the governance structure by virtue of going through the same gateway.
This is not a theoretical design. One portfolio company we worked with came into an engagement with AI governance spread across four different use cases, each owned by a different business unit. Each unit had built its own oversight layer: one used a cloud provider's built-in content filtering, one had written custom logging middleware, one relied on post-hoc database queries, and one had no structured oversight at all. When compliance asked for an audit trail for a regulatory inquiry, the company spent three weeks manually correlating four different log formats. A team of people doing work that should have been a single query against a unified system.
After moving to a gateway architecture, the answer to "show me every decision this system made in the past 90 days with these input characteristics" is a query that returns in seconds. Not because the data changed, but because it is now in one place, in one format, under one access model.
What Distributed Governance Actually Means
Per-model governance means each model integration owns its own controls. If you have six different AI applications, each one handles logging, rate limiting, and policy enforcement independently.
The appeal is real. Each team can configure their governance to exactly what they need without negotiating shared infrastructure. There is no single point of failure. Teams ship faster because they are not waiting for a central platform to support their specific use case.
The cost is fragmentation. Every policy update requires multiple code changes across multiple codebases. Every audit requires stitching logs in different formats from different systems. Every new model integration starts governance from scratch. Token spend is invisible at the portfolio level because there is no view that crosses application boundaries.
The distributed model also creates an accountability gap that is hard to see until it becomes a problem. When governance is per-model, nobody owns the total picture. The product team owns their integration. The compliance team owns the policy document. The operating partner owns the portfolio. None of them can see the full AI surface area. That gap is exactly what regulators are looking for.
The Three-Layer Model That Clarifies the Decision
The governance architecture question becomes cleaner when you separate the three functional layers that any governance system needs to cover.
The control plane is where decisions about routing, policy, and cost are made before a model call happens. This is where you set rules like: certain query types go to cheaper models, certain portcos operate under stricter content policies, certain applications have monthly token budgets. The control plane is the policy engine that sits upstream of every inference event.
The runtime is where the actual inference happens. This includes the model call, any tool use, the context window assembly, and the response. The runtime layer is where you measure latency, token consumption, and model performance on a per-task basis.
The audit layer is the immutable record of what happened. Every control plane decision, every runtime execution, every token consumed gets written to the audit layer. This is what compliance auditors query. This is what you use to debug a misbehaving workflow. This is what you point to when a regulator asks you to demonstrate that your AI system is operating within defined parameters.
A distributed architecture means each application manages its own version of all three layers independently. A centralized architecture means all three layers are shared infrastructure that any application in the portfolio can plug into.
The reason centralized wins on governance specifically, even when distributed wins on developer velocity for other considerations, is the audit layer. A fragmented audit trail is not an audit trail. It is several logs that were never designed to be read together. When the moment comes that you need to produce a coherent account of what your AI systems decided and under what controls, a distributed architecture makes that question hard to answer. A centralized architecture makes it trivial.
Where Token Economics Reinforce the Architecture Decision
The governance architecture question intersects with cost management in a way that is not always obvious at the outset. Governance is usually framed as a compliance overhead. When it is centralized at the gateway, it also becomes a cost lever.
A gateway can observe all inference traffic before it reaches a model. That position lets the gateway do several things that per-model architectures cannot do efficiently.
First, the gateway can route requests to the cheapest model that can handle the task. A query that only requires simple classification does not need to go to a flagship reasoning model. A query that requires complex multi-step inference does. The gateway, sitting in front of all models, can make that routing decision based on query characteristics. Each individual application, operating in its own governance silo, usually routes everything to a single model because that is simpler to implement. The routing logic never gets built because there is no layer that can see all the traffic.
Second, the gateway can cache responses to identical or near-identical queries. Portfolio companies often have multiple applications making semantically similar requests across different portcos or use cases. A per-model architecture cannot see this duplication. A gateway can, and can return a cached response instead of paying for the same inference twice.
Third, the gateway can enforce portfolio-level token budgets. Without centralized visibility, portcos hit budget surprises because no individual application owner sees the total spend. The gateway changes this: one dashboard, one set of budgets, one enforcement mechanism across every AI application in the portfolio.
In engagements where we have moved portcos from per-model governance to a gateway architecture, the token savings from routing optimization and caching have been material. This is not a governance-or-economics tradeoff. It is governance architecture that also improves the economics. The control plane pays for itself.
How the Architecture Answer Drives the Vendor Decision
Now you can go back to the vendor evaluation. But the evaluation looks different because you have answered the architecture question first.
If you have determined that your architecture is centralized gateway-based, the vendor question becomes: what is the best gateway to serve as your control plane, and what do you run on top of it for the runtime and audit layers? Your evaluation criteria are: does this gateway support all the models your portcos currently use? Does it have native audit logging that meets your compliance requirements? Can it enforce the token routing rules your cost model depends on? Can it scale to the number of portcos in your portfolio without per-portco configuration overhead? You are selecting infrastructure that your entire portfolio will share, so stability, vendor lock-in risk, and integration breadth matter more than feature velocity.
If you have determined that your architecture is distributed, the vendor question becomes: what is the lightest-weight tooling that lets each team implement governance consistently without building from scratch? You are selecting something that multiplies team productivity, so developer experience, fast integration, and a strong default configuration matter more than centralized visibility features you will never use.
The vendor evaluation is fundamentally different depending on the architecture answer. Running the evaluation before answering the architecture question means you might select a platform that is excellent for centralized governance and spend a year trying to make it work in a distributed model, or select a distributed-first tool and then discover that your compliance requirements actually demand a unified audit trail. The RFP outcome and the architecture need to match. When they do not, the implementation fights itself.
What to Actually Do
Before you issue the RFP or schedule the vendor demos, answer one question: when a regulator or a board member asks you to demonstrate what your AI systems decided last quarter and under what controls, where does that answer come from?
If the answer is a single query against unified infrastructure, you have a centralized architecture or you are on the path to one. If the answer is "we would have to pull logs from several places and reconcile them manually," you are in a distributed architecture whether you planned for it or not.
The follow-up question is: does that distributed structure serve your portcos better than a gateway would, given your portfolio's actual footprint, team structure, and compliance obligations?
If the answer is yes, the vendor evaluation is about picking the right tools for distributed governance. If the answer is no, the vendor evaluation is about picking a gateway, and the path forward is to move applications through it as they come up for renewal or new integration.
Most portfolios, when they work through this with their actual regulatory exposure and the number of models they are running across portcos, land on centralized. Not because decentralized is wrong in principle, but because the audit requirement alone usually forces the issue. A portco that cannot produce a unified audit trail from its AI systems in a regulated environment does not have a technology gap. It has an infrastructure gap.
The governance platform that answers that specific gap is the one worth buying. The vendor decision follows from the architecture decision. That ordering is not bureaucratic. It is the only sequence that produces an implementation you can actually audit.