17 November 2026 · 8 min read

import-model

Why this matters now

For some organisations, the strongest reason to run AI on private infrastructure isn’t cost. It’s control: over where data goes, which laws reach it, how fast it answers and what it costs next year. In 2026 all four of these moved at once.

This post looks at each pressure in turn, then at how a private GPU cluster running Nutanix Enterprise AI (NAI) answers it.

Caveat: A PoC built around a hyperscaler’s native agent framework, knowledge base, and guardrail services cannot necessarily be lifted and shifted on-premises. These dependencies would need to be assessed and, where necessary, replaced or re-engineered, adding development, integration, and testing effort.

GDPR: the transfer question never quite goes away

Every prompt sent to a public API is a data transfer. If it holds personal data, GDPR applies to where it goes, who can access it, and on what legal basis. For health, genetic or biometric data (GDPR Article 9 special categories) the bar is higher still.

The legal protections for data transfer between the EU and the US may change.

Data Privacy Framework Program (DPF) was developed to provide U.S. organizations with reliable mechanisms for personal data transfers to the United States from the European Union, United Kingdom, and Switzerland ensuring data protection that is consistent with EU, UK, and Swiss law.

While the DPF is still in force today, and the European Commission still supports it, a potential future ruling by the European Court of Justice, targeting the validity of the current Framework, would remove the main legal basis for most EU–US personal data transfers overnight.

There’s a second, separate issue: the US CLOUD Act. It lets US authorities compel a US provider to hand over data, wherever the servers are. An EU region from a US hyperscaler or AI vendor doesn’t remove that exposure. The DPF doesn’t address it either.

What this means for AI: Cloud AI can introduce cross-border transfer and jurisdictional considerations when personal data is processed externally. EEA-hosted inference on infrastructure under your control removes the external inference transfer path, simplifying data-sovereignty and GDPR compliance.

The EU AI Act: delayed, not removed

The Digital Omnibus on AI has delayed the introduction of high-risk controls, but didn’t change what those rules require.

Date What applies
2 August 2028 High-risk AI embedded in regulated products (Annex I: medical devices, machinery)
2 December 2027 Standalone high-risk AI (Annex III: employment, credit, education, critical infrastructure, biometrics)
Since 2 August 2025 General-purpose AI model obligations (on model providers)
Since 2 February 2025 Prohibited AI practices

For organisations that deploy a high-risk system, the Act expects things infrastructure teams will recognise:

  • Traceability: keep the logs the system generates, and know which model version produced which output.
  • Human oversight: people who can monitor, interpret and override the system.
  • Monitoring and incident reporting: watch the system in operation and report serious incidents.
  • Control of input data: make sure what you feed the system is relevant to its purpose.

None of this legally requires on-premises hosting. But each obligation is easier to prove when you control the model version, the logs and the data path. With a public API, the model behind an endpoint can change under you, and your audit trail depends partly on the vendor’s.

The delay is runway, not relief. Teams that build their governance now won’t be queuing for conformity assessments with everyone else in late 2027.

Note: Hyperscalers will continue to develop controls that allow EU customers to meet increasingly stringent compliance requirements. However, the potential for data exposure remains a compliance and sovereignty risk. For highly sensitive environments, there is still something uniquely powerful about having the ability, quite literally, to disconnect the external network and physically isolate the environment.

Latency: consistency matters more than raw speed

The latency case for private inference is about predictability and proximity, not beating a hyperscaler’s hardware.

  • Distance to the data. A retrieval-augmented (RAG) app pulls documents from internal systems, sends them to the model and waits. When the model sits next to the data, those round trips stay inside the data centre.
  • No shared queue. A public API runs on shared provider infrastructure, where rate limits, service load and provider-side incidents can introduce latency outside your direct control.
  • Sites with weak links. Labs, factories, hospitals and branches can keep running AI locally when the WAN is slow or down.
  • Tuning you own. Nutanix Enterprise AI (NAI) 2.6 added KV-cache-aware routing (tech preview), which sends a request to the GPU that already holds its context. That cuts time to first token for multi-turn chats and long documents.

Be honest about the trade-offs. A frontier model running on a hyperscaler’s latest GPU infrastructure may deliver higher raw token throughput than a modest on-premises deployment. The private-inference advantage is therefore not necessarily peak speed, but predictable latency, data proximity and control. There is also an economic trade-off: privately owned GPUs need sufficient utilization to justify their cost. If they spend much of their time idle, consumption-based cloud APIs can be considerably more cost-effective.

Price and model changes you don’t control

With a closed API, one vendor controls the price, the availability and the behaviour of the model, and can change any of them on its own schedule.

This isn’t hypothetical. Frontier labs revise their pricing models as their products evolve, and those changes don’t wait for your budget cycle. In April 2026, for example, Anthropic moved its Claude enterprise plan from fixed to usage-based pricing; for heavy users, a change like that can shift the bill mid-quarter. DeepSeek scheduled two of its API models for retirement in July 2026. Model lifecycles are measured in months; enterprise hardware lifecycles in years.

The switching cost isn’t the API call. It’s everything tuned to one model’s behaviour:

  • prompts and system instructions
  • evaluation sets and guardrails
  • embeddings in your vector store
  • tool and function definitions
  • the business sign-off on how the output behaves

When a model is retired, all of that gets revalidated, on the vendor’s deadline.

Private inference changes the terms. You hold open-weight model files, so a version runs until you retire it. Nutanix Enterprise AI (NAI) is licensed by GPU memory (or vCPU) capacity, not by tokens, so heavier use doesn’t change the licence bill. And you can test a new model alongside the old one before switching.

Note: API prices have mostly fallen over time. Private infrastructure protects you from increases, but it can also lock you out of decreases.

Where Nutanix Enterprise AI fits

import-model

NAI is a private model-as-a-service layer. It runs open-weight models from NVIDIA and Hugging Face on your own GPU (or CPU) clusters and serves them as secured API endpoints that developers call much as they would a public API. It runs on the Nutanix Kubernetes Platform, on Nutanix infrastructure or on public-cloud Kubernetes.

Pressure What NAI on a private cluster gives you
GDPR transfers and the CLOUD Act Prompts, retrieved documents and outputs stay on infrastructure you own, in the country you choose.
AI Act traceability Model and API monitoring, GPU usage monitoring, event auditing and role-based access, with logs that stay in-house.
AI Act oversight and versioning You pin model versions and decide when they change.
Latency Inference sits next to your data; KV-cache-aware routing and vLLM speculative decoding (2.6) improve response times.
Vendor price and model changes Licensing by GPU memory or vCPU, not tokens; open-weight models you keep as long as you need.
Cost control The AI Gateway (tech preview) adds per-user token rate limits as a financial guardrail.

The AI Gateway matters because it doesn’t force an all-or-nothing choice. It gives one API in front of both private models and external providers, with the same authentication and visibility. Sensitive workloads stay private; others can use a public model, through the same front door.

Caveats: First, sovereignty depends on the whole stack, not one product: where the hardware sits, who operates it, and what telemetry or support access is enabled. Check those settings for strict environments. Second, private inference means open-weight models. For some tasks, frontier closed models are still ahead.

Use cases and industries

The case for private inference is strongest where sensitive data, regulation and steady demand meet.

Industry Example use cases Main driver
Healthcare and diagnostics Summarising lab and clinical reports, coding clinical documents, searching internal procedures Health data is a GDPR special category; AI in medical devices falls under Annex I (2028)
Pharma and life sciences Searching R&D and trial documents, drafting regulatory submissions Patient data plus valuable IP
Financial services Credit and risk analysis support, contract review, KYC document processing Credit scoring is Annex III high-risk; DORA scrutiny of ICT third-party concentration
Public sector Case-file summaries, citizen correspondence, internal knowledge search Sovereignty mandates and citizen data
Manufacturing Maintenance assistants on the plant floor, technical documentation search Edge sites, weak WAN links, design IP
Legal and professional services Searching privileged documents, drafting from precedent Client confidentiality

A useful test for any use case: would you be comfortable explaining to a regulator, or a customer, exactly where its prompts went?

The case for hybrid AI: sovereignty where it matters, cloud where it makes sense

Private inference comes with real trade-offs. GPUs require significant upfront investment, cloud-native AI workloads may need additional development to run on-premises, and the necessary infrastructure and AI skills are still scarce, and an underutilized GPU cluster still carries most of the same capital and operating costs as a busy one.

The practical model is to route by data and by workload:

  1. Sensitive or regulated data (personal, health, financial, IP): private models on your own cluster.
  2. Steady, high-volume workloads: keep them public where significant cloud-native development already exists; consider private where GPU infrastructure is available and predictable demand can justify the dedicated capacity.
  3. Bursty, experimental or frontier-model tasks with non-sensitive data: public API, through the same gateway.

It’s the same pattern infrastructure teams already use for reserved and on-demand cloud capacity. The difference is that the deciding factor is the data first, and cost second.

Views are my own.

tags: [AI, Nutanix, Tokenomics, AgenticAI, AI]

Sources


<
Previous Post
From Tokens to GPUs: Controlling Enterprise AI Costs with Nutanix Agent Gateway
>
Blog Archive
Archive of all previous blog posts