Running your own AI models: What it really costs to get everything up and running

"We'll just host the model ourselves" — this sentence is quickly uttered in many bank strategy discussions when it comes to data protection and sovereignty. What it actually costs and how long the path to production operation takes often remains surprisingly vague. Yet, reliable market data is now available: for hardware, cloud rentals, personnel, electricity, and project timelines. An inventory with figures, as of August 2026.

The Shopping List: Hardware

The basic building block of productive AI infrastructure is the 8-GPU server. An HGX system with eight NVIDIA H100s costs between $250,000 and $320,000 according to a market overview by Mercatus; NVIDIA's complete DGX systems range from $350,000 to over $500,000, while the newer B200 generation typically costs around $450,000. With ancillary costs, it quickly adds up: the AI Infrastructure Cost Guide by Haink calculates total hardware costs of around $471,000 for an entry-level cluster consisting of an 8× H100 server, storage, switch, and rack. Added to this are delivery times: 36 to 52 weeks are cited for certain HGX configurations, which is why purchase scenarios should plan for three to six months of infrastructure lead time.

The model determines how much hardware is needed. A 70B model requires around 140 GB of VRAM in full precision — i.e., two H100s or one H200 —, and only 36 to 38 GB in 4-bit quantization. The largest open models of the 600B class fit quantized onto a single 8× H100 server. The alternative to purchasing is renting: cloud GPU prices for the H100 range from $1.38 to $12.29 per GPU hour (median around $2.29) according to the IntuitionLabs price index — specialized providers are several times cheaper than hyperscalers. A rented 8× GPU node running 24/7 costs about $35,000 to $50,000 per month depending on the provider.

The Underestimated Item: People

Hardware is only half the bill. An independent analysis estimates the share of chips and personnel combined to be 70 to 80 percent of total deployment costs. Running a medium-sized inference cluster is estimated to require 0.5 to 1 MLOps engineer — no small feat with German salaries: machine learning engineers in this country earn a median of around €65,000 to €75,000 gross, with senior profiles with cloud architecture experience earning over €100,000. Larger organizations that supply multiple product teams with AI features typically build dedicated platform teams of 8 to 15 people. Those who cannot fill the operational responsibility do not have a cost problem, but a feasibility problem.

Electricity: A Factor of Its Own in Germany

An 8× H100 system draws around 10.2 kilowatts under load — the GPUs themselves account for only just over half of this, with the rest going to CPU, network, power supplies, and cooling. At the German industrial electricity price of around 16 cents per kilowatt-hour (BDEW electricity price analysis, January 2026), a continuously running system incurs pure electricity costs of about €14,300 per year; including data center overhead (PUE around 1.4), it is around €20,000 per year — derived as a calculation from consumption and electricity price. Important for the industry: the subsidized industrial electricity price of around 5 cents starting in 2026 only applies to particularly energy-intensive sectors — banks and financial service providers do not regularly fall under this. Additionally, there are structural requirements: H100 racks require 15 to 32 kilowatts of connected load, significantly more than classic colocation racks provide; the Blackwell generation requires liquid cooling.

Time to Production: Months, Not Weeks

Proof of concept is quickly achieved — getting a model running with vLLM or TGI takes one to four weeks with experienced teams. Production readiness is something else. The operator checklist from Allganize estimates 8 to 16 weeks for an experienced platform team to reach the first production-ready cluster — including monitoring from day one, SSO in front of the inference API, audit logging, canary deployments with automatic rollback, and eval gates against quality regressions. Teams that cut corners in this process spend the same amount of time afterwards in incident response, according to the same analysis. Across multiple sources, the consensus for full production readiness is two to six months, depending on compliance requirements and team experience — in regulated institutions, it is rather at the upper end.

When the Math Flips

Is it worth it compared to the API? The cost asymmetry is substantial at high utilization: self-hosted inference on modern hardware costs $0.02 to $0.11 per million tokens according to SemiAnalysis benchmarks, while commercial APIs for comparable classes charge $3 to $30. The catch: these values only apply to high, continuous utilization. An example calculation for a 70B model on four A100 GPUs results in a total TCO of $7,300 to $12,300 per month with purchased hardware (depreciation, personnel, electricity) compared to $16,500 to $24,000 with rented cloud GPUs. Rules of thumb have been established: under 10,000 queries per day, cloud APIs are clearly more economical; above 100,000 queries per day, self-hosting usually wins; 60 to 80 percent of continuous GPU utilization is considered the utilization threshold. The majority of use cases in medium-sized institutions simply lie below these thresholds.

As of August 2026; all sources are linked in the text.

The Consequence: Build in Agility

The realistic overall calculation for a serious in-house operation — hardware starting in the several hundreds of thousands of dollars or five-figure monthly rentals, one to several FTEs of operational personnel, around €20,000 in electricity per server and year, two to six months to production readiness — is not an argument against self-hosting. It is an argument against premature commitments in either direction. Because the parameters of this calculation are constantly changing: GPU prices are falling, API prices are falling faster, open models are becoming more efficient, and your own query volume is growing.

This is precisely why we built our Assistant at Acceleraid to be model-agnostic: the underlying model can be changed at any time — from cloud API to self-hosted open model when volume and compliance requirements justify it, or back if the math changes. Knowledge, contexts, and configurations are fully preserved in the process. The location decision for the model thus becomes an economic optimization rather than a one-way street.

Anyone preparing this decision concretely should know three figures: their own daily query volume across all AI use cases, the operational roles that can realistically be filled internally — and the price that the current API provider is likely to charge in twelve months. Only when these three figures are on the table can the question "Buy, rent, or API?" be answered seriously.

Illustration: AI-generated. AI-supported content: We use AI technologies and automated agents, including those from Microsoft, Google, OpenAI, Anthropic, and other providers, to create our posts. Topics, professional orientation, and final approval remain with our team.