Running Local LLMs for Security Work: What Self-Hosted Inference Actually Buys You
Self-hosted LLM inference — running models locally on your own hardware rather than calling a hosted API — is often pitched on cost or framed as a privacy-enthusiast hobby. For security work in a regulated context, both framings miss the actual value. The real win is a control boundary: data that provably never leaves your infrastructure, and a model whose behaviour and availability you fully own. That is not a hobby; it is a defensible architectural position with specific, real advantages — and specific, real costs that the enthusiast framing tends to skip.
This is an honest look at where local models win for security work, and where they do not.
The control-boundary argument
The strongest case for local inference in security work is not cost and not even privacy in the abstract — it is the control boundary. When you run a model locally, you can state with certainty where the data went: nowhere. It was processed on infrastructure you control, and it never crossed a boundary into someone else’s system.
For security work this matters acutely, because the data security work touches is often the most sensitive data there is:
- Incident data — details of an active or past breach, which you absolutely do not want traversing third-party infrastructure.
- Log and detection content — which reveals your environment, your coverage, and your gaps.
- Vulnerability and assessment findings — a map of where you are weak.
- Client data, in a consulting context, often under contractual or regulatory residency constraints.
Sending any of that to a hosted inference API means it crossed a boundary, and now you must reason about the provider’s handling, retention, and jurisdiction. Running it locally means the question does not arise — the data never left. For a regulated financial entity under DORA’s third-party and data-residency pressures, “the data never crossed our boundary” is a far cleaner answer than “we have a data-processing agreement with the provider.” The control boundary is the product.
What else local inference genuinely buys
Beyond the control boundary, real advantages for security work specifically:
Behavioural ownership. A local model’s behaviour is yours — you control the version, the configuration, the system prompts, and you are not subject to a provider changing the model underneath you. For workflows you depend on, that stability and control is valuable. A hosted model can change behaviour between calls; a local one changes when you change it.
Availability independence. A local model does not depend on a provider’s uptime, rate limits, or continued offering of a given model. For security tooling that needs to run regardless of external service status — including potentially during an incident when you may not want external dependencies in the loop — local inference removes a dependency. There is something architecturally clean about incident-response tooling that has no external API in its critical path.
No per-call cost ceiling on volume. Once the hardware is paid for, high-volume processing — triaging large log sets, bulk analysis — does not meter per token. For sustained high-volume security workloads, the economics can favour local after the capital cost, though this is genuinely workload-dependent and not the main argument.
Experimentation freedom. For research and tooling development, a local model you can probe, configure, and run unlimited times without metering or external logging is a better laboratory. You can iterate without watching a meter or sending experimental prompts off-premises.
Where local models honestly lose
A credible assessment names the costs, and they are real:
Capability gap. The frontier hosted models are, generally, more capable than what you can run locally on reasonable hardware. For tasks needing maximum capability, local models on accessible hardware may not match a hosted frontier model. You are trading some capability for the control boundary, and for some tasks that trade is wrong — the local model simply cannot do the harder work as well.
Operational burden. You own the infrastructure: the hardware, the deployment, the updates, the availability. That is real work and real cost that a hosted API amortises across a provider. The control boundary is paid for in operational responsibility, and if you cannot carry that responsibility, the boundary is illusory.
Capital cost and obsolescence. Capable local inference needs capable hardware, which is a meaningful upfront cost — and the hardware ages as models and requirements advance. Hosted inference converts that into operating expense and offloads the obsolescence. The local capital bet only pays off at sufficient sustained use.
You own the safety properties too. A hosted model comes with the provider’s safety layers. A local model’s behaviour — including its failure modes — is entirely yours to manage. For security tooling this cuts both ways: more control, but also more responsibility for ensuring the model behaves appropriately in your workflows.
The decision framework
Rather than “local good, hosted bad,” the actual decision is a routing question per workload:
- Does this data have a control-boundary or residency requirement? If yes, that pushes hard toward local regardless of other factors — the boundary is the point. Incident data, sensitive client data, anything under residency constraint.
- Does this task need frontier capability? If yes, and the local model cannot match it, that pushes toward hosted — accepting the boundary cost — unless the data sensitivity forbids it. When both are true (frontier-need and residency-required), that tension is a real architectural problem with no free answer.
- Is this high-volume, sustained, or availability-critical? Those favour local on economics and independence grounds.
- Can I actually carry the operational burden? If not, the local control boundary is theoretical — a badly-run local deployment is worse than a well-governed hosted one.
The mature posture is usually both: local inference as the default for sensitive security data where the control boundary matters and the capability is sufficient, hosted for tasks that genuinely need frontier capability on data that permits it. Route per workload, not per ideology.
The takeaway
Self-hosted LLM inference for security work is not primarily about cost or privacy-as-hobby. It is about a control boundary — provable data containment and behavioural ownership — that is genuinely valuable when the data is incident details, detection content, vulnerability findings, or client data under residency constraints. That boundary is a defensible architectural position, especially under EU financial regulation.
But it is paid for: in capability gap, operational burden, capital cost, and ownership of the model’s safety properties. The reframe to carry: local inference buys you a control boundary, and you pay for it in capability and operational responsibility — so route each workload by whether it needs the boundary more than it needs the frontier. For sensitive security data, the boundary usually wins. For the hardest tasks on shareable data, it usually does not. Decide per workload, and own the tradeoff honestly.
An independent piece by johlem.net — IT security consulting, Luxembourg. Self-hosted AI and data-sovereign infrastructure for regulated work.