Security Data Lake vs SIEM: Stop Paying to Delete Your Own Evidence
Aug 31, 2026
SIEM ingest pricing forces teams to drop the telemetry they need most. When a security data lake beats a SIEM, when it doesn't, and how the hybrid works.
Most SIEM pricing is metered on ingest: per gigabyte per day, per event per second, or per "workload unit" that works out to the same thing. The invoice tracks how much you collect. So every team running one eventually holds the same meeting: which telemetry do we stop collecting to keep the bill flat?
The answer is usually the high-volume sources — DNS queries, VPC flow logs, endpoint process events, load balancer access logs. Each line is individually boring. Together they are exactly the material you need to reconstruct what an attacker did.
That is the structural problem with metered ingest: you are paying to not collect evidence. And the logs you dropped are reliably the ones the incident needed. Intrusions are routinely discovered weeks or months after initial access, so a 30- or 90-day retention window — set by budget, not by investigative need — means the start of the intrusion has already rolled off by the time you find the end of it.
What is a security data lake?
A security data lake is security telemetry stored in cheap object storage, in open formats, with query compute you attach only when you need it. Each clause carries weight:
- Object storage — S3, GCS, Azure Blob, or MinIO on your own hardware — costs on the order of a couple of cents per GB per month at standard tiers, less in archive tiers. Effective SIEM cost per GB, once licence, compute, and retention are rolled in, commonly runs two to three orders of magnitude higher.
- Open table formats — Parquet files organised into Apache Iceberg or Delta Lake tables — mean the data belongs to you, not to a query engine. DuckDB on a laptop, Trino or Spark on a cluster, and serverless engines like Athena all read the same tables. Switching tools is a config change, not a migration.
- Decoupled compute makes retention and query cost separate decisions. Keeping three years of DNS logs costs storage only. You pay for compute during the hours you are actually investigating, which for most of that data is close to never — until it is the only thing that matters.
- Normalisation is where the "security" part earns its name. Raw syslog dumped into a bucket is a swamp. A usable lake normalises sources into a consistent schema — OCSF is becoming the default — so one query about authentication events spans Okta, Windows, and your VPN without per-source archaeology.
None of this is exotic. It is the lakehouse architecture data engineering settled on years ago, applied to the one dataset where long retention is forensically and often legally load-bearing.
Security data lake vs SIEM: what each is actually good at
The comparison is usually framed as a product bake-off. It is really two different jobs.
| Dimension | SIEM | Security data lake |
|---|---|---|
| Pricing model | Metered ingest (per GB / per event) | Flat cheap storage + on-demand compute |
| Realistic retention | 30–90 days hot; archives awkward to search | Years, searchable, at low flat cost |
| Real-time correlation | Core strength — seconds from event to alert | Batch or micro-batch — minutes, not seconds |
| Detection content | Mature, vendor-maintained rule packs | Bring your own (Sigma, detection-as-code) |
| Compliance dashboards | Built in | You build them |
| Ad-hoc investigation at scale | Slow and expensive across long time ranges | Strong — scan years of Parquet for pennies |
| Data portability | Export is painful by design | Open formats, any engine |
| Operational burden | Low, especially SaaS | Real: pipelines, schemas, table maintenance |
Read the table honestly and a pattern falls out: a SIEM is an alerting product, a lake is evidence infrastructure. The SIEM's job is to tell you something is happening right now and hand an analyst a workflow. The lake's job is to guarantee that when you ask "what did this host talk to in the last eighteen months," the answer exists and costs dollars, not a professional-services engagement against cold archives.
When a plain SIEM is the right call
If your daily telemetry volume is modest, your alerting needs are covered by vendor rule packs, and you have no engineers to own ingestion pipelines, a SaaS SIEM's metered pricing is a fair trade for never operating any of this. The lake's economics only matter once the ingest bill is forcing you to drop data you know you should keep.
The failure mode on the other side is just as real. A lake with no owner degrades into a swamp: schema drift, broken pipelines nobody notices until an incident, and terabytes of unqueryable JSON. Cheap storage of data you cannot query is not a security capability, it is a compliance liability with an S3 bill.
The hybrid pattern: lake as the system of record
Most searches for "SIEM alternatives" end somewhere more boring and more effective than rip-and-replace:
- Everything lands in the lake first. Full-fidelity telemetry, normalised, retained for years. This is the system of record.
- A high-signal subset forwards to the SIEM — authentication events, EDR detections, cloud control-plane changes. The SIEM keeps doing what it is genuinely good at: real-time correlation, case management, compliance reporting. It just does it over a fraction of the raw volume, which is what shrinks the ingest bill.
- Scheduled detections run over the lake. Sigma-style rules compiled to SQL on a 5–15 minute cadence cover the detections that never needed sub-second latency — which, if you audit your rule set honestly, is most of them.
The lake becomes where investigations and hunts happen; the SIEM becomes a real-time alerting layer over curated inputs. Neither is pretending to be the other.
A lake nobody hunts over is just cheap storage
The honest weakness of the hybrid pattern is labour. Years of queryable telemetry only pay off if someone is actually asking it questions, and threat hunting has always been the first activity cut when the SOC is drowning in alert triage.
This is the part we built Helix Protect around. Protect stands up a security data lake as the system of record for your security events, wires into the monitoring and alerting you already run — Prometheus, Grafana, PagerDuty, your existing SIEM — and adds purpose-built detection triggers on top. Then agents do the work that never gets staffed: continuous threat hunting over the full retention window, attack-surface discovery, and investigation of the signals your current stack raises but nobody has time to chase.
Because hunting means handling real attacker artifacts, the agents run open-weight models on infrastructure you control — hostile telemetry never leaves your network, and no third-party guardrail refuses the analysis mid-incident. That failure mode is real and documented; we wrote it up in Autonomous Security Teams.
Keep the SIEM for what it is good at. Put the evidence somewhere you can afford to keep it. Then put something to work over it — see Helix Cyber and the AI SOC use case for how we run that loop.