Getting SecDataOps going is less grand than it sounds. What it takes is ten systems, a spreadsheet and roughly a quarter of steady, unexciting effort. This guide is the version for people who would rather follow numbered steps than read theory. It assumes you run or support a company’s IT and want the logs your security people rely on to be present, trustworthy and affordable.
The steps follow the outline in the Ops Atlas page on SecDataOps, which describes a discipline for engineering and governing security telemetry. The Atlas also says plainly that the label is new and vendor-led, so treat the steps below as a practical reading of its advice, not a certified method.
Step 1: Choose the ten feeds you cannot afford to lose
Do not begin with a tool. Begin with a list. Jot down the data feeds you would hate to find blank once an incident starts. Typical candidates: your identity provider, email platform, firewall, DNS resolvers, cloud audit trail, key servers, endpoint agents, VPN and the database holding customer data. Keep it to ten. The whole plan depends on finishing a small scope properly rather than half-covering fifty sources.
Step 2: Score each source from zero to five
Create a register with one row per source. Score each on five dimensions: those five are device coverage, field coverage, timeliness, consistency and retention, or in plain terms: do all machines report, are the needed fields there, does data show up promptly, is the format stable, and is history kept long enough. The Atlas borrows a zero-to-five scale from a well-known detection-coverage method. Be honest. A row of fours that nobody has ever tested is worth less than a row of twos you have measured.
Step 3: Write a short contract for each source
One page, same template every time. Include:
- the named owner;
- the schema or format the source follows;
- the fields that must always be present;
- the freshness target;
- which storage tier it goes to and how long it stays;
- a canary event that proves it is alive;
- the monthly cost;
- gaps you already know about.
For the numbers, borrow the Atlas’s example and adjust it: freshness of about 300 seconds at the 95th percentile, a parse rate near 0.995, completeness around 0.98, and a pager alert once 900 seconds pass without data. If you cannot meet a target today, write the target you can meet and a date to tighten it.
Step 4: Add a health monitor that notices silence
An empty feed rarely throws an error. Build a monitor that counts events per source per interval and compares them with the contract. Alert when the count drops to zero or falls out of its usual band. The Atlas warns about averages: a healthy overall total can hide the one stream that stopped, so watch each source separately.
Step 5: Treat parsers like software
Store parsing rules in version control. For each source keep a small set of real, scrubbed sample records as fixtures, and add contract tests to your build so a change that breaks field extraction fails before reaching production. Two traps from the Atlas are worth testing explicitly. First, the Vector pipeline tool coerces a null integer to zero, so junk can look like traffic denied on port zero; an anchored pattern with a quarantine route avoids that. Second, its default memory buffers will not survive a crash, so choose disk buffers where loss matters. And remember plain syslog cannot acknowledge delivery, so deduplicate on a stable event identifier or replays will double-count.
Step 6: Keep raw, normalise forward
Choose a common schema before the data fans out. OCSF is one open option, and the Atlas mentions ECS and a few vendor models as alternatives. Whatever you choose, store the raw record in the cheapest tier and produce the normalised copy from it. Parsers make mistakes and schemas deprecate fields between releases, so deleting raw data after normalising removes your ability to fix errors later. Record event time and processing time separately, in UTC.
Step 7: Price every stream
Add a cost column to the register. Then look at your three largest streams. The Atlas shows the sort of sum worth doing. Firewall flows of 400 GB daily, all hot, run to about 400 cost units a day; the same data as 8 GB of hot summaries with raw parked in a 400-day lake comes to roughly 164, and the raw record is retained far longer than before. One caution: some cloud tiers have inclusion periods, such as 31 days in one analytics service, so shortening retention below that saves nothing.
Step 8: Cut streams with a playbook, not a hunch
When you reduce a stream, follow a fixed order.
- Name every detection rule that leans on the stream.
- Sort its fields into needed, useful and noise.
- Pick one lever: trim fields, summarise, route elsewhere or sample.
- Replay a week of raw data through the changed pipeline.
- Confirm every dependent rule still fires.
Skip step 4 and you will learn about the damage during an incident. That is the whole reason for it.
Step 9: Run your first silence test
Pick a lab or low-risk source. Stop it on purpose. Start a timer. See how long until someone is alerted, then compare the result with the contract. You will probably find a gap, and that gap is the most valuable output of the exercise. Repeat monthly on a different source.
Step 10: Set retention on purpose
The Atlas calls a blind 90-day default an anti-pattern, pointing to an annual casework review published by a breach-response consultancy, covering 2025, where attackers typically lingered about two weeks before discovery yet stayed far longer in espionage-style cases, with one campaign close to 400 days. Public guidance varies: a US federal memo asks for 12 months active plus 18 months cold, Australian guidance suggests a year for administrative logs as a floor, and India’s 2022 CERT-In directions say 180 days. Read the rules that apply to you, then write down your own number and the reason for it.
Common mistakes to avoid
The Atlas keeps a list of anti-patterns, and three deserve a warning here. The first is the habit of ingesting everything and deciding later; Australian-led guidance it cites says over-collection makes effective rules harder to write. The second is assuming the SIEM licence also covers log management, when a SIEM only searches what reaches it. The third is forcing a single schema on every consumer, which tends to satisfy nobody. Another small trap sits in Windows event forwarding, where the rendered text format roughly doubles each event’s size for no analytical gain.
Cloud audit services have their own fine print. On the largest cloud, logging of object-level storage activity must be switched on and is charged on its own. Snowflake’s ACCESS_HISTORY can lag by up to 180 minutes, needs the Enterprise edition and omits failed queries, so QUERY_HISTORY with its shorter 45-minute lag fills that hole. Put these notes in the contract’s known-gaps field and nobody gets surprised.
A calendar you can copy
Days 1 to 30: register, scores, retention policy, health monitors on the top ten. Days 31 to 60: contracts, fixtures, build checks and canaries for those same ten, then a price on every stream. Days 61 to 90: move the three biggest streams to cheaper tiers with replay checks, hold the first silence drill and issue a brief baseline report. The Atlas frames its own 30-60-90 outline the same way, and it suits a small team.
Where the money goes and what to tell leadership
Expect resistance on two fronts. Finance will ask why logging costs so much, and security will ask why you want to drop anything. Your register answers both, because it shows each stream’s cost beside what depends on it. For a wider view of budgeting your digital programme, see this roadmap of digital services for new clients, and for the money side, the walkthrough on catching spending leaks and extending runway.
Finish the quarter by writing a one-page summary: sources covered, scores before and after, money saved, gaps still open. Then do it again for the next ten.
