Ask three detection engineers on the same team to build the same rule. You get three different rules. Nobody is wrong. The process is unwritten, so each person runs their own version of it. One checks for a duplicate rule first, one does not. One baselines the volume over 30 days, one looks at yesterday. One writes down why an exclusion exists, two do not. Three humans who talk to each other can survive this. It stopped being survivable when I added a coding agent, because the agent starts every session from zero. If the process lives in someone's head, the agent invents a fresh process each morning, and every one of them looks reasonable. A coding agent amplifies whatever process you already have. A written process gets faster. An unwritten process gets faster at being inconsistent.
So I wrote the process down and packaged it as a Claude Code plugin. The plugin contains four commands that walk a rule from draft to merge request: /draft-rule, /validate-rule, /test-rule, and /submit-rule. It contains a router skill that selects the correct command from a plain-language request. It contains about twenty markdown files that document the real parser fields for each log source. It contains three Python scripts for the steps that have exactly one correct answer. This post covers what a detection engineering lifecycle should look like in the first place, how the plugin maps onto it, the design decisions that made it usable by my team rather than just me, and the failures that shaped it.
What good detection engineering looks like
Before the tooling, here is the process it serves. The plugin's shape only makes sense against the lifecycle it automates.
A detection rule is a small piece of software. Like software, it has a lifecycle that starts before the first line of query and continues long after deployment. Most of the pain in a detection program comes from treating the query-writing as the whole job. The lifecycle I hold my own work to has eight stages.
1. Intake. Rule ideas come from somewhere specific: an incident where a rule should have fired and did not, a threat intel report that describes a technique you have infrastructure for, a hunt that found behavior worth watching permanently, a red team finding, or a coverage-gap review against ATT&CK. The output of intake is a hypothesis plus a reason. The hypothesis is "behavior X in log source Y is worth alerting on". This stage is cheap. Everything after it is not, so the reason gets written down before anything gets built. A rule whose origin nobody can reconstruct is a rule nobody can confidently tune or retire later.
2. Research. Confirm the telemetry exists, and learn its real shape. What event types does the source emit for this behavior? Which fields are populated, and in what format? Is the country a full name or an ISO code? Is the hostname short or fully qualified? Does the event you expect even reach the SIEM? A rule written against field names from vendor documentation instead of live events is the most reliable way to ship a rule that parses cleanly and never fires.
3. Draft. Write the query that finds the behavior. Wrap it in the metadata the platform and the repository require. This is the only genuinely creative stage in the lifecycle.
4. Validate. Confirm the rule will not break when the platform runs it. Do the brackets balance? Is the tactic in ID form? Is the technique a child of that tactic? Is the lookback zero, or longer than the platform allows? Does the file carry the metadata the repository requires? Each of these has exactly one right answer.
5. Test. Run the query over real history and look at what comes back. There are two halves. The first half is the count. Zero results over 30 days means the rule can never fire. Four thousand results means you have written a report. The second half is reading the actual rows. Who are these users? Is this the same service account 40 times? Most teams do only the first half. The second half is where the bad rules get caught.
6. Review. A second human reads the rule before it goes live. Is the query defensible against the stated intent? Does the severity match the measured volume? Is the MITRE mapping accurate? Is there a runbook for the analyst who receives the alert at 3 a.m.?
7. Deploy. The rule goes live, and its lifecycle state gets recorded. You need to be able to query which rules are live, since when, and what state each one is in. The next stage depends on that record.
8. Operate. Triage outcomes flow back into the rule. False positives become tuning changes. Those changes re-enter the lifecycle at the draft stage and go through every gate again. Volume drift triggers a review. Eventually the telemetry changes shape, or the behavior stops mattering, and the rule gets retired deliberately instead of rotting in place. A rule nobody revisits after deployment decays silently. Parsers change, infrastructure changes, and the rule keeps passing syntax checks while it detects nothing.
None of this is novel. It is roughly what any detection-as-code write-up describes. I spell it out because each stage exists to prevent a specific failure, and every stage that lives only in an engineer's head gets skipped exactly when it matters: under time pressure, after an incident, on a Friday. That inconsistency was tolerable friction with humans. An agent does the mechanical work at ten times the pace, so an unreliably-executed lifecycle produces unreliable rules ten times faster.
The plugin automates stages 2 through 7. Research is folded into drafting: the drafting command must read the live schema documentation before it writes any query text. Review and deploy ride on the merge request. A human reviews, and the merge is the deployment. Intake and operation stay human-driven, but the plugin feeds them. The volume baseline from the test stage is what the operate stage compares against later. That leaves four commands, one per automated gate: draft, validate, test, submit.
Three kinds of file
The whole setup is one Claude Code plugin. Inside it there are three kinds of file. The distinction between them carries most of the design:
| Kind | What it is | When it runs |
|---|---|---|
| Skill | Knowledge and routing, as markdown | Loaded automatically when the task matches its description |
| Command | A named, ordered procedure, as markdown | Only when a person invokes it: /draft-rule |
| Script | A Python helper that does one deterministic thing | Only when a skill or command calls it |
siem-master-plugin/
├── plugin.json
├── commands/
│ ├── draft-rule.md # the four workflow stages,
│ ├── validate-rule.md # one command each
│ ├── test-rule.md
│ ├── submit-rule.md
│ ├── hunt.md # ad-hoc search
│ └── setup.md # credential setup
├── skills/
│ ├── detection-engineering/ # the router: which command, in what order
│ ├── logscale-queries/ # query syntax + ~20 per-source field docs
│ ├── dac-detections/ # read-only questions about the rule repo
└── scripts/
├── siem_query.py # run a search. read-only
├── dac_repo.py # repo mechanics: status, pull, branch, push, validate
└── gitlab_api.py # open a merge request
The plugin is how the process reaches other people. A skill on my laptop helps me. A skill in a plugin my teammates install means everybody resolves the same field names and the same format standard. That has a direct consequence for how the files are written. The drafting command says, in as many words: do not read the field reference from a local clone in somebody's home directory, and do not depend on a personal memory file, because neither exists on a teammate's machine. If a fact matters to the process, it ships in the plugin.
Commands are for procedures. Skills are for knowledge. A command is a sequence with a beginning and an end, and a person chooses to start it. A skill is a body of knowledge that should surface whenever it is relevant, without anybody asking. A procedure in a skill fires when you did not want it. Knowledge in a command is unavailable unless you remember to run it.
The scripts exist because of one test I apply to every step of the lifecycle: does this step have exactly one correct output for a given input? If yes, it becomes a script, and the command's instruction shrinks to a single line: run it. If no, it stays with the model, and the skill documents how to think about it. The tempting shortcut is to write "check the rule has an entity to group by, a severity, and a valid technique ID" into a skill and let the model check by eye. The model gets that right most of the time. "Usually right" is not good enough for a check that runs on every rule the team ships.
Stage one: draft
/draft-rule is the longest of the four commands. Drafting is the stage with the most judgement in it, so it needs the most guidance.
The command starts by checking that the repository clone exists, using dac_repo.py status, because half of what follows depends on files in that repository. Then it collects what it needs from the analyst: the behavior, the log source, the MITRE tactic and technique, the severity, and the expected volume. If any of those are missing, it asks for them in one consolidated question rather than guessing. It refuses to assume a severity above medium without the analyst saying so. Severity determines who gets paged, and paging decisions belong to humans.
Before it writes any query text, it reads three things:
- The field reference for that specific log source, which ships inside the query skill as one markdown file per source. This is the research stage of the lifecycle, made mandatory.
- An existing rule that acts as the canonical template for field order and shape, so new rules look like old rules.
- The house format standard, which is written inline in the command itself.
The third one is a deliberate choice. The format standard is short and mandatory, so the authoritative copy lives in the command rather than in a reference file that could drift out from under it. Every rule's query has to enrich the pivot address with geolocation and ASN, group by one entity, collect the fields the analyst needs in the alert, and build a human-readable description string. Those four requirements make an alert triageable in our environment. An analyst who opens the alert sees who did what from where, without running a second query. The linter checks the same four requirements again in the next stage.
After the query, the command renders the YAML and writes it to the rules directory. It refuses to overwrite an existing file. Then it runs the validator once and fixes what it can before it hands back to the analyst.
Commands are stateless, and the file says so
Early on, two related rules from the same incident got drafted back to back in one session. The second rule inherited the first one's ticket and severity. Both looked correct in isolation. I had to unpick it after the merge request was already open. The fix was a section near the top of /draft-rule that has nothing to do with detections:
This command is stateless across invocations. Even if you just finished drafting another rule in this same session, treat this invocation as a clean start. Do not reuse the ticket from a previous run. Do not carry over severity, tactic, technique, or allow lists from a sibling rule. Re-read the context files rather than relying on memory of what they said last time.
An agent late in a long session behaves like a colleague who is sure they remember, and unlike the colleague, it will not hedge. If a stage must start clean, the file has to say so. It also has to say what specifically must not be carried over.
Stage two: validate
The validator is not in the plugin. It lives in the detection repository, next to the rules, as scripts/validate_rules.py. The plugin only wraps it:
python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dac_repo.py" validate --rule <path>
The wrapper filters the JSON report down to the one rule. It exits 0 on a pass and 3 on violations. Keeping the linter in the rules repository matters more than it sounds. The standard travels with the rules. CI runs the same code the agent runs. A person who has never installed the plugin can still check their work. The plugin is a convenience layer over the standard, not the home of it.
The checks have names, and the names appear in the output. That lets the command act on them programmatically instead of interpreting prose:
$ python3 .../dac_repo.py validate --rule rules/okta_mfa_reset.yml
FAIL rules/okta_mfa_reset.yml
[house_format] filter has no asn() on the pivot address
[lookback_zero] search.lookback is 0; rule will never match a window
[mitre_technique] T1110.003 is not a child of TA0006
3 violations
The command carries two explicit lists. The first list holds the violations the agent fixes automatically and then re-validates: a missing enrichment or grouping call, a zero lookback, a tactic written as a name instead of an ID, a log source key one fuzzy match away from a valid one, and a missing threat-intel confidence threshold on a rule that calls a lookup. Each of those has a single correct fix, so there is nothing to decide.
The second list holds the violations the agent reports and leaves alone. A duplicate rule ID or name is a genuine conflict that needs an author's decision. Maybe the new rule should replace the old one, or maybe it should not exist. A technique that does not map to the stated tactic depends on what the author meant to cover: change the technique, or change the tactic? Unbalanced brackets in the query are ambiguous. It is not clear where the closing bracket belongs, and a wrong guess produces a rule that parses, runs, and is silently incorrect. That is the worst possible outcome for a detection. Everything on this second list goes back to the author with the violation named.
Stage three: test
/test-rule pulls the query out of the YAML and runs it against real history. It uses the same read-only search script the hunting workflow uses:
python3 "${CLAUDE_PLUGIN_ROOT}/scripts/siem_query.py" '<query>' \
--start 7d --limit 500 --json
It reports the total, the per-day rate, the first and last match, and the top groups. Then it compares the rate against the severity the author chose. The thresholds are mine and entirely organization-dependent:
| Severity | Expected rate | Verdict if higher |
|---|---|---|
| Informational | No cap | Always fine |
| Low | Up to 20 per day | Warn |
| Medium | Up to 5 per day | Warn |
| High | Up to 1 per day | Warn |
| Critical | About 2 per week | Warn |
The table converts an argument about severity into an volume check. "This feels like a high" becomes "a high-severity rule that produces 43 alerts a day is either a low-severity rule or a broken query". Nobody has to have that conversation twice. The table also closes the loop with the operate stage of the lifecycle. The volume measured here is the baseline the rule gets judged against after a month in production.
The command then prints three to five sample rows. This is the half of testing that catches problems the count cannot. Forty hits is a plausible count for a real detection. Forty hits that are all the same backup service account is an exclusion waiting to be written. A count cannot tell you which one you have. A person who reads five rows can tell in ten seconds. This is also where the description string from the format standard pays off. If the sample rows are unreadable, the alert will be too.
Zero results gets its own verdict, and the verdict is deliberately suspicious. Either the behavior really is that rare, or the query is broken. The command tells the author to prove the rule fires against a known event before going further: replay one, or find a historical true positive. A rule that has never been seen to fire is a hypothesis, not a detection.
Stage four: submit
Submission carries no security judgement at all. It is pure convention: branch naming, commit message, ticket reference, MR description, and reviewer assignment. That makes it the best candidate for full automation. Before the plugin, it was also the stage where people diverged from each other most.
/submit-rule refuses to do anything until validation passes. There is no reason to open a merge request for a rule that CI will reject. Then it pulls the latest main so the diff is clean. It creates a branch named from the rule and the submitter, so the author is readable from the branch list. It commits the rule file, pushes with a shared token, and opens a merge request with a generated description:
python3 .../dac_repo.py pull
python3 .../dac_repo.py branch <slug> # prints submission/<user>-<slug>-<timestamp>
python3 .../dac_repo.py push <branch>
python3 .../gitlab_api.py open-mr --source <branch> --target main \
--title "submission: <rule name>" --description "<generated>" \
--assignee-username <reviewer>
The description is built from the rule metadata and the volume baseline from the test stage. It ends with the review checklist from the lifecycle's review stage: is the query defensible against the stated intent, does the severity match the measured volume, is the MITRE mapping accurate, and is there a runbook. The reviewer gets everything they need in the MR itself, including the numbers. The review is about judgement, not about re-running the tests.
The command's last action is to print the merge request URL and stop. The agent never merges. A human reads the diff, and the merge is what applies the rule to the platform.
The merge is also what marks the rule as deployed in my lifecycle tracking, not the merge request opening. I had that backwards at first, and the effect was subtle. Every tuning change to a live rule opened an MR. The MR opening re-flagged the rule as in-review. The rule got dragged backwards through its own lifecycle while it was still running in production. State transitions belong on the event that actually changes the world. For a repo-based workflow, that event is the merge.
The router skill
Nothing above forces the four stages to happen in order. Without a router, the agent picks whichever stages look relevant to the request and quietly skips the rest. It skips the duplicate search because the analyst did not mention duplicates. It skips the baseline because the query looked fine.
The router is a skill rather than a command because it has to load itself. The analyst says "this rule is too noisy" and never types a slash command at all. That sentence has to land in the right workflow anyway. The skill's job is to recognize the situation and route:
---
name: detection-engineering
description: >
Use when the user is authoring, tuning, testing, or submitting a correlation
rule. Routes through the detection-as-code commands rather than the vendor UI.
---
Use this skill when the user mentions any of:
"new rule", "detection rule", "tune", "too noisy", "false positive",
"MITRE mapping", "deploy a detection", "push a rule".
The team's workflow is repo-based, not UI-based.
| Command | When to suggest it |
|-----------------|-----------------------------------------------------|
| /draft-rule | The user wants a new detection. Start here. |
| /validate-rule | After drafting, or after any hand edit of a YAML. |
| /test-rule | Baseline the volume before submitting. |
| /submit-rule | Open the MR for review. |
Typical flow: draft → validate → test → submit.
Each command suggests the next one on success.
## When NOT to use this skill
- Investigating existing events → the hunt command
- Inspecting a deployed rule's details → the correlation-rules skill
- Reconstructing activity on a host → the timeline command
Notice that "tune" and "too noisy" route into the same commands as "new rule". A tuning change is a draft-validate-test-submit cycle like any other. It is the operate stage of the lifecycle re-entering at the draft stage. Routing it through the same gates is what stops a quick exclusion from skipping the baseline that would have shown it excluded too much.
Two details in the file do more work than their size suggests. First, each command names the next one at its own end, so the ordering survives even in a session where the router never loads. Second, the "when not to use this" section is as load-bearing as the routing table. A skill with a broad description and no negative cases gets pulled into every adjacent task and gives confident advice about work it was not written for. The description field is the routing hint the agent sees before the full skill loads. Write it as "when to use this", not "what this is". I relearned that lesson every time a vaguely-described skill either failed to trigger or triggered everywhere.
Reference material ships with the skill
The query skill is more than syntax rules. It carries about twenty markdown files, one for each log source we ingest. Each file documents the real parser field names, the real event types, and the format of the values: the difference between the event.action values you assumed and the ones the parser actually emits. This is the research stage of the lifecycle, cached. Every schema fact an engineer would otherwise rediscover by querying the SIEM is written down once, next to the skill that uses it. There are three reasons the files ship in the plugin:
- Everyone resolves the same field name for the same source. A rule I write and a rule a teammate writes refer to the same thing.
- The agent reads the file for the source in front of it, and nothing else. Twenty log source documents in context on every task would be wasteful. One, on demand, is cheap.
- A change to the reference is a change to a file in version control, which somebody reviews. Schema knowledge stops being tribal.
Personal memory files are still useful, and I keep plenty. They are just the wrong home for anything the team relies on. The first version of the drafting command read from a documentation clone that existed only in my home directory. I found out when a teammate installed the plugin on a clean machine. That clean-machine test is the fastest way to find this whole class of problem. I now treat it as part of shipping any plugin change.
These files also rot faster than code. A parser changes, a platform limit moves, or an event type is renamed, and the markdown keeps asserting the old value with total confidence. There is no failing test to flag it. The reference needs an owner and an occasional re-check against live data, the same way the rules themselves do.
Design rules for scripts an agent will call
A script written for a human is not automatically safe for an agent. Humans read error messages charitably, retry with variations, and give up gracefully. An agent does exactly what the output tells it to, forever. A few habits make the difference.
Exit non-zero on failure, with distinct codes. Mine exits 3 specifically for validation violations, so the caller can tell "the rule is bad" apart from "the tool broke". Those two conditions demand opposite responses: fix the rule versus fix the environment. A generic exit 1 collapses them.
Print the raw upstream error. If the platform returns HTTP 400 with a parse position, print that. A script that catches the exception and prints "query failed" has deleted the one piece of information that would have fixed the problem on the next attempt. The agent will burn three retries rediscovering it.
Offer --json. The flag costs ten lines and stops the model from parsing your table layout. The model will otherwise parse the table, usually correctly and occasionally not, and the occasional failures are the expensive ones.
Name the missing thing, and the fix. The query script exits with MISSING_CREDENTIAL, and the command that calls it knows to route the user to the setup command. An error that says what to run next does not need a human to interpret it.
Never prompt for input. An interactive confirmation stops an agent forever. Watch git in particular. It will happily prompt for credentials, an editor, or a merge message, and each one is a hung session.
Read-only by default. The search script cannot create, modify, or delete anything, and its credential is scoped to search. The one script that writes is the one that opens merge requests, and it has its own token. If the agent goes wrong during a hunt or a test, the blast radius is a wasted query.
Write large outputs to disk and return the path. The query script does this automatically. A 40,000-row result should never go into the context window. The agent gets the summary and the path, and reads slices of the file if it needs detail.
Reference scripts from markdown, never restate them. A command that describes what a script does will drift from the script the first time the script changes. The command names the script and its exit codes and stops there.
Ask, do, enforce
A skill asks for the standard and a script performs it. Neither one enforces it, because a model can always decide not to call the script. The third layer has to live outside the conversation.
CI is the reliable version. The same validator runs on every merge request, and a rule that fails cannot merge, whoever or whatever wrote it. Most teams already have the mechanism and only need to point it at the linter. Hooks are the fast version. Some harnesses can run a command whenever the agent writes a file. Kongsgård's team at DNB (mentioned later) uses a post-write hook that validates YAML the instant the agent saves it. That turns a CI failure twenty minutes later into a correction two seconds later. A standard that exists only in the markdown is a request. The enforcement layer makes it a property of the system.
The gap none of this closes
Everything above proves the query parses, returns sensible rows from historical data, and arrives in the repository in the right shape. None of it proves the detection fires when somebody actually performs the behavior. Those are different claims. A rule can pass every stage here and still fail in production. The lab hostname does not match a pattern in the rule. The data source it references was retired last year. A summarization job lags behind the search window.
Kongsgård's post on end-to-end detection validation is the natural next step. It is the same shape as everything here, run one stage further. The agent reads a detection and writes a minimal attack simulation in TTPForge YAML with a defanged payload. It detonates the simulation on an isolated lab host over SSH. Per-step checks confirm the technique actually worked. Then it polls the SIEM, first for the raw telemetry and then for the detection match. On failure, a diagnose-only agent classifies which stage broke: silent host, missing event, late arrival, or rule rejection. In lifecycle terms, this extends the test stage from "the query returns sensible history" to "the whole pipeline turns the behavior into an alert". That second claim is the one the test stage was always approximating.
Building this in your environment
Do it in this order. Each step is useful on its own before you begin the next one, so there is no point where you have invested a month and have nothing.
- Write down your lifecycle. It may not be my eight stages, and your automated segment may not be my four. Open a file and describe the last rule you built, step by step, including the steps you skipped. That file is your first command. Its wrongness is now visible and fixable, which it never was in your head.
- Turn the deterministic steps into scripts. Start with the linter. Put it in the repository with the rules, not in the plugin, so CI and the agent run the same code.
- Make the scripts agent-safe. Distinct exit codes, raw errors,
--json, no prompts, credentials from the OS keyring. This is an afternoon of work that removes a whole class of confusion. - Add read-only SIEM access. Use a scoped credential with search permission and nothing else. This unlocks the research and test stages, which are where the agent saves the most time.
- Write one command per stage, and have each one name the next. This is what keeps the chain intact in a session where the router never loads.
- Add the router skill. Give it trigger phrases, a table of commands, and a "when not to use this" section.
- Package it. The moment a second person needs the process, everything it depends on has to ship together. Test on a clean machine. Mine failed the first time.
- Add enforcement. Put the linter in CI first. Add a write hook after that, if your harness has them.
Read the list again and there is very little about artificial intelligence in it. A written lifecycle. A linter that lives with the rules. Three small scripts with honest exit codes. Four written procedures. A packaging step so the team gets the same copy. Enforcement in CI. Every one of those is something a detection team should have built before anyone shipped a coding assistant. Most teams, mine included, had not.
What changed is the return on the work. Documentation that a human reads twice a year is hard to justify. Documentation that is read on every single task, by something that follows it exactly, pays for itself in a week.
