This post describes the detection agent we run in our SOC. Given a detection rule, it gathers the rule's alert history, decides whether the rule should be tuned, writes the change, validates and backtests it, and opens a merge request. A detection engineer reviews each MR and merges it or closes it. The agent can't merge or deploy anything itself. I'll cover the architecture, the components with some alternatives to each, enough code to build a minimal version, and the problems we hit in production. Product names are mostly left out; the design doesn't depend on any particular vendor. ## Prerequisites Before the agent existed, our detections were already stored in Git as YAML. Changes went through merge requests, CI validated them and deployed them to the SIEM, and every alert became a case in our case management system with a closure disposition. Tuning was already partly automated. An engineer would work through a noisy rule with an AI coding assistant, using written instructions on how we tune and a commit history with several hundred prior tuning changes as examples. The agent automates a process that was already documented. If your tuning decisions aren't written down anywhere, an agent has nothing to learn them from, and that's the first thing to fix. ## Why in-house Tuning a rule well needs the rule's source and change history, how analysts closed its alerts and what they wrote, what normal activity looks like in your own telemetry, and the reasons behind existing exclusions. Most of that is internal. Commercial products can get some of it through integrations, but they won't know why a given exclusion exists. Building it ourselves also meant the agent's output goes through the same merge request and CI process as every other detection change, so there's no separate path to production to secure. ## Overview ```mermaid flowchart TD A[Analyst requests a tune] --> D[Detection agent] B[Nomination from alert triage] --> D C[Script or API call] --> D D --> F[Merge request] F --> G{Human review} G -->|merge| H[CI deploys to SIEM] ``` Requests arrive from three places: - An analyst clicks "request tuning" on a case, picks the rule and adds a sentence explaining why. The SOAR platform posts that to the agent. - Our alert triage automation, which is a separate system, nominates a rule when it keeps closing that rule's alerts as benign. Each nomination carries a category such as service account, expected admin activity, scanner or broad logic. - Engineers and tooling can post a request directly. All three produce the same request format, and the agent treats them identically after validation. ## Components Every component here is a commodity service. The right-hand column lists alternatives. | Role | What it does | Options | |---|---|---| | Intake | Sends tune requests from the case page to the agent | Tines, Splunk SOAR, Cortex XSOAR, Torq, Shuffle, n8n, or a webhook from your case manager | | Front door | Validates the request, enqueues it, returns immediately | A serverless function (Cloud Functions, AWS Lambda, Azure Functions) or a small FastAPI/Flask service | | Queue | Serialises requests, retries on infrastructure failure | Google Cloud Tasks, AWS SQS, Azure Service Bus, Pub/Sub, Celery or RQ on Redis | | Worker | Runs the pipeline | Cloud Run, ECS/Fargate, Azure Container Apps, Kubernetes jobs | | Model | Tool-use LLM | Any frontier model with tool calling, through the vendor API or a cloud provider's managed endpoint (Bedrock, Vertex AI, Azure) | | State | Rate-limit markers, run records | Object storage (S3, GCS, Blob), DynamoDB, Redis | | Change control | Branches, MRs, CI | GitLab, GitHub, Bitbucket | Some specifics on how ours is configured: The front door does no work beyond validation. Most SOAR platforms time out on slow webhooks, so it enqueues and returns in milliseconds. The queue runs one request at a time. A tuning run takes several minutes and runs many SIEM queries, and serialising them keeps load on the SIEM and model spend predictable. The queue calls the worker with a signed identity token, so the worker isn't publicly reachable. The worker runs under its own service identity, which is the only identity with access to the Git token. The reasons are covered under the Rule of Two below. The model is called through the cloud provider's managed endpoint and authenticated with the worker's identity, so there's no model API key to store. ## The pipeline The agent can change detections, so most of its steps are deterministic code. The model is used for two decisions. ```mermaid flowchart TD A["Validate the request<br/>(code)"] --> B["Gather the evidence<br/>(code)"] B --> C["Tune it or not?<br/>(model)"] C --> D["Write the change<br/>(model)"] D --> E["Validate and backtest<br/>(code)"] E --> F["Open the MR<br/>(code)"] ``` ### Request validation (code) A request identifies a rule by repo path, SIEM rule ID or exact name, and can include a note from the requester. The rule is resolved against an index built from the main branch, with no fuzzy matching and no paths outside the rules directory. The requester must have a company email address. The note is limited to 4,000 characters, with control and zero-width characters removed. Invalid requests get a 400 so the queue doesn't retry them. ### Evidence (code) Before the model is involved, code collects: - the rule YAML from the main branch; - 30 days of closed cases for the rule, each classified from its closure status as true positive, benign or false positive, with titles, summaries and analyst comments; - how many of those closures happened after the rule's most recent substantive change; - a baseline backtest of current alert volume. Large 30-day counts can time out or return truncated results, so the backtest first runs a timed one-day probe and uses its duration to pick a 30-, 7- or 1-day window. It then runs a count and rejects any result the SIEM marks as truncated. Time-window conditions that are relative to the rule's own execution are removed before counting. ### Decision (model) The model receives the rule, the evidence and the requester's note, and calls `submit_triage` with one of three outcomes: tune, no tune needed, or not tunable. It can run read-only SIEM queries during this step and has no other tools. Requester notes and case text are wrapped in `<untrusted>` tags, and the policy prompt explains how to treat them. ### Authoring (model) For a tune, the model calls `submit_tune` with structured fields rather than a diff. Trimmed: ```json { "type": "object", "properties": { "filter": {"type": "string", "description": "The complete new query filter"}, "lookback": {"type": "string", "description": "Optional, e.g. 1h"}, "schedule": {"type": "string", "description": "Optional"}, "severity": {"type": "string", "enum": ["low", "medium", "high", "critical"]}, "tactic": {"type": "string", "description": "ATT&CK tactic ID, e.g. TA0006"}, "technique": {"type": "string", "description": "ATT&CK technique ID, e.g. T1110.003"}, "note": {"type": "string", "description": "Appended to the rule's notes: what changed and why"}, "rationale": {"type": "string", "description": "Goes in the MR description"} }, "required": ["filter", "note", "rationale"] } ``` Each field is checked in code against an allowlist, format rules and length limits. ### Edit, gates and MR (code) Code applies the fields to the YAML and confirms the result parses back to the intended rule. The gates described below then run. If a hard gate fails, the failure is returned to the model with two more attempts, plus one extra attempt specifically for exceeding the volume cap. If it still fails, the outcome is "not tunable". Otherwise code opens the MR. ## Prompt and tools The system prompt is built from separate markdown files: - `environment.md` describes the organisation, its estate and what normal activity looks like. - `query_essentials.md` covers the SIEM's query language and the mistakes models commonly make in it. - `tuning_policy.md` describes how we tune: prefer narrow exclusions on stable identifiers, never exclude on a value that appears in a true positive, explain the change in the rule's notes, and so on. The model's tools are deliberately few: run a read-only SIEM query, fetch our documentation for a log source, and the two submit tools. The log-source documentation tool reduced bad queries more than any prompt change. Models know the query language in general but not your field names or parsers. Query results pass through a shaping step that caps their size and drops fields the model doesn't use, which keeps each turn smaller and cheaper. ## The loop Both model steps use the same loop: run tools until the model calls the step's submit tool, validate the submission in code, and return any problem as a tool error in the same conversation. A stripped-down version using the Anthropic Python SDK follows; the structure is the same with other tool-use APIs. ```python import json from anthropic import Anthropic # AnthropicBedrock / AnthropicVertex are drop-in alternatives client = Anthropic() MODEL = "your-model-id" CACHE = {"type": "ephemeral"} def with_cache(messages): # Mark the newest block so each turn reads earlier turns from cache. last = messages[-1] blocks = list(last["content"]) blocks[-1] = {**blocks[-1], "cache_control": CACHE} return messages[:-1] + [{**last, "content": blocks}] def run_step(prompt, system_prompt, tools, handlers, submit_tool, check, max_turns=25, max_fixes=2): messages = [{"role": "user", "content": [{"type": "text", "text": prompt}]}] fixes = 0 for _ in range(max_turns): resp = client.messages.create( model=MODEL, max_tokens=8000, system=[{"type": "text", "text": system_prompt, "cache_control": CACHE}], tools=tools, messages=with_cache(messages), ) # Return the assistant turn unchanged. messages.append({"role": "assistant", "content": resp.content}) calls = [b for b in resp.content if b.type == "tool_use"] if not calls: messages.append({"role": "user", "content": [ {"type": "text", "text": f"Call {submit_tool} to finish."}]}) continue results = [] for call in calls: if call.name == submit_tool: problem = check(call.input) # None means valid if problem is None: return call.input fixes += 1 if fixes > max_fixes: return None results.append({"type": "tool_result", "tool_use_id": call.id, "content": problem + " Resubmit the complete input.", "is_error": True}) else: out = handlers[call.name](**call.input) results.append({"type": "tool_result", "tool_use_id": call.id, "content": json.dumps(out, default=str)[:20000]}) messages.append({"role": "user", "content": results}) return None # no valid submission: report "failed" and leave it for a person ``` A tool definition, simplified: ```python tools = [ { "name": "run_siem_query", "description": "Run a read-only query against the SIEM. Read the log " "source docs first if you're unsure of field names.", "input_schema": { "type": "object", "properties": { "query": {"type": "string"}, "lookback": {"type": "string", "description": "e.g. 1h, 24h, 7d"}, }, "required": ["query"], }, }, {"name": "submit_tune", "description": "Submit the rule change.", "input_schema": SUBMIT_TUNE_SCHEMA}, ] ``` Notes on the loop: - There are two cache breakpoints, on the system prompt and on the newest message. Earlier turns are never edited, so the cached prefix stays identical between calls. A tuning run makes many model turns, and caching roughly halves the cost. - `resp.content` goes back unchanged. Models that return reasoning blocks require this. - A bad submission is corrected in the same conversation. Raising an exception instead would cause the queue to retry the whole run from the start. - If the step can't produce a valid submission, the run ends as "failed" and is left for a person, not retried. ### One model or two The agent writes many SIEM queries, so we tested delegating query writing to a smaller, cheaper model. Across four rules in a live comparison, the large model alone cost $1.88 and took 20 minutes. The large model with a smaller query worker cost $3.12 and took 35 minutes. Verdicts were identical. The handoffs between models cost more than the cheaper tokens saved, so delegation is disabled by default. ## Merge request controls The agent's changes go through the same merge request review and CI as human changes. We added several controls around that. The worker holds a Git token that technically has merge permission. The Git client the agent uses exposes five methods: ```python class GitWriter: def create_branch(self, name): ... # only under tune/agent-* def commit_rule(self, branch, path, content, message): ... # one rules/*.yml file def create_mr(self, branch, title, body, labels): ... def comment_mr(self, mr_iid, body): ... def list_open_agent_mrs(self): ... ``` There is no merge method, and a unit test fails if one is added. The MR body escapes HTML, quick-action commands (lines beginning with `/`) and `@mentions` in any model or requester text. The diff is also checked against the rendered YAML. A tune may change the query filter, lookback, trigger mode, schedule, severity, status, ATT&CK tactic and technique, and review dates, and may append to the notes. Changes to any other field fail the check. ### Rule of Two and service separation We used Meta's [Rule of Two](https://ai.meta.com/blog/practical-ai-agent-security/) when deciding where credentials live. It says an agent shouldn't simultaneously process untrusted input, have access to sensitive systems, and be able to change state. Our alert triage automation processes raw alerts, which contain attacker-influenced content. The detection agent runs as a separate service under its own identity, and that identity is the only one with access to the Git token secret, granted on that single secret rather than project-wide. The triage side can't read the token. The detection agent never receives raw alert payloads, only case history, closure comments and the requester's note, and that text is still treated as untrusted. ### Gates and flags Hard gates stop the pipeline after the repair attempts above: - the edit applies and parses back correctly; - the diff only touches allowed fields; - the CI validator reports no new findings compared with the original rule; - the new query runs against an hour of live data without errors. Flags don't block. They're listed at the top of the MR and add a `needs-decision` label: - `excludes_tp_entity`: a value the change excludes appeared in a true-positive case. - `possible_injection`: a newly excluded value appears in the requester's note or case text but is supported by fewer than two benign closures. - `regression_to_zero`: the rule would stop firing entirely. - `severity_raise`, `allowlist_edit` and `weak_evidence`. `possible_injection` is computed from the evidence, not by asking a model whether text looks like an injection. If a case comment asks for user X to be excluded and the closure history doesn't support it, the MR is flagged. ## Validation Validation covers four separate checks. The agent runs a vendored copy of the validator used by the detection repo's CI, so a pass means the same thing in both places. The copy drifted from CI once; there's now a script that refreshes it. The new query must run against an hour of real data without errors. A timeout is flagged rather than treated as a failure. The backtest compares alert counts before and after using the windowed counting described above. A drop to zero is flagged. When the agent proposes a new rule rather than a tune, it also writes a `control_filter`: the same search with the conditions that identify malicious activity removed. If the control returns events and the rule returns a few, the data exists and the rule is selective. If the control returns nothing, the log source isn't being collected and the rule can't fire. The model must build the control by deleting clauses from the rule. Without that constraint it sometimes wrote an unrelated query that happened to return results. Code rejects a control identical to the filter or containing clauses the filter doesn't have, and the model gets one attempt to correct it. ## Lessons from production ### Shadow mode for nominations Nominations from alert triage were first enabled in shadow mode, controlled by an environment variable with three values (off, shadow, on). In shadow mode the nomination is logged but not sent. Of the first 114 shadow nominations, 90 were for a single rule owned by a third-party cloud security product. That rule isn't in our repo, so the agent couldn't act on it. Restricting nominations to SIEM correlation rules fixed it. Nominations are rate-limited to one per rule per day, or up to three if the categories differ. The limit is tracked with a JSON marker per rule in object storage, written with a conditional write so two concurrent writers can't both send. ### Split history at the last change In early testing, six of the ten rules with the most benign closures over 30 days had already been tuned in the previous week and had produced no benign alerts since. The evidence step now reports how many closures happened after the rule's last substantive change, ignoring commits from our sync bot. ### Intentional noise Some identity rules have allowlists that look like tuning targets. In one case the excluded accounts belong to known infostealer victims, and the rule is meant to alert when one of their logins succeeds. An agent optimising for lower volume would remove that. We documented the reasoning in the rule's notes and in the tuning policy prompt, and any change to an allowlist is flagged. ### Analyst requests are instructions Early versions argued with analysts' requests and frequently returned "no tune needed" when an analyst had asked for a specific change. The policy now treats an explicit analyst request as the decision. The agent declines only when the change is impossible or would increase alert volume, and otherwise writes the narrowest change that does what was asked, with flags if the evidence disagrees. ### Output length limits An early rejection of a proposed detection was stored truncated at 1,000 characters, cutting off at the point where it named the missing telemetry, which was the reason for the rejection. The limit exists to stop runaway output and is now 6,000 characters. ### Model upgrades Newer models are often cheaper per token and better at agentic tasks, but upgrading needs testing: - Some newer models have stronger cybersecurity safeguards and may refuse rather than answer. Detection content includes malware command lines and attack tooling, so check the stop reason and have a fallback model. - Some always produce reasoning before answering. Those tokens bill as output, so the effort setting becomes the main cost control, and `max_tokens` needs room for the reasoning. - Reasoning blocks must be returned unchanged in tool loops. Our message translation layer currently keeps only text and tool calls, so it would need changing first. Put the model choice behind an environment variable, replay a set of past tune requests whose MRs were reviewed, and compare outcomes, cost and refusal rate before switching. ### Reuse existing infrastructure The detection agent started as a proof of concept and reused infrastructure we already ran: the container image, queue setup, model client, SIEM query tools and validator. The only new resources were the ones the threat model required: a separate service and identity for the Git credential, and a queue for tune requests. Further hardening can follow once it's proven useful. ## Building your own A suggested order: 1. Put detections in Git with MR review and CI. Make sure case dispositions are recorded consistently, and that you have API access to the SIEM and the case manager. 2. Write down your tuning policy, and collect past tuning commits as examples. 3. Build the evidence step in plain code: rule from main, closures by disposition split at the last change, and a baseline count. Check its output by hand on a few rules. 4. Add the decision step with a `submit_triage` tool and read-only SIEM access. Run it in dry-run mode and compare its decisions with what your engineers would choose. 5. Add the authoring step with a structured `submit_tune` tool, the YAML edit, CI's validator, a backtest and the diff check. 6. Give it a Git client that can only branch, commit one rule file, open and comment on MRs, under its own identity. 7. Add flags to the MR, then enable real MRs for analyst requests. 8. Add nominations from your alert pipeline in shadow mode, review them for a couple of weeks, then enable them. ## Readiness checklist - Detections are in Git, and changes go through review and CI. - Closed cases have meaningful dispositions. - The SIEM can be queried programmatically. - Tuning practices are documented. - Someone is assigned to review the agent's MRs. - Agent activity and cost are logged. Each of these is worth having without the agent. ## Next steps The main gap is confirming that a rule detects the behaviour it targets. Current checks cover format, execution, volume and data availability, but not whether the technique itself triggers the rule. We have a lab that runs techniques against test machines, and connecting it to the pipeline is the next piece of work. We also plan to turn reviewed MRs into test cases so the agent is measured against reviewer decisions over time. The eval harness exists and needs data.