Every SOC I've worked in has the same rule. You know the one. It fires forty times a day, every alert gets closed as benign, and everybody on the team could tell you roughly what the fix is. Nobody has the afternoon to actually do it. Multiply that by a few dozen rules and you have most of your queue. Over the last few months we built an AI agent to go after that problem. It triages our alerts, notices when it keeps closing the same rule's alerts as benign, and opens a merge request that tunes the rule. A person reviews the MR and merges it, or doesn't. This post is the long version. I'll cover how it works, the actual stack, enough code that you could build your own, and the mistakes we made. The mistakes are honestly the most useful part. ## The starting point matters more than the model Before any of this, our detections already lived in a Git repo as YAML. Changes went through merge requests, CI validated them and deployed them to the SIEM (CrowdStrike NG-SIEM), and every alert ended up as a case in TheHive with a disposition when someone closed it. Tuning itself was already semi-automated. One of us would sit down with an AI coding assistant, point it at a noisy rule, and work through it together. We had written "skills" describing how we tune, and the repo had hundreds of `tune:` commits showing what good looked like. So when we built the agent, we weren't inventing a process. We were taking one that was already written down and letting it run without someone at the keyboard. That's the most important thing I can tell you. If your tuning judgment only lives in people's heads, an agent won't find it. Write it down first. ## Why build it yourself? There are plenty of "AI SOC analyst" products out there. But think about what a detection agent actually needs to know to tune a rule well: - the rule's source and everything that's ever changed in it, - how your analysts closed its alerts, and what they wrote when they did, - what normal looks like in *your* telemetry, - and why that weird exclusion on line 14 is there on purpose. Almost all of that is internal. A vendor can get part of the way with integrations, but they're never going to know that a particular allowlist exists because those accounts belong to infostealer victims we're watching. The other reason is change control. Building it in-house means the agent's output lands in the same merge request process every human change goes through. It isn't a new side door into production. It's another contributor to the repo. ## The big picture ```mermaid flowchart TD A[Alert becomes a case] --> B[Triage agent investigates] B --> C[Notes written back to the case] B -->|"keeps closing this rule?"| D[Tuning agent] E[Analyst clicks 'Request tuning'] --> D D --> F[Merge request] F --> G{Human review} G -->|merge| H[CI deploys to SIEM] ``` Two agents. The triage agent reads every alert and writes up what it found. The tuning agent fixes rules, either because triage nominated one or because an analyst asked. Everything the tuning agent produces is a merge request, and nothing ships until a person merges it. ## The stack Here's what's actually running. None of it is exotic, and that was deliberate. - **SOAR: Tines.** It already created our cases, so it's where the agent hooks in. When a case is created or an alert merges into one, a Tines story posts the case to our cloud endpoint. When the agent finishes, Tines receives the verdict on a callback webhook and writes it to the case. - **Front door: a small Python Cloud Function** (GCP, gen2). It does nothing clever. It checks the request, works out which "lane" it belongs to (triage, tuning, hunting and so on), and drops it on a queue. It returns in milliseconds, which matters because Tines doesn't like waiting. - **Queues: Cloud Tasks.** One queue per lane. The triage queue dispatches **one task at a time** with up to five retries. That sounds slow, but it keeps state handling simple and cost predictable, and triage isn't latency-critical at the scale of seconds. Cloud Tasks signs each call with an OIDC token from a dedicated invoker service account, so the workers don't need to be public. - **Workers: Python on Cloud Run.** Same container image for every lane, different entry points. The tuning agent runs as its own Cloud Run service with its own service account. More on why later. - **Model: Claude on Vertex AI**, through the official Anthropic SDK's Vertex client. Opus for the decisions, and optionally a cheaper Sonnet model as a query-writing helper in the tuning lane. Auth is the service account's own identity, so there's no API key to leak. - **State: a GCS bucket.** Small JSON objects for locks, per-case dedup state and nomination markers. No database. - **Observability: structured JSON log lines** in Cloud Logging, one per finished run, plus result rows sent back to Tines for a dashboard. - **Change control: GitLab.** The tuning agent opens MRs on the detection-as-code repo, and the repo's existing CI does the rest. Here's how a single triage request moves through it: ```mermaid flowchart TD A[Tines posts the case] --> B[Receiver function] B --> C[Cloud Tasks queue] C --> D[Worker on Cloud Run] D <--> E[Claude on Vertex] D <--> F[Read-only tools] D --> G[Tines callback writes the case] ``` ## Anatomy of a triage run A typical run looks like this. These are real numbers from a run today. 1. The worker gets the case: the alert, the TheHive case ID, and the previous AI verdict if this case was triaged before. 2. It builds a system prompt and starts a tool-use loop with the model. 3. The model calls tools: looks up the correlation rule, pulls the user's details and devices, reads our docs for that log source, and runs a couple of SIEM queries. 4. After six model turns and about a minute, it calls `submit_verdict`. 5. Code validates the verdict, sends it to the Tines callback, and logs a completion line with the cost. That run cost $1.35. ### The system prompt is assembled, not written We don't have one giant prompt. We have a stack of markdown files, and each lane composes the ones it needs: - `environment.md`: who we are, what our estate looks like, what's normal here. - `cql_essentials.md`: how to write correct queries in our SIEM's query language, and the mistakes models commonly make in it. - `investigator.md`: how to triage. - Optional add-ons: one for cloud-security alerts, one for when the agent should consider whether the rule needs tuning, and one for "update mode," when an alert merges into a case that's already been triaged. The first two are shared by every lane. That turns out to matter a lot for prompt caching, which I'll get to. ### Tools are where the real work happens The triage agent has read-only tools across most of the places an analyst would look: - the SIEM (run a query, look up a correlation rule), - the EDRs, - identity providers, - read-only access to our cloud accounts, - our cloud security platform, - threat intel, - past cases in TheHive, - our wiki and ticketing, through MCP servers, - and a tool that fetches our own documentation for each log source. That last one is underrated. The model knows the query language in general. It doesn't know our field names or which parser we use for a given source. Giving it a "read the docs for this log source" tool cut down on bad queries more than any prompt tweak. Two things I'd tell anyone building tools: **Shape the results.** Raw SIEM output is huge. Every tool result goes through a shaping step that caps its size and drops fields the model never uses. Smaller results mean cheaper turns and a model that isn't drowning. **Make every tool read-only.** The triage agent can't close a case, can't isolate a host, and can't disable a user. It recommends. A human or a separate, non-AI step does the doing. ### The verdict is a tool call with a schema The agent finishes by calling a tool called `submit_verdict`. Its input schema *is* the verdict format. Here's a trimmed version: ```json { "type": "object", "properties": { "disposition": {"type": "string", "enum": ["close", "escalate"]}, "confidence": {"type": "string", "enum": ["low", "medium", "high"]}, "justification": {"type": "string"}, "evidence": {"type": "string", "description": "One line per tool call: '<tool> -> <finding>'"}, "recommendations": {"type": "string", "description": "Required when escalating"}, "status": {"type": "string", "enum": ["FalsePositive", "True positive, not malicious", "True positive, no impact"]}, "tuning_candidate": {"type": "boolean"}, "tuning_category": {"type": "string", "enum": ["service_account", "expected_admin_activity", "scanner_or_tooling", "broad_logic", "duplicate_rule", "other"]}, "critical": {"type": "boolean", "description": "Active, ongoing harm only. A downstream system pages a human."} }, "required": ["disposition", "confidence", "justification", "evidence"] } ``` A few things are going on here. **The descriptions are prompts.** The real schema has long descriptions on every field ("first line is a one-sentence summary, then 2–6 bullets..."). The model reads them every time it calls the tool, right when it matters. **Code checks the rules the schema can't express.** For example, `recommendations` is required when escalating, `status` is required when closing, and `update_summary` is required in update mode. If the model gets one wrong, the error goes back to it as a tool result and it tries again. **`tuning_candidate` is how triage talks to the tuning agent.** When the model closes an alert and thinks the *rule* is the problem, not just this one alert, it says so and picks a category. That's the nomination. ### The loop itself Here's the core loop, stripped down. Ours has more going on (iteration budgets, nudges, audit records), but this is the shape, and it's genuinely most of what you need. ```python import json from anthropic import AnthropicVertex client = AnthropicVertex(project_id="your-project", region="global") MODEL = "claude-opus-4-8" # what we run today; see the upgrade lesson below CACHE = {"type": "ephemeral"} def with_cache(messages): # Cache everything up to the newest block, so each turn re-reads # earlier turns at the cache-read price. last = messages[-1] blocks = list(last["content"]) blocks[-1] = {**blocks[-1], "cache_control": CACHE} return messages[:-1] + [{**last, "content": blocks}] def triage(alert, system_prompt, tools, handlers, check_verdict, max_turns=25): messages = [{"role": "user", "content": [ {"type": "text", "text": "Investigate this alert:\n" + json.dumps(alert)}]}] for _ in range(max_turns): resp = client.messages.create( model=MODEL, max_tokens=8000, system=[{"type": "text", "text": system_prompt, "cache_control": CACHE}], tools=tools, messages=with_cache(messages), ) # Pass the assistant turn back exactly as it came. messages.append({"role": "assistant", "content": resp.content}) calls = [b for b in resp.content if b.type == "tool_use"] if not calls: messages.append({"role": "user", "content": [ {"type": "text", "text": "Call submit_verdict to finish."}]}) continue results = [] for call in calls: if call.name == "submit_verdict": problem = check_verdict(call.input) # None means it's valid if problem is None: return call.input results.append({"type": "tool_result", "tool_use_id": call.id, "content": problem, "is_error": True}) else: out = handlers[call.name](**call.input) results.append({"type": "tool_result", "tool_use_id": call.id, "content": json.dumps(out, default=str)[:20000]}) messages.append({"role": "user", "content": results}) return None # no verdict: escalate to a human, don't retry ``` And a tool definition looks like this (simplified): ```python tools = [ { "name": "run_logscale_query", "description": "Run a read-only CQL query against NG-SIEM. Read the log " "source docs first if you're unsure of field names.", "input_schema": { "type": "object", "properties": { "query": {"type": "string"}, "lookback": {"type": "string", "description": "e.g. 1h, 24h, 7d"}, }, "required": ["query"], }, }, {"name": "submit_verdict", "description": "Submit your final verdict.", "input_schema": VERDICT_SCHEMA}, ] ``` A few notes on that loop, because each line is there for a reason: - **There are two cache breakpoints:** one on the system prompt, and one on the newest message. Tools are sent before the system prompt, so the tool list gets cached with it. - **The conversation is append-only.** We never edit or trim earlier turns. That's what keeps the cached prefix byte-identical. - **We pass `resp.content` back as-is.** That matters even more on newer models, which return reasoning blocks that have to be sent back unchanged. - **A bad verdict gets corrected in the same conversation.** More on why in the lessons. - **Running out of turns means a human looks at it,** not another retry. ## Let the code decide, let the model propose The tuning agent is where the design gets more careful, because it's the one that can change something. ```mermaid flowchart TD A["Gather the evidence<br/>(code)"] --> B["Tune it or not?<br/>(model)"] B --> C["Write the change<br/>(model)"] C --> D["Validate and backtest<br/>(code)"] D --> E["Open the MR<br/>(code)"] ``` The full pipeline has more stages than that, but the split is what matters: only two of them involve the model. **Request validation (code).** A tune request names a rule, by repo path, Falcon ID or exact name, and carries some optional context from the analyst. The rule is always resolved against an index built from `main`. There's no fuzzy matching and no paths outside the rules directory. The requester has to be a company email. The context is capped at 4,000 characters, with control and zero-width characters stripped. Anything that fails returns a 400, so Cloud Tasks doesn't retry it. **Evidence (code).** Before the model sees anything, code pulls: - the rule YAML from `main`; - 30 days of closed cases for the rule from TheHive, each classified from its closure status as true positive, benign or false positive, along with titles, summaries and analyst comments; - how many of those closures came *after* the rule's last real change; - a baseline backtest of how often the rule fires right now. The backtest is a bit fiddly. A big SIEM count over 30 days can time out or come back truncated, so we run a timed one-day probe first and use its timing to choose a 30-, 7- or 1-day window. Then we run a terminal `count()` and refuse any aggregate the SIEM marks as truncated. We also strip out any time-window checks the rule makes relative to its own run, so they don't distort the count. **Triage step (model).** The model gets the rule, the evidence and the analyst's note, and calls `submit_triage` with one of: tune, no tune needed, or not tunable. It can run SIEM queries to dig in, but it can't do anything else. Analyst notes and case text are wrapped in `<untrusted>` tags in the prompt, and the policy prompt says what that means. **Author step (model).** If it's a tune, the model calls `submit_tune` with a structured spec: the new filter, and optionally a new lookback, schedule, severity, MITRE tags and a note. It isn't a free-form diff. It's a set of fields, and each one is validated: field allowlist, format checks on lookback, schedule, severity and MITRE IDs, and length limits. **Edit, gates and MR (code).** Code applies the spec to the YAML text, then checks that the result parses back to exactly the intended rule. Then come the gates (next section). If a hard gate fails, the model gets the failure back and two more tries, plus one extra try specifically for blowing the volume cap. After that, the answer is "not tunable." If it passes, code opens the MR. ### One model or two? The tuning agent spends a lot of its time writing SIEM queries. We tried giving that job to a cheaper Sonnet "query worker" that the main Opus model delegates to. On a live comparison across four rules: - Opus on its own cost $1.88 and took 20 minutes; - Opus plus a Sonnet query worker cost $3.12 and took 35 minutes; - both gave the same verdicts. So it's off by default. Delegation sounds cheaper on paper, but the handoffs cost more than they saved. Measure before you split. ## The merge request is the trust boundary We didn't build an approval system. We already had one. The tune worker holds a GitLab token, and technically that token *could* merge. What stops it is that the GitLab client we gave it only has five methods: ```python class GitLabWriter: def create_branch(self, name): ... # only under tune/agent-* def commit_rule(self, branch, path, content, message): ... # one rules/*.yml file def create_mr(self, branch, title, body, labels): ... def comment_mr(self, mr_iid, body): ... def list_open_agent_mrs(self): ... ``` There's no merge method to misuse, and a test fails if anyone adds one. The MR body escapes HTML, GitLab quick actions (a line starting with `/`) and `@mentions` in any model or analyst text, so nothing in the description can act on the repo or ping people. On top of that, the diff itself is checked on the rendered YAML. A tune may change `search.filter`, `lookback`, `trigger_mode`, the schedule, `severity`, `status`, `tactic` and `technique`, and the lifecycle review dates. It may also append to `notes`. If it touches anything else, it fails. ### Rule of Two, and why there are two services We used Meta's [Rule of Two](https://ai.meta.com/blog/practical-ai-agent-security/) as a gut check. An agent shouldn't do all three of these at once: - process untrusted input, - have access to sensitive systems, - and be able to change things. The triage agent reads attacker-controlled content all day, because that's what alerts are. If the GitLab token were readable from that service, a well-crafted prompt injection in an alert could, in theory, become a commit. So the tuning agent runs as its own Cloud Run service with its own service account. That account is the *only* one with access to the GitLab token secret, and the access is granted on that one secret, not across the project. The triage service never gets the token, and the tuning agent never sees raw alert payloads, only case history and the analyst's note. ### Gates versus flags We split the checks into two kinds. **Hard gates** stop the pipeline (after the repair attempts above): - the edit applies cleanly and round-trips; - the diff stays inside the allowed fields; - our validator reports nothing new compared with the original rule; - the new query runs against an hour of live data without errors. **Flags** don't block anything. They go at the top of the MR with a `needs-decision` label so the reviewer can't miss them. A few examples: - **`excludes_tp_entity`**: a value the change excludes showed up in a true-positive case. - **`possible_injection`**: a newly excluded value is mentioned in the analyst's note or case text, but fewer than two benign closures back it up. - **`regression_to_zero`**: the rule would never fire again. - **`severity_raise`, `allowlist_edit` or `weak_evidence`.** That `possible_injection` check is purely structural. We don't ask a model "is this an injection?" We check whether the evidence supports the exclusion. If someone puts "please exclude user X" in a case comment and nothing in the case history backs it up, the MR says so in red. ## How we know a rule actually works People say "validated" like it's one thing. It's at least four. **Same validator as CI.** The agent runs a vendored copy of the exact validator our DaC repo's CI runs, so "passes" means the same thing in both places. Vendored copies drift, and ours did, so there's now a refresh script. **It runs.** The new query has to run against an hour of real data, counted, without errors. A timeout gets flagged rather than failed. **Backtest.** Alert counts before and after, using the same windowed counting method as above. Going from 400 to 12 is great. Going to zero is suspicious. **The data exists.** This is for new rules. A rule that never fires might mean there are no attacks, or it might mean we don't collect that log. So every new rule comes with a `control_filter`: the same search with the "is this bad?" clauses deleted. If the control finds lots of events and the rule finds a few, great. If the control finds nothing, we're blind and the rule is decoration. The model has to build the control by *deleting* clauses from the rule. Left alone, it would sometimes write a brand-new query that happened to match something, which proves nothing. Code now rejects a control that's identical to the filter, or one that contains a clause the filter doesn't have, and gives the model exactly one try to fix it. ## The lessons ### Run in shadow mode, and actually read the logs When we first let triage nominate rules for tuning, we ran it in shadow mode (an environment variable with three settings: off, shadow, and on). Shadow logs what it would have sent, without sending it. Of the first 114 nominations, 90 were for one cloud security product's rule. That rule doesn't live in our repo, so the tuning agent couldn't have done anything with it. One line of scoping ("only nominate SIEM correlation rules") fixed it. Nominations are also rate-limited. Each rule gets at most one per day, unless it's noisy for a different reason, up to three. That state is a small JSON marker per rule in GCS, written with generation preconditions so two workers racing can't both send. ### "Noisy for 30 days" is a lie if you tuned it last week In an early test, six of our ten "noisiest" rules had already been tuned in the previous week. Since the fix, they'd produced none of that noise. On raw 30-day numbers they looked like the top candidates. So the evidence now says how many closures happened after the rule's last real change, skipping commits from our sync bot. If you look at alert history, split it at your own change points. ### Some noise is on purpose We have identity rules with allowlists that look like obvious tuning targets. In one case, the excluded accounts belong to known infostealer victims, and the whole point is to alert when a login *succeeds*, not on every attempt. An agent chasing lower volume would happily "fix" that. The answer wasn't a smarter model. It was writing that context down where the agent reads it, in the rule's notes and the standing policy prompt, and flagging any edit to an allowlist. The other side of this: when an analyst explicitly asks for a tune, the agent should do it. Early versions pushed back on analysts, and nobody wants a tool that argues with the people who work the queue. Now it only declines when the change is impossible or would increase alert volume. ### Most of the cost was duplicates When I dug into where triage money was going, one burst explained a lot of it. A single rule fired 64 times into one case, and every one of those alerts kicked off a full re-triage. Across 202 re-triages we looked at, only 2 changed the outcome, and both happened because a *different* kind of alert joined the case. So we put a gate in front of the model. When an alert merges into a case the agent has already triaged: ```mermaid flowchart TD A[New alert joins a triaged case] --> B{Seen this exact one before?} B -->|yes| C[Skip it] B -->|no| D{High or critical?} D -->|yes| E[Re-triage now] D -->|no| F[Wait 10 minutes,<br/>then re-triage once<br/>for everything that arrived] ``` "Seen this exact one" is a fingerprint: ```python def fingerprint(alert): key = json.dumps([rule_key(alert), entities(alert), severity(alert)]) return hashlib.sha256(key.encode()).hexdigest()[:24] ``` `rule_key` is the rule's ID or name. `entities` is a sorted list of users, hosts, IPs, hashes and cloud resources pulled from the alert. Severity is normalised to a label. IDs, timestamps and titles are deliberately left out, because they change on every copy. One product even rewrites the title each time. Each case gets one small JSON object in GCS. It holds the fingerprints already covered, the latest verdict, and any alerts parked waiting for a batch. Because the triage queue runs one task at a time, the read-modify-write needs no lock. The batching trick I like most is the delayed re-triage task, which gets a *name* derived from the case and the time window: ```python name = f"retriage-{case_id}-{int(time.time() // 600)}" ``` Cloud Tasks refuses a second task with the same name. So when 30 alerts land in the same ten minutes, 30 workers try to schedule the re-triage, one succeeds, and the rest get "already exists" and move on. That's deduplication for free from the queue. When the batch fires, the model gets the prior verdict plus every parked alert in one prompt and re-assesses them together. If the batch can't be scheduled for some reason, we triage immediately instead. Failing open here costs money, but never coverage. We replayed a week of real traffic, 1,099 alerts, through it. Re-triages on already-triaged cases went from about 487 to about 161, and total model runs from 1,099 to 773. We didn't change a single prompt. ### Turn on prompt caching An agent doesn't make one model call. It loops. Every turn re-sends the system prompt, the tool list, the alert and everything that's happened so far. Without caching, you pay full price for all of it, every turn. We added the two cache breakpoints shown in the loop above to our shared model client. It was about twenty lines, plus a kill switch environment variable. Here's a real completion log line, trimmed: ```json {"event": "investigation_complete", "usage": {"model_turns": 6, "tokens": {"input": 12, "cache_write": 167732, "cache_read": 350558, "output": 4996}, "cost_usd": 1.35, "cost_uncached_usd": 2.72, "cache_savings_usd": 1.37}, "disposition": "close", "confidence": "high", "duration_seconds": 67.2, "tools_used": ["cs_correlation_rules", "get_user_details", "fetch_logsource_docs", "run_logscale_query", "get_user_devices", "submit_verdict"]} ``` Twelve tokens at the normal input price. Everything else is a cache write or a cache read. Across 475 triage runs, we spent about $528 where we would have spent about $1,058. That's half the bill, for an afternoon's work. The catch is that it only works if the harness never edits earlier turns. If your framework trims history, or slips a timestamp in near the top of the prompt, the cache misses every time and you'll never notice. Log cache reads and writes on every run so you can see it working, and log cost per run while you're at it. Someone will ask. ### When the agent fails, hand it to a person About once an hour, the model would submit a verdict that was missing a required field. That raised an exception, the task returned a 500, and Cloud Tasks retried it from scratch, as a whole new investigation, up to five times, while everything behind it in the one-at-a-time queue waited. Now validation errors go back to the model as an `is_error` tool result in the same conversation, with a hint telling it to resubmit the *complete* verdict, not just the missing field. It gets two tries. If it still can't, the case escalates to a human with whatever the agent found, including the list of tools it ran, so the analyst knows there's partial evidence to look at. The general rule: once work has started, a failure should hand off to a person, not retry. ### SOAR has race conditions too Not everything was about the model. Our SOAR used to build case titles by gluing alert names together, so you'd get "Alert A / Alert B / Alert C / Alert D" as a case title. We replaced that with a Tines LLM action that writes one sensible title whenever a new alert merges in. It returns `{changed, title}`, so most merges leave the title alone. It worked great until two alerts merged into the same case at the same moment. Both read the old title from the earlier lookup, and the second write clobbered the first. The fix was an HTTP step that re-reads the case right before the LLM call. Anything that reads, modifies and writes a case deserves the same suspicion you'd give a database. ### Don't truncate the part that matters Early on, the agent turned down a new detection, and the stored explanation got cut off at 1,000 characters. It stopped mid-sentence, right where it was about to name the missing telemetry, which was the entire reason for the rejection. Caps are there to stop runaway output, not to summarize. Ours is 6,000 characters now. ### Upgrading the model is a migration Newer models are often cheaper per token and better at agentic work, but for a security tool, read the migration notes carefully: - Some newer models ship with stronger cybersecurity safeguards and can return a refusal instead of an answer. SOC work is full of malware command lines and phishing payloads. Your code has to check the stop reason, and you want a fallback model. - Some always reason before answering, and those reasoning tokens bill as output, so effort level becomes your main cost lever. - Your output limit has to leave room for the reasoning. - The reasoning blocks have to be passed back unchanged in tool loops. Our translation layer keeps only text and tool calls from each reply today, so that's a change we'd have to make first. Put the upgrade behind an environment variable, replay a few dozen real alerts, and compare verdicts, cost and refusal rate before you switch. ### Reuse what you have, harden later This started as a proof of concept, and the fastest way to get there was reusing everything the triage agent already had: the same container image, Cloud Tasks plumbing, model client, SIEM tools, audit trace and validator. The only new resources were the ones the threat model required: the separate service and service account for the write credential, and a queue for the new lane. Everything else can get hardened once the thing has earned its keep. ## Build your own: the minimum viable version If I were starting from scratch, this is the order I'd do it in. 1. **Get the foundations right.** Detections in Git, MRs and CI, case dispositions that mean something, and SIEM API access. Don't skip this. 2. **Build triage first, read-only.** A receiver, a queue, a worker and the loop above, with three or four tools: SIEM query, rule lookup, user lookup and log-source docs. Post the verdict back to your case manager as a comment. Don't let it close anything yet. 3. **Make the verdict a tool with a schema,** and validate it in code. Send errors back to the model in the conversation. 4. **Turn on prompt caching on day one,** and log tokens and cost on every run. 5. **Add the duplicate gate** before you scale up. Fingerprint the alert, and skip what you've already covered. 6. **Add `tuning_candidate` to the verdict** and log nominations in shadow mode for a couple of weeks. Read them. 7. **Build the tuning agent as a separate service.** Have code gather the evidence, use structured `submit_triage` and `submit_tune` tools, a restricted Git client, CI's own validator, a backtest, and flags on the MR. 8. **Turn on MRs for real** once the shadow output looks like what your best detection engineer would write. ## Should you build one? My honest checklist: - Your detections are in Git, and changes go through review and CI. - Your closed cases have real dispositions on them. - You can query your SIEM programmatically. - Your tuning judgment is written down somewhere. - Someone will actually review the MRs. - You'll be able to see what it's doing and what it costs. If you're missing a few of those, go fix them first. Every one of them is worth having even if you never build the agent. ## What's next for us The big gap is proving a rule actually catches what it claims to. We check format, execution, volume and whether the data exists, but not whether the attack itself would fire it. We have a lab for running techniques against test machines, and wiring that into the pipeline is next. We also want every reviewed MR to become a test case, so the agent gets measured against our own reviewers' decisions over time. There's already an eval harness. It just needs to be fed. ## Wrapping up The agent didn't replace anyone's judgment. It made our judgment cheaper to apply. The research one of us used to do for a single noisy rule, on a good afternoon, now happens for every nominated rule, with the evidence already attached and waiting for a reviewer. And the parts that made it trustworthy weren't really AI at all. A short list of verdicts, a Git client with five methods, a validator shared with CI, a merge button only a person can press, and an agent that's comfortable saying "no tune needed."