Threat Hunting 102: Detection Engineering: Writing Rules That Actually Catch Things

Because hoping your SIEM will magically spot the bad guys is not, technically speaking, a strategy.

Introduction: From Hunting to Harvesting

In Threat Hunting 101, we covered how to proactively search for threats that have slipped past your defences. That's brilliant, but let's be honest -- nobody has the time or the staffing to manually hunt for everything, all the time. At some point, the things you find need to become automated detections. Otherwise, you're just a very expensive human grep command.

This is where detection engineering enters the picture. It's the discipline of turning threat intelligence, hunting findings, and hard-won experience into reliable, maintainable, and -- crucially -- actionable detection rules. Think of it as the bridge between "I found something dodgy" and "the system will catch this next time without me losing a weekend."

Detection engineering is also the art of restraint. Anyone can write a rule that fires on everything. The real skill is writing one that fires on the right things. Get it wrong, and you'll drown your SOC analysts in a sea of false positives until they start ignoring alerts entirely -- which, ironically, is exactly what the attackers are counting on.

The Four Horsemen of Detection Formats

Not all detections are created equal, and not all of them look at the same data. Here's the lineup.

Sigma Rules: The Rosetta Stone of SIEM Detection

Sigma is to SIEM detection what Snort is to network IDS -- a portable, vendor-agnostic format for writing detection rules. Created by Florian Roth and Thomas Patzke, Sigma lets you write a detection rule once and convert it to work with Splunk, Elastic, Microsoft Sentinel, QRadar, or whatever SIEM you've been contractually obligated to use.

The format is YAML-based, which means it's human-readable (mostly) and version-control friendly. Here's what a Sigma rule looks like in practice:

title: Suspicious PowerShell Download Cradle
id: 3b6ab547-8ec2-4991-b9c2-2a93b2c4e2a8
status: experimental
description: >
    Detects PowerShell commands commonly used as
    download cradles by malware and offensive tools
author: Tech Corsair
date: 2026/10/15
references:
    - https://attack.mitre.org/techniques/T1059/001/
logsource:
    category: process_creation
    product: windows
detection:
    selection_process:
        Image|endswith: '\powershell.exe'
    selection_commands:
        CommandLine|contains:
            - 'Invoke-WebRequest'
            - 'IWR'
            - 'wget'
            - 'curl'
            - 'Net.WebClient'
            - 'DownloadString'
            - 'DownloadFile'
    condition: selection_process and selection_commands
falsepositives:
    - Legitimate administrative scripts
    - Software deployment tools
    - SCCM/Intune script execution
level: medium
tags:
    - attack.execution
    - attack.t1059.001

The beauty of Sigma is the logsource abstraction. You describe what you're looking for conceptually, and the backend converter handles the specific field mappings for your SIEM. In theory. In practice, you'll still spend a fair bit of time tweaking converted rules, because every SIEM parses Windows event logs slightly differently, and none of them are sorry about it.

The SigmaHQ repository contains thousands of community-contributed rules mapped to MITRE ATT&CK, making it an excellent starting point for any detection programme. Don't reinvent the wheel -- stand on the shoulders of the community and then add your own flourishes.

YARA Rules: Pattern Matching for the Paranoid

If Sigma is for log-based detection, YARA is for file and memory scanning. Originally described as "the pattern matching Swiss Army knife for malware researchers," YARA lets you write rules that identify files based on textual or binary patterns, file metadata, and logical conditions.

YARA rules are used by malware analysts, incident responders, and threat hunters to scan endpoints, network shares, email attachments, and memory dumps for indicators of compromise. Here's an example:

rule Suspicious_Macro_Document
{
    meta:
        author = "Tech Corsair"
        description = "Detects Office documents with suspicious macro indicators"
        severity = "medium"
        reference = "https://attack.mitre.org/techniques/T1566/001/"
        date = "2026-10-15"

    strings:
        $ole_magic = { D0 CF 11 E0 A1 B1 1A E1 }
        $auto_open = "AutoOpen" ascii nocase
        $auto_exec = "Auto_Open" ascii nocase
        $shell_exec = "Shell" ascii nocase
        $wscript = "WScript.Shell" ascii nocase
        $powershell = "powershell" ascii nocase
        $hidden_window = "vbHide" ascii nocase
        $download = "XMLHTTP" ascii nocase

    condition:
        $ole_magic at 0 and
        any of ($auto_open, $auto_exec) and
        2 of ($shell_exec, $wscript, $powershell, $hidden_window, $download)
}

The condition section is where the logic lives. This rule fires when a file has the OLE magic bytes at offset 0 (confirming it's an Office document), contains an auto-execution trigger, and matches at least two of the suspicious capability strings. It's specific enough to avoid flagging every macro-enabled spreadsheet in Accounting, but broad enough to catch most weaponised documents.

YARA's real power comes from combining patterns with logic. You can match on hex sequences, regular expressions, file size, entry point offsets, imported functions, and even PE section characteristics. Tools like YARA-X (the Rust rewrite) and community rule sets from providers like Elastic, ReversingLabs, and the YARA-Rules project give you a solid foundation.

Snort and Suricata: Watching the Wire

For network-level detection, Snort and Suricata remain the workhorses. Both use rule-based signatures to inspect network traffic in real time, looking for patterns that match known attack traffic, exploit attempts, or command-and-control communications.

Suricata has largely eaten Snort's lunch in production environments thanks to multi-threading support, native JSON logging (EVE logs), and built-in protocol parsing for HTTP, TLS, DNS, SMB, and more. Both tools consume rules in a compatible format:

alert http $HOME_NET any -> $EXTERNAL_NET any (
    msg:"ET MALWARE Possible Cobalt Strike Beacon C2 Activity";
    flow:established,to_server;
    http.uri; content:"/api/v1/submit";
    http.header; content:"Cookie:";
    pcre:"/Cookie:\s*[A-Za-z0-9+\/]{60,}={0,2}/";
    classtype:trojan-activity;
    sid:2030001; rev:1;
)

The emerging threat (ET) rule sets from Proofpoint provide thousands of network signatures updated daily. Pair these with JA3/JA4 TLS fingerprinting and DNS monitoring, and your network detection layer starts looking rather formidable.

SIEM Correlation Rules: Connecting the Dots

Where individual detections spot atomic events, correlation rules identify patterns across multiple events over time. This is the SIEM's party piece -- taking those individual breadcrumbs and recognising that together, they spell trouble.

Common correlation patterns include:

• Threshold-based: More than 10 failed logins to the same account within 5 minutes
• Sequence-based: Account creation followed by privilege escalation followed by data access within 1 hour
• Statistical: Login from a country where the user has never authenticated before
• Temporal: Service account activity outside business hours

Correlation rules are where the SIEM earns its (considerable) licensing fee. They're also where most organisations struggle, because good correlation requires good data normalisation, and good data normalisation requires someone to have had the foresight to standardise field names across forty-seven different log sources. Good luck with that.

Detection-as-Code: Version Control Your Paranoia

If your detection rules live exclusively inside your SIEM's GUI, edited by hand, with no version history and no review process, you are -- and there's no gentle way to put this -- doing it wrong.

Detection-as-code applies software engineering principles to detection management:

• Version control: Every detection rule lives in a Git repository with full change history
• Peer review: New detections go through pull requests before deployment, just like application code
• Automated testing: Rules are validated against known-good and known-bad samples in CI/CD pipelines
• Infrastructure as code: SIEM configurations, dashboards, and detection rules are deployed programmatically
• Documentation: Each rule includes context on what it detects, why it matters, known false positives, and investigation steps

This approach brings enormous benefits. When a detection fires at 3 AM, the on-call analyst can look at the Git history to understand why the rule exists, who wrote it, what threat it addresses, and how to investigate. Compare this to the alternative: a cryptically named rule called "ALERT_SUSPICIOUS_v3_FINAL_FINAL_USE_THIS_ONE" with no documentation and an author who left the company eighteen months ago.

Tools like Sigma CLI, Detection Lab, and Atomic Red Team make it straightforward to build a testing pipeline. Write a rule, validate its syntax, run it against known-bad telemetry from an ATT&CK simulation, check for false positives against baseline data, and deploy with confidence.

The Detection Engineering Lifecycle

Detection engineering isn't a one-and-done activity. It's a continuous loop, much like threat hunting itself, but with more YAML and fewer existential crises.

1. Research the Threat

Start with threat intelligence. What are adversaries actually doing? Read advisories, study malware reports, review ATT&CK technique updates, and pay attention to what your threat hunters are finding. The goal is to understand the adversary's tradecraft well enough to identify observable artefacts.

2. Write the Detection

Translate the threat behaviour into a detection rule. Choose the appropriate format (Sigma for logs, YARA for files, Suricata for network) and focus on the behaviour rather than specific IOCs. IP addresses and domain names change daily. PowerShell using Invoke-Expression with a Base64-encoded payload? That's a behaviour, and it's far more durable.

3. Test Against Known-Bad

Validate your detection against real attack samples. Use tools like Atomic Red Team to simulate the technique in a lab environment and confirm the rule fires correctly. If it doesn't trigger on a known-bad sample, it certainly won't catch a real attacker.

4. Tune for False Positives

This is where most detections are won or lost. Deploy the rule in a "detection-only" mode (no automated response) and monitor for false positives. Every environment is different -- what's suspicious in a financial firm might be perfectly normal in a software development shop. Tune the rule's conditions, add exclusions for known-good activity, and adjust severity levels based on your environment's baseline.

5. Deploy to Production

Once tuned, promote the detection to production with appropriate response actions -- alert generation, ticket creation, automated enrichment, or (if you're feeling brave) automated containment. Ensure the rule is tagged with MITRE ATT&CK mappings, severity, and investigation runbooks.

6. Maintain and Retire

Detections are not fire-and-forget. Review performance metrics regularly: How often does it fire? What percentage are true positives? Has the underlying threat evolved? Rules that consistently produce false positives without true positives should be tuned or retired. Rules for threats that no longer apply to your environment should be archived. A lean, high-fidelity detection set beats a bloated one every single time.

Alert Fatigue: The Silent Killer of SOC Analysts

Here's an uncomfortable truth: too many detections is worse than too few.

A SOC analyst drowning in 3,000 alerts per day -- most of them false positives -- will inevitably start ignoring alerts. This isn't a character flaw; it's basic human psychology. When 95% of what you see is noise, your brain starts treating everything as noise. The remaining 5% of genuine alerts get lost in the deluge, triaged hours or days late, or simply closed without investigation because the analyst has already handled 47 "critical" alerts that turned out to be someone in Marketing running a scheduled task.

The research consistently shows that alert fatigue is one of the top contributors to missed breaches. Organisations with fewer, higher-fidelity detections consistently outperform those with thousands of noisy rules.

How to fight alert fatigue:

• Prioritise quality over quantity. Ten well-tuned, high-fidelity detections are worth more than a thousand noisy ones
• Tier your detections. Not everything needs to be "CRITICAL." Use severity levels that reflect actual risk, not theoretical worst-case scenarios
• Automate the boring bits. Use SOAR playbooks to enrich alerts automatically before they reach an analyst. If the alert can be contextualised and closed by automation, let the robots handle it
• Track metrics. Monitor your true positive rate per detection. If a rule hasn't produced a true positive in six months, question whether it's earning its keep
• Build feedback loops. Analysts who investigate alerts should be able to flag false positives easily, and that feedback should flow directly back to detection engineers

The ideal detection programme is a curated collection of sharp, well-maintained rules -- not an ever-growing pile of everything someone once thought might be suspicious. Think quality wine cellar, not hoarder's garage.

Putting It All Together

Detection engineering is where threat hunting meets software engineering, and the result is something greater than either discipline alone. It transforms one-off hunting discoveries into persistent, automated surveillance. It applies engineering rigour to what was previously an ad-hoc process of copying Splunk queries from blog posts and hoping for the best.

The key principles worth remembering:

1. Detect behaviours, not indicators. IOCs are ephemeral; TTPs are durable
2. Write detections as code. Version control, peer review, and automated testing are not optional luxuries
3. Choose the right tool. Sigma for logs, YARA for files, Suricata for network, correlation rules for multi-event patterns
4. Test relentlessly. Against known-bad, against your own baseline, and against evolving adversary tradecraft
5. Curate ruthlessly. A detection that produces only noise is worse than no detection at all

The ultimate goal isn't to have the most detections. It's to have the right detections -- the ones that fire when something genuinely bad is happening, with enough context for an analyst to take swift, decisive action. Everything else is just theatre.

And if your detection set is more than 60% community rules you downloaded and never tuned? Well, at least you've got something. But you might want to schedule some quality time with your SIEM before an attacker schedules it for you.

---

This is part 2 of the Threat Hunting series on techcorsair.co.uk. Next up: building a threat hunting lab environment where you can break things without your CISO having a breakdown.

---

Category: Threat Hunting | Tags: detection engineering, sigma rules, yara, suricata, siem, detection-as-code, alert fatigue, mitre att&ck