Malware Analysis 101: The Workflow: Poking Suspicious Files With a Very Long Stick

Category: Digital Forensics

Tags: malware-analysis, sandboxing, reverse-engineering, powershell, static-analysis, dynamic-analysis

There are two types of people in cybersecurity: those who see a suspicious executable and immediately want to run it, and those with functioning self-preservation instincts. Malware analysis is the discipline that bridges the gap -- a structured approach to understanding what malicious software does, how it does it, and who built it, all without letting it anywhere near production systems.

This article covers the practical malware analysis workflow from initial triage through static and dynamic analysis, including the fine art of unravelling obfuscated PowerShell -- which is now the delivery mechanism of choice for approximately 90% of threats and 100% of incidents that ruin long weekends.

The Analysis Pipeline

Every malware sample follows the same basic workflow:

Sample Received -> Sandboxing -> Static Analysis -> Dynamic Analysis -> Report

Each phase builds on the last. Sandboxing gives a quick behavioural overview, static analysis reveals the inner workings without execution, and dynamic analysis confirms what actually happens when the thing runs. Skip phases at your peril -- sandbox evasion is trivially common, and skipping static analysis means missing indicators that dynamic analysis might not trigger.

Phase 1: Sandboxing -- The Quick and Dirty Assessment

Before spending hours in a disassembler, throw the sample at an automated sandbox and let it do the tedious work of initial classification. Sandboxes execute samples in instrumented virtual environments and record everything: network connections, file system changes, registry modifications, process creation, and API calls.

Sandbox Options

What Sandboxes Reveal

• File hashes compared against known malware databases
• Network callouts -- C2 domains, download URLs, DNS queries
• Process behaviour -- child processes spawned, injection attempts
• File system changes -- what gets dropped, modified, or deleted
• Registry modifications -- persistence mechanisms
• Detection signatures from multiple AV engines

Sandbox Limitations

This is where it gets important:

• Sandbox evasion is common. Modern malware checks for virtual environments (VM artefacts, suspicious MAC addresses, unrealistic mouse movement), sets sleep timers to outlast sandbox execution windows, and checks for user interaction before detonating. A clean sandbox result does not mean a clean file.
• False positives happen. Legitimate software sometimes triggers heuristic detections. Installers that modify the registry and create services look suspicious to automated analysis.
• Never upload sensitive samples to public sandboxes. If the sample came from a client engagement or contains proprietary code, use a private instance. What goes into VirusTotal stays in VirusTotal -- and is searchable by everyone, including the threat actor who sent it.

Phase 2: Static Analysis -- Reading Without Running

Static analysis examines the sample without executing it. Think of it as a forensic pathologist examining evidence before opening the sealed bag -- non-destructive, reversible, and considerably safer.

Strings Extraction

The simplest and often most revealing technique. Hardcoded strings reveal URLs, IP addresses, registry keys, file paths, error messages, and sometimes the malware author's questionable taste in variable names.

strings -n 7 malware.exe          # ASCII strings (min 7 chars)
strings -el malware.exe            # Unicode/UTF-16LE strings

What to look for:

• URLs and IP addresses (C2 infrastructure)
• File paths (drop locations, targets)
• Registry keys (persistence mechanisms)
• API function names (capabilities)
• Encrypted/encoded blobs (second-stage payloads)
• Debug strings (sometimes attackers forget to strip these)
• Mutex names (used to prevent multiple instances)

Import Analysis

The DLLs and functions a binary imports reveal its capabilities before you ever run it. A PE file that imports CreateRemoteThread and VirtualAllocEx is telling you exactly what it plans to do with its evening.

Tools for PE analysis:

• PEStudio -- excellent for initial triage, highlights suspicious imports and sections
• CFF Explorer -- detailed PE header analysis
• DIE (Detect It Easy) -- packer and compiler detection
• Ghidra / IDA Pro -- full disassembly and decompilation

Packer Detection

If the binary is packed (compressed or encrypted), strings and imports will be minimal. Common packers include UPX, Themida, VMProtect, and custom packers. DIE and PEiD can identify known packers. UPX-packed binaries can often be unpacked with upx -d; anything more exotic requires dynamic analysis or manual unpacking.

Phase 3: Dynamic Analysis -- Running the Thing (Safely)

Dynamic analysis means executing the sample in a controlled, isolated environment and observing what happens. This reveals runtime behaviour that static analysis might miss -- decrypted payloads, reflective DLL loading, process hollowing, and network communication with live C2 infrastructure.

Essential Monitoring Tools

Dynamic Analysis Checklist

1. Snapshot your analysis VM before execution
2. Start monitoring tools -- ProcMon, Wireshark, FakeNet-NG
3. Take a Regshot baseline
4. Execute the sample and let it run for 5-10 minutes minimum
5. Interact with the system -- some malware waits for mouse clicks or keyboard input
6. Note all observations: processes spawned, files dropped, connections attempted
7. Take a second Regshot and compare
8. Revert to snapshot when finished

What to Watch For

• New processes -- especially children of Office applications, wscript, cscript, or PowerShell
• Process injection -- a process suddenly connecting to the network that has no business doing so
• File drops -- new executables in Temp, AppData, or system directories
• Scheduled tasks and services -- persistence installation
• DNS queries -- domain generation algorithms (DGA) produce distinctive patterns
• Outbound connections -- especially to unusual ports or with regular beacon intervals

---

PowerShell Deobfuscation: Unwrapping the Russian Doll

Modern malware delivery chains increasingly rely on PowerShell -- it's pre-installed on every Windows system, it can execute entirely in memory, and obfuscation techniques can make a single command line look like the output of a cat walking across a keyboard.

Common Obfuscation Layers

Layer 1: Base64 encoded command

powershell -nop -exec bypass -EncodedCommand SQBFAFgAIAAoAE4AZQB3...

The -EncodedCommand parameter accepts base64-encoded UTF-16LE text. Decoding is straightforward:

echo "SQBFAFgAIAAoAE4AZQB3..." | base64 -d | iconv -f utf16 -t utf8

Layer 2: Compressed and encoded (the nesting doll)

$d=[Convert]::FromBase64String('H4sIAAAA...');
$s=New-Object IO.MemoryStream(,$d);
$z=New-Object IO.Compression.GzipStream($s,
    [IO.Compression.CompressionMode]::Decompress);
IEX (New-Object IO.StreamReader($z)).ReadToEnd()

This is base64 wrapping gzip wrapping the actual payload. Each layer must be peeled off in order.

Layer 3: String concatenation and case randomisation

powerSheLL -WiNDowstyl 1 -EXe BypaSs -coMMAn
  . ( $VerBOsepReFerenCE.tosTrIng()[1,3]+'x'-join'')

This exploits PowerShell's case insensitivity and flexible parameter parsing. It's annoying rather than technically complex.

The Decoding Pipeline

For multi-layer encoded PowerShell, a systematic approach:

# Step 1: Extract and decode the outer base64 layer
cat payload.txt | grep -oE '[a-zA-Z0-9/+=]{40,}' | base64 -d

# Step 2: Convert from UTF-16LE to UTF-8
| iconv -f utf16 -t utf8

# Step 3: Extract inner base64 and decompress
| grep -oE '[a-zA-Z0-9/+=]{40,}' | base64 -d | gunzip

# Complete pipeline
cat payload.txt | grep -oE '[a-zA-Z0-9/+=]{40,}' | base64 -d | \
  iconv -f utf16 -t utf8 | grep -oE '[a-zA-Z0-9/+=]{40,}' | \
  base64 -d | gunzip

CyberChef: The Visual Approach

For those who prefer a GUI, CyberChef handles multi-layer decoding with drag-and-drop operations:

1. From Base64 -- strip the outer encoding
2. Decode text (UTF-16LE) -- convert Microsoft's encoding
3. Regular expression -- extract the inner base64 blob
4. From Base64 -- decode the inner layer
5. Gunzip -- decompress
6. From Hex -> XOR -- if shellcode is hex-encoded with XOR key

Where to Find Encoded PowerShell in the Wild

Source	Event ID	What to Look For
System Event Log	7045	Service ImagePath containing encoded commands
PowerShell Operational	4104	Script Block Logging captures decoded content
Windows PowerShell	400, 403	Engine start/stop with HostApplication field
Scheduled Tasks	N/A	Task XML action elements
WMI Subscriptions	N/A	Consumer CommandLineTemplate

---

Identifying Cobalt Strike

Cobalt Strike is the most commonly encountered commercial C2 framework in real-world incidents. After decoding obfuscated PowerShell, Cobalt Strike payloads typically reveal:

• Named pipe patterns: \\.\pipe\status_XXXX or \\.\pipe\msagent_XXXX
• Beacon configuration extractable with tools like CobaltStrikeParser or SentinelOne's parser
• Characteristic shellcode headers -- identifiable byte patterns in the first few hundred bytes
• Malleable C2 profiles -- customised HTTP request patterns that mimic legitimate traffic

JARM fingerprinting can also identify Cobalt Strike team servers from their TLS configuration, though operators increasingly randomise these.

---

Common Malware Delivery Chains

Understanding typical delivery chains helps with attribution and scoping:

Stage	Example Chain
Phishing email	.docm with macro, .iso with .lnk
First-stage loader	Emotet, Qakbot, IcedID, BazarLoader
Second-stage framework	Cobalt Strike, Brute Ratel, Sliver
Final payload	Ryuk, Conti, BlackCat, LockBit

The attacker doesn't build everything from scratch. These chains represent a criminal supply chain -- different groups specialise in access brokering, framework development, and ransomware operations.

---

Building an Analysis Lab

A proper malware analysis environment needs:

• Isolated VM -- no bridged networking, no shared folders with the host
• Snapshot capability -- revert after every analysis session
• FlareVM or REMnux -- pre-built analysis distributions with tools installed
• FakeNet-NG or INetSim -- network simulation to catch C2 attempts safely
• Host-only networking -- communication only between analysis VMs

The cardinal rule: never analyse malware on a system connected to your production network. Not even briefly. Not even "just this once." Not even if you're "pretty sure" the sample is dead. The moment you think "it'll be fine" is the moment it isn't.

---

What's Next in the Series

• Windows Forensics 102: Execution Artifacts -- find evidence that malware ran, even after it's been cleaned
• Incident Response 103: Ransomware -- what to do when the malware has already won
• PowerShell Security Cheatsheet -- logging and detection for PowerShell abuse
• Threat Hunting 101: Methodology -- proactively find malware before it finds you