rava://case/SYSMON-ML-2026

$ load sysmon_dataset.csv

✓ 6,258 events · 3,236 benign · 3,022 burst

$ train RandomForestClassifier()

fittingdone

Accuracy on the test set

Precision 100%Recall 100%Errors 0

MISLEADING

RAVA / ISSUE 001 · INTERACTIVE INVESTIGATION

Misleading Performance in Sysmon-Based Machine Learning

A detector just scored perfectly. You have six questions between you and the reason it learned almost nothing.

Open the case file ↓Jump to verdict

100% PERFECT→≈61% SHORTCUTS REMOVED→82% BEHAVIORAL RESULT

Abstract

Near-perfect accuracy in Sysmon-based ransomware detection can be a measurement artifact of event-level representation. Identifier features carry most predictive weight; removing them collapses performance to roughly 55–61%. Behavioral aggregation is more honest, but compresses 3,022 attack events into only three malicious windows—a phenomenon this work identifies as temporal collapse. Balanced evaluation yields 82%, the first trustworthy figure in the study.

Research questionDoes the representation of Sysmon telemetry determine apparent model performance?

Primary findingRepresentation—not algorithm choice—dominates the evaluation outcome.

Reader taskCommit to each prediction before revealing the corresponding evidence.

Question / 01

Why was it 100%?

A confusion matrix can look flawless while the model learns the wrong thing. Feature importance reveals what the headline metric conceals.

View as

EVIDENCE AFeature reliance

ThreadId46%

ProcessId41%

EventId7%

Other6%

ThreadId and ProcessId carry 87% of the importance. They identify processes; they do not describe behavior.

EVIDENCE B“Perfect” matrix

Pred. benignPred. maliciousActual benign✓TN0FPActual malicious0FN✓TP

On repetitive event-level telemetry, zero errors are the first clue—not the final proof.

So the score rests on two identifier columns.

Remove the leak. Then what?

↓

Q2 · LAB / 01REMOVE IDENTIFIERS

Commit before reveal

What happens when ProcessId and ThreadId are removed?

View as

Accuracy after removing IDs100%Choose a hypothesis, then run the experiment.

ThreadIdREMOVED

ProcessIdREMOVED

EventId1.00 importance

The result remains sealed until the reader commits to a prediction.

Question / 03

Why can’t events carry meaning?

The class counts look balanced, but behavior lives in time. One ransomware-like action exploded into thousands of near-identical events inside roughly three seconds.

View as

FULL TIMELINE · ONE POINT PER EVENT

Benign · distributed6,258 / 6,258 events Burst · ≈3 seconds

THE INSIGHT

Event frequency is not behavioral diversity. Thousands of copies of one action over-represent that pattern and hand the model repetition to memorize.

The fix: summarize events inside fixed time windows.

Aggregate behavior. What survives?

↓

LAB / 02Predict what survives a 10-second window.

View as

6,258 EVENTS ↓ GROUPED INTO BEHAVIORAL WINDOWS

Original events6,258

Behavioral windows130

Benign windows127

Malicious windows3

Temporal collapseA high-frequency attack produces thousands of events but occupies too little time to survive aggregation as a meaningful number of training samples.

Question / 05

Train on the collapsed data.

With 127 benign windows and only three malicious windows, what will the model’s attack recall be?

LAB / 03 · PREDICT MALICIOUS RECALL

Two perfect scores. Zero real learning.

Balance it, and meet the truth.

↓

View as

RANDOM FOREST · 10 S WINDOWS127 : 3

Accuracy100%MISLEADING

Malicious precision—

Malicious recall0.00

False positives0

Missed attacks3

All three malicious windows are missed. The headline remains perfect only because the majority class overwhelms the metric.

Investigation / 06

The pattern survives every model.

Select any result. The repeated arc rules out algorithm choice as the primary explanation.

ModelEvents + IDsEvents − IDsWindows rawWindows balanced

Random Forestprimary classifier

Logistic Regressionlinearity check

SVM (RBF)kernel method

XGBoostcomplexity check

Isolation Forestunsupervised baseline

Random Forest100%

Identifier leakage dominates the perfect event-level score.

Verdict

Representation governs performance. Not the algorithm.

You watched a perfect score collapse when two identifier columns vanished, watched 3,022 events compress into three windows, and watched imbalance manufacture a second fake 100%. The cross-model evidence repeats the same result.

01Event-level modeling misleadsIdentifiers and repetition let models bind labels to shortcuts.

02Aggregation is honest, but collapsesBehavior enters the representation while burst samples disappear.

03Balanced evaluation reveals the task82% is useful because its errors are visible and interpretable.

Research replay

Months of investigation, compressed into twelve decisions.

The clean result was not the first result. Replay the wrong turns, premature celebration, and representation changes that produced it.

CHAPTER 01 / 12

The lab

Build a safe crime scene.

An isolated Windows VM captures process, network, file, and registry activity while a controlled PowerShell script produces the ransomware burst.

VM → Sysmon → EVTX → CSV

Methods & boundaries

What produced the evidence.

Controlled collection

An isolated virtual machine captured Sysmon process creation, network connection, file creation, and registry activity while a PowerShell script simulated ransomware behavior.

Four representations

Models were evaluated on event-level data with identifiers, event-level data without identifiers, unbalanced aggregated windows, and balanced aggregated windows.

Stress tests

Five-, ten-, and thirty-second windows were compared across Random Forest, Logistic Regression, SVM, XGBoost, and an Isolation Forest baseline.

Limitations

The dataset is controlled and compact, the simulated burst lasts roughly three seconds, and oversampling does not reproduce the diversity of real malicious behavior. External validation remains necessary.

Cite this work

Keep the article connected to its source.

Esmaeili, K., & Ghafarian, A. (2026). “Misleading Performance in Sysmon-Based Machine Learning: The Impact of Data Representation and Temporal Structure.” University of North Georgia.

Open access · CC BY-NC-SA 4.0 · Under review