Misleading Performance in Sysmon-Based Machine Learning
Why did this happen? A ransomware detector trained on Windows Sysmon events just scored a perfect 100% on every metric, and the paper behind this experience proves it learned almost nothing. You have six questions between you and the answer.
Why was it 100%?
First move in any investigation: ask the suspect what it was paying attention to. Feature-importance analysis interrogates the Random Forest directly, which columns actually drove its predictions?
What the model relied on
ThreadId and ProcessId carried nearly all the predictive weight. Those are identifiers, labels the operating system assigns, not behaviour. The model matched IDs to classes the way you'd memorize answers, not learn material.
The “perfect” confusion matrix
Zero errors in both classes. In event-level Sysmon data, a flawless matrix is not a triumph, it's the first clue.
Event-level leakage
The reliance of machine learning models on identifier-based or repetitive event features (e.g., ProcessId, ThreadId) that correlate with class labels but do not represent meaningful behavioural patterns.
Remove the leak. Then what?
The experiment: delete ProcessId and ThreadId, retrain on the same balanced event-level data. Before you run it, commit to a prediction. That's how science works.
Accuracy without identifiers
Where the importance went
Confusion matrix · Random Forest
Accuracy crashed to ~61% (RF), ~55% (LR), ~60% (SVM, XGBoost). And the model didn't start learning behaviour, it just grabbed the next shortcut, putting all of its weight on EventId (1.0). The perfection was never real.
Why can't events carry meaning?
Look at the crime scene itself. The dataset's counts are balanced, 3,236 benign vs 3,022 burst. But behaviour lives in time, and time tells a completely different story: one ransomware-like action exploded into thousands of near-identical file-creation events inside roughly 3 seconds.
Event frequency is not behavioural diversity. Treated as independent samples, thousands of copies of one action over-represent that pattern and hand the model repetition to memorize. Events are components of behaviours, so the representation should describe behaviours, not events.
Aggregate behaviour. What survives?
Behavioural aggregation groups events into 10-second windows, each described by summary features, total event count, number of unique event types. Predict the outcome before you watch it.
Train on the collapsed data.
Random Forest, aggregated dataset, natural class distribution: 127 benign windows, 3 malicious. Last prediction before the twist.
The result
The model predicted every single window as benign. With 127 of 130 windows in one class, that alone buys ~perfect accuracy. Evaluation even warned that only a single label appeared in the predictions.
Confusion matrix · all predictions benign
The 100% from the cold open and this 100% are two different failures, leakage there, imbalance here, with the same root cause: representation.
Balance it, and meet the truth.
Random up-sampling duplicates the minority class until both sides hold 127 windows. No new information is invented, it simply stops the majority class from drowning the signal. Press the button.
Random Forest · balanced aggregated (10 s)
82%, and it's the best number in this entire investigation. Zero false positives. Nine malicious windows genuinely missed. Real precision–recall trade-offs are what learning actually looks like. Window size barely matters: ~80% at 5 s, ~75% at 30 s.
An honest confusion matrix, at last
23 TN · 19 TP · 9 FN · 0 FP. Imperfect, informative, trustworthy.
You just concluded what the paper concludes: representation governs performance. Not the algorithm.
You watched a perfect score collapse when two ID columns vanished, watched 3,022 events compress into 3 windows, and watched imbalance manufacture a second fake 100%. Nobody told you the conclusion, the evidence did. The paper's cross-model experiments confirm it below.
Corroboration · five models, four representations, tap any cell
Replay the research.
The investigation you just ran took the authors months. Watch the journey unfold the way it actually happened, including the celebration that turned out to be premature.
Three findings worth carrying out of the case file.
Event-level modelling misleads
Feature leakage and structural bias let models bind identifiers and repetition to labels, near-perfect metrics with no real detection capability.
Aggregation is honest, but collapses
Behavioural windows force learning over time, yet compress high-frequency attacks into a handful of samples: temporal collapse and severe imbalance.
Consistent across all five algorithms
Random Forest, Logistic Regression, SVM, XGBoost, Isolation Forest, same pattern everywhere. Data representation, not model selection, is the dominant factor.
Open leads for future investigators
Adaptive or variable-length windows to preserve temporal structure without crushing bursts; sequence models (RNNs, transformers) and graph-based approaches linking processes, files and network events; larger real-world attack datasets; and continued work on leakage reduction and interpretability.