Isolation Forest: Anomalies Need Fewer Random Splits
Isolation Forest does not first learn what a normal cluster looks like. It repeatedly chooses a random feature and a random split value, then measures how many splits are needed to isolate each observation. The useful output is an anomaly ranking; the business threshold comes later.
Define the anomaly unit before fitting the forest
In the synthetic APM dataset, one row contains CPU, memory, latency, and throughput for one span. Five spans share the same trace and synthetic anomaly label. This creates two possible analysis units:
- Span level: score each service span independently.
- Trace level: aggregate or combine the five spans and score the request as a unit.
That choice changes both the feature table and the evaluation. If the target is a trace-level incident, a random row split is unsafe because sibling spans from one trace can enter both training and test data. Splitting by trace_id keeps an entire request on one side.
Random partitions create an isolation path
Each isolation tree is built from a random subsample:
- Select a feature at random.
- Select a split value between that feature’s minimum and maximum values.
- Send observations to the left or right child.
- Repeat until an observation is isolated or the tree reaches its height limit.
The path length is the number of edges from the root to the terminating node. Short paths indicate observations that were easy to separate from the rest of the sample.
Average path length becomes an anomaly score
Across multiple trees, the score is:
s(x, n) = 2^(-E[h(x)] / c(n))
E[h(x)]: average path length of observationxacross the forest.c(n): expected path length of an unsuccessful search in a binary search tree.n: subsample size.
A shorter average path produces a score closer to 1. A longer path produces a score closer to 0.
The tree-building logic is small
This pseudocode keeps the original recursive idea visible:
def build_iTree(X, height_limit, current_height=0):
if current_height >= height_limit or len(X) <= 1:
return ExNode(size=len(X))
q = randomly_select_feature(X)
p = randomly_select_value(min(X[:, q]), max(X[:, q]))
X_left = X[X[:, q] < p]
X_right = X[X[:, q] >= p]
return InNode(
left=build_iTree(X_left, height_limit, current_height + 1),
right=build_iTree(X_right, height_limit, current_height + 1),
split_attribute=q,
split_value=p,
)
The same observation is passed through every tree, and its path lengths are averaged before calculating the score.
def path_length(x, tree, current_height=0):
if isinstance(tree, ExNode):
return current_height
if x[tree.split_attribute] < tree.split_value:
return path_length(x, tree.left, current_height + 1)
return path_length(x, tree.right, current_height + 1)
Subsampling is part of the method
The original method uses small random subsamples; 256 observations is a common default when the dataset is larger than that. Small samples make trees cheaper to build and reduce the chance that many normal observations hide an anomaly inside a dense region.
The main parameters control different parts of the result:
- Number of trees controls how many random partitions contribute to the average.
- Subsample size controls the population seen by each tree.
- Maximum features controls which dimensions are available to each tree.
- The score threshold controls how many observations are finally labeled anomalous.
The four APM features also have different distributions. CPU and memory are bounded percentages, throughput is a count, and latency has a much wider upper range in the published data. Isolation Forest is less directly scale-sensitive than a Euclidean-distance method such as LOF, but skewed features and extreme ranges still affect where random partitions can isolate points. Feature distributions should be inspected rather than scaled by habit or ignored by habit.
A score is not yet an alert
Isolation Forest produces a ranking or score before it produces a business decision. A fixed threshold such as 0.5 or 0.6 is not automatically correct for every dataset. The useful threshold depends on the expected anomaly rate and, more importantly, how many false alerts can be reviewed.
This separation matters in practice: the forest generates anomaly evidence; the operating threshold decides what the team must investigate.
Evaluate the ranking at trace level and at the review budget
The synthetic is_anomaly field is evaluation ground truth, not a model feature. Since anomalies are the minority, accuracy can remain high even when the detector misses them. Evaluation should include precision and recall, a precision–recall curve, and performance at the number of alerts the team can actually review.
Span-level metrics answer whether individual service operations were ranked correctly. Trace-level metrics answer whether the request containing the anomaly was surfaced. Reporting both prevents five anomalous spans from being mistaken for five independent incidents.
Isolation explains separability, not root cause
A short path tells us that a point was easy to separate through random partitions. It does not prove which feature caused a production incident, and standard feature importance from a supervised model does not automatically apply. Root-cause analysis requires a separate explanation layer and evidence from the relevant trace or logs.
For this project, Isolation Forest is therefore one component: it ranks unusual metric observations. Trace reconstruction, threshold selection, and root-cause evidence remain separate decisions.