Entropy/IP: report for dataset S2
(Server IPv6 Addresses)
(How did we find this? Click to show the full report)
Entropy vs. 4-bit Aggregate Count Ratio (ACR)
First, we estimate the Entropy for each nybble in the IPv6 addresses, across the whole dataset. For example, if the last nybble is highly variable, then the corresponding Entropy will be high. Conversely, the Entropy will be zero for nybbles that stay constant across the dataset. Below we plot the normalized value of Entropy for each of the 32 nybbles, along with the 4-bit Aggregate Count Ratio, which was introduced in Plonka and Berger, 2015.
Second, we group adjacent nybbles with similar Entropy to form larger segments, with the expectation that they represent semantically different parts of each address. We label these segments with letters and mark them with dashed lines in the plot below.
Segment Mining
Next, we search the segments for the most popular values and ranges of values within them. For that purpose, we use statistical methods for detecting outliers and the DBSCAN machine learning algorithm. We analyze distribution and frequencies of values inside address segments.
Below we present the results, with ranges of values shown as two values in italics (bottom to top). The last column gives the relative frequency across the whole dataset. The /32 prefixes are anonymized.
A: bits 0-32 (hex chars 1- 8)B: bits 32-40 (hex chars 9-10)
- 20010db8 21.90%
- 30010db8 10.64%
- 40010db8 9.50%
- 50010db8 7.73%
- 60010db8 4.21%
- 70010db8 0.30%
- 80010db8 0.21%
- 90010db8 8.12%
- a0010db8 6.95%
- b0010db8 5.62%
- c0010db8 4.92%
- d0010db8 3.87%
- * e0010db8-e0010dbf 14.89%
- * f0010db8-f0010dbf 1.14%
C: bits 40-44 (hex chars 11-11)
- 00 75.21%
- 01 6.88%
- 4c 2.48%
- 20 2.35%
- 24 1.82%
- 10 1.80%
- 0c 1.45%
- * 02-ff 8.00%
D: bits 44-48 (hex chars 12-12)
- 0 53.90%
- 1 20.96%
- 2 7.04%
- f 2.74%
- 7 2.68%
- 3 2.40%
- * 4-e 10.27%
E: bits 48-52 (hex chars 13-13)
- a 12.13%
- 0 10.38%
- 8 9.39%
- 4 8.20%
- 7 6.47%
- d 6.07%
- * 1-f 47.36%
F: bits 52-56 (hex chars 14-14)
- 0 95.91%
- f 1.99%
- 9 1.10%
- 8 0.81%
- * 1-7 0.18%
G: bits 56-60 (hex chars 15-15)
- 1 41.50%
- 2 24.26%
- 0 13.59%
- 3 11.41%
- 4 5.21%
- * 5-f 4.04%
H: bits 60-64 (hex chars 16-16)
- 8 53.80%
- 9 25.55%
- 0 14.04%
- a 5.55%
- b 0.41%
- 1 0.41%
- f 0.12%
- 2 0.11%
I: bits 64-96 (hex chars 17-24)
- 0 18.21%
- 1 9.40%
- 2 7.30%
- 3 6.61%
- 4 6.44%
- * 5-f 52.03%
J: bits 96-112 (hex chars 25-28)
- 00000000 96.50%
- 84000000 0.08%
- 9b000000 0.08%
- 9a000000 0.07%
- 92000000 0.07%
- 96000000 0.06%
- 91000000 0.06%
- 89000000 0.06%
- 83000000 0.06%
- 95000000 0.06%
- * 0000003f-f88b5b88 2.90%
K: bits 112-116 (hex chars 29-29)
- 0000 87.48%
- 58dd 0.49%
- 5f65 0.44%
- adcd 0.34%
- b833 0.29%
- 0210 0.29%
- 48f6 0.24%
- 0216 0.22%
- 451f 0.22%
- b832 0.21%
- * 17c0-17de 0.63%
- * 0001-ffd8 9.16%
L: bits 116-128 (hex chars 30-32)
- 1 31.05%
- 0 30.36%
- 2 28.86%
- * 3-f 9.73%
- d77 2.15%
- 168 1.84%
- 768 1.61%
- e59 1.38%
- aca 1.37%
- b13 1.04%
- 15e 1.04%
- c1a 1.00%
- 2ef 1.00%
- fc4 0.94%
- * 6d0-ddb 41.70%
- * 000-ffb 44.93%
Bayesian Network Structure
Next, we search for statistical dependencies between the segments. For that purpose, we train a Bayesian Network (BN) from data.
Below we show structure of the corresponding BN model. Arrows indicate direct statistical influence. Note that directly connected segments can probabilistically influence each other in both directions (upstream / downstream). Under some conditions, segments without direct connection can still influence each other through other segments: e.g., A can influence C through B if C depends on B and B depends on A (even if there is no direct arrow between A and C).
Learning BN structure from data is in general a challenging optimization problem. Hence, there might be more than one possible BN structure graph for the same dataset.
Conditional Probability Browser
Finally, below we show an interactive browser that decomposes IPv6 addresses into segments, values, ranges, and their corresponding probabilities. The browser lets for exploring the underlying BN model and see how certain segment values probabilistically influence the other segments.
Try clicking on the colored boxes below. You should see the colors changing, which reflects the fact that some segment values can make the other values more (or less) likely. For instance, in the Sample Report, you may find that clicking on J1 (i.e., the first value in segment J) makes segments C, D, F, H, and I largely predictable (see our paper for more examples).
You may condition the model on many segment values. Clicking on selected values un-selects them. Clicking on the red "Clear" above the color map un-selects them all. Below the browser we show the estimated proportion of the addresses matching your selection (vs. the dataset). If the browser cannot estimate the probabilities in a reasonable time, it asks before trying harder.
Candidate Target Addresses
Using the BN model, below we generate a few candidate target IPv6 addresses matching the selection above. Note that we anonymize the IPv6 addresses in this report.
As we show in the paper, this technique allowed us to successfully scan IPv6 networks of servers and routers, and to predict the IPv6 network identifiers of active client IPv6 addresses.