# When entropy meets Shannon

**[splunk.com/en_us/blog/tips-and-tricks/when-entropy-meets-shannon.html](https://www.splunk.com/en_us/blog/tips-and-tricks/when-entropy-meets-shannon.html)**

April 21, 2016

By [Splunk April 21, 2016](https://www.splunk.com/en_us/blog/author/admin.html)

This is the third post on URL analysis, please have a look at the two other posts for more
context about what can be done with Splunk to analyze URLs:

[Splunking 1 million URLs](http://blogs.splunk.com/2016/03/22/splunking-1-million-urls/)
[Hunting that evil typosquatter](http://blogs.splunk.com/2016/04/01/hunting-that-evil-typosquatter/)

You will find in this article information on how one can detect DNS tunnels. While you can
find [lots of very useful apps on Splunkbase to help you analyze DNS data, it is always good](https://splunkbase.splunk.com/apps/#/page/1/search/dns/order/relevance)
for curious individuals to discover some techniques being used underneath.

A lot of captive portals are bypassed everyday by anyone able to run a DNS request, if
someone can run on their machine the following command:
```
 $ host splunk.com
 splunk.com has address 54.69.58.243
 ...

```
Without being authenticated on the captive portal, then they can use any service on the
[internet using a DNS tunnel. There are a lot](https://github.com/iagox86/dnscat2) [of](http://wiki.kartbuilding.net/index.php/Dnstunnel) [tools](http://code.kryo.se/iodine/) [out](https://www2.deloitte.com/fr/fr/pages/risque-compliance-et-controle-interne/articles/cyber-academy.html/ressources/outils/dns2tcp/.html) [there to create those tunnels. And](https://github.com/FedericoCeratto/dnscapy)
[for a great paper on the topic, I encourage you to read the Detecting DNS Tunneling from](https://www.sans.org/reading-room/whitepapers/dns/detecting-dns-tunneling-34152)


-----

SANS Institute.

## Claude Shannon to the rescue!

Long time
ago, the
venerable
Claude E.
Shannon
wrote the
paper “A
Mathematic
[al Theory of](http://math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf)
Communicat
ion“, which I
strongly
encourage
to read for
its clarity

Claude Shannon

and

By Jacobs, Konrad - [https://opc.mfo.de/detail?photo_id=3807,](https://opc.mfo.de/detail?photo_id=3807) [CC BY-SA 2.0 de,](https://creativecommons.org/licenses/by-sa/2.0/de/deed.en) [Link](https://commons.wikimedia.org/w/index.php?curid=45380422)

amazing
source of
information.


[He invented a great algorithm known as the Shannon Entropy which is useful to discover](https://en.wikipedia.org/wiki/Entropy_(information_theory))
the statistical structure of a word or message.

If you consider a word, being a discrete source of the finite number of characters type which
can be considered, for each possible character there will be a set of probabilities which
would produce various outputs. There will be an entropy for each character. This entropy on
the chosen word is defined as the average of the output weighted on the probability of
occurrence of the characters.

The previous paragraph can easily be translated into the following Python code (taken from
the excellent URL Toolbox on Splunkbase:
```
def shannon(word):
  entropy = 0.0
  length = len(word)

```

-----

```
  occ {}
  for c in word :
    if not c in occ:
      occ[ c ] = 0
    occ += 1
  for (k,v) in occ.iteritems():
    p = float( v ) / float(length)
    entropy -= p * math.log(p, 2) # Log base 2
  return entropy

```
Which can be run directly from any word you can have in Splunk:

As you can see, the score is pretty high, which makes sense since there is a high variety of
frequency over those data. If we click on the ut_shannon field to sort in reverse order, this is
what you could get:


-----

As one can see, words of low characters distribution get a low score.

## Catching DNS tunnels from subdomains in URLs
```
If we run the following query, interesting results are shown:
sourcetype="isc:bind:query" | eval list="mozilla" | `ut_parse(query, list)` |
`ut_shannon(ut_subdomain)` | table ut_shannon, query | sort ut_shannon desc

```

-----

As you can see in the results here, the high score come from tunnels made to the domain
ip-dns.info as well as something which is unknown but could also be a tunnel: traffic
towards greencompute.org

I hope this post helps you to see tools and methodologies one can use to find out unusual
activity strictly based on the DNS traffic. More to come…

---------------------------------------------------Thanks!
Sebastien Tricaud

Posted by

**[Splunk](https://www.splunk.com/en_us/blog/author/admin.html)**


-----