Shannon Entropy Formula

I am a regular reader of the occasional posts on SportsTrader. The posts are signed Matekus and after a little searching I found a defunct blog about US horse racing by an "applied mathematician" called Matt Matekus. The first post on SportsTrader carries a scant biography.

According to my career script, I expected to become a successful, self-employed, consultant and to retire in late middle-age to the west coast and study mathematics. Actually, I achieved the initial career goal relatively quickly and lived the lifestyle I had anticipated for a short while only to be trumped by entropy.

(I have since discovered a forum poster by the name of JF Dinneen referring to the SportsTrader blog as his own so either Matt Matekus is a pseudonym or I am mistaken. It has no bearing on this article.)

Coincidentally, having been "trumped by entropy" a recent post by Matekus concerns entropy, which Wikipedia so handily defines as "a measure of the number of microscopic configurations Ω that correspond to a thermodynamic system in a state specified by certain macroscopic variables". Does that have you begging for more? Okay, then we shall continue.

There is a brief mention of entropy in Epstein's The Theory of Gambling and Statistical Logic, which also mentions my early work with genetic programming in the EDDIE project. Having read Epstein's brief mention of entropy in his book I wondered if there was any use for it in sports trading. I didn't see anything so we will delve further into what Matekus has to say but first a little more about what entropy is.

Initially, the term entropy was used in physics to describe the dissipation of useful energy within a natural system. Since then the term entropy has been used in all manner of ways. In physics entropy can be used to describe the heat death of the universe whereby stars convert the energy in hydrogen into heat, light and heavier elements. This process continues until all energy is dissipated (but not lost - First Law of Thermodynamics), the universe cools to near absolute zero and goes dark. There is no Restaurant at the End of the Universe as that would violate the first law.

The term entropy was borrowed by Claude Shannon the "father of information theory" (who gave rise or assistance to a host of gamblers) as a measure of information content. Matekus uses Shannon Entropy to determine a Wisdom of the Crowd Index. I am all for traders creating their own metrics rather than using the "out of the box" financial trading studies you find in third party sports trading software. Those studies have no place in sports trading (with the exception of the moving average) and it's debateable if they have any place in financial trading either.

Shannon uses entropy to measure how much information can be got out of a system. Imagine a baby who has just learned to say "da". The baby has a vocabulary of just one word and uses it to say, "da da da da da" all day long. There is little to no information in that single word. Does it means "dad", "hello" or "I just filled my nappy"?

As the baby grows older they will add words to their vocabulary and might eventually say, "Me hungry". Not gramatically correct but the increasing vocabulary allows for more information to be conveyed. Finally, the child will become an adult and on a business trip might phone to hotel reception and say, "Can you bring my breakfast to room 542 at 8:30AM. I would like toast, cereal, orange juice and coffee. Thank you." The adult vocabulary has increased further with a much greater information content.

Shannon Information is given by


where p is the probability of the ith piece of information occuring.

For a baby with a vocabulary of one word the probability that the baby will say that one word is 1.00 therefore the Shannon Information is zero.

If we have an unbiased coin the Shannon Information is

p(heads) = 0.5 & p(tails) = 0.5

Therefore H= -[(0.5 log2 0.5) + (0.5 log2 0.5)]

= 1.0

For a biased coin

p(heads) = 0.6 & p(tails) = 0.4

Therefore H= -[(0.6 log2 0.6) + (0.4 log2 0.4)]

= 0.97

Interesting? No, I don't think so either. The reason being is that this measure is a trailing indicator. It makes no prediction about future prices, which is what we are trying to do when finding edge. After all, if you know a coin is biased then you are going to jump on every bet with Kelly Criterion and to hell with the "entropy of a closed system".

Joseph Buchdahl, author of How to Find a Black Cat in a Coal Cellar, delved deeper into the effectiveness of Shannon Information with an article that compared opening and closing odds.

All the Wisdom of the Crowd Index tells us is how mixed up the market prices are. From zero, if all the odds are the same to 1.0 when a winner is known. I don't see how entropy is related to the Wisdom of the Crowd. They are two different things altogether.

Wisdom of the Crowd is the act of information entering into a system, the more the better. A single bettor won't add much information to a betting market, thousands will. Entropy does not take into account the number of bettors. A single bettor could place a few bets and move prices away from their true values. What we know from Efficient Market Hypothesis is that by the time the event has started the large number of bettors in a betting market have generated a pretty good idea as to what the true odds should be. 

Whilst I enjoy reading SportsTrader, sometimes the website appears to be just a series of thought exercises rather than anything useful. Still, each and every post makes me think that little bit harder.

No comments:

Post a Comment