Be Wary When Streaming Betfair Data

Back in January of this year, I mentioned that I had become one of Betfair's beta testers for their new data streaming service. I had a look but the more I thought about it, the more I thought it might not be such a good idea for the majority of traders and maybe not for Betfair either.

Previously, Betfair's data had to be requested from its servers via a process called polling. In polling, a timed process on the client side (trader's software) asks Betfair for the latest market data at a fixed time period (e.g. once a second). With streaming, a client now requests a stream for a market and the data is pushed by the server to the client whenever the data has been updated, which can be at any rate depending on market updates (e.g. the minute plus rate for markets far from going in-play to the millisecond rate for markets about to go in-play or that are in-play).

For the user, data streaming offers almost instantaneous market updates without having to request them from the server, save for the initial creation of the stream. This approach offers less work on the client side but for Betfair the workload increases because it now has to broadcast data continually rather than answering requests. I only hope that Betfair is upto the task.

At present, I have no need for streamed data. Some of my trading models are based on a one minute polling cycle and a few on a one second cycle. Having data streamed to my trading models at a rate far faster than they have been optimised will cause framing problems. In my latest book, Betfair Trading Techniques, I discuss the framing problem whereby using different rates at which data is sampled during model testing and model implementation can affect profitability. 

A model based on polling data at the one minute rate could spook a trader, trading a model whilst sitting by the screen watching data fly by at the one second rate or faster. The model will still be profitable at the one minute rate but if a manual trader gets nervous looking at the noise at the one second rate they might trade out for a loss too often and ruin the profitability of the model.

Think of it in terms of a signal to noise ratio. A trend or oscillation is the signal but noise in the form of sudden counter moves at the sub-second level might make create a false signal, forcing a trading model to trade out when the underlying signal is still valid. You have to be wary when told that streaming offers you data "ten times faster" as a badly programmed bot can end up losing your money ten times faster.

Anything faster than one second is for algorithmic trading and then only if the server running an algo-bot is optimally placed to process the stream and act upon it before others. I can't imagine a manual trader being able to look at data flashing by in less than a second being able to make any sense of it.

With data streaming, Betfair has ushered in the high-frequency trading era for sports trading and that is not good for the future of manual traders. The same has come to pass for financial trading. I believe that sports exchanges are going the way of financial exchanges, where the majority of trading is performed electronically. Manual trading will be for the casual punter or the low-frequency fire and forget bettor.

High-frequency trading will offer opportunities for a new breed of trader, the technical trader who has no interest in the underlying sport and just trades the numbers and market structure. Algorithmic trading offers a route into sports trading for those with a logical, mathematical mind, used to programming and problem solving.

The more algorithmic traders on the exchanges the better, as it will increase liquidity. Algorithmic trading offers the challenge of pitting your skills against others, experimenting with new ideas and new ways of looking at the trading problem. This is precisely why I have published two books, Programming for Betfair and Betfair Trading Techniques, to provide tools and to give aspiring algorithmic traders a head start.

Further Reading

Betfair Trading Techniques - Trading Models, Machine Learning, Money Management, Monte Carlo Methods & Algorithmic Trading. Betting exchanges are becoming ever more like financial markets. This has seen the rise of technical traders who find new and inventive ways of trading, little of it having anything to do with the underlying sports. Manual traders are having to give way to automation and algorithmic trading.

To stay ahead, the most successful traders are resorting to systematic and automated methods to build and trade their strategies. This book demonstrates techniques for sports trading, including; fundamental and technical trading, statistical arbitrage, money management, Monte Carlo methods, machine learning and the increasing necessity for algorithmic trading.

Included in the book is application code so that traders can build two trading tools to assist in their pursuit of edge. Contents List

Programming for Betfair - A Guide to Creating Sports Trading Applications with API-NG. The Betfair exchange, coupled with its API, permits a suitably skilled trader to code complex trading applications, which would not look out of place in the financial markets. This book offers a sports trader the chance to build their own trading applications, regardless of their programming ability.

Each chapter of Programming for Betfair contains snippets of code that combine to create a complete trading application. The application is geared towards horse racing but can easily be adapted to other sports on Betfair's exchange. Using Microsoft's Visual Studio (downloadable for free) the reader is shown how to code an application that will gather prices for any market on Betfair's exchange and then place bets into that market. 

The reader is shown how to automate their trading so that they can remove emotion from their trades and scale up their trading for increased profits. Further development of the application permits it to save data from Betfair onto the reader's hard drive for offline analysis and visualisation in a spreadsheet for the purpose of building trading algorithms. 

Also covered is an enhancement of Betfair's charts so that charts can be automatically updated and compared. The final chapter of the book discusses ideas for taking the application and the reader's skills to the next level. Topics discussed include constructing your own trading indicators, volume analysis, trend following, arbitrage, low-latency trading and many more. Contents List

Betfair Trading Techniques

I have recently published Betfair Trading Techniques, which I hope will be of interest to sports traders. The book is directed at a wide range of abilities and tastes. Beginners will be expected to know the basics of sports trading and betting. Intermediate traders and bettors can further improve their skills and I hope even experts will find something new to think about.


Betting exchanges are becoming ever more like financial markets. This has seen the rise of technical traders who find new and inventive ways of trading, little of it having anything to do with the underlying sports. Manual traders are having to give way to automation and algorithmic trading.

To stay ahead, the most successful traders are resorting to systematic and automated methods to build and trade their strategies. This book demonstrates techniques for sports trading, including; fundamental and technical trading, statistical arbitrage, money management, Monte Carlo methods, machine learning and the increasing necessity for algorithmic trading.

This book comes with application code so that readers can program a money management system using Monte Carlo methods and a machine learning application for building trading models. All programming is done within Microsoft's free of charge Visual Studio environment. Spreadsheet software (Excel or openware) will be required too.

Amazon - Betfair Trading Techniques

Screenshots from the applications.



Chapter 1 – Information is Power 

Betting and Trading
Fundamental and Technical Trading
Is All Sports Trading and Betting Arbitrage?
Data Requirements

Chapter 2 – Tools of the Trade 

Spreadsheet Software
Microsoft Visual Basic
Web Scraping With HTML Agility Pack
Trading Software
Additional Skills

Chapter 3 – Trading Models 

Entries and Exits
Data Lore
Market Efficiency
Wisdom of the Crowd
In-Play Trading
Measuring a Model's Predictive Ability
     Return and Yield 
     Strike Rate 
     Winning and Losing Streaks 
     Sharpe Ratio 
     Walk Forward Analysis 
Time Frames
Monitoring and Refinement
Protecting Edge

Chapter 4 – Fundamental Models 

Dutch Masters
Horse Racing
     Creating a Dutch Book 
     Finishing Times 
     Exotic Arbitrage 
Association Football
Remarks on Fundamental Modelling

Chapter 5 – Technical Models 

     Weight of Money 
     Weight of Flow 
     VWAP & TWAP 
     Moving Averages 
What Moves Prices?
Market Dynamics
     Price Tick Transitions 
     Resistance Lines 
     Associated Markets 
Bonus Arbitrage
Remarks on Technical Modelling

Chapter 6 – Money Management 

Trade Sizing
     Level Stakes 
     Percentage Stakes 
     Kelly Criterion 
Martingale and Other Madness
Analysis of Staking Methods
Market Capacity
     The Myth of Compounding 
Monte Carlo or Bust
     Using RiskSimulator 
     Further Experiments 
Which Staking System Should Be Used?
Portfolio Trading

Chapter 7 – Machine Learning 

With a Little Help From My (Human) Friend
Combinatorial Explosion
Genetic Algorithm
     Implementation – GA Optimiser 
     Pre-Processing the Training Data 
     Running the Application 
     Fitness Measures 
     Local Maxima 
Expanding the Application Further

Chapter 8 – Algorithmic Trading 

Why Trade Algorithmically?
Emotional Trading
First to Market Advantage
     Algorithmic Trading Software 
     Dedicated Trading Servers 
Scalable Income
Closing Comments


Microsoft Visual Studio Quick Reference
Starting a New Project
Adding Components to a Form
Adding Forms, Modules and Classes to an Application
Running the Application from the IDE
Creating a Standalone Application

Is there an eBook?

There is no eBook and no likelihood of there being one.

Need help with the book?

If you are looking for help on this page then you are reading the book too quickly. Start again from the very first page, read carefully and you will find the URL for the book's support web page.

Is it Luck or Undefined Skill?

On my daily surf through various sports trading websites I noticed a number of articles related to luck. I believe the author was attempting to answer accusations of being lucky. As a poker playing friend would say after the umpteenth bad beat handed to me, "I'd rather be lucky than good." The truth is that my friend is a very good poker player, as his $1,000,000 plus winnings during his 14 year career testifies to. His statement was just his way of brushing off inferiors.

As far as our sports trading friend is concerned, he is doing himself an injustice. He is not lucky. If he is lucky then he can expect to regress to the mean. That is the point of being lucky, it swings both ways. A fair coin tossed ten times might show nine tails and one head. If someone is gambling with a fair coin and putting their money on tails then they are lucky because in the long run they are going to break-even, the more they play the game.

A player at a roulette wheel might get lucky in the short-term but we know that in the long-run they are going to lose because the house has the edge. If our sports trader is a consistent winner over many years then it is safe to say they have a positive edge. Calling it luck means they are not able to define their skill. Something they need to address or they can fob people off like my poker playing friend with "I'd rather be lucky than good."

Modelling Market Efficiency With A Coin

I feel as though I am still not getting through to some people with regards to market efficiency. What can you do to convince people who possess a reality distortion field and who think they can beat games in which they have no edge? It is undeniable that in the long run the wisdom of the crowd predicts the correct probability for selections in a sports betting market.

Another problem I have is getting people to understand what "the long run" means. We can't judge the efficiency of a single betting market, any more than we can judge whether a coin is biased or not after a single toss. The markets and the coin have to be repeatedly tested.

To demonstrate what the long run is and its implications I created a simple coin tossing simulator in a spreadsheet.

The spreadsheet determines the ratio of heads to tails and then charts are created for various simulation trial lengths. The random number generator for the simulator is set up to simulate a fair coin and so we would expect the ratio of heads to tails to be even (1:1) but only in the long run.

If we look at a chart after 10 trials we see the ratio is nowhere near 1:1 (1.0 on the chart). In fact the chart looks more like 2:1, refusing to go below the 2.0 line. What can we say about the coin after 10 trials? Nothing. The same with 10 horse races. Our system might be winning after 10 races but is that down to luck or skill?

The simulation was continued for 50, 100, 500 and 1000 trials with charts displaying the heads to tails ratio. After 50 trials the chart was showing a trend towards 1:1 but could we now say the coin is fair? The actual ratio might be below the 1:1 line.

After 100 trials the chart appears to be settling above 1:1. Is the coin biased towards heads?

After 500 trials the chart is now showing a ratio for heads to tails of 1:1. The coin has reverted to the mean expected ratio. Our horse racing strategy might now be at breakeven but losing through commission payments.

And again after 1000 trials. We might be confident that the coin is fair and has regressed to the mean. However, run the simulation again and you might not get 1:1 after 1000 trials of a fair coin. I have run this simulation many times and after 1000 trials it can settle far from 1:1. I know that I have created the sim to be fair but it will take many more trials than 1000 to get consistent results. You can never be too sure about the coin.

These observations are true about sports betting markets too. With just a few races in a data set nothing can be said about the efficiency of market prices. Over hundreds of thousands of races, over many years, it is shown that prices tend towards their true value. Those who bet on horse races and who use starting prices should beware.

A bettor creating a strategy must understand that prices tend towards fair during the course of a market's lifespan. Most who think they have tamed the markets are either very lucky or haven't been trading or betting long enough for mean reversion to drag them back to breakeven (minus commission - i.e. a loss).

Most traders and bettors create systems with a limited life span and so they won't get the chance to take advantage of the long run. A losing system can have good luck or a winning system can suffer bad luck because they haven't been traded with long enough. You need an edge to have a chance of beating a market in the long run. As the simulation shows a negative edge can initially look like a positive edge and vice versa but in the long run you will mean revert if there is no edge and you will lose through commission losses.

Another way of looking at the charts above is that they are time lines towards the start of horse races. When the line reaches the 1:1 ratio the market has happened upon the long run efficient prices. Before then everything is up for grabs. You can bet on value or you can trade the mean reversion. But what if you don't know if the coin is fair or not. What then? Maybe the coin is biased 2:1 in favour of heads but you don't know that it is.

This is the main reason why I do not waste my time studying form and betting on fundamentals. I much prefer technical trading, working with mean reversion rather than against and preying upon other traders who get caught up in an exuberant market. I don't know what a price should be, I only know that when a race is about to start, prices will be fair in the long run but until the race starts there is the opportunity to take advantage of the indecision of the crowd until wisdom takes over.

There are some good fundamental traders who can determine the optimal time to bet. Usually they have access to private data and the odds are against you being such a person.

Lucy, Lie Down and Take a Pill Love

Following on from my recent article on How to Read a Sports Trading Website I shall take a look at a prime example. We have all heard the old marketing maxim, "Sex Sells!" If you want to sell something to men then the best way to do it is with a louche lady. Take a look at Lucy Collins, with her head cocked disdainfully. I bet you'd read anything she wrote.

Of course, that's if Lucy is a lady. You see, online people can be anyone they want to be. A man can be a lady. A lady can be a man. An ageing Crystal Palace fan can even be a 17th century mathematician, if they want.

Luckily, Lucy has given us a link to her website, Already, the alarm bells are beginning to ring with a URL like that. Lucy generously provides us with an About web page, in which she tells us that she is from a "horsey background". Immediately, I have images in my mind of playful gymkhanas on her father's estate. Lucy has also kindly supplied us with an address to see the estate, which I enter into Google maps. Oh. I see father has fallen on hard times and is now living in a rather run-down semi in Sheffield. Never mind, upwards... and occasionally downwards.

There is also a business name attached to the address and like all businesses they must file accounts so off we go to do a company check on CityMan Business Solutions Ltd. However, I see no mention of dear Lucy. Could Nigel be having some identity problems? I am doubting the identity of Lucy even more. Let's have a look at the content of Lucy's website.

Oh dear. Page after page after page of affiliate links. Only if something can be linked to via an affiliate link is it "reviewed". Of course, the reviews are always gushing, even if the system is obviously rubbish. Without a positive review nobody is going to click the affiliate link and lucky Lucy doesn't get to hear, "kerching, another idiot mugged!" There is also the classic eBook that can only be bought through the vendor's website and not through an independently run site like Amazon.

I have no idea who is behind this Lucy character but if it's gambling related then it is probably a male. A male who doesn't know much about sports betting and trading but who does know how to get mileage from affiliate programs. The business name suggests that "Lucy" is associated with the business owner. I can't believe an IT contractor would waste their time on this drivel unless they are not very good at IT so maybe it's a son.

As I said in How to Read a Sports Trading Website if a website deletes pages then you can use the Wayback Machine to look for them. However, any attempt at using the Wayback Machine for will not return any pages as the owner of the site has set up a robots.txt file to stop certain spiders and robots from indexing the site and that includes the Wayback Machine. Something to hide, maybe?

I am sure the majority of you looking at the site will regard it as amateurish and an obvious attempt at making money from the gullible. Of course, all publicity is good publicity so I've upped the page view count for the young lad over the next week or so. But as you will know by now, the website is just a click bait website. Nothing to see here. Move along.

Was it Something I Said?

A certain CEO of a multi-billion dollar global empire (not Cassini) has again deleted his trading blog and, this time, his Twitter account too. Most likely he came to his senses, realising that earning obscene amounts of money as a corporate CEO is a lot easier than throwing in the "job" and living off his wits as a sports trader, earning about 1/1000th of a CEO's net salary.

If it was something I said then I am truly...... glad I said it. I don't write articles to suck up or gloss over. It's part of what I am. The subtitle of this website is "Helping sports traders to achieve their potential" and you don't help people by telling them trading is easy, that anyone can do it and that everyone is a winner.

Assuming the average player on Betfair pays 5% commission then if we are "all winners" then we are all still losers to the tune of 5%. If we are all to do equally well then we all break even minus the juice Betfair takes for itself. It's as simple as that. Every profitable winner needs more than one loser to keep their head above water.

If I think someone is talking out of their A, be they traders, vendors or teachers (unqualified) then I will say so. I know that I have ruffled feathers without mentioning names and yet, I have seen more than one trader, vendor or "teacher" get upset. Why is that, if I don't mention any names? A guilty conscience?

Not everyone can trade. In fact, most can't. Anyone who says they can take a random person off the street and make a trader out of them is lying. Look at the Million Dollar Traders video again. The majority hadn't a clue, even after training. The ones that could trade didn't make a fortune. To be paid off, a big winner needs many losers.

Trading, as with face to face gambling such as poker, is an unpleasant game. You are there to clean people out. I had no real friends in the casinos in which I played poker during the 1990s. I knew poker playing actors. I knew Sir Clive Sinclair. I knew Victoria Coren. We were cordial. We put on our masks. We went to work trying to mug each other. When I see a new bot getting cleaned out because its programmer is incompetent, I don't think "Poor chap", I think, "I want some of that." Just as I did when a beginner or tourist sat down at the poker table.

The only time I am considerate is when I write articles or books. In these situations I am a teacher (qualified - graduate teacher at university and adult education college) and I am attempting to impart knowledge without handing over my edge on a silver platter. If I don't think I can produce content that is any good enough for you, the reader, then I don't bother. You won't find me making a scalping video and then saying, "See what I did? Easy! Now go away and replicate what I did." (Whilst I delete the videos that didn't work out the way I wanted them to.)

Another crossover I notice from poker is that many people like to boast about their winnings but never talk about their losses. Rarely do you hear people talking about consecutive losing trades any more than a poker player talks about losing runs. No, all you hear about is the winning. You don't sell product if you list its failures. Negative advertising doesn't work in this game.

Trading blogs come and go but mostly go. There is no hiding. You can always use the WayBack Machine to reminisce about those who fell by the wayside. Especially if they make a come back then you can point out to them all their previous losses that they have decided to ignore. I'm looking forward to a new blog, maybe by someone who has now moved up through the boardroom to become the owner of a multi-billion dollar global corporation. And yet, they will always have that yearning to be roughing it with us plebs.

Some feel they have a boastful duty to tell everyone that trading is easy. Maybe they have something to gain (financially or egotistically) from saying so. I feel I must tell people how difficult it is with 90% being the often quoted figure as the failure rate for traders. Strange how those who tell you how easy it is to profit from trading never give you that figure. Why not ask them why?

I Can't Do It When You're Watching Me!

A curious post "Luck and Skill in Trading..." from Peter Webb on his vendor blog who says, "Of course, you can’t guarantee that you get it right and also talking and trading is pretty hard. When trading you need no distractions to be able to trade effectively. A room full of people is a major distraction!" 

I wonder how traders in financial markets manage to trade sitting next to each other in huge trading rooms with all the noise, hustle and bustle, let alone the former open outcry traders pushing and shoving in their trading pits? Come bonus time and a trader doesn't get any extra for his year's work can he say to his manager, "I would have made more but everyone was looking at me!"

[EDIT: Two can play at this game. Financial traders have noisy squawk boxes, television screens and other traders shouting orders into phones whilst they themselves trade. 14 winning trades in a row is nothing if you have been trading for a long time. I have tossed a coin and gotten 14 heads in a row but that doesn't make me a genius tosser. Just a tosser!]

Jospeh Buchdahl discusses luck and skill in his excellent book Squares & Sharps, Suckers & Sharks and, as usual, he backs his statements up with facts and figures rather than anecdotes. What Joseph more than amply demonstrates is that at the top of the game, be it in sport or gambling on sport, is that as the skill gap closes the luck factor increases. 

Soccer has always been a little backward in going forward, which you can read about in Soccernomics. Sport science and big data has only recently entered the realm of that sport. Putting aside Leicester's outlier in the 2015/16 season (Leicester has since mean reverted) the vast sums of television money that now buys players and support staff means a bunching up at the top of the Premier League by those teams owned by the super-rich. There could well be a lucky moment or two this season that decides who takes the league title.

In trading too, with all the available data and the technological improvements to process that data it gets harder every year to find an edge. The future will probably see markets predominantly filled with algo-traders and high-frequency operations, less of the old-school manual traders "scalping" a tick or two. As with pit traders in finance, the manual trader in sports will become marginalised. Reactions to late-breaking information will be more rapid than today and as in the Premier League it will be the wealthy operations with the latest technology that truly profit.

Another snippet from Mr Webb's post mentions, "If you assume that my ability to get a winning trade is just that of a coin toss, a 50-50 chance, then what is the chance of me getting all 14 traits (trades) correct?" I can't help but think that the little imp has been reading my website again and suffering from a little cognitive bias in jumping to a conclusion that isn't there.

I did say in my article Scalping and The Winning Combination that in a directionless market a (noise) trader has only a 50% chance of success. If a market has direction then the chance of trading market movements is no longer 50/50 but then I did say, "If they had asked, 'In an upwardly trending market.' then I might have said something different."