Gambling giant takes a punt on cross-database monitoring tool

Betting giant Betfair is using DBTuna from British developer Application Performance to boost its SQL Server, Oracle and MySQL database performance.

Betfair's complex collection of database platforms comes as consequence of being the world’s leading online betting exchange, a concept it pioneered. Betfair allows punters to choose their own odds and bet against each other, even after an event has started. Its multinational operation processes five million transactions a day and more than 300 bets a second.

Nigel Noble, senior performance DBA at Betfair, says that with this kind of usage even a 15-second performance problem can have profound effect on business, especially when one considers the huge number of bets placed just before the start of a major sporting event.

"Over the years, we have reviewed a number of database monitoring tools but each time have been disappointed to find that the best granularity they could provide in our busy production environment was a 15 minute time slice," says Noble. "Although this information is interesting when looked at over the long term, when applied to our short duration performance problems, you realise that they completely miss the information we need and effectively leave us blind. DBTuna, in contrast, enables us to select the level of granularity we require for each of our databases; we routinely set the time slice to 10 seconds to give us the granularity and detail we require."

Another key requirement was for a tool that could monitor all three major database platforms. Betfair.it, for example, runs Betfair’s Sportbook betting site for Italy. It is a very busy site backed by a large SQL Server 2005 database. When CPU consumption peaked to 100% at random times of day and for quite lengthy periods, DBTuna was able to provide deep-dive visibility into exactly what was occurring and helped locate the root cause of the issue.

The tool was able to provide a performance profile of each and every SQL statement and stored procedure executing within the SQL Server instance and assess the percentage of CPU consumed by each. It quickly pinpointed two stored procedures consuming the majority of the CPU resource, but this was only part of the story. What DBTuna clearly showed was that the number of executions was high, yet fairly static, for the two offending stored procedures.

However, the number of logical I/O operations had increased four-fold during the problematic time periods due to a shift in query execution plans. It was this that had caused the CPU to spike. Further investigation revealed that a batch operation preceding the problem had deleted many rows and had triggered the shift in execution plan. Correlation of these two events was key in understanding what had occurred and vital for resolving the issue.

Oliver Cook, engineering services manager at Betfair, says: "We are working closely with Application Performance R&D in order to ensure that future DBTuna releases will become even easier to use. It is vital that we can delegate more performance and capacity testing to each of our development teams rather than relying solely on our specialised central team. By making the development teams more self-sufficient we will be able to increase product delivery throughput and hence support business growth and maintain our market-leading position."