Rather like England’s victory against Paraguay in its opening World Cup match last year, Betfair’s success in processing millions of transactions, ‘was not without its challenges’, says Rorie Devine, chief technology officer (CTO) for the online gambling exchange.
However, Betfair’s preparations for frenzied betting activity on the site where members bet against each other, as one sets and the other accepts, have paid off.
On the day of England’s first match, 4.4 million bets were placed and only 20 took more than one second to process. It means 99.999 per cent of bets were placed in less than one second.
‘To get five nines is fantastic,’ says Devine. ‘The reason why performance is so important is that lots of activity is done on the site in a short timeframe.’
Last summer, Betfair worked with its database supplier Oracle to increase the target percentage of bets processed in less than one second from 99.1 per cent to 99.9 per cent.
Devine says estimating opportunity cost for delays in processing transactions is not an exact science, but he suggests the mathematics is impressive.
‘Reducing the amount of time it takes to process a bet by 0.5 seconds – on five million bets per day – saves 694 hours, or 28 days of total customer waiting time per day,’ he says.
‘This delivers a better customer experience and returns the money to the customer for re-use, more quickly.’
Some traffic, however, is more welcome than others. A recent project at the business attempted to shape response times for visitors using automated trading applications – see link below.
Better traffic management means Betfair can delay new investment plans for some of its infrastructure, including extra front-end servers.
However, it is likely that the traffic to the company’s web site will double in the next 12 months, making continued IT investment essential.
‘Everything doubles every year,’ says Devine. ‘We have a two-year capacity plan for delivering each layer of infrastructure, but building the infrastructure is not cheap and if we can get smarter, we can save money on bottom-line costs.’
Not cheap is an understatement. Betfair has invested more than £30m in technology and employs more than 300 engineers to support its infrastructure.
Last year it upgraded its servers to Sun Fire T1000 and T2000 servers, which host the company’s Oracle database.
‘They are very cost-effective in terms of power drawn,’ says Devine. ‘The big challenge is space, power and cooling, not bandwidth. They have brought into question the need for blade servers which give you density, but a lot of heat.’
The organisation uses different types of storage technology for different data uses – particularly important is how quickly information is needed to meet processing targets.
‘There is the usual trade-off between speed, access and costs,’ says Devine. ‘We put the time-sensitive information close to the servers, and more static data on the networked attached storage.’
When it comes to securing data, there can be no trade-off. Betfair has a large team of security specialists, described by Devine as one of the company’s best teams, responsible for combating hackers and denial of service (DoS) attacks.
In 2004, Russian criminals attempted a series of DoS attacks aimed at extorting online betting sites.
‘It felt like a bandwagon with European criminals thinking they could make money extorting betting sites,’ says Devine.
He says the threat has not gone away. ‘There are less but more specialised DoS attacks. The really big ones are getting bigger, but the industry has moved on in two years and we have to make sure there are easier targets for criminals. It is like having the best car alarm – it might make a criminal pick on another car.’
As Betfair handles sensitive financial data, the government also has a say over security, and the company must comply with the Department of Culture, Media and Sport’s regulatory framework for remote gambling.
Outside audits from payment card providers also help push information security to the top of the agenda.
Firms such as Betfair, that process debit and credit card payments, must meet security standards set out in the Payment Card Industry Data Security Standard, devised by Visa and MasterCard to minimise the risk of financial information being stolen.
‘Our site is subject to external penetration tests every quarter,’ says Devine. ‘We must meet the standards they expect.’
He says Betfair’s site gets so much traffic and deals with such huge amounts of information, that it is run by one of the five hottest Oracle databases in the world.
The relationship with Oracle is, therefore, a strategic part of Betfair’s business, but it is mutually beneficial as Oracle, like other major suppliers, can learn about its technology’s capabilities from Betfair.
‘We spend a lot of time talking to Oracle’s development team,’ says Devine. ‘We push technology to its limits. If it will break, it will break here.’
One thing Devine wants to ensure is the site does not break – when the site is the business, disaster recovery is critical.
‘Disaster recovery is very important. We have spent a lot of time and effort on our main data centre and we have a standby disaster recovery data centre in a separate location,’ says Devine.
Considering his claim that Betfair handles twice as many debit transactions as any other European web site, and expects to continue growing, dealing with sensitive financial data makes the firm reliant on getting the best out of technology.
However, Devine is not a gambling man when it comes to unnecessary risk.
‘We need to take advantage of upgrades, but we also want to be at the right place on the risk curve – not the bleeding edge,’ he says.
Do you agree?
Have your say on this article