It was a warm spring afternoon in September last year when Hosea Malope was on his way to the capital city, Pretoria in South Africa to entertain key clients at a rugby game. It all turned pear shaped before he even arrived at the stadium.
Malope, of Zonke Monitoring Systems (ZMS), was set to leave the next day for Las Vegas in the United States to exhibit the Central Electronic Monitoring System (CEMS), a new system to monitor Limited Pay-out Machines (LPM). His company was trying to get into the lucrative American market.
ZMS is a supplier of management and monitoring systems for gambling machines. It monitors around 8,000 of these machines for the National Gambling Board.
Back in Pretoria, two other colleagues had left for Las Vegas the day before. Malope had just arrived in the city, from a funeral in Hammanskraal, just outside of Pretoria, when he took a call from a client traveling in Europe. He was informed that some of the LPM’s on his sites had broken down and they could not get them to work. It meant Malope had to fix them.
To make matters worse, this happened at a time when ZMS had just won a contract with the client to implement the monitoring system for machines on sites in Swaziland and Zimbabwe.
“When he phoned me I said, ‘Well, let me not disappoint the man, let’s find out what’s happening’. My other IT engineer was there, he was coming to the rugby with us. He’s the one that looks after infrastructure and I asked, ‘What is happening?’ He was not aware,” says Malope.
The IT engineer phoned the help desk, who said the problem was the machines were not sending the necessary instructions.
“For a gambler, he or she wants to be assured that the machines they are playing on pay according to regulations and they’ve been tested, they’ve been certified and the return-to-player percentage is configured correctly,”adds Malope.
If there are certain errors on a gambling machine it will make the machine unplayable. The only way to fix it is to use a dial up to communicate with the device on site that will enable that machine.
“We thought it was a simple issue so we waited a bit, then an hour later, this thing was still not resolved. And this customer was flying around Europe,” says Malope.
“Whenever the client landed somewhere, he would call me to say ‘By the way…’,” constantly alerting Malope that the problem was still not fixed.
Malope had to leave the party immediately after the final whistle signalling a win for the South African team against Australia in the Tri-Nations, to try and solve the problem. At least South Africa beat Australia on the day. But Malope faced further frustration.
When he arrived at the ZMS offices in Johannesburg, he and the IT engineer went down to a building that hosts their systems only to find that some cables were cut. The day before, Telkom, the country’s premier telecommunications company, were doing maintenance work and managed to cut some cables by mistake.
“When the machines are down it means no revenue for the site; the pub owner, for the route operator, no taxes for the provinces, no VAT (value added tax) for government and no revenue for us, because we charge a fee for providing monitoring services,” says Malope.
In South Africa alone, the various provincial gambling boards make around $1.5 million and the treasury makes around $1.6 million in VAT every month.
Only a few weeks before, they had tested their systems in a disaster recovery plan which worked well. If one Internet Service provider (ISP) failed, such as Internet Solutions or Data Pro, another would kick-in and continue operations.
“But what we neglected to do in that process was consider Telkom lines being cut off. We never actually tested the situation where the system is still intact at Internet Solutions, but there’s no communication, there’s no Telkom lines,” says Malope.
He went home just before midnight that evening with everything still not fixed. The next day, just over 24 hours since their systems went offline, they managed to get two lines working through Data Pro.
“It was end of the month and that’s one of the weeks where we really perform well,” he says.
By the afternoon, ZMS had about 10 to 12 lines running, instead of the usual 30 and Telkom could only fix the cables on Monday. The problem was only solved on Monday evening.
Following this, ZMS beefed up its disaster recovery plan and made sure they had enough lines at Data Pro.
“We split our Telkom lines between the two centers and made sure we can use any of those lines at any given point in time. We are at a stage now where we think our system is solid,” says Malope.
With everything running smoothly now, Malope says the company is trying to expand into other African markets and around the world.
To this day, ZMS fears Telkom workers bearing sharp tools.