IBM Blames Contractors for Aussie e-Census StumbleAs Chief Contractor, IBM Is Now in Compensation Discussions
The legislative hearings this week into why Australia's online census went offline despite years of planning illustrates one core fact: When it comes to highly visible, failed IT projects, the blame rolls downhill (see IBM Blamed for Australian Census Debacle).
The Australian government has sought to pinpoint the technical problems behind the online census, which suffered from distributed denial-of-service attacks that triggered other knock-on equipment hiccups. The embarrassing incident shook confidence in the government's ability to execute large-scale IT projects.
The online census form was offline for nearly two days shortly after 7:30 p.m. on Aug. 9, disrupting the collection of crucial data. Although Australia had experimented with collecting data online for past censuses, a large push was made this year to have citizens fill it out online, which was expected to save the government millions of dollars over tabulating paper forms.
Those savings have partially been consumed. The Australian Bureau of Statistics has said the technical problems cost as much as $30 million (U.S. $23 million), eating into the ABS's expected $100 million savings.
IBM, the lead contractor on the $9.6 million project to coordinate the online census, is now in confidential settlement talks with the government to compensate it for lost taxpayer dollars.
"IBM and I personally unreservedly apologize to the Australian public and the Australian Commonwealth for the inconvenience caused on census night," IBM Managing Director Kerry Purcell told a Senate committee on Oct. 25. "We were head contractor in this matter, and as such take full responsibility for our role."
No one has been fired over the debacle, Purcell says. Just a day after the census failed, Prime Minister Malcolm Turnbull warned that "heads will roll."
The technical problems are outlined in a series of written reports from contractors submitted to the Senate Economics References Committee, which will release its findings on Nov. 24.
Early on, planners were concerned about DDoS attacks, which aim to cripple services by sending a barrage of data traffic. The plan to counter such attacks, dubbed "Island Australia," involved geo-blocking, an effective but blunt strategy that would only allow internet traffic from within Australia to reach the census form.
After a second DDoS attack on the morning of Aug. 9, IBM directed two ISPs - NextGen Group and Telstra - that provided network links to the census form, to turn on geo-blocking. A minor, third attack followed and then a fourth, which caused a snowball of failures that have now sparked fierce disagreement between companies.
NextGen's connectivity relied in part on two upstream providers, Vocus Communications and NTT. According to IBM, Vocus failed to geo-block a traffic route coming from Singapore, which allowed through DDoS traffic from the fourth attack that caused later ill-effects to census systems.
"Vocus admitted the error in a teleconference with IBM, NextGen and Telstra around 11 p.m. on 9 August 2016," IBM's submission says.
Vocus acknowledges an issue with the Singapore link, but says the fourth attack wasn't large by DDoS standards, peaking at only about 563 megabits per second. DDoS attacks average between 1 Gbps and 15 Gbps.
"Such attacks would not usually bring down the census website, which should have had relevant preparations in place to enable it to cater for the expected traffic from users as well as high likelihood of DDoS attacks," Vocus says in its submission.
A computer security expert based in Singapore, who asked not to be named, says it doesn't appear any DDoS attacks were directed at Australia on Aug. 9. Rather, he says it's more likely that a traffic anomaly was misinterpreted as a DDoS attack, a common mistake.
The Australian Federal Police is still investigating who may have been behind the attacks.
IBM saw its problems begin after the fourth attack started at 7:27 p.m. One of its performance monitoring systems displayed messages showing a spike in outbound traffic.
The warnings, which later proved erroneous, indicated that data was being pulled from census systems, raising concerns that hackers may be at work. Those fears caused IBM to then voluntarily shut down the website.
Then, a firewall became overloaded with data, which required the rebooting of a router. Solving that problem took 80 minutes, due to a configuration error.
IBM also implemented another defensive measure, only allowing people who had already logged into the census site to complete their forms. That move disrupted a second router, which also needed to be rebooted.
Vocus maintains it was IBM's router issues and the subsequent voluntary shutdown of the census site that caused the main disruptions. NextGen contends it followed IBM's instructions on how to handle a DDoS disruption and had little input in countering attacks.
Reached for comment, an IBM spokeswoman referred to Purcell's Senate testimony and the company's written submission. NextGen and Vocus officials didn't have a comment beyond their submissions.
The technical failures on census night largely pushed what had been a fierce, census-related debate over privacy into the background. The ABS made some key changes for this year's census that rankled privacy activists, worried over the safety of data (see Australia in Privacy Furor Over Census).
Those changes included extending the period in which the ABS retains names and addresses from 18 month to four years. Unless census takers opt otherwise, Australia destroys the names and addresses, unlike most other countries.
The ABS argued that the extension, which was subject to a public inquiry, was needed so the agency could conduct a more in-depth analysis that would further help shape public policy. The government plans to anonymize individuals' data in a way that also allows it to be connected to other large data sets.
But experts contend that those anonymization methods, which involve so-called statistical linkage keys, aren't necessarily secure and could be reversed, enabling attackers to tie the collected census data to real people.