An engineer at a South African Internet service provider accidentally deleted important network settings while rushing out for a cigarette break, causing the largest Internet outage on the African continent at the time.
A few decades ago, one of South Africa’s leading Internet service providers faced an unprecedented crisis due to a seemingly minor mistake by an employee. An engineer named Paton worked as a «backbone network engineer» — a position that required high responsibility and attention to detail.
The company Paton worked for played a key role in keeping the Internet running not only in South Africa but also in neighboring countries. The provider’s DNS servers were authoritative for thousands of domains, including the national top-level domains of several African countries.
One day, Paton was tasked with updating the network blocks
When Paton returned from the break, the office was in real chaos. The network operations center was flooded with calls from angry customers. It turned out that the largest Internet outage on the African continent at the time had occurred.
To make matters worse, an unknown person claiming to be a hacker contacted a local tech publication and claimed to be involved in the incident. The news spread quickly, creating additional problems for the company’s management.
An investigation revealed that there had been no security breach. Paton, in his haste, accidentally replaced all existing access control lists instead of simply adding new network units. This resulted in a complex system of routing Internet traffic for a large part of Sub-Saharan Africa ceasing to function.
After the incident, Paton not only restored the ACLs and updated the network blocks, but also developed the company’s first change management protocol — a set of rules and procedures that govern the process of making changes to IT systems to avoid incidents or disruptions.
Source: TheRegister