Closing out 2018, in December the Oracle Internet Intelligence team observed Internet disruptions in countries around the world due to power outages, government direction, technical faults, and possible issues relating to satellite connectivity. While these causes have become relatively common, it is interesting to note that other common reasons for Internet disruptions, including severe weather (such as typhoons and hurricanes), concerns over cheating on exams, and denial-of-service attacks did not appear to drive significant Internet disruptions observed in Oracle’s Internet Intelligence Map during the month. And while we tend to focus on Internet disruptions, it is also important to highlight that after several rounds of testing, nationwide mobile Internet access was finally activated across Cuba.
Cuba
In three tranches (based on the first two digits of a subscriber’s mobile phone number) over December 6, 7, and 8, ETECSA, Cuba’s national telecommunications company, enabled nationwide mobile Internet access. The rollout was reportedly stable, in contrast to the congestion experienced during the trials conducted several months prior. The figure below shows the gradual adoption of this newly available connectivity through changes in the DNS Query Rate. As seen in the graph, the query rate was comparatively low in the days ahead of December 6, with peak levels growing gradually each day as the rollout took place, ultimately settling into a fairly consistent rate. (It is unclear what caused the sudden drops evident in the graph late in the day on December 7 and 10.)
However, this new mobile Internet service is based on 3G, delivering slower connection speeds than those experienced in countries that have deployed 4G/LTE-based services. Cuba’s slower mobile connections come at a higher cost as well, with published reports indicating “The new service will cost about 10 cents per megabyte, with packages ranging from 600 megabytes for about $7 to four gigabytes for about $30.”
ETECSA also found itself turning to social media to advise its subscribers in a number of areas related to Internet usage including data usage, security, and child safety – topics very familiar to users and network providers with more well established connectivity. In one Tweet posted by @ETECSA_Cuba, the provider cautions users that mobile data is charged by volume (as opposed to by connection time, as with local Wi-Fi connections), and that they should be cognizant of things like video format, image resolution, and the applications that they are signed into. In another Tweet, embedded below, ETECSA reminds users that while security is a shared responsibility, it is mainly the responsibility of the users. And in a Facebook post, ETECSA provided recommendations to parents around safe Internet usage for children.
@ETECSA_Cuba le recuerda que la seguridad en internet es responsabilidad de las plataformas tecnológicas, de los prestadores de servicios, de los centros de asistencia, y principalmente de los usuarios, actores de las operaciones en internet. pic.twitter.com/UaZAJZnM29
— ETECSA (@ETECSA_Cuba) December 7, 2018
<script async="async" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>
Power Outages
Similar to issues seen in July, August, and October, power issues in Venezuela disrupted Internet connectivity in the country during December. On December 14, Venezuela’s national electric company posted a Tweet noting that “strong fires” under high voltage lines was causing service interruptions in Aragua (a state in the north-central region of the country). The impact of these power outages is visible in the figure below, with two noticeable declines in the Traceroute Completion Ratio graph during the latter half of the day and into the 15th, and the DNS Query Rate graph declining during the same period, without exhibiting the second peak visible during previous days.
The power outage’s impact is also visible in the Traffic Shifts graph for AS21826 (Inter / Corporación Telemic C.A.), a network provider that services customers in Aragua. The figure below also shows two noticeable declines in the number of completed traceroutes to endpoints within the network during the latter half of the 14th and early on the 15th. Interestingly, the combined latency appears to drop concurrent with the power outage, but this is likely due to the lower number of completed traceroutes. (That is, higher latency paths were interrupted by the power outage, so the average combined latency becomes lower.)
Government Directed
On December 20, advocacy group Access Now posted a Tweet regarding reports of an Internet shutdown in Sudan. Multiple responses noted that access to social media platforms had been blocked across several network providers in the country, and several responses claimed that a complete Internet shutdown would occur on the following day. As shown in the figure below, an Internet disruption was observed in Sudan on December 21, with a brief but significant drop measured in the Traceroute Completion Ratio metrics, as well as a more prolonged drop in the number of routed networks. In addition, the DNS Query Rate metric remained largely flat on December 21, after seeing lower peak levels during the previous two days – these lower peaks could have been related to blocked access to social media platforms.
Completed traceroutes to endpoints in Sudatel (Sudan Telecom), a major telecommunications provider in the country, also experienced a brief but significant drop on December 21, as shown in the figure below.
At the end of December, Internet connectivity (and SMS services) were disrupted in the Democratic Republic of the Congo in advance of the announcement of results from the prior weekend’s presidential elections. The figure below shows that the disruptions appears to have started around mid-day (GMT) on December 31 and continued into the first week of 2019.
Technical Faults
On December 12, Dauphin Telecom, a telecommunications provider in Saint Martin, posted the update shown above to their Facebook page letting subscribers know of degraded service due to a cable break in Puerto Rico. It is likely that the post was referencing an issue with the Southern Caribbean Fiber system, as it connects a number of island nations where Internet disruptions were observed through Oracle’s Internet Intelligence Map.
The figures below show Country Statistics graphs for Saint Martin and Saint Barthelemy during the period of degraded service. In both countries, the graphs show that the issue disrupted all three measured metrics.
The impact of the cable break is also evident at a network level, as shown in the Traffic Shifts graphs below for AS36511 (Dauphin Telecom Guadeloupe) and AS18895 (Nustream Communications). The Dauphin Telecom graph shows the shift to a higher latency backup route during the latter half of the day on December 12. The shift seen at Nustream Communications, a Puerto Rican network provider, was shorter in duration and less pronounced.
On December 15, a Tweet posted by @InternetIntel referenced a large Internet outage in Kenya that started the prior evening, noting that local provider Wananchi Group was among the affected providers. This observed disruption aligns with Tweets to @ZukuOfficial and @Zuku_WeCare sent by users and/around December 14 complaining about Internet availability – Zuku is a brand name used by Wananchi Group for fiber and satellite TV services in East African countries, including Kenya. The figures below show the impact of the outage for the three measured metrics at a country level for Kenya, as well as on traceroutes to endpoints in AS15399 (Wananchi Group). The Traffic Shifts graphs indicate that the observed outage may have been due to an issue with upstream provider Seacom.
On December 16, a number of responses were posted by @ZukuOfficial that stated that the company was “working to restore internet service after a technical fault.”
Hi George, we are working to restore internet service after a technical fault. Sorry for the inconvenience. ^WW
— Zuku (@ZukuOfficial) December 16, 2018
<script async="async" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>
Starting late in the day on December 23, Comoros experienced a significant Internet disruption that stretched into the better part of the day on December 24. The Country Statistics figure below illustrates the significant impact seen across all three measured metrics. In line with that, the Traffic Shifts figure below for AS36939 (Comores Telecom) shows that there were no successfully completed traceroutes to endpoints within the network for the period of the disruption, until they failed over to a backup path across West Indian Ocean Cable Company. This backup path was used for several hours until connectivity through BICS (Belgacom ICS) was restored.
Just a couple of days after Christmas, major network services provider CenturyLink posted the following Tweet:
Our network is experiencing a disruption affecting customer services. We understand some customers are currently unable to generate tickets through the CenturyLink help portal. We know how important these services are, we are working to restore services as quickly as possible.
— CenturyLink (@CenturyLink) December 27, 2018
<script async="async" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>
Published reports (CNET, LightReading) indicate that the disruption, which started several hours before CenturyLink posted that Tweet, impacted customers across multiple U.S. states, including disrupting access to 911 emergency services, and CenturyLink’s status portal reported impacted connectivity to “external network services” for a number of the company’s cloud data centers. Impacted users exchanged information about the disruptions on forums including Reddit and the Outages mailing list. The issue was reportedly caused by “a faulty network management card from a third-party equipment vendor”. The card’s misbehavior ultimately “congested controller card CPUs (central processing units) network-wide, causing functionality issues and rendering many nodes unreachable.”
The figure below shows the Traffic Shifts graph for AS11530 (CenturyLink Communications), one of many autonomous systems associated with CenturyLink, and a significant day-long decline in the number of completed traceroutes to endpoints within the network is clearly evident.
As would be expected, the issue impacted connectivity for networks that have upstream connectivity from CenturyLink. The Traffic Shifts figures below illustrate the impact on three (of many) such networks. In each case, traceroutes failed over to backup paths across other network providers, including (interestingly) Level 3, which is owned by CenturyLink, but was apparently unaffected by the outage.
Possible Satellite Connectivity Issues
Over the last ten days of December, we observed Internet disruptions across the Solomon Islands (December 22), Guyana (December 27), and Kiribati (December 30). The impacts of the disruptions can be seen in the figures below and were visibly more pronounced in the Solomon Islands and Kiribati.
Examining the disruptions at a network level, it appears that they may have been related to problems with upstream connectivity through O3b, a “network communications service provider building and operating a medium Earth orbit (MEO) satellite constellation primarily intended to provide voice and data communications to mobile operators and Internet service providers” that is a wholly owned subsidiary of SES S.A. The Traffic Shifts figures below for networks in the three impacted countries show that O3b is the primary upstream provider across all three, and the disruptions evident in these graphs align with the Country Statistics graphs above.
Conclusion
The Oracle Internet Intelligence Map provides insight into the impact of Internet disruptions around the world, both at a national and network level. Although the disruptions have varied in severity and duration, there have been a number of common root causes – some natural (typhoons, hurricanes), some technical (cable cuts, power outages), and some intentional (exams, elections). Going forward into 2019, the Oracle Internet Intelligence team will continue to track Internet disruptions and their associated causes – given the activity seen during the first week of January, things (unfortunately) show no sign of slowing down.
︎
</figure>
