Throughout the working day approx. 50% of Tebay users were able to access local and remote network resources but unable to access the public internet in order to use Email / Microsoft Teams and other web enabled functionality.
Data Centre services, Craven House users and remote VPN users were not affected.
The issue has been traced to a top level routing issue between Tebay and AFGlobal's other sites (notably the Allen Data Centre in Texas and the UK Woking Data Centre).
Essentially the traffic of the affected users was reaching a junction in the network and its destination was sign posted as being equidistant in BOTH directions at once. Where humans might be able to deal with this computers cannot.
It would seem that the timing of the issue (now trackable as 11:56PM on Friday) coincides with upgrades that were made to both the UK Data Centre firewall (11:38PM) and the Tebay firewall (11:58PM) by Amsys on Friday evening.
We've asked Amsys to conduct a root cause analysis on this incident so that we can better understand the cause and take steps to prevent it occurring again in future.
This issue is now resolved and business as usual is expected tomorrow.