| Summary: |
| You can look at the progress of a priority problem in more detail here. |
| Problem: [62178] Data centre problems |
Status: Closed |
Problem description
Added on: 03/06/2010 @ 02:15
At approximately 4am, we suffered a partial outage affecting one of our data centres in Sheffield. The problem followed shortly after a line card failed in one of our core routers. This will be affecting access to certain services for some customers.
If you were logged in to the portal, you could associate your username with this problem.
Comment
updated on: 03/06/2010 @ 10:26
Monitoring flags numerous problems between 3am-4am.
Comment
updated on: 03/06/2010 @ 10:26
Investigation suggests faulty line card at one of our Sheffield data centres is contributing to the problem.
Comment
updated on: 03/06/2010 @ 10:26
Engineer arrives onsite at 5.30am to reseat affected card.
Comment
updated on: 03/06/2010 @ 10:27
Work begins at 6.00am to bring affected servers back into service.
Comment
updated on: 03/06/2010 @ 10:28
Second card failure shortly after 9.00am, case raised with supplier.
Comment
updated on: 03/06/2010 @ 11:34
We're continuing to investigate routing problems on the network.
A replacement line card is sceduled to arrive on site shortly after midday.
Comment
updated on: 03/06/2010 @ 12:00
Email has been queuing on our anti-spam appliances since about 9.00am this morning. This will be leading to delays in the arrival of email for anybody who has spam filtering switched on for their account.
Comment
updated on: 03/06/2010 @ 12:43
Customers may be experiencing IMAP connection errors when logging into Webmail.
Comment
updated on: 03/06/2010 @ 13:49
Restoring the new card caused a short storage blip that will have affected connectivity to our website and some other services.
Unfortunately, this has created an issue with access to our Plesk/PAYH hosting platform that is currently under investigation.
Comment
updated on: 03/06/2010 @ 18:07
Access to our Plesk/PAYH platforms has been restored although we will need to recut the database as it has become corrupted.
Comment
updated on: 03/06/2010 @ 18:09
Webmail access restored and all mail has been dequeued from our anti-spam appliances.
This email still needs to filter through to customers' mailboxes and until it does access to our mail collection servers may take longer than usual and you may still see some emails arrive that have been delayed in transit.
Comment
updated on: 03/06/2010 @ 19:11
We have now restored all service, we hace had to temporarily disable POP for the following VISPs, however they are still available over IMAP and Webmail:
globalnet.co.uk
freenetname.co.uk
totalise.co.uk
care4free.net
madasafish.com
Comment
updated on: 03/06/2010 @ 20:16
We have restored service to all VISPs now.
Comment
updated on: 03/06/2010 @ 20:20
Unfortunately this has recurred and we have had to stop POP for the same VISPs, as before IMAP and webmail should still be working.
Comment
updated on: 04/06/2010 @ 18:33
We have now restored POP access to all VISPs
Go to current priority problems | Go to archived priority problems
|
