First of all I must apologise for the downtime customer had over Friday & Saturday, the downtime was due to work carried out at the Netsource Datacentre (Cloud Network)

Were were issue a warning that work was going to be carried out...

DESCRIPTION:

NetSource is performing repairs to the NetSource Cloud environment. These repairs are to correct instability in the environment which could manifest into a larger issue. We will be replacing and upgrading a Power Distribution Unit and adding an additional layer of power redundancy to some areas of this infrastructure.

Urgency: Planned / EMERGENCY

Date Notice Sent: 10/23/17

Maintenance Window: 22:00 on October 26th - 02:00 On October 27th (Central Time)

Reason for Maintenance:

A PDU is showing signs of failure and has failed a recent self-test. We can predict that it will likely fail and should be preemptively replaced. During the maintenance window, NetSource will be removing that device and replacing it with an upgraded unit. We will also be reconfiguring the power input to some of the infrastructure components.

Expected Impact:

The majority of the environment will be in full maintenance mode. The network(s) will be down and due to the nature of the components involved, most of the virtual machine will be shut off during the maintenance period. No systems outside of the NetSource Cloud will be effected.

EMERGENCY CONTACT: 630-778-1212

See Ticket 00117845 for real-time updates.

Please pass this message on to others in your organization that may need to know.

Thank you,

Joe Ayala
Director of Data Center Operations Midwest
630-778-1212
2368 Corporate Lane,  Suite 112
Naperville IL 60563
www.netsource.com
support@netsource.com


We were prepared for the downtime and ready to answer any customer queries and assure them that we were monitoring the downtime and were in contact with the datacentre...then the following notifications came out from Netsource...


Oct 26, 2017 21:46:48 NetSource:
The maintenance is on schedule and will start in 15 minutes at 10PM local time (CDT).

Oct 26, 2017 22:00:32 NetSource:
Admins are starting to turn off VM's and servers. Everything will be down at this time.

Oct 27, 2017 01:29:23 NetSource:
VM's are starting. We are continuing to work on getting all VM's back up.

Oct 27, 2017 02:44:05 NetSource:
Roughly all VM's are up and running, still have a few remaining that we are working on.


Oct 27, 2017 03:56:55 NetSource:
We are still working on the VM's that are not up yet.

Oct 27, 2017 06:42:25 NetSource:
NetSource administrators continue to work on getting the remaining handful of virtual machines back up and running. The process is taking much longer than we anticipated. All our efforts are focused on returning all services back to normal.  

We will update this ticket with new information as we proceed.

Thank You, Support
support@netsource.com

Oct 27, 2017 08:35:11 NetSource:
Thanks you for your patience during this issue. The administrators continue to work on getting the remaining handful of virtual machines back up and running. The process is taking much longer than we anticipated. All our efforts are focused on returning all services back to normal.  

We will update this ticket with new information as we proceed.

Thank You, Support
support@netsource.com

Oct 27, 2017 09:39:24 NetSource:
if you still have a server that is down, and a separate ticket is not opened for you, please feel free to contact support@netsource.com and let us know.

At this time, we are working through solutions but cannot offer an ETA. We will once we have more information.

We will update this ticket with new information as we proceed.

Thank You, Support
support@netsource.com

Oct 27, 2017 13:02:40 NetSource:
On the backend, our administrators are making progress, however, you might need see any change for a bit. We'll try to come to a resolution or,a t least, provide an ETA as soon as possible.

Thanks, Support
support@netsource.com

Oct 27, 2017 15:29:43 NetSource:
Our administrators continue making progress, however, you might need see any change to your server for a bit. The process is time consuming, however, we continue to work on this with no pauses.

We'll try to come to a resolution or, at least, provide an ETA as soon as possible.

Thanks, Support
support@netsource.com

Oct 28, 2017 11:30:25 NetSource:
All services are restored except one customer VM. We are working directly with that customer. That specific case has some additional factors not present for any other customers.

At this time, the emergency response is complete.

We appreciate your patience during this unexpected event. As part of the system recovery, our administrators added additional layers of redundancy and made configuration changes to prevent this sort of extended outage. Our account management team will follow up accordingly.

Regards, Support
support@netsource.com

This was the Tui server where some of our customers and reseller accounts are hosted on.


Oct 28, 2017 11:36:39 NetSource
All services are restored.


We appreciate your patience during this unexpected event. As part of the system recovery, our administrators added additional layers of redundancy and made configuration changes to prevent this sort of extended outage. Our account management team will follow up accordingly.

Regards, Support
support@netsource.com


Customers on the Tui server finally came back online.


I've now asked why this has happened and for an explanation of what went wrong as this was very similar to the same outage we had with them over 6 months ago and we were promised then that extra redundancy measures would be put in place. Currently I'm still waiting for that reply.