2021-09-1
md
Web Site Offline in Previous Three Days

Actions by someone with motives not yet divulged brought down a number of servers belonging to the web hosting provider that I use. There is no doubt about the malevolence of this individual because he began by erasing the contents of backup servers before doing the same with production servers. The intention to cause irreparable damage was clear. The fallout appears to be as follows,

I am one of the lucky ones, as my website was on one of the servers that was completely restored. So after being offline for about three days, my site is running again. My apologies to the thousands who were unable to reach their favourite site.

The only major problem encountered after the restoration, was that a few htaccess files blocked access to all web pages. When emails started to come in, I thought to use Filezilla (i.e. SFTP) to view the content of my account and saw that the web site was restored. Then the penny dropped. The 403 errors I was getting when trying to open web pages on my site did not correspond to the custom 403 error pages I had installed so I just removed the htaccess files and found that my website can be accessed using HTTPS or HTTP. I wonder if a different Web server is being used now which could explain why the htaccess files no longer worked correctly?

Over 90% of requests from web crawlers (spiderbots or web search engines if preferred) received a 403 result on August 31 when my site was restored. I assume that many who tried to reach this site yesterday ran into the same problem, but I do not have equivalent statistics for requests from "real" people. In any case, apologies to everyone that could not find a page on the site. To put that facetious comment about thousands unable to access my web pages into perspective: there were 216 different visitors to the site on the 31st, not counting bots.

I am experiencing some SSL/TLS certificate conflicts. It just may be that the provider changed all its certificates in response to the malevolent attack, I will be examining this issue later on. In any case it should have no effect for anyone trying to access posts and other resources on this site. If there is a problem, please advise me by email; there is a link at the bottom of each post for this purpose.

Given that last paragraphs, it is obvious that my email accounts are also operational. Of course, any email sent during the last 3 days never reached my account inbox but hopefully the sender will have received a message reporting that the sent email could not be forwarded to my account. The message also contained a rather cryptic error message, which hopefully everyone ignored. The good news is that, aside for emails sent during those last three days, no emails have been lost. I am also very grateful that my personal calendar was preserved because I had no backup. The first thing I did when I got access to it was to make a local copy. A top priority is to find a way to automatically make backups of the calendar which is shared with most of my personal devices.

What follows is the messages from the web hosting provider as its team of technicians worked to fix the problem. It reads a bit like one of those British procedural whodunit. Who knows the computer security team backed by the forensic computer analysts may become a new genre on TV. Names have been changed to preserve the innocent.

LIVE: Major Incident in Progress

August 28, 2021: a major incident is impacting the availability of web hosting and reseller hosting accounts on several systems in our datacenter. We currently have all hands on deck working on the problem but the situation is serious.

So, what happened?

Based on our investigation to date, the morning of August 28 at approximately 6AM, an individual with a third-party service provider used their privileged account access to connect to one of our datacenter’s management portals and without authorization, initiated server reimaging on some of our backup servers, then on some of our production servers.

Within only hours our incident response team had identified the issue and disabled access to the source account, preventing any further damage. The environment was secured, the individual fully locked out, and our disaster recovery plan immediately kicked into action but damage was already done.

The tally of the incident, however, was and still is important: a few major systems, including some production servers and some backup servers were damaged, with a large number of web hosting and reseller hosting accounts affected, resulting in possible permanent data loss.

After nearly 2 days of tedious work and a combination of external datacenter backup restores and system-level storage rebuilding, our team was able to successfully recover (or is in the process of recovering) over 50% of those lost accounts. We can confirm that Cloud, Dedicated, Weebly and Managed WordPress accounts were largely unaffected.

Unfortunately, at the moment, I can tell you that several production servers and their respective backup servers are still in an unrecoverable state and the data recovery experts assisting us believe that the recovery potential is very low. As such, the focus for these accounts has shifted from data recovery to starting fresh on new, clean accounts. In parallel we will continue to attempt to recover any data we can.

Update August 28, 2021 - 1:40PM

We were able to successfully restore service to Robert2 server, which is back up and running. We are advancing in our efforts to restore other systems.

Update August 28, 2021 - 1:25PM

Systems Bob & Peter have confirmed working backups and we are proceeding with full backup restore. The following systems are still down:

We are continuing to investigate recovery strategy for these systems and will update soon.

Update August 28, 2021 - 1PM

We can confirm that multiple systems are impacted by a major incident right now. We are attempting to recover source data and restore service functionality and will post updates as soon as they become available.

We sincerely apologize for any inconvenience caused and appreciate your patience as we continue to actively work on understanding and resolving this incident.

Update August 28, 2021 - 7:50PM

We have now proceeded to restore the servers:

The following servers are being restored from disaster recovery backups and accounts are gradually coming back up as they restore (full process will exceed 24 hours):

The following servers have suffered from more severe data loss and we are investigating advanced data recovery steps:

If you have your own local backup of your account/website (hosted on one of the 5 servers below) you can request activation of a new account on a different server, by contacting us via live chat. This service will be provided at no extra charge to you. However, please note that ONLY those with backups will have new accounts created, as it will essentially be replacing your existing account.

While limited data recovery is still possible on our end, the likelihood of a full account recovery in the next 24 hours is low. We are nonetheless continuing our investigation and making this restoration our number 1 priority.

Update August 28, 2021 - 5:10PM

We have now proceeded to restore the servers:

The following servers are being restored from disaster recovery backups:

We are still investigating the situation for the following servers and do not yet have an update at this time:

Update August 28, 2021 - 3PM

We have begun the process for disaster recovery from external backups for the following servers:

Considering this is a full system backup restore, restoration can last up to 24 hours before all the data is available again. We will provide updates as they become available.

We are still investigating the situation for the following servers and do not yet have an update at this time:

Update August 28, 2021 - 9:00PM

As our team continues to work overnight to restore accounts, please note that live updates will be paused until tomorrow morning at 10AM so we can focus on the restoration efforts. We’ll then be able to give a more detailed account of what has been accomplished.

If you have your own local backup of your account/website (hosted on one of the 5 servers below) you can request activation of a new account on a different server, by contacting us via live chat. This service will be provided at no extra charge to you. However, please note that ONLY those with backups will have new accounts created, as it will essentially be replacing your existing account.

While limited data recovery is still possible on our end, the likelihood of a full account recovery in the next 24 hours is low. We are nonetheless continuing our investigation and making this restoration our number 1 priority.

Servers with more severe data loss are shown below:

Update August 29, 2021 - 10:00AM

Thank you once again for your patience. We continued to work overnight on restoring accounts.

Here are the updates for each of our restored servers so far:

Here are the updates for servers currently in the restoration process:

Services on the above servers should be fully functional within the next 12-24 hours.

Please note: We will be posting another update at 11AM for the more severely impacted servers.

Update August 29, 2021 - 11:00AM

Thank you everyone for your patience and support during what has been a very trying 24 hours. Our team has been working tirelessly to restore service to affected clients and have had some success but the news isn’t all good, so here is where we stand.

Servers that have been fully or partially recovered

We have successfully recovered the following servers:

Accounts on servers with high likelihood of data recovery

We have 4 systems whose external backup servers have not been impacted, and these are actively being used to stream account restorations right now. We expect to be able to fully restore all backed up accounts by Monday evening and hopefully sooner. Here is where they are in the restore process:

A full restore process for a system of this size generally takes 24 hours. As soon as an account is restored, its website and email should become available so for most users on these machines you should have your services restored before the end of the day Sunday.

Accounts on servers with low likelihood of data recovery

The following 5 systems have had their external backup servers also partially destroyed, and we have been as of yet unsuccessful at recovering their data:

For clients on these machines, we are pursuing 3 recovery strategies in parallel:

  1. Re-upload your content from your own backup
    If you have a local version of your website and/or files on your computer and can re-upload them, please contact support and they will activate a fresh account for you on a new server immediately.
  2. Partial data recovery on production servers
    We have had some success in recovering hosting accounts data files (excluding databases) for 2 of the affected servers (Heaven & Bernie for now) and may be able to perform a partial restoration of these accounts within the next 48-72 hours. This process is slow and tedious and unlikely to complete before Tuesday.
  3. Manual data recovery
    We’ve contracted an external firm to assist us with data recovery efforts on our backup servers and they started their work on Saturday night. Manual data recovery is a tedious and slow process, but we expect to get a report on what can be salvaged before day’s end. This approach will likely take the longest to result in recovered data, with a timeline as long as 1-2 weeks.

We will be posting more information along with a post-mortem once the immediate operational issues have been addressed.

We have mobilized our full team to assist in this effort and we currently have dozens of system administrators, support technicians and data recovery engineers assisting with the recovery effort since yesterday. We understand this incident may seriously impact your business and are doing everything in our power to mitigate the damage.

More information will follow as we have it.

Update August 29, 2021 - 5:00PM

As our team continues to work to restore accounts, please note that live updates are paused until tomorrow morning at 10AM so we can again focus on the restoration efforts. We’ll then be able to give an account of what has been accomplished overnight.

Accounts on servers being restored now

The restoration process is advancing well and we hope to be able to fully recover all accounts on these servers from backups by end of day Monday, and hopefully sooner. Here’s the latest progress:

Accounts on servers with damaged backups

The following servers have had both their local storage and their external backup storage heavily damaged, making the recovery process extremely difficult:

Our initial attempt to repair the data on our backup servers has failed and at this point the likelihood of successfully restoring account data from these servers is very low.

As a result, we are working with data recovery experts to attempt to restore data from the source server, but this process is extremely long and tedious, potentially requiring file-by-file analysis for systems with millions of files and may require months to complete.

Here is how you can move forward if your account was on one of those servers:

For further assistance

Please use our Live Chat or tickets for help requests. Phone lines are temporarily closed to better support restoration efforts. We thank you for your continued patience and understanding.

Update August 29, 2021 - 11:00PM

Accounts on servers with damaged backups

With our initial data recovery process unsuccessful, we’re now recommending that all affected clients that have experienced data loss consider using a new, temporary hosting account that’s already created in their Client Area.

To use this account, click into it to create your emails and upload website data just as you would on your regular hosting account.

Once this is done, just link your domain to this account, update your domain’s DNS to:

arc1.hwcc.biz
arc2.hwcc.biz

Once the DNS changes propagate (can take several hours) your new account can be used to start accepting and sending new email and upload new website copy quickly. These accounts will remain free of charge until at least January 1, 2022 and are intended as a stopgap measure until a more permanent solution is found. Accounts on servers being restored now

The restoration process is advancing well and we hope to be able to fully recover all accounts on these servers from backups by end of day Monday, and hopefully sooner. Here’s the latest progress:

Update August 30, 2021 - 9:00AM

The restoration process is advancing for the 4 servers where backups are available:

For clients on servers with damaged backups, we recommend you initiate your disaster recovery plan ASAP. We have created hosting accounts in your Client Area to help you get up and running while we explore long-term data recovery options. The impacted servers are:

Update August 30, 2021 - 3:00PM

The restoration process is advancing for the 4 servers where backups are available:

There is no update at this time for clients on servers with damaged backups. We continue to recommend that you use the new account to help continue your business operations. We will make additional resources available soon to you to help with the rebuild process.

Update August 30, 2021 - 10:00PM

Servers that have been fully or partially recovered

The restoration process is advancing for the 3 servers where backups are available:

Update August 30, 2021 - 6:00PM

Servers that have been fully or partially recovered

We have successfully recovered the following servers:

Accounts on servers with high likelihood of data recovery

The restoration process is advancing for the 3 servers where backups are available:

Accounts on servers with damaged backups

The following servers have had both their local storage and their external backup storage heavily damaged, making the recovery process extremely difficult:

Here is how you can move forward if your account was on one of those servers:

Update August 31, 2021 - 11:00AM

Servers that have been fully or partially recovered

We have successfully recovered the following servers:

Accounts on servers being restored now

The restoration process is continuing to advance and we hope to be able to fully recover all accounts on this server from backups by 3pm. Here’s the latest progress:

Accounts on servers with damaged backups

For all customers on servers that have had both their local storage and their external backup storage heavily damaged, we are in the process of developing an informative Rescue Portal. This resource will provide you with all the information required to help you quickly and immediately start over in your new account.

Update August 31, 2021 - 2:30PM

We have now successfully completed 99.9% of backup restorations of all the backups we had in an undamaged state. This means that if your account was hosted on either of the following servers, your website and email should be up and running now:

If it's not, there's a technical issue and you should please reach out to support for assistance. 100% of Sibername accounts had recoverable backups and have now been restored.

Our efforts and attention is now shifting to servers with unrecoverable backups. The situation is still critical, but there are some (limited) positive developments. On 2 of the affected servers, specifically Heaven & Bernie, we were able to retrieve most of the files in users' accounts. This unfortunately excludes databases, but should include email content.

In the next 24-48 hours, we expect to make available fresh production servers (to replace the 5 we have lost) to which we will start streaming available data, as we find it. For the other servers, we're working with top data recovery teams to attempt to extract what we can, but the timeline for this is likely weeks, not days, and the results are far from certain.

Update August 31, 2021 - 7:30PM

We have now successfully completed 100% of backup restorations of all the backups we had in an undamaged state. This means that if your account was hosted on either of the following servers, your website and email should be up and running now:

If it's not, there's a technical issue and you should please reach out to support for assistance. 100% of Sibername accounts had recoverable backups and have now been restored.

Our efforts and attention has now shifted to servers with unrecoverable backups. Tonight, we'll finish activating replacement servers for the 5 we've lost, and by tomorrow morning your original hosting accounts will be once again available (although without data). For clients on Heaven & Bernie, we'll be automatically restoring the files that we managed to recover, and this should include all your emails but not databases. In the meanwhile, you can safely continue using your new accounts.

For those hosted on servers Kent, Knight and Firth, we continue to work with the data recovery teams but these efforts are likely to last weeks, not days.

Update September 1, 2021 - 11:45AM

Accounts on fully-recovered servers (Peter, Lake and Bob) should be in working order now. There were problems with wrong dedicated IPs after we announced their full recovery yesterday, but this issue has since been addressed by our team overnight and everything should be working on these servers now. If you have any remaining issues, please reach out to customer support via live chat or ticket.

Please note: Customer support is experiencing a spike in chat and ticket requests and are doing their best to reply as quickly as possible. We will be providing an FAQ resource that will help to answer the large majority of questions that continue to pop up from our clients. This should help to reduce longer response times from our customer experience team. We, once again, want to apologize for the extra delay and thank you for your continued patience here.

Update September 1, 2021 - 1:00PM

Here are the latest updates on the 6 of servers still impacted:

KentFirth and Knight servers: new empty accounts are now available in your Client Area, under your original service, to allow you to start rebuilding. If you are already using a new account, you can continue using it without issues. In parallel, we are working with top disaster recovery teams to attempt to recuperate what we can but the progress is going to be slow and unlikely to result in a full recovery of data. You can use your new accounts knowing that they will not in any way hinder our recovery efforts.

Heaven and Bernie servers: we have secured files (but not databases) and are in the process of restoring them. They will be made available to you as possible (ETA: later today)

In addition, some data loss occurred for a small subset of resellers on server Robert. To help stabilize existing accounts, we are scheduling a migration of all data & accounts tonight and will be including data from this server in our recovery efforts. Resellers with missing data there are strongly encouraged to restore functionality from local backups.