Krypt，KT美国洛杉矶机房失火发生在2023年5月21日下午4点左右，当时易秋网络老易正在大亚湾海边沙滩，突然收到几个客户的QQ，说机子挂了，我前一秒还在给机房续费账单，后一秒机房官网挂了。今天是火灾发生后的第13天，总算有点眉目了。可能大家都是第一次遇到机房火灾，灾后处理的效率，确实不尽人意。在开始的一周，大家都在观望，什么时候恢复机子，干网络的对时间，对机子在线率，要求特别高，但是机房也没说啥时候能恢复。在第二周才陆续有实操的，易秋老易这边找机房拿了一些备用机子，临时给客户用，洛杉矶独立服务器，拿了圣何塞的标准线路的独立服务器供客户免费用1个月先；ion的洛杉矶挂了的，用ion圣何塞同配置的机子替换；kt原始vmware虚拟技术的VPS，也有拿KT东部VPS替换，我这做的工作只能是琐碎的，尽量减少大家损失的，一切还有待机房的安排。

以前断断续续的的KT洛杉矶失火更新参考：https://www.eeqiu.com/index.php/announcements/59/%E5%85%B3%E4%BA%8EKT%E7%BE%8E%E5%9B%BD%E6%9C%BA%E6%88%BF%E6%95%85%E9%9A%9C%E8%AF%B4%E6%98%8E%E8%B0%A2%E8%B0%A2%E7%90%86%E8%A7%A3%E5%92%8C%E6%94%AF%E6%8C%81%E9%BA%BB%E7%83%A6%E8%80%90%E5%BF%83%E7%AD%89%E5%BE%85.html

2023年6月3日半夜1点50收到机房正规邮件说明：

As previously shared, our Los Angeles LAX10 colocation suite (“Colo 4”) at 2260 W El Segundo Blvd was affected by an event due to causes beyond our reasonable control on Sunday, May 21, 2023, resulting in fire and water damage rendering Colo 4 inoperative. In response, we expedited the installation of new cabinets, power circuits, and network fiber connectivity in an adjacent colocation suite within the same building, reducing the standard installation process by several weeks. 
 
Our colocation, cloud, and bare metal customers in Colo 4 were directly affected by this incident.  As of today, colocation customers have been provided the infrastructure necessary to migrate their environments, cloud customers with offsite disaster recovery services have been restored at a remote facility, and bare metal server environments are being moved to a new location for data access. Unfortunately, our local cloud storage and backup systems in Colo 4 were significantly affected by the incident and cloud customers without offsite disaster recovery services are still affected. We have replaced hardware for both the local storage and backup systems, but several hard drives remain damaged, affecting data recovery efforts. We are actively working with drive manufacturers on data recovery solutions.
 

如前所述，我们位于2260 W El Segundo Blvd的洛杉矶LAX10主机代管套房（“Colo 4”）于2023年5月21日（星期日）因超出我们合理控制范围的原因受到事件影响，导致火灾和水损坏，使Colo 4无法运行。作为回应，我们加快了在同一栋建筑内相邻主机代管套件中安装新机柜、电源电路和网络光纤连接的速度，将标准安装流程缩短了几周。

我们在Colo 4的主机代管、云和裸金属客户直接受到了这一事件的影响。截至目前，主机代管客户已获得迁移其环境所需的基础架构，具有异地灾难恢复服务的云客户已在远程设施中恢复，裸金属服务器环境正在移动到新的位置以进行数据访问。不幸的是，我们在Colo 4的本地云存储和备份系统受到了该事件的严重影响。

易秋老易简短对上面一段话的理解，意思是大部分的lax10独立服务器大概率可以恢复，恢复只是时间问题。VPS客户的话，受影响比较大，因为机房几块硬盘坏了。有异地备份的ion VPS已经恢复，没异地备份的ion VPS还要等待硬盘厂商看看能不能恢复数据。kt老的vmare虚拟技术不限流量的VPS，这次影响也很大。

截止2023.6.4上午10点，洛杉矶lax10独立服务器，易秋网络这里已经恢复了25个机子，还有个别机子，机房说通电的，但是不通，还需要耐心等等。

截止2023.6.23上午11点，KT洛杉矶失火后恢复工作，也进入一个阶段了。目前易秋网络老易这的机子，独立服务器已经恢复一大半了，ion也恢复不少了，还剩下vmware虚拟技术的VPS一个也没恢复。下面是机房的一点更新：意思是老的vmware虚拟技术的VPS全部团灭，恢复无望。

Dear Valued Customers:

As previously shared, our Los Angeles LAX10 colocation suite (“Colo 4”) at 2260 W El Segundo Blvd was affected by an event due to causes beyond our reasonable control on Sunday, May 21, 2023, rendering Colo 4 inoperative.

Over the past several weeks we have worked with third-party data recovery vendors who have now exhausted their data recovery efforts. Unfortunately, the incident at our Los Angeles data center impacted both primary and local backup services and cloud customers that did not have remote backup or offsite disaster recovery services will not be able to recover their data.

We are now in the process of bringing up a new cloud environment. Cloud customers that have been affected will be set up on this new environment. We encourage you to consider remote backup services and explore additional disaster recovery services. Please visit krypt.com or contact your sales representative for more information.

如前所述，我们位于2260 W El Segundo Blvd的洛杉矶LAX10主机代管套房（“Colo 4”）在2023年5月21日（星期日）因超出我们合理控制范围的原因受到事件影响，导致Colo 4无法运行。

在过去的几周里，我们与第三方数据恢复供应商合作，他们现在已经耗尽了数据恢复工作。不幸的是，我们洛杉矶数据中心的事件影响了主备份和本地备份服务，没有远程备份或异地灾难恢复服务的云客户将无法恢复其数据。

我们现在正在建立一个新的云环境。受影响的云客户将在此新环境中进行设置。我们鼓励您考虑远程备份服务，并探索其他灾难恢复服务。