Update - The block reclamation revalidation has completed successfully at this time. The array continues to remain stable at this time. We are continuing to closely monitor and assess with the vendor.
Jul 3, 09:16 EDT
Jul 3, 09:16 EDT
Update - The array has continued to remain stable with minimal latency while the background process continues to run. We will continue to monitor along with our vendor at this time.
Jul 2, 10:28 EDT
Jul 2, 10:28 EDT
Update - Storage latency has remained at expected levels throughout the day today. At the request of our vendor, we are starting a process to assist with block reclamation. This change is non-service affecting and is not expected to create additional latency and is only planned to run for a limited period of time. The process will be stopped if any unexpected issues occur.
Jul 1, 22:09 EDT
Jul 1, 22:09 EDT
Update - The lower latency we saw this morning has continued throughout the day today and the system has remained stable. We are continuing to monitor and adjust based on recommendations from the storage vendor's engineering team.
Jun 30, 19:16 EDT
Jun 30, 19:16 EDT
Update - We are continuing to monitor and adjust based on recommendations from the storage vendor's engineering team. Further work overnight and into this morning has led to a significant further reduction in latency on the array and the system remains stable at this time.
Jun 30, 12:21 EDT
Jun 30, 12:21 EDT
Update - At the request of the storage vendor we are postponing this planned failover event. We will update prior to a future scheduled failover.
Jun 29, 17:22 EDT
Jun 29, 17:22 EDT
Update - Overall latency has been stable, but still elevated. We are continuing to move workloads off of the affected array. We are forcing a failover of the affected SAN's controllers at 5:30 PM EDT today. Customers will experience high disk latency as the controllers failover and IO is transferred to the standby controller.
Jun 29, 16:48 EDT
Jun 29, 16:48 EDT
Update - Latency has improved, but it still elevated. We are continuing to work with the storage vendor while moving workloads off of the affected array.
Jun 29, 07:47 EDT
Jun 29, 07:47 EDT
Update - We continue to work with the storage vendor to address the latency issue. Latency has been reduced further overnight but is still elevated. While we continue to work with the storage vendor, we are still moving workloads off of the affected array.
Jun 29, 05:44 EDT
Jun 29, 05:44 EDT
Update - We continue to work with the storage vendor to address the latency issue. Latency has been reduced, but is still elevated. While we continue to work with the storage vendor, we are still moving workloads off of the affected array.
Jun 28, 23:27 EDT
Jun 28, 23:27 EDT
Update - The original bug that this array encountered when the latency issue started was run into again. We are continuing to work with the storage vendor's engineering team to resolve the issue. While that work continues we are also moving additional workloads off the array to try and stabilize the performance of the array.
Jun 28, 18:58 EDT
Jun 28, 18:58 EDT
Investigating - SAN latency had been stable throughout the day. However, at approximately 4:30PM the SAN experienced an unexpected service restart. This has increased latency while the services catch back up. We are working with the vendor to determine the cause of the latest service restart.
Jun 28, 16:48 EDT
Jun 28, 16:48 EDT
Update - Latency has remained stable since the last changes. We are continuing to monitor the status while we work to move workloads off of this array.
Jun 28, 08:43 EDT
Jun 28, 08:43 EDT
Monitoring - The recent changes have had a positive impact on the array and performance has improved. We are continuing to monitor and review with the vendor.
Jun 28, 04:58 EDT
Jun 28, 04:58 EDT
Update - We continue to work the storage vendor's engineering department. An additional change has been recommended. We are applying the change at 3:30AM. This requires a restart of the data service. Customers may experience slightly increased latency as the services restart.
Jun 28, 03:14 EDT
Jun 28, 03:14 EDT
Update - The changes have improved latency. The latency is improving more over time. We are taking additional measures to reduce overall IO on the array as well. We will provide additional updates as conditions change.
Jun 27, 18:35 EDT
Jun 27, 18:35 EDT
Update - The storage vendor has identified a setting that can help reduce the latency. We will be implementing the change at 5PM EDT. Customers may see slightly higher latency as services are restarted to apply the change.
Jun 27, 16:53 EDT
Jun 27, 16:53 EDT
Update - We are continuing to work with the storage vendor to resolve the issue. It has been escalated to their engineering team to continue working on a resolution. We have implemented some workarounds to try and reduce the IO on the array until the root cause can be resolved. We are reviewing potential additional changes to reduce the latency even further. Additional updates will be provided as they become available.
Jun 27, 16:12 EDT
Jun 27, 16:12 EDT
Update - Latency is still elevated and sporadic. We are continuing to work through the issue. More information will be provided as it becomes available.
Jun 27, 14:31 EDT
Jun 27, 14:31 EDT
Investigating - We have continued to monitor the status and review the cause with our storage vendor. Disk latency has started to slowly rise over the last 30 minutes. We are reviewing additional steps to get the latency back to appropriate levels.
Jun 27, 13:58 EDT
Jun 27, 13:58 EDT
Monitoring - The failover completed successfully and we have seen latency remain stable. We are continuing to monitor the situation. We will continue to work with the vendor to investigate the root cause and actions to prevent this issue in the future.
Jun 27, 12:53 EDT
Jun 27, 12:53 EDT
Investigating - Per vendor's recommendation we will be initiating a fail-over of the controllers at 12:15PM EDT today to try and alleviate the continued latency issues. During the failover latency may increase briefly as IO is moved to the redundant controller. An update will be sent out when that is completed.
Jun 27, 12:05 EDT
Jun 27, 12:05 EDT
Update - We continue to see fluctuations in latency and are continuing to troubleshoot with the vendor. More information will be provided as it becomes available.
Jun 27, 11:37 EDT
Jun 27, 11:37 EDT
Update - We started to see an increase in storage latency again. We are continuing to investigate the issue with the storage vendor.
Jun 27, 10:58 EDT
Jun 27, 10:58 EDT
Monitoring - Disk latency is returning to normal. We are still investigating the root cause and monitoring the situation. Additional updates will be provided as available.
Jun 27, 10:45 EDT
Jun 27, 10:45 EDT
Update - The active SAN controller experienced a software failure that caused it to crash and fail-over. The fail-over should not cause high latency, we are reviewing the issue with the vendor. The controllers are in a fully redundant state. Additional details will be provided as available.
Jun 27, 10:18 EDT
Jun 27, 10:18 EDT
Investigating - We are currently investigating a storage latency issue with one of our arrays. Customers on the affected array may see slow response times for for storage. We will provide more information as we have it.
Jun 27, 09:43 EDT
Jun 27, 09:43 EDT
Cloud Applications
Operational
LogicMonitor Monitoring
Operational
Solarwinds Monitoring
Operational
Proofpoint E-Mail Security
Operational
Mailstore Mail Archiving
Operational
FileCloud File Sharing
Operational
PIT01 (Pittsburgh, PA)
Degraded Performance
Virtual Infrastructure
Operational
Network Infrastructure
Operational
Storage Infrastructure
Degraded Performance
Internet Connectivity
Operational
IdealConnect Connectivity
Operational
IdealVoice SIP Connectivity
Operational
STL01 (St. Louis, MO)
Operational
Virtual Infrastructure
Operational
Network Infrastructure
Operational
Storage Infrastructure
Operational
Internet Connectivity
Operational
IdealConnect Connectivity
Operational
PHL01 (Philadelphia, PA)
Operational
Virtual Infrastructure
Operational
Network Infrastructure
Operational
Storage Infrastructure
Operational
Internet Connectivity
Operational
IdealConnect Connectivity
Operational
PHX01 (Phoenix, AZ)
Operational
Virtual Infrastructure
Operational
Network Infrastructure
Operational
Storage Infrastructure
Operational
Internet Connectivity
Operational
IdealConnect Connectivity
Operational
TPA01 (Tampa, FL)
Operational
Virtual Infrastructure
Operational
Network Infrastructure
Operational
Storage Infrastructure
Operational
Internet Connectivity
Operational
IdealConnect Connectivity
Operational
BNA01 (Nashville, TN)
Operational
Virtual Infrastructure
Operational
Network Infrastructure
Operational
Storage Infrastructure
Operational
Internet Connectivity
Operational
IdealConnect Connectivity
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.