Internet outage affecting connectivity to BlackLine.
Incident Report for BlackLine
Resolved
All environments are restored to pre-event status. Thank you for your patience and understanding.
Posted Jul 20, 2024 - 22:54 UTC
Update
BlackLine GCP Production and Sandbox environments for both US and EU have been restored.

The BlackLine US2 and AMS Production environments are now fully operational.

The BlackLine US2 Sandbox environment is also fully restored. There may still be isolated instances of functionality that are not operational for the EU Sandbox.

All core systems should now be available. We are working to ensure full redundancy across all services.
Posted Jul 20, 2024 - 20:08 UTC
Update
BlackLine GCP Production and Sandbox environments for both US and EU have been restored.

The BlackLine US2 and AMS Production environments are now fully operational.

The BlackLine US2 Sandbox environment is 99% restored. There may be isolated instances of functionality that is not operational. We are working to complete these restorations.

All core systems should now be available. We are working to ensure full redundancy across all services.
Posted Jul 20, 2024 - 18:01 UTC
Update
We are continuing to work on a fix for this issue.
Posted Jul 20, 2024 - 17:37 UTC
Update
BlackLine GCP Production and Sandbox environments for both US and EU have been restored.

The BlackLine US2 and AMS Production environments are now fully operational.

The BlackLine US2 Sandbox environment is 99% restored. There may be isolated instances of functionality that is not operational. We are working to complete these restorations.
Posted Jul 20, 2024 - 17:29 UTC
Update
Saturday Morning Update:

The BlackLine US2 and AMS Production environments are now fully operational. There are still very isolated pieces of functionality that may not be fully operational, and we are working to resolve those issues.

GCP Environments are 95% operational. Users should be able to log in and utilize the majority of functions. Sandbox environments are 90% or more, and we are making progress this morning.
Posted Jul 20, 2024 - 15:00 UTC
Update
The BlackLine US2 and AMS Production environments are now fully operational.

The BlackLine US2 Sandbox environment is 95% restored. However, some services are still non-operational, and we are working to make them operational.

We are actively working through the GCP environments; however, we do not have a completion estimate.
Posted Jul 20, 2024 - 02:24 UTC
Update
The BlackLine US2 Production environment is now fully operational.

The BlackLine US2 Sandbox environment is 95% restored. However, some services are still non-operational, and we are working to make them operational.

AMS environments are approximately 80% restored, with some services impacted.

We are actively working through the GCP environments; however, we do not have a completion estimate.
Posted Jul 19, 2024 - 23:40 UTC
Update
BlackLine’s US2 environments are now at 95% restored. However, some services, such as Account Analysis, Excel Imports, and Journal services, are still non-operational. We are working to get those services operational.

AMS environments are approximately 80% restored, with some services impacted.

We are actively working through the GCP environments; however, we do not have a completion estimate.
Posted Jul 19, 2024 - 21:50 UTC
Update
BlackLine’s US2 environments are about 90% restored. However, some services, such as Account Analysis, Excel Imports, and Journal services, are still non-operational. We are working to get those services operational.

AMS environments are approximately 80% restored, with some services impacted.

We are actively working through the GCP environments; however, we do not have a completion estimate.
Posted Jul 19, 2024 - 19:59 UTC
Update
BlackLine is utilizing all available resources to restore services to our customers in the wake of the global issue that surfaced last night. While the root cause has been fully understood, we appreciate your patience as BlackLine personnel must manually confirm each affected server, which is a complicated effort. Please be aware that there is no issue with BlackLine software; all data is fully secure, and we are working at top priority to provide complete functionality as quickly and safely as possible.

On a related note, please be advised that all scheduled BlackLine release and update activities for this weekend have been postponed.
Posted Jul 19, 2024 - 17:45 UTC
Update
US2 PROD+SB clients may be able to log in however not all services are online and can affect imports, job processes, reporting, etc. BlackLine engineers are currently triaging and working to restore service.
Posted Jul 19, 2024 - 12:38 UTC
Update
We are continuing to work on a fix for this issue.
Posted Jul 19, 2024 - 11:53 UTC
Update
BlackLine engineers are currently triaging and working to restore service.
Posted Jul 19, 2024 - 08:20 UTC
Update
BlackLine has identified an issue with our anti-malware software impacting Windows systems. The team is currently triaging and working to restore service.
Posted Jul 19, 2024 - 07:33 UTC
Identified
BlackLine has identified an issue with our anti-malware software impacting Windows systems. The team is currently triaging and working to restore service.
Posted Jul 19, 2024 - 06:55 UTC
Update
Due to a larger outage across the internet some customers may experience issues with connectivity to BlackLine. We are monitoring and will continue to work with providers to address connectivity.
Posted Jul 19, 2024 - 06:28 UTC
Investigating
Customers may be experiencing an issue where BlackLine Production instances are unavailable. BlackLine teams are investigating the issue. We will provide regular updates on this page until the issue is resolved.
Posted Jul 19, 2024 - 06:06 UTC
This incident affected: Financial Close - Production (PROD US2, PROD EU1, PROD NA3, PROD EU3, PROD AP1), Financial Close - Sandbox (SB US2, SB EU1, SB NA3, SB EU3, SB AP1), Invoice-to-Cash - Sandbox (US, EU, AU), Invoice-to-Cash - Production (US, EU), Intercompany - Production (US), and Intercompany - Sandbox (US).