Incident Summary

On August 12, 2026, between approximately 10:31 AM and 10:45 AM PDT, the Lever platform experienced partial unavailability. During this 14-minute window, some customers were unable to access the platform, and others experienced elevated error rates for API requests.

 Detection

The issue was identified through internal monitoring. A Datadog alert for high database page faults triggered at 10:37 AM PDT, notifying the engineering team. This allowed responders to isolate the specific database nodes experiencing high load and identify the responsible operations.

 Root Cause

The incident was triggered by a safety mechanism within an internal administrative tool designed to prevent expensive database queries. When an operator searched for unindexed data, this safety check requested a "plan" from the database to see if the query was safe to run. However, because the data was unindexed, the database attempted to evaluate every possible retrieval method, resulting in the same large account being scanned 11 times for a single request. These diagnostic checks bypassed standard time limits, consuming the database read capacity required for normal customer traffic.

 Resolution

Engineering teams identified and terminated the long-running diagnostic operations, which immediately restored database performance and platform stability. Following the cleanup, a database-level safeguard was applied to force these specific queries into a single, efficient path, preventing the database from attempting multiple expensive scans in the future.

 Preventative Measures

• Redesigning Safety Checks: We are replacing the current diagnostic method with a new system that verifies query safety by inspecting database definitions directly, eliminating the need to run expensive tests on live data.
• Advanced Alerting: We are implementing new monitoring for database read churn and long-running operations to detect and resolve similar issues before they affect customers.
• Enhanced Logging: We are improving internal audit logs to ensure faster identification and attribution of administrative database activity.
• Database Optimization: We are adding new indexes to properly support these search patterns, ensuring they are inherently efficient.