The database connection pool exhaustion on prod-db-03, where all 200 connections were in use and 47 were waiting (ctx://meridian/splunk-events/evt_f7c7ee7a962e92a8), was directly preceded by critical memory pressure.
Investigated in 6.5s·24 artifacts read·7 entities traversed
Impact
summarizing business impact…
Severity
critical
Events in chain
9
Window
57s
from 20:45:21
Entities affected
6
I·2 · B·4 · C·0
Actions
3
2 high / crit
Root cause hypothesis
agent-meridian-reasoner
The database connection pool exhaustion on prod-db-03, where all 200 connections were in use and 47 were waiting (ctx://meridian/splunk-events/evt_f7c7ee7a962e92a8), was directly preceded by critical memory pressure. Host metrics show memory utilization reached 94% (ctx://meridian/splunk-events/evt_3700c48ba9c04cf7), exceeding the 85% warning threshold and likely causing performance degradation that prevented timely release of database connections.
Causal chain · 9 events
ordered by USC temporal
20:45:21 UTC
metrics:host on prod-db-03: {"metric":"mem.used_pct","value":94,"rss_gb":14.85,"threshold_warn":85,…
ctx://meridian/splunk-events/evt_3700c48ba9c04cf7
20:45:30 UTC
metrics:host on prod-db-03: {"metric":"mem.used_pct","value":92.6,"rss_gb":14.6,"threshold_warn":85…
ctx://meridian/splunk-events/evt_029a43057ff891bd
20:45:57 UTC
metrics:host on prod-db-03: {"metric":"mem.used_pct","value":89.8,"rss_gb":14.1,"threshold_warn":85…
ctx://meridian/splunk-events/evt_9adbff8759d53edb
20:46:00 UTC
metrics:host on prod-db-03: {"metric":"mem.used_pct","value":87,"rss_gb":13.6,"threshold_warn":85,"…
ctx://meridian/splunk-events/evt_7c1b5dbf2de33693
20:46:13 UTC
metrics:host on prod-db-03: {"metric":"mem.used_pct","value":88.4,"rss_gb":13.85,"threshold_warn":8…
ctx://meridian/splunk-events/evt_ddaf0188128f2ecd
20:46:16 UTC
metrics:host on prod-db-03: {"metric":"mem.used_pct","value":91.2,"rss_gb":14.35,"threshold_warn":8…
ctx://meridian/splunk-events/evt_02915d9f6a7e81d5
20:46:18 UTC
db:error on prod-db-03: {"level":"ERROR","component":"conn_pool","pool_max":200,"pool_in_use":200,"…
ctx://meridian/splunk-events/evt_a5fc9d55c150c15b
20:46:18 UTC
db:error on prod-db-03: {"level":"ERROR","component":"conn_pool","pool_max":200,"pool_in_use":200,"…
ctx://meridian/splunk-events/evt_b499f11d9a886302
20:46:18 UTC
db:error on prod-db-03: {"level":"ERROR","component":"conn_pool","pool_max":200,"pool_in_use":200,"…
01criticalImmediately investigate and mitigate the source of excessive memory usage on prod-db-03, as metrics show mem.used_pct at 94% (ctx://meridian/splunk-events/evt_3700c48ba9c04cf7).
02highAnalyze database configuration and application connection logic to determine why connections were not being released, leading to the exhausted pool (ctx://meridian/splunk-events/evt_f7c7ee7a962e92a8).
03mediumNotify all entities within the blast radius, including service-a and clients under the Alpha 99.9% Uptime SLA, about the service interruption.