Fix Security Log tab loading the entire multi-GB log into memory

parse_security_log() did f.readlines() on ASTERISK_LOG before slicing the
last 5000 lines - that reads the ENTIRE file into memory first. That log
is Asterisk's unrotated console/security output, and this tab polls it
every 30 seconds from the browser. Confirmed live: on a 1GB-RAM droplet
with a 1.4GB log file, this ballooned the dashboard (explicitly meant to
be a lightweight stdlib-only process) to 677MB RSS / 1.8GB peak swap,
which left CrowdSec unable to even start (boot timeout) and directly
contributed to the droplet becoming unresponsive.

Fixed by reading only a bounded ~2MB tail from the end of the file
(seek + fixed-size read) instead of the whole thing - memory use is now
constant regardless of how large the log grows. Tested against a 180MB
synthetic log: memory delta dropped from being proportional to file size
to ~7MB, in 0.05s.

Also added log rotation (services/asterisk-digital-ocean.sh) for that same
file, which had no rotation at all and reached 1.4GB in about 3 days -
copytruncate avoids needing to signal the containerized Asterisk process
to reopen its log handle.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ho9mZgAkVpdz7S5wJkg8Nf
This commit is contained in:
Claude
2026-07-23 04:44:19 +00:00
parent 78456e6a3a
commit 616dcdcfbb
2 changed files with 53 additions and 3 deletions
+25
View File
@@ -270,6 +270,29 @@ _asterisk_do_refresh_vendor_files() {
fi
}
# ── Shared: log rotation for logs/full (unbounded otherwise) ──────────────
# Confirmed live: with no rotation, this file grew to 1.4GB in about 3 days
# on a busy box (SIP scanning noise is constant on the public internet) —
# a real disk-exhaustion risk on a small droplet, and separately made the
# Security Dashboard balloon to 600+MB RAM/GBs of swap reading it every 30s
# before that was fixed to only read a bounded tail (see
# services/security-dashboard.sh). copytruncate avoids needing to signal
# Asterisk to reopen its log file — it has a long-held file descriptor on
# this path and no reload mechanism this installer can reach from the host.
_asterisk_do_write_logrotate() {
local _ea_dir="$1"
cat > /etc/logrotate.d/asterisk-digital-ocean << LOGROTATE
$_ea_dir/logs/full {
size 100M
rotate 5
compress
missingok
notifempty
copytruncate
}
LOGROTATE
}
# ── Shared: docker-compose.yml ─────────────────────────────────────────────
# Same reasoning as above — one copy of the template used by both fresh
# installs and updates. Must be called with $PWD already at $EA_DIR.
@@ -379,6 +402,7 @@ install_asterisk-digital-ocean() {
_asterisk_do_refresh_vendor_files
_asterisk_do_write_compose
_asterisk_do_write_logrotate "$EA_DIR"
log_info "Rebuilding and restarting containers..."
if docker compose up -d --build --force-recreate; then
@@ -447,6 +471,7 @@ install_asterisk-digital-ocean() {
cd "$EA_DIR" || return 1
_asterisk_do_refresh_vendor_files
_asterisk_do_write_logrotate "$EA_DIR"
# ── DigitalOcean droplet detection ────────────────────────────────────────
# A droplet's own public IP/ID are readable, unauthenticated, from the
+28 -3
View File
@@ -659,18 +659,43 @@ TIER_RE = re.compile(r"^(internal|restricted|full)$")
NUMBER_RE = re.compile(r"^\d{11}$")
SECURITY_LOG_TAIL_BYTES = 2 * 1024 * 1024 # comfortably enough for 5000 lines
def parse_security_log(limit=200):
"""Tail ASTERISK_LOG and return the most recent SecurityEvent lines,
newest first, as dicts. Missing file / no lines -> empty list, never an
error — this is a convenience view, not load-bearing."""
error — this is a convenience view, not load-bearing.
Reads only a bounded byte window from the END of the file, not the whole
thing — this log is Asterisk's unrotated console/security output and can
grow to multiple GB. The previous version did f.readlines() (loads the
ENTIRE file into memory) before slicing the last 5000 lines, and this
tab polls every 30 seconds from the browser. Confirmed live: on a 1GB-RAM
droplet with a 1.4GB log file, that ballooned this "stdlib only,
deliberately lightweight" process to 677MB RSS / 1.8GB peak swap, which
left CrowdSec unable to even start (boot timeout) and contributed
directly to the droplet becoming unresponsive. Bounding this to a fixed
~2MB window keeps memory use constant regardless of how large the log
file grows.
"""
if not ASTERISK_LOG or not os.path.isfile(ASTERISK_LOG):
return []
events = []
try:
with open(ASTERISK_LOG, "r", errors="replace") as f:
lines = f.readlines()[-5000:] # cap how much we ever scan
with open(ASTERISK_LOG, "rb") as f:
f.seek(0, os.SEEK_END)
size = f.tell()
start = max(0, size - SECURITY_LOG_TAIL_BYTES)
f.seek(start)
data = f.read()
except OSError:
return []
text = data.decode("utf-8", errors="replace")
lines = text.splitlines()
if start > 0 and lines:
lines = lines[1:] # first line is likely truncated mid-line
lines = lines[-5000:]
for line in lines:
if "SecurityEvent=" not in line:
continue