Fix Security Log tab loading the entire multi-GB log into memory
parse_security_log() did f.readlines() on ASTERISK_LOG before slicing the last 5000 lines - that reads the ENTIRE file into memory first. That log is Asterisk's unrotated console/security output, and this tab polls it every 30 seconds from the browser. Confirmed live: on a 1GB-RAM droplet with a 1.4GB log file, this ballooned the dashboard (explicitly meant to be a lightweight stdlib-only process) to 677MB RSS / 1.8GB peak swap, which left CrowdSec unable to even start (boot timeout) and directly contributed to the droplet becoming unresponsive. Fixed by reading only a bounded ~2MB tail from the end of the file (seek + fixed-size read) instead of the whole thing - memory use is now constant regardless of how large the log grows. Tested against a 180MB synthetic log: memory delta dropped from being proportional to file size to ~7MB, in 0.05s. Also added log rotation (services/asterisk-digital-ocean.sh) for that same file, which had no rotation at all and reached 1.4GB in about 3 days - copytruncate avoids needing to signal the containerized Asterisk process to reopen its log handle. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ho9mZgAkVpdz7S5wJkg8Nf
This commit is contained in:
@@ -270,6 +270,29 @@ _asterisk_do_refresh_vendor_files() {
|
||||
fi
|
||||
}
|
||||
|
||||
# ── Shared: log rotation for logs/full (unbounded otherwise) ──────────────
|
||||
# Confirmed live: with no rotation, this file grew to 1.4GB in about 3 days
|
||||
# on a busy box (SIP scanning noise is constant on the public internet) —
|
||||
# a real disk-exhaustion risk on a small droplet, and separately made the
|
||||
# Security Dashboard balloon to 600+MB RAM/GBs of swap reading it every 30s
|
||||
# before that was fixed to only read a bounded tail (see
|
||||
# services/security-dashboard.sh). copytruncate avoids needing to signal
|
||||
# Asterisk to reopen its log file — it has a long-held file descriptor on
|
||||
# this path and no reload mechanism this installer can reach from the host.
|
||||
_asterisk_do_write_logrotate() {
|
||||
local _ea_dir="$1"
|
||||
cat > /etc/logrotate.d/asterisk-digital-ocean << LOGROTATE
|
||||
$_ea_dir/logs/full {
|
||||
size 100M
|
||||
rotate 5
|
||||
compress
|
||||
missingok
|
||||
notifempty
|
||||
copytruncate
|
||||
}
|
||||
LOGROTATE
|
||||
}
|
||||
|
||||
# ── Shared: docker-compose.yml ─────────────────────────────────────────────
|
||||
# Same reasoning as above — one copy of the template used by both fresh
|
||||
# installs and updates. Must be called with $PWD already at $EA_DIR.
|
||||
@@ -379,6 +402,7 @@ install_asterisk-digital-ocean() {
|
||||
|
||||
_asterisk_do_refresh_vendor_files
|
||||
_asterisk_do_write_compose
|
||||
_asterisk_do_write_logrotate "$EA_DIR"
|
||||
|
||||
log_info "Rebuilding and restarting containers..."
|
||||
if docker compose up -d --build --force-recreate; then
|
||||
@@ -447,6 +471,7 @@ install_asterisk-digital-ocean() {
|
||||
cd "$EA_DIR" || return 1
|
||||
|
||||
_asterisk_do_refresh_vendor_files
|
||||
_asterisk_do_write_logrotate "$EA_DIR"
|
||||
|
||||
# ── DigitalOcean droplet detection ────────────────────────────────────────
|
||||
# A droplet's own public IP/ID are readable, unauthenticated, from the
|
||||
|
||||
@@ -659,18 +659,43 @@ TIER_RE = re.compile(r"^(internal|restricted|full)$")
|
||||
NUMBER_RE = re.compile(r"^\d{11}$")
|
||||
|
||||
|
||||
SECURITY_LOG_TAIL_BYTES = 2 * 1024 * 1024 # comfortably enough for 5000 lines
|
||||
|
||||
|
||||
def parse_security_log(limit=200):
|
||||
"""Tail ASTERISK_LOG and return the most recent SecurityEvent lines,
|
||||
newest first, as dicts. Missing file / no lines -> empty list, never an
|
||||
error — this is a convenience view, not load-bearing."""
|
||||
error — this is a convenience view, not load-bearing.
|
||||
|
||||
Reads only a bounded byte window from the END of the file, not the whole
|
||||
thing — this log is Asterisk's unrotated console/security output and can
|
||||
grow to multiple GB. The previous version did f.readlines() (loads the
|
||||
ENTIRE file into memory) before slicing the last 5000 lines, and this
|
||||
tab polls every 30 seconds from the browser. Confirmed live: on a 1GB-RAM
|
||||
droplet with a 1.4GB log file, that ballooned this "stdlib only,
|
||||
deliberately lightweight" process to 677MB RSS / 1.8GB peak swap, which
|
||||
left CrowdSec unable to even start (boot timeout) and contributed
|
||||
directly to the droplet becoming unresponsive. Bounding this to a fixed
|
||||
~2MB window keeps memory use constant regardless of how large the log
|
||||
file grows.
|
||||
"""
|
||||
if not ASTERISK_LOG or not os.path.isfile(ASTERISK_LOG):
|
||||
return []
|
||||
events = []
|
||||
try:
|
||||
with open(ASTERISK_LOG, "r", errors="replace") as f:
|
||||
lines = f.readlines()[-5000:] # cap how much we ever scan
|
||||
with open(ASTERISK_LOG, "rb") as f:
|
||||
f.seek(0, os.SEEK_END)
|
||||
size = f.tell()
|
||||
start = max(0, size - SECURITY_LOG_TAIL_BYTES)
|
||||
f.seek(start)
|
||||
data = f.read()
|
||||
except OSError:
|
||||
return []
|
||||
text = data.decode("utf-8", errors="replace")
|
||||
lines = text.splitlines()
|
||||
if start > 0 and lines:
|
||||
lines = lines[1:] # first line is likely truncated mid-line
|
||||
lines = lines[-5000:]
|
||||
for line in lines:
|
||||
if "SecurityEvent=" not in line:
|
||||
continue
|
||||
|
||||
Reference in New Issue
Block a user