fix(agent-office): alive를 heartbeat staleness로 판정 + 다운/복구 전이 발송실패 시 재시도 (최종 리뷰 I1·I2)

I1: collect_status - heartbeat 키 존재 여부가 아닌 ts age 기반으로 alive 판정.
    age > NODE_STALE_THRESHOLD_SEC(90s, env 주입 가능)이면 키 있어도 dead.
    config.py에 NODE_STALE_THRESHOLD_SEC=90 추가.
I2: check_and_alert - 다운/복구 전이 시 send_raw 실패하면 _node_state 갱신 보류.
    다음 사이클에서 동일 전이 재감지 → 재발송 시도 (다운 이벤트 유실 방지).
테스트: _hb 헬퍼 현재 시각 기본값으로 수정 + 신규 2개 (stale→dead, I2 재시도 회귀).
14 passed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019LV86jBozkNhSFXJA412fq
This commit is contained in:
2026-06-29 18:50:45 +09:00
parent b49cc14ef3
commit 94cddccaa7
3 changed files with 72 additions and 13 deletions

View File

@@ -2,7 +2,7 @@
from __future__ import annotations
import datetime as dt, json, logging
import redis.asyncio as aioredis
from .config import REDIS_URL, NODE_ALERT_DEADLETTER_THRESHOLD
from .config import REDIS_URL, NODE_ALERT_DEADLETTER_THRESHOLD, NODE_STALE_THRESHOLD_SEC
logger = logging.getLogger("agent-office.node_monitor")
@@ -66,10 +66,13 @@ async def collect_status(redis=None) -> dict:
if raw:
try:
hb = json.loads(raw)
info.update(alive=True, state=hb.get("state"),
jobs_done=hb.get("jobs_done", 0), jobs_failed=hb.get("jobs_failed", 0),
last_job_at=hb.get("last_job_at"),
last_beat_age_s=_beat_age(hb.get("ts") or "", now))
age = _beat_age(hb.get("ts") or "", now)
info["last_beat_age_s"] = age
info["alive"] = age is not None and age <= NODE_STALE_THRESHOLD_SEC
info["state"] = hb.get("state")
info["jobs_done"] = hb.get("jobs_done", 0)
info["jobs_failed"] = hb.get("jobs_failed", 0)
info["last_job_at"] = hb.get("last_job_at")
if w["kind"] == "watcher" and hb.get("mode"):
out["paused_reason"] = hb["mode"]
except (json.JSONDecodeError, UnicodeDecodeError):
@@ -114,19 +117,24 @@ async def check_and_alert(status=None) -> list[str]:
return []
sent: list[str] = []
for w in st["workers"]:
name, alive = w["name"], w.get("alive", False)
name = w["name"]
alive = w.get("alive", False)
prev = _node_state.get(name)
transition_send_failed = False
if prev is True and not alive:
text = f"🔴 [{name}] 워커 다운"
if (await send_raw(text=text)).get("ok"):
add_log("node_monitor", f"{name} 다운", "warning")
sent.append(text)
add_log("node_monitor", f"{name} 다운", "warning"); sent.append(text)
else:
transition_send_failed = True
elif prev is False and alive:
text = f"🟢 [{name}] 워커 복구"
if (await send_raw(text=text)).get("ok"):
add_log("node_monitor", f"{name} 복구", "info")
sent.append(text)
_node_state[name] = alive
add_log("node_monitor", f"{name} 복구", "info"); sent.append(text)
else:
transition_send_failed = True
if not transition_send_failed:
_node_state[name] = alive
dl = w.get("dead_letter", 0)
if dl >= NODE_ALERT_DEADLETTER_THRESHOLD and dl != _dl_notified.get(name, 0):
text = f"❌ [{name}] 실패 누적 {dl}건 (dead-letter)"