Compare commits

..

24 Commits

Author SHA1 Message Date
29ea82821d fix(naver-fetch): ingest 응답 필드 matched→queued (BE async 8e28ce9 정합) 2026-07-10 11:26:12 +09:00
8e64c4feb8 fix(naver-fetch): realestate base :18800 + 동별 ingest + 180s no-retry 2026-07-10 10:55:51 +09:00
0b4e4c2e0e feat(naver-fetch): Dockerfile(Playwright) + compose 서비스(18716) + .env.example 2026-07-10 10:21:54 +09:00
56c13cbcbd feat(naver-fetch): FastAPI lifespan + heartbeat 배선 + /health 2026-07-10 10:21:04 +09:00
e3c7f24794 feat(naver-fetch): fetch_worker (run_cycle/loop/heartbeat) 2026-07-10 10:21:03 +09:00
03deb12bd1 feat(naver-fetch): NaverClient (Playwright 토큰 + httpx fetch) 2026-07-10 10:21:03 +09:00
b0155830f2 feat(naver-fetch): NAS realestate 클라이언트 (targets/ingest) 2026-07-10 10:21:03 +09:00
6b0e91342c feat(naver-fetch): 스캐폴딩 + config 2026-07-10 10:21:03 +09:00
9abca3eeab docs: /infra 워커 관측 규칙 + trade-monitor climax 정합 반영
- CLAUDE.md: 모든 WSL docker 워커 /infra 관측 필수 규칙(BE 팀규칙) +
  services 행에 trade-monitor(:18715) 반영
- README.md: sell_climax 정합·36 tests·env 우선순위 문구 갱신

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N83vbXEA8h83GMXQcg8fxD
2026-07-03 11:15:36 +09:00
5dbb11ac83 fix(trade-monitor): sell_climax holdings_intel 정합
BE 회신(holdings_intel.py:109-118)에 맞춰 반전 기준을
price<day_open → price<day_high×climax_close_pct(윗꼬리)로 변경.
- kis_client.get_quote에 day_high(stck_hgpr) 추가
- monitor._build_ctx가 day_high를 ctx로 전달
- climax_vol_x·climax_close_pct를 monitor-set exit_params에서 읽기
  (fallback: TM_CLIMAX_VOL_MULT/0.97)
- 테스트 36/36 (climax exit_params 2건 추가)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N83vbXEA8h83GMXQcg8fxD
2026-07-03 11:15:27 +09:00
8e1b20190d docs(readme): trade-monitor 워커 섹션 + /infra 관측 규칙 반영 2026-07-03 11:05:57 +09:00
fa6ef6c5c8 feat(trade-monitor): Dockerfile + compose 서비스(18715) + .env.example 2026-07-03 01:48:14 +09:00
12aa55ed14 feat(trade-monitor): FastAPI lifespan + heartbeat 배선 + /health 2026-07-03 01:47:34 +09:00
ce8983c1b9 feat(trade-monitor): monitor 오케스트레이션 (run_cycle/loop/state_fn) 2026-07-03 01:47:34 +09:00
04aff34883 feat(trade-monitor): NAS trade-alert 클라이언트 (monitor-set/report) 2026-07-03 01:46:37 +09:00
d761716e00 feat(trade-monitor): KIS 자체 토큰 + quote + 일봉 클라이언트 2026-07-03 01:46:37 +09:00
241ce41a6a feat(trade-monitor): 매수/매도 조건 로직 (§6 8개 조건) 2026-07-03 01:45:41 +09:00
366a9160d5 feat(trade-monitor): 순수 지표 모듈 (sma/rsi/highest_high) 2026-07-03 01:45:41 +09:00
141209ad42 feat(trade-monitor): 스캐폴딩 + config 2026-07-03 01:44:25 +09:00
03e50d2be1 fix(task-watcher): _shared를 빌드 컨텍스트에 포함 (heartbeat import 크래시 수정)
task-watcher는 build context가 ./task-watcher라 services/_shared가 이미지에
없었음 → A3가 추가한 `from _shared.heartbeat import` ModuleNotFoundError로
컨테이너 즉시 크래시(재시작 후 alive=false). render 워커 패턴대로
context=. + COPY _shared /app/_shared + PYTHONPATH=/app 로 수정.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019LV86jBozkNhSFXJA412fq
2026-07-01 02:26:57 +09:00
54fca07d43 feat(ai_trade): NAS Redis heartbeat (trader market_open/closed)
- ai_trade/heartbeat.py: build_trader_payload() + heartbeat_loop() 자체 미니 헬퍼
  (Windows 호스트 실행이라 _shared import 경로 달라 독립 구현, 계약은 동일)
- ai_trade/main.py: lifespan에 hb_task spawn + shutdown 시 cancel
  state_fn = scheduler._is_market_day & _is_polling_window(KST now) 조합
  signals = len(state.signals) 실시간 주입
- requirements.txt: redis>=5.0 추가
- ai_trade/tests/test_heartbeat.py: build_trader_payload 3케이스 TDD 검증

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019LV86jBozkNhSFXJA412fq
2026-07-01 01:07:00 +09:00
574b5712c3 feat(task-watcher): heartbeat 발신 (state=mode, paused 이유 노출)
- watcher_loop 에서 mode 판정 직후 worker:task-watcher:heartbeat SET EX 45
- payload: build_payload(state=mode, extra={"mode": mode})
- LOOP_INTERVAL 30s < TTL 45s → 만료 전 주기적 갱신
- conftest.py 추가: services/ 를 sys.path에 주입해 _shared import 가능
- tests/test_watcher.py: payload kind/state/mode 필드 검증 (1 passed)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-01 00:59:28 +09:00
2ff31b2e76 feat(render-workers): 4 render 워커 heartbeat 배선 + poll_once 카운터
- services/_shared/heartbeat.py (A1) WorkerStats/utc_now_iso/heartbeat_loop 소비
- image-render / video-render / music-render / insta-render 각 worker.py:
  stats = WorkerStats() 모듈 레벨 추가, poll_once에서 dispatch 전 busy=True,
  ack 후 jobs_done+1 / fail 후 jobs_failed+1 + last_job_at + busy=False
- 각 main.py: lifespan에 aioredis(decode_responses=False) + heartbeat_loop 태스크 spawn,
  종료 시 cancel + aclose
- 각 tests/test_worker.py: test_poll_once_increments_jobs_done 추가
  (image:flux / video:sora / music:suno / insta:_process_one mock)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019LV86jBozkNhSFXJA412fq
2026-07-01 00:52:57 +09:00
d1b9ff570d feat(_shared): 워커 heartbeat 모듈 (worker:<name>:heartbeat TTL SET)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-01 00:43:01 +09:00
66 changed files with 4913 additions and 26 deletions

View File

@@ -15,12 +15,14 @@ Windows AI 머신 (AMD 9800X3D + RTX 5070 Ti 16GB) 의 두 신호 파이프라
| `ai_trade/` | 자동매매 메인 (구 `signal_v2` 2026-05-19 rename) — Chronos-bolt + 분봉 모멘텀 + KIS WebSocket + 신호 생성 | `:8001` | **Phase 4 완료 (2026-05-17)**, Phase 5 대기 | | `ai_trade/` | 자동매매 메인 (구 `signal_v2` 2026-05-19 rename) — Chronos-bolt + 분봉 모멘텀 + KIS WebSocket + 신호 생성 | `:8001` | **Phase 4 완료 (2026-05-17)**, Phase 5 대기 |
| `legacy/start_v1.bat` | (deprecated) V1 진입점 — root `start.bat`에서 이동됨. 자동 실행 차단 | — | **OFF** | | `legacy/start_v1.bat` | (deprecated) V1 진입점 — root `start.bat`에서 이동됨. 자동 실행 차단 | — | **OFF** |
| `ai_trade/start.bat` | 자동매매 진입점 | — | `ai_trade/main.py` uvicorn 실행 | | `ai_trade/start.bat` | 자동매매 진입점 | — | `ai_trade/main.py` uvicorn 실행 |
| `services/` | (예정) NAS↔Windows 분산 worker — insta-render·music-render·video-render·task-watcher | 18710~ | **Plan-B-Insta 작업 중** | | `services/` | NAS↔Windows 분산 worker — insta·music·video·image-render·task-watcher·**trade-monitor(:18715, 실시간 매매 알람)** | 18710~ | 운영 |
| `.env` | 환경변수 (`KIS_REAL_*`, `TELEGRAM_*`, `STOCK_API_URL`, `WEBAI_API_KEY`, `LOG_LEVEL`) | — | | | `.env` | 환경변수 (`KIS_REAL_*`, `TELEGRAM_*`, `STOCK_API_URL`, `WEBAI_API_KEY`, `LOG_LEVEL`) | — | |
| `requirements.txt` | 공용 의존성 | — | torch, chronos-forecasting, fastapi, httpx, websockets 등 | | `requirements.txt` | 공용 의존성 | — | torch, chronos-forecasting, fastapi, httpx, websockets 등 |
`.venv`**구조적으로 깨짐**: `pyvenv.cfg` 가 한글 사용자 경로(`C:\Users\박재오\...`) 를 포함하여 콘솔 코드페이지가 roundtrip 못함. 테스트는 시스템 Python 으로 실행: `C:\Users\jaeoh\AppData\Local\Programs\Python\Python312\python.exe -m pytest ai_trade/tests -q`. `.venv`**구조적으로 깨짐**: `pyvenv.cfg` 가 한글 사용자 경로(`C:\Users\박재오\...`) 를 포함하여 콘솔 코드페이지가 roundtrip 못함. 테스트는 시스템 Python 으로 실행: `C:\Users\jaeoh\AppData\Local\Programs\Python\Python312\python.exe -m pytest ai_trade/tests -q`.
> **분산 워커 /infra 관측 규칙 (팀 규칙, BE 제정)**: 모든 WSL docker 워커는 ① Redis `worker:<name>:heartbeat`(EX45) 발신 ② BE `node_monitor.WORKER_REGISTRY` 등재 ③ `/api/agent-office/nodes`·web-ui `/infra` 노출이 필수. trade-monitor는 kind=trader로 등재됨.
--- ---
## 서버 시작 방식 ## 서버 시작 방식

View File

@@ -3,7 +3,7 @@
Windows AI 머신(AMD 9800X3D + RTX 5070 Ti 16GB)에서 동작하는 두 영역의 서비스: Windows AI 머신(AMD 9800X3D + RTX 5070 Ti 16GB)에서 동작하는 두 영역의 서비스:
1. **ai_trade** — Confidence Signal Pipeline V2. NAS stock 백엔드와 KIS Open API를 결합해 매수/매도 신호를 생성하는 FastAPI 워커. 1. **ai_trade** — Confidence Signal Pipeline V2. NAS stock 백엔드와 KIS Open API를 결합해 매수/매도 신호를 생성하는 FastAPI 워커.
2. **services** — NAS↔Windows 분산 렌더링 워커(인스타 카드 / 음악 / 영상 / 이미지) + task-watcher. 2. **services** — NAS↔Windows 분산 워커: 렌더링(인스타 카드 / 음악 / 영상 / 이미지) + task-watcher + **trade-monitor**(실시간 매매 알람).
상위 워크스페이스 컨텍스트는 `../CLAUDE.md`, 본 디렉토리 상세는 `CLAUDE.md`, 운영 체크포인트는 `CHECK_POINT.md` 참조. 상위 워크스페이스 컨텍스트는 `../CLAUDE.md`, 본 디렉토리 상세는 `CLAUDE.md`, 운영 체크포인트는 `CHECK_POINT.md` 참조.
@@ -20,6 +20,7 @@ Windows AI 머신(AMD 9800X3D + RTX 5070 Ti 16GB)에서 동작하는 두 영역
| `services/video-render/` | sora / veo / kling / seedance 4 provider 영상 생성 게이트웨이. `queue:video-render` 소비. | `:18712` | | `services/video-render/` | sora / veo / kling / seedance 4 provider 영상 생성 게이트웨이. `queue:video-render` 소비. | `:18712` |
| `services/image-render/` | gpt_image / nano_banana / flux(ComfyUI 로컬) 3 provider. `queue:image-render` 소비. | `:18714` | | `services/image-render/` | gpt_image / nano_banana / flux(ComfyUI 로컬) 3 provider. `queue:image-render` 소비. | `:18714` |
| `services/task-watcher/` | 박재오 작업 시간대에 `queue:paused` 토글 → 워커 일시 정지. | `:18713` | | `services/task-watcher/` | 박재오 작업 시간대에 `queue:paused` 토글 → 워커 일시 정지. | `:18713` |
| `services/trade-monitor/` | 실시간 매매 알람. monitor-set pull → KIS 시세 + TA 조건(§6 8종) → report + heartbeat(kind=trader). 무상태. | `:18715` |
| `legacy/signal_v1/` | ⚠ **DEPRECATED** (2026-05-19). LSTM 봇. 자동 실행 차단됨. | OFF | | `legacy/signal_v1/` | ⚠ **DEPRECATED** (2026-05-19). LSTM 봇. 자동 실행 차단됨. | OFF |
--- ---
@@ -130,6 +131,60 @@ redis-cli -h 192.168.45.54 KEYS 'processing:*'
--- ---
## trade-monitor — 실시간 매매 알람 워커 (신규 2026-07-03)
NAS stock 백엔드(`:18500`)에서 `monitor-set`을 60초마다 pull하고, KIS 실시간/일봉 시세로 TA 조건을 평가해 발화집합을 `report`로 전송. NAS가 edge diff로 신규 알림만 텔레그램/프론트에 노출. **워커 무상태**(dedup은 NAS 영속). 포트 `:18715`, WSL2 Docker. 상세 설계·조건 규칙은 `services/trade-monitor/DESIGN.md`.
### 루프 (1분)
1. `GET /api/webai/trade-alert/monitor-set` (X-WebAI-Key) → buy_targets(watchscreener) + sell_targets(보유 avg_price/qty/holding_high) + buy_params/exit_params + session.
2. `session=="closed"`면 KIS 호출 0, idle. **비-KRX(알파벳) 티커 skip**(워커 책임).
3. KIS quote + 일봉 250봉 → 지표 계산 → 조건 평가(종목 단위 실패 격리).
4. `POST /api/webai/trade-alert/report {as_of, firing:[...]}` — 빈 배열도 전송(edge clear).
5. heartbeat `worker:trade-monitor:heartbeat` EX45 (kind=trader, state=market_open|market_closed|idle + last_alert_at). 60초 루프 > TTL45 만료갭 회피 위해 **15초 독립 태스크**.
### 조건 (§6 — `condition` 문자열이 FE 라벨/뱃지로 그대로 매핑됨)
- **매수**: `buy_ma20_pullback`(정배열 + ma20 근접 반등), `buy_breakout`(20봉 고점 돌파 + 거래량 배수), `buy_rsi_bounce`(RSI 과매도 반등, 무상태).
- **매도**: `sell_stop_loss`, `sell_take_profit`, `sell_trailing_stop`, `sell_ma_break`(ma50/ma200 severity), `sell_climax`(거래량 급증 + `price<day_high×0.97` 윗꼬리 — holdings_intel 정합됨, `exit_params` 파라미터화).
### 핵심 파일
| 파일 | 책임 |
|------|------|
| `config.py` | Settings (`TM_` 접두사, ai_trade와 분리) |
| `indicators.py` | 순수: `sma` / `rsi_series`(Wilder) / `highest_high` |
| `conditions.py` | 순수 §6: `evaluate_buy` / `evaluate_sell` |
| `kis_client.py` | KIS **자체 OAuth 토큰** + `get_quote` + `get_daily_ohlcv` + 0.5s throttle |
| `nas_client.py` | monitor-set / report (X-WebAI-Key + retry) |
| `monitor.py` | `run_cycle` / `monitor_loop` / `make_state_fn` |
| `main.py` | FastAPI lifespan + `_shared.heartbeat_loop` 배선 + `/health` |
### 환경 변수
| 변수 | 기본 | 설명 |
|------|------|------|
| `NAS_BASE_URL` | `http://192.168.45.54:18500` | stock 백엔드 |
| `WEBAI_API_KEY` | (필수) | X-WebAI-Key |
| `REDIS_URL` | `redis://192.168.45.54:6379` | heartbeat |
| `TM_KIS_APP_KEY` / `TM_KIS_APP_SECRET` / `TM_KIS_ACCOUNT` | (필수) | KIS **전용** 자체 토큰(ai_trade와 분리 발급 → 토큰 상호 무효화·EGW00201 회피) |
| `TM_KIS_IS_VIRTUAL` | `0` | 실전/모의 |
| `TM_LOOP_INTERVAL` | `60` | 루프 주기(초) |
| `TM_CLIMAX_VOL_MULT` | `3.0` | sell_climax 거래량 배수 fallback (우선순위: monitor-set `exit_params.climax_vol_x` > 이 값) |
### 상태
⏳ 구현·머지 완료(테스트 36/36, sell_climax holdings_intel 정합 포함), **미배포**. 배포 전: ① 전용 KIS 앱키 발급·주입(박재오 진행 중) ② 첫 운영 KIS 필드 검증(stck_hgpr 등). BE가 `node_monitor.WORKER_REGISTRY`에 등재 완료 → 배포 시 `/api/agent-office/nodes`·web-ui `/infra`에 trader 노드 자동 노출(미배포 동안 down, 무경보).
### 시작 (NAS, WSL2 Docker)
```bash
cd services
docker compose up -d trade-monitor
```
---
## 환경 변수 (ai_trade) ## 환경 변수 (ai_trade)
| 변수 | 기본 | 설명 | | 변수 | 기본 | 설명 |
@@ -168,6 +223,7 @@ cd services/insta-render && python -m pytest tests/ -q
cd services/music-render && python -m pytest tests/ -q cd services/music-render && python -m pytest tests/ -q
cd services/video-render && python -m pytest tests/ -q cd services/video-render && python -m pytest tests/ -q
cd services/image-render && python -m pytest tests/ -q cd services/image-render && python -m pytest tests/ -q
cd services/trade-monitor && python -m pytest tests/ -q # 36 tests
``` ```
**`.venv` 한글 사용자 경로 깨짐**으로 시스템 Python(`C:\Users\jaeoh\AppData\Local\Programs\Python\Python312\python.exe`) 사용 권장. 또는 `py -3.12 -m pytest …`. **`.venv` 한글 사용자 경로 깨짐**으로 시스템 Python(`C:\Users\jaeoh\AppData\Local\Programs\Python\Python312\python.exe`) 사용 권장. 또는 `py -3.12 -m pytest …`.
@@ -183,6 +239,8 @@ cd services/image-render && python -m pytest tests/ -q
5. **services worker_id** — env로 명시 권장. hostname 기반 default는 컨테이너 재기동 시 바뀌어 orphan 분실 위험. 5. **services worker_id** — env로 명시 권장. hostname 기반 default는 컨테이너 재기동 시 바뀌어 orphan 분실 위험.
6. **dead-letter 누적**`redis-cli LLEN dead_letter:*` 정기 점검 필요. 6. **dead-letter 누적**`redis-cli LLEN dead_letter:*` 정기 점검 필요.
7. **Dockerfile build context**`services/` 루트 (각 worker 디렉토리 아님). compose 변경 동반. 7. **Dockerfile build context**`services/` 루트 (각 worker 디렉토리 아님). compose 변경 동반.
8. **분산 워커 /infra 관측 필수 (팀 규칙)** — 모든 WSL docker 워커는 heartbeat(`worker:<name>:heartbeat` EX45) + BE `node_monitor.WORKER_REGISTRY` 등재 + `/infra` 노출이 필수. trade-monitor는 kind=trader로 등재됨.
9. **trade-monitor KIS 앱키 분리** — ai_trade와 **다른 전용 app_key**(`TM_KIS_*`) 사용. 같은 app_key 공유 시 토큰 상호 무효화 + EGW00201. 실전 최대 89앱 발급 가능.
--- ---

57
ai_trade/heartbeat.py Normal file
View File

@@ -0,0 +1,57 @@
"""ai_trade heartbeat — NAS Redis로 worker:ai_trade:heartbeat SET.
Global Constraints 계약 1: kind=trader, state=market_open|market_closed.
ai_trade는 Windows 호스트 실행이라 _shared import 경로가 달라 자체 미니 헬퍼로 둔다.
"""
from __future__ import annotations
import asyncio
import datetime as dt
import json
import logging
import os
import redis.asyncio as aioredis
logger = logging.getLogger(__name__)
REDIS_URL = os.getenv("REDIS_URL", "redis://192.168.45.54:6379")
KEY = "worker:ai_trade:heartbeat"
INTERVAL = int(os.getenv("HEARTBEAT_INTERVAL", "15"))
TTL = int(os.getenv("HEARTBEAT_TTL", "45"))
def build_trader_payload(state: str, signals: int = 0) -> str:
"""JSON 문자열 반환. state: 'market_open' | 'market_closed'."""
return json.dumps({
"name": "ai_trade",
"kind": "trader",
"state": state,
"ts": dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
"last_job_at": None,
"jobs_done": signals,
"jobs_failed": 0,
})
async def heartbeat_loop(state_fn) -> None:
"""Redis에 HEARTBEAT_INTERVAL마다 SET EX TTL.
Args:
state_fn: () -> (state: str, signals: int). 호출자가 폴링 윈도우 판정 주입.
"""
redis = aioredis.from_url(REDIS_URL, decode_responses=False)
try:
while True:
try:
state, signals = state_fn()
payload = build_trader_payload(state, signals)
await redis.set(KEY, payload, ex=TTL)
logger.debug("ai_trade heartbeat sent: state=%s signals=%d", state, signals)
except asyncio.CancelledError:
raise
except Exception:
logger.exception("ai_trade heartbeat 실패 — 다음 주기에 재시도")
await asyncio.sleep(INTERVAL)
finally:
await redis.aclose()

View File

@@ -3,9 +3,12 @@ from __future__ import annotations
import asyncio import asyncio
import logging import logging
from contextlib import asynccontextmanager from contextlib import asynccontextmanager
from datetime import datetime
from zoneinfo import ZoneInfo
from fastapi import FastAPI from fastapi import FastAPI
from ai_trade import heartbeat as _hb
from ai_trade import state as state_mod from ai_trade import state as state_mod
from ai_trade.chronos_predictor import ChronosPredictor from ai_trade.chronos_predictor import ChronosPredictor
from ai_trade.config import get_settings from ai_trade.config import get_settings
@@ -13,8 +16,11 @@ from ai_trade.kis_client import KISClient
from ai_trade.kis_websocket import KISWebSocket from ai_trade.kis_websocket import KISWebSocket
from ai_trade.pull_worker import poll_loop, make_asking_price_callback from ai_trade.pull_worker import poll_loop, make_asking_price_callback
from ai_trade.rate_limit import SignalDedup from ai_trade.rate_limit import SignalDedup
from ai_trade.scheduler import _is_polling_window, _is_market_day
from ai_trade.stock_client import StockClient from ai_trade.stock_client import StockClient
_KST = ZoneInfo("Asia/Seoul")
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -23,6 +29,7 @@ class AppContext:
dedup: SignalDedup | None = None dedup: SignalDedup | None = None
shutdown: asyncio.Event | None = None shutdown: asyncio.Event | None = None
poll_task: asyncio.Task | None = None poll_task: asyncio.Task | None = None
hb_task: asyncio.Task | None = None
kis_client: KISClient | None = None kis_client: KISClient | None = None
kis_ws: KISWebSocket | None = None kis_ws: KISWebSocket | None = None
chronos: ChronosPredictor | None = None chronos: ChronosPredictor | None = None
@@ -87,9 +94,27 @@ async def lifespan(app: FastAPI):
) )
) )
def _trader_state() -> tuple[str, int]:
"""scheduler의 실제 폴링 윈도우 판정으로 market_open/market_closed 결정."""
now = datetime.now(_KST)
is_open = _is_market_day(now) and _is_polling_window(now)
state_str = "market_open" if is_open else "market_closed"
signals = len(state_mod.state.signals)
return state_str, signals
_ctx.hb_task = asyncio.create_task(_hb.heartbeat_loop(_trader_state))
yield yield
# Shutdown # Shutdown heartbeat task
if _ctx.hb_task is not None:
_ctx.hb_task.cancel()
try:
await _ctx.hb_task
except asyncio.CancelledError:
pass
# Shutdown poll task
if _ctx.shutdown is not None: if _ctx.shutdown is not None:
_ctx.shutdown.set() _ctx.shutdown.set()
if _ctx.poll_task is not None: if _ctx.poll_task is not None:

View File

@@ -0,0 +1,38 @@
"""Tests for ai_trade heartbeat payload builder."""
import json
import pytest
def test_trader_payload_market_open():
from ai_trade.heartbeat import build_trader_payload
p = json.loads(build_trader_payload("market_open", signals=2))
assert p["name"] == "ai_trade"
assert p["kind"] == "trader"
assert p["state"] == "market_open"
assert p["ts"].endswith("Z")
assert p["jobs_done"] == 2
def test_trader_payload_market_closed():
from ai_trade.heartbeat import build_trader_payload
p = json.loads(build_trader_payload("market_closed"))
assert p["name"] == "ai_trade"
assert p["kind"] == "trader"
assert p["state"] == "market_closed"
assert p["jobs_done"] == 0
assert p["jobs_failed"] == 0
assert p["last_job_at"] is None
def test_trader_payload_ts_format():
"""ts 필드가 ISO 8601 UTC 형식 (YYYY-MM-DDTHH:MM:SSZ)인지 확인."""
from ai_trade.heartbeat import build_trader_payload
import re
p = json.loads(build_trader_payload("market_open"))
assert re.match(r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z", p["ts"]), (
f"ts={p['ts']!r} does not match expected UTC format"
)

View File

@@ -7,6 +7,7 @@ pytest>=8.0
pytest-asyncio>=0.23 pytest-asyncio>=0.23
respx>=0.21 respx>=0.21
websockets>=12 websockets>=12
redis>=5.0
# Phase 3b dependencies (Chronos-2 + ML) # Phase 3b dependencies (Chronos-2 + ML)
transformers>=4.40 transformers>=4.40
chronos-forecasting>=1.4 chronos-forecasting>=1.4

View File

@@ -0,0 +1,55 @@
"""분산 워커 heartbeat — worker:<name>:heartbeat SET (TTL). Global Constraints 계약 1."""
from __future__ import annotations
import asyncio, datetime as dt, json, logging, os
logger = logging.getLogger(__name__)
DEFAULT_INTERVAL = int(os.getenv("HEARTBEAT_INTERVAL", "15"))
DEFAULT_TTL = int(os.getenv("HEARTBEAT_TTL", "45"))
class WorkerStats:
"""worker_loop가 갱신, heartbeat_loop가 읽는 가변 카운터."""
def __init__(self):
self.busy = False
self.jobs_done = 0
self.jobs_failed = 0
self.last_job_at = None # ISO str | None
def utc_now_iso() -> str:
return dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def build_payload(name: str, kind: str, state: str, stats: WorkerStats, extra: dict | None = None) -> str:
payload = {
"name": name, "kind": kind, "state": state, "ts": utc_now_iso(),
"last_job_at": stats.last_job_at,
"jobs_done": stats.jobs_done, "jobs_failed": stats.jobs_failed,
}
if extra:
payload.update(extra)
return json.dumps(payload)
async def render_state(redis, stats: WorkerStats, paused_key: str = "queue:paused") -> str:
if await redis.get(paused_key) == b"1":
return "paused"
return "busy" if stats.busy else "idle"
async def heartbeat_loop(redis, name, kind, stats, *, interval=DEFAULT_INTERVAL,
ttl=DEFAULT_TTL, paused_key="queue:paused", state_fn=None):
key = f"worker:{name}:heartbeat"
logger.info("heartbeat 시작 name=%s ttl=%ds", name, ttl)
while True:
try:
if state_fn is not None:
state, extra = await state_fn(redis, stats)
else:
state, extra = await render_state(redis, stats, paused_key), None
await redis.set(key, build_payload(name, kind, state, stats, extra), ex=ttl)
except asyncio.CancelledError:
raise
except Exception:
logger.exception("heartbeat 발신 실패 name=%s", name)
await asyncio.sleep(interval)

View File

@@ -0,0 +1,46 @@
"""Tests for _shared.heartbeat — Task A1."""
import json
import sys
from pathlib import Path
import pytest
# Make `_shared` importable (same pattern as test_reliable_queue.py)
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent))
from _shared.heartbeat import WorkerStats, build_payload, render_state
def test_build_payload_has_contract_fields():
s = WorkerStats(); s.jobs_done = 3; s.last_job_at = "2026-06-29T00:00:00Z"
payload = json.loads(build_payload("image-render", "render", "idle", s))
assert payload["name"] == "image-render"
assert payload["kind"] == "render"
assert payload["state"] == "idle"
assert payload["jobs_done"] == 3
assert payload["last_job_at"] == "2026-06-29T00:00:00Z"
assert payload["ts"].endswith("Z")
def test_build_payload_merges_extra():
payload = json.loads(build_payload("task-watcher", "watcher", "free", WorkerStats(), extra={"mode": "free"}))
assert payload["mode"] == "free"
class _FakeRedis:
def __init__(self, paused): self._paused = paused
async def get(self, key): return b"1" if self._paused else None
@pytest.mark.asyncio
async def test_render_state_paused_overrides_busy():
s = WorkerStats(); s.busy = True
assert await render_state(_FakeRedis(paused=True), s) == "paused"
@pytest.mark.asyncio
async def test_render_state_busy_then_idle():
s = WorkerStats(); s.busy = True
assert await render_state(_FakeRedis(paused=False), s) == "busy"
s.busy = False
assert await render_state(_FakeRedis(paused=False), s) == "idle"

View File

@@ -82,7 +82,8 @@ services:
task-watcher: task-watcher:
build: build:
context: ./task-watcher context: .
dockerfile: task-watcher/Dockerfile
container_name: task-watcher container_name: task-watcher
restart: unless-stopped restart: unless-stopped
ports: ports:
@@ -126,3 +127,52 @@ services:
interval: 60s interval: 60s
timeout: 5s timeout: 5s
retries: 3 retries: 3
trade-monitor:
build:
context: .
dockerfile: trade-monitor/Dockerfile
container_name: trade-monitor
restart: unless-stopped
ports:
- "18715:8000"
environment:
- TZ=Asia/Seoul
- REDIS_URL=${REDIS_URL:-redis://192.168.45.54:6379}
- NAS_BASE_URL=${NAS_BASE_URL:-http://192.168.45.54:18500}
- WEBAI_API_KEY=${WEBAI_API_KEY:-}
- TM_KIS_APP_KEY=${TM_KIS_APP_KEY:-}
- TM_KIS_APP_SECRET=${TM_KIS_APP_SECRET:-}
- TM_KIS_ACCOUNT=${TM_KIS_ACCOUNT:-}
- TM_KIS_IS_VIRTUAL=${TM_KIS_IS_VIRTUAL:-0}
- TM_LOOP_INTERVAL=${TM_LOOP_INTERVAL:-60}
- TM_CLIMAX_VOL_MULT=${TM_CLIMAX_VOL_MULT:-3.0}
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
interval: 60s
timeout: 5s
retries: 3
naver-fetch:
build:
context: .
dockerfile: naver-fetch/Dockerfile
container_name: naver-fetch
restart: unless-stopped
ports:
- "18716:8000"
environment:
- TZ=Asia/Seoul
- REDIS_URL=${REDIS_URL:-redis://192.168.45.54:6379}
- NAS_BASE_URL=${NAVER_NAS_BASE_URL:-http://192.168.45.54:18800}
- INTERNAL_API_KEY=${INTERNAL_API_KEY:-}
- FETCH_INTERVAL_HOURS=${FETCH_INTERVAL_HOURS:-2.5}
- DAYTIME_START=${DAYTIME_START:-08:00}
- DAYTIME_END=${DAYTIME_END:-20:00}
- NAVER_REAL_ESTATE_TYPE=${NAVER_REAL_ESTATE_TYPE:-APT:ABYG:JGC:PRE}
- NAVER_HEADLESS=${NAVER_HEADLESS:-1}
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
interval: 60s
timeout: 5s
retries: 3

View File

@@ -3,11 +3,14 @@ from __future__ import annotations
import asyncio import asyncio
import logging import logging
import os
from contextlib import asynccontextmanager from contextlib import asynccontextmanager
import redis.asyncio as aioredis
from fastapi import FastAPI from fastapi import FastAPI
import worker import worker
from _shared.heartbeat import heartbeat_loop
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(name)s %(levelname)s %(message)s") logging.basicConfig(level=logging.INFO, format="%(asctime)s %(name)s %(levelname)s %(message)s")
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -16,15 +19,19 @@ logger = logging.getLogger(__name__)
@asynccontextmanager @asynccontextmanager
async def lifespan(app: FastAPI): async def lifespan(app: FastAPI):
worker_task = asyncio.create_task(worker.worker_loop()) worker_task = asyncio.create_task(worker.worker_loop())
hb_redis = aioredis.from_url(os.getenv("REDIS_URL", "redis://192.168.45.54:6379"), decode_responses=False)
hb_task = asyncio.create_task(heartbeat_loop(hb_redis, "image-render", "render", worker.stats))
logger.info("image-render lifespan 시작") logger.info("image-render lifespan 시작")
try: try:
yield yield
finally: finally:
worker_task.cancel() for t in (worker_task, hb_task):
try: t.cancel()
await worker_task try:
except asyncio.CancelledError: await t
pass except asyncio.CancelledError:
pass
await hb_redis.aclose()
logger.info("image-render lifespan 종료") logger.info("image-render lifespan 종료")

View File

@@ -67,3 +67,25 @@ async def test_poll_once_returns_false_on_timeout(monkeypatch):
assert handled is False assert handled is False
fake_queue.ack.assert_not_awaited() fake_queue.ack.assert_not_awaited()
fake_queue.fail.assert_not_awaited() fake_queue.fail.assert_not_awaited()
# ----- heartbeat stats 카운터 -----
class _OneJobQueue:
def __init__(self): self.acked = False
async def dequeue(self, timeout=5):
if self.acked: return None
return ({"job_type": "flux_generation", "task_id": "t1", "params": {}}, b"raw")
async def ack(self, raw): self.acked = True
async def fail(self, raw, payload): pass
@pytest.mark.asyncio
async def test_poll_once_increments_jobs_done(monkeypatch):
worker.stats.jobs_done = 0
monkeypatch.setattr(worker, "run_flux_generation", lambda task_id, params: None)
handled = await worker.poll_once(_OneJobQueue())
assert handled is True
assert worker.stats.jobs_done == 1
assert worker.stats.busy is False
assert worker.stats.last_job_at is not None

View File

@@ -18,6 +18,7 @@ from providers.gpt_image import run_gpt_image_generation
from providers.nano_banana import run_nano_banana_generation from providers.nano_banana import run_nano_banana_generation
from providers.flux import run_flux_generation from providers.flux import run_flux_generation
from _shared.reliable_queue import ReliableQueue from _shared.reliable_queue import ReliableQueue
from _shared.heartbeat import WorkerStats, utc_now_iso
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -25,6 +26,8 @@ REDIS_URL = os.getenv("REDIS_URL", "redis://192.168.45.54:6379")
QUEUE_KEY = "queue:image-render" QUEUE_KEY = "queue:image-render"
PAUSED_KEY = "queue:paused" PAUSED_KEY = "queue:paused"
stats = WorkerStats()
# string names so `unittest.mock.patch` / `monkeypatch.setattr` on `worker.<name>` # string names so `unittest.mock.patch` / `monkeypatch.setattr` on `worker.<name>`
# is correctly intercepted by getattr(sys.modules[__name__], ...) # is correctly intercepted by getattr(sys.modules[__name__], ...)
_DISPATCH_TABLE = { _DISPATCH_TABLE = {
@@ -59,14 +62,21 @@ async def poll_once(queue: ReliableQueue) -> bool:
if result is None: if result is None:
return False return False
payload, raw = result payload, raw = result
stats.busy = True
try: try:
await asyncio.to_thread(_dispatch, payload) await asyncio.to_thread(_dispatch, payload)
except Exception: except Exception:
logger.exception("dispatch unhandled exception task_id=%s", logger.exception("dispatch unhandled exception task_id=%s",
payload.get("task_id")) payload.get("task_id"))
await queue.fail(raw, payload) await queue.fail(raw, payload)
stats.jobs_failed += 1
stats.last_job_at = utc_now_iso()
stats.busy = False
return True return True
await queue.ack(raw) await queue.ack(raw)
stats.jobs_done += 1
stats.last_job_at = utc_now_iso()
stats.busy = False
return True return True

View File

@@ -3,12 +3,15 @@ from __future__ import annotations
import asyncio import asyncio
import logging import logging
import os
from contextlib import asynccontextmanager from contextlib import asynccontextmanager
import redis.asyncio as aioredis
from fastapi import FastAPI from fastapi import FastAPI
import card_renderer import card_renderer
import worker import worker
from _shared.heartbeat import heartbeat_loop
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(name)s %(levelname)s %(message)s") logging.basicConfig(level=logging.INFO, format="%(asctime)s %(name)s %(levelname)s %(message)s")
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -20,15 +23,19 @@ async def lifespan(app: FastAPI):
await card_renderer.init_browser() await card_renderer.init_browser()
# 큐 워커 백그라운드 시작 # 큐 워커 백그라운드 시작
worker_task = asyncio.create_task(worker.worker_loop()) worker_task = asyncio.create_task(worker.worker_loop())
hb_redis = aioredis.from_url(os.getenv("REDIS_URL", "redis://192.168.45.54:6379"), decode_responses=False)
hb_task = asyncio.create_task(heartbeat_loop(hb_redis, "insta-render", "render", worker.stats))
logger.info("insta-render lifespan 시작") logger.info("insta-render lifespan 시작")
try: try:
yield yield
finally: finally:
worker_task.cancel() for t in (worker_task, hb_task):
try: t.cancel()
await worker_task try:
except asyncio.CancelledError: await t
pass except asyncio.CancelledError:
pass
await hb_redis.aclose()
await card_renderer.shutdown_browser() await card_renderer.shutdown_browser()
logger.info("insta-render lifespan 종료") logger.info("insta-render lifespan 종료")

View File

@@ -230,3 +230,27 @@ def test_make_queue_redis_socket_timeout_exceeds_block():
c = worker.make_queue_redis() c = worker.make_queue_redis()
st = c.connection_pool.connection_kwargs.get("socket_timeout") st = c.connection_pool.connection_kwargs.get("socket_timeout")
assert st is not None and st > 5 # blmove 블록(5s)보다 커야 안정 assert st is not None and st > 5 # blmove 블록(5s)보다 커야 안정
# ----- heartbeat stats 카운터 -----
class _OneJobQueueInsta:
def __init__(self): self.acked = False
async def dequeue(self, timeout=5):
if self.acked: return None
return ({"task_id": "t1", "params": {"slate_id": 1, "theme": "default"}}, b"raw")
async def ack(self, raw): self.acked = True
async def fail(self, raw, payload): pass
@pytest.mark.asyncio
async def test_poll_once_increments_jobs_done(monkeypatch):
worker.stats.jobs_done = 0
async def fake_process(client, payload): pass
monkeypatch.setattr(worker, "_process_one", fake_process)
async with httpx.AsyncClient() as client:
handled = await worker.poll_once(_OneJobQueueInsta(), client)
assert handled is True
assert worker.stats.jobs_done == 1
assert worker.stats.busy is False
assert worker.stats.last_job_at is not None

View File

@@ -14,9 +14,11 @@ import redis.asyncio as aioredis
from card_renderer import render_slate from card_renderer import render_slate
from _shared.reliable_queue import ReliableQueue from _shared.reliable_queue import ReliableQueue
from _shared.heartbeat import WorkerStats, utc_now_iso
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
stats = WorkerStats()
REDIS_URL = os.getenv("REDIS_URL", "redis://192.168.45.54:6379") REDIS_URL = os.getenv("REDIS_URL", "redis://192.168.45.54:6379")
NAS_BASE_URL = os.getenv("NAS_BASE_URL", "http://192.168.45.54:18700") NAS_BASE_URL = os.getenv("NAS_BASE_URL", "http://192.168.45.54:18700")
@@ -89,12 +91,19 @@ async def poll_once(queue: ReliableQueue, client: httpx.AsyncClient) -> bool:
if result is None: if result is None:
return False return False
payload, raw = result payload, raw = result
stats.busy = True
try: try:
await _process_one(client, payload) await _process_one(client, payload)
except Exception: except Exception:
await queue.fail(raw, payload) await queue.fail(raw, payload)
stats.jobs_failed += 1
stats.last_job_at = utc_now_iso()
stats.busy = False
return True return True
await queue.ack(raw) await queue.ack(raw)
stats.jobs_done += 1
stats.last_job_at = utc_now_iso()
stats.busy = False
return True return True

View File

@@ -7,12 +7,15 @@ from __future__ import annotations
import asyncio import asyncio
import logging import logging
import os
from contextlib import asynccontextmanager from contextlib import asynccontextmanager
import redis.asyncio as aioredis
from fastapi import FastAPI, HTTPException from fastapi import FastAPI, HTTPException
from pydantic import BaseModel from pydantic import BaseModel
import worker import worker
from _shared.heartbeat import heartbeat_loop
from providers.sync_ops import ( from providers.sync_ops import (
generate_lyrics, get_credits, generate_lyrics, get_credits,
get_timestamped_lyrics, generate_style_boost, get_timestamped_lyrics, generate_style_boost,
@@ -25,15 +28,19 @@ logger = logging.getLogger(__name__)
@asynccontextmanager @asynccontextmanager
async def lifespan(app: FastAPI): async def lifespan(app: FastAPI):
worker_task = asyncio.create_task(worker.worker_loop()) worker_task = asyncio.create_task(worker.worker_loop())
hb_redis = aioredis.from_url(os.getenv("REDIS_URL", "redis://192.168.45.54:6379"), decode_responses=False)
hb_task = asyncio.create_task(heartbeat_loop(hb_redis, "music-render", "render", worker.stats))
logger.info("music-render lifespan 시작") logger.info("music-render lifespan 시작")
try: try:
yield yield
finally: finally:
worker_task.cancel() for t in (worker_task, hb_task):
try: t.cancel()
await worker_task try:
except asyncio.CancelledError: await t
pass except asyncio.CancelledError:
pass
await hb_redis.aclose()
logger.info("music-render lifespan 종료") logger.info("music-render lifespan 종료")

View File

@@ -167,3 +167,25 @@ async def test_poll_once_returns_false_on_timeout(monkeypatch):
dispatch_mock.assert_not_called() dispatch_mock.assert_not_called()
fake_queue.ack.assert_not_awaited() fake_queue.ack.assert_not_awaited()
fake_queue.fail.assert_not_awaited() fake_queue.fail.assert_not_awaited()
# ----- heartbeat stats 카운터 -----
class _OneJobQueue:
def __init__(self): self.acked = False
async def dequeue(self, timeout=5):
if self.acked: return None
return ({"job_type": "suno_generation", "task_id": "t1", "params": {}}, b"raw")
async def ack(self, raw): self.acked = True
async def fail(self, raw, payload): pass
@pytest.mark.asyncio
async def test_poll_once_increments_jobs_done(monkeypatch):
worker.stats.jobs_done = 0
monkeypatch.setattr(worker, "run_suno_generation", lambda task_id, params: None)
handled = await worker.poll_once(_OneJobQueue())
assert handled is True
assert worker.stats.jobs_done == 1
assert worker.stats.busy is False
assert worker.stats.last_job_at is not None

View File

@@ -21,6 +21,7 @@ from providers.suno import (
) )
from providers.local import run_local_generation from providers.local import run_local_generation
from _shared.reliable_queue import ReliableQueue from _shared.reliable_queue import ReliableQueue
from _shared.heartbeat import WorkerStats, utc_now_iso
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -28,6 +29,8 @@ REDIS_URL = os.getenv("REDIS_URL", "redis://192.168.45.54:6379")
QUEUE_KEY = "queue:music-render" QUEUE_KEY = "queue:music-render"
PAUSED_KEY = "queue:paused" PAUSED_KEY = "queue:paused"
stats = WorkerStats()
# Maps job_type → module-level function name (string). # Maps job_type → module-level function name (string).
# _dispatch resolves the name via globals() at call time so unittest.mock.patch # _dispatch resolves the name via globals() at call time so unittest.mock.patch
# on "worker.<name>" is correctly intercepted. # on "worker.<name>" is correctly intercepted.
@@ -74,6 +77,7 @@ async def poll_once(queue: ReliableQueue) -> bool:
if result is None: if result is None:
return False return False
payload, raw = result payload, raw = result
stats.busy = True
try: try:
# sync provider 함수 — thread로 실행해서 이벤트 루프 블로킹 방지 # sync provider 함수 — thread로 실행해서 이벤트 루프 블로킹 방지
await asyncio.to_thread(_dispatch, payload) await asyncio.to_thread(_dispatch, payload)
@@ -81,8 +85,14 @@ async def poll_once(queue: ReliableQueue) -> bool:
logger.exception("dispatch unhandled exception task_id=%s", logger.exception("dispatch unhandled exception task_id=%s",
payload.get("task_id")) payload.get("task_id"))
await queue.fail(raw, payload) await queue.fail(raw, payload)
stats.jobs_failed += 1
stats.last_job_at = utc_now_iso()
stats.busy = False
return True return True
await queue.ack(raw) await queue.ack(raw)
stats.jobs_done += 1
stats.last_job_at = utc_now_iso()
stats.busy = False
return True return True

View File

@@ -0,0 +1,17 @@
# naver-fetch — realestate 매물 fetch 워커
REDIS_URL=redis://192.168.45.54:6379
# realestate-lab 직결 (targets/listings-ingest). compose가 NAVER_NAS_BASE_URL로 주입
NAVER_NAS_BASE_URL=http://192.168.45.54:18800
# render 워커 공유키 (X-Internal-Key)
INTERNAL_API_KEY=
# 스케줄 (KST)
FETCH_INTERVAL_HOURS=2.5
DAYTIME_START=08:00
DAYTIME_END=20:00
# 네이버
NAVER_REAL_ESTATE_TYPE=APT:ABYG:JGC:PRE
NAVER_HEADLESS=1
# NAVER_TOKEN_URL= # 기본값(new.land.naver.com houses) 사용

View File

@@ -0,0 +1,152 @@
# naver-fetch 워커 — 구현 설계
> **2026-07-10 · web-ai 소유.** realestate 매물 알림 파이프라인의 가정-IP fetch 워커.
> **권위 계약(원본 스펙)**: web-backend repo `realestate-lab/docs/superpowers/specs/2026-07-09-naver-listing-worker-design.md` §4.
> 이 문서는 그 계약을 Windows Docker 워커로 구현하기 위한 **구현 설계**.
---
## 1. 역할 · 경계
네이버가 NAS(데이터센터) IP를 429/ReadTimeout으로 막기 때문에, **가정 IP(LAN 192.168.45.x)에서 네이버 매물을 fetch해 NAS로 전달**하는 얇은 프록시 워커. trade-monitor·render 워커와 동일한 FastAPI + asyncio 관례. WSL2 Docker.
**책임**
1. 2~3h 주간 주기로 NAS `targets` 조회(감시 동 목록).
2. Playwright로 네이버 Bearer 토큰 획득 → httpx로 동별 매물 fetch(페이지네이션).
3. raw article을 그대로 NAS `listings-ingest`로 전달(**파싱·매칭·알림은 NAS**).
4. Redis heartbeat(`worker:naver-fetch:heartbeat` EX45, kind=fetcher).
5. 토큰/네트워크 오류를 사이클·동 단위로 격리.
**경계 밖(안 함)**: article 파싱, 매물 매칭, 텔레그램 발송(전부 NAS), dedup.
### Spike 검증 완료 (2026-07-10)
- Playwright 헤드리스 → `new.land.naver.com` 로드 → `/api/articles` XHR의 **Bearer JWT 토큰 인터셉트 성공**.
- 인터셉트 토큰+쿠키를 httpx로 재사용 → `GET /api/articles?cortarNo=1159010100&...` **HTTP 200, articleList 20건**(isMoreData 페이지네이션). 토큰 있으면 이 LAN에서 429 회피됨.
- raw article 키: `articleNo, articleName, tradeTypeName, dealOrWarrantPrc, areaName, buildingName, floorInfo` 등 → NAS로 그대로 전달.
---
## 2. 모듈 분해
| 파일 | 책임 | 인터페이스 |
|------|------|-----------|
| `main.py` | FastAPI + lifespan. `fetch_loop` + `heartbeat_loop` 스폰, `/health` | 부작용 |
| `fetch_worker.py` | `FetchState`, `run_cycle`, `fetch_loop`, `heartbeat_loop`(전용) | 부작용 |
| `nas_client.py` | `get_targets()` / `post_ingest(fetched_at, batches)` (X-Internal-Key + retry) | 부작용(HTTP) |
| `naver_client.py` | `NaverClient`: `acquire_token()`(Playwright) + `fetch_articles(cortar_no, page)`(httpx) | 부작용(브라우저/HTTP) |
| `config.py` | `Settings` — env 로드 | 순수 |
순수 로직이 적은 워커(얇은 프록시)라 파싱은 없음. 테스트는 HTTP mock(respx) + fetch_worker 조립 중심.
---
## 3. 데이터 흐름 (run_cycle)
```
state.state = "fetching"
await naver.acquire_token() # Playwright 1회 (사이클당)
tg = await nas.get_targets() # {dongs:[{dong,cortar_no}], deal_types, page_limit}
for d in tg.dongs: # 동 단위 try/except 격리 (fetch/ingest 각각)
articles = []
for page in 1..tg.page_limit:
resp = await naver.fetch_articles(d.cortar_no, page) # httpx GET
articles += resp.get("articleList", [])
if not resp.get("isMoreData"): break
# ⚠️ 동별로 즉시 ingest — 전체 batch를 한 POST로 보내면 NAS 동기 처리(매물 수 비례)가
# >180s 타임아웃. 동 단위(각 수십건)로 쪼개 각 POST가 timeout 내 완료 + 부분 실패 격리.
r = await nas.post_ingest(fetched_at=iso_utc_now, batches=[{"dong": d.dong, "articles": articles}])
state.listings_pushed += len(articles) # {received,new,matched}
state.last_fetch_at = epoch_now
state.state = "idle"
```
> ⚠️ **ingest 분할**: 스펙은 batches(복수)를 한 POST로 예시하지만, 첫 실행 대량(6동×~40건)이 NAS에서 >180s 걸려 타임아웃 → **동별 1 POST**로 분할(batches 리스트 계약 호환, 각 POST는 batches 1개). nas_client의 ingest는 timeout 180s·재시도 없음(재전송 시 NAS 중복 알림 방지). BE에 분할 통지 필요.
> ⚠️ **시간 포맷 2종**: ingest `fetched_at`은 **ISO UTC 문자열**(스펙 §4 ingest 스키마, 예 `2026-07-09T05:44:23Z`), heartbeat `ts`/`last_fetch_at`은 **epoch 정수**(스펙 §4.4). 혼동 주의.
`heartbeat_loop`(별도 15초)이 `FetchState`를 읽어 §4.4 페이로드 발신.
---
## 4. naver_client (Playwright 토큰 + httpx fetch)
- `async acquire_token()`: `async_playwright`로 chromium 헤드리스 기동 → `new.land.naver.com/houses?ms=...`(설정 `NAVER_TOKEN_URL`) 로드 → `page.on("request")`로 첫 `Authorization: Bearer ...` 헤더 캡처 → `self._token`, `self._cookies`(context.cookies) 저장 → 브라우저 종료. 실패 시 예외(사이클 skip).
- **토큰 수명 정책**: 사이클마다 fresh 획득(2~3h당 1회). JWT 만료 무관, 캐시/갱신 로직 없음(단순).
- `async fetch_articles(cortar_no, page) -> dict`: httpx GET `https://new.land.naver.com/api/articles?cortarNo=<cortar_no>&order=rank&realEstateType=<NAVER_REAL_ESTATE_TYPE>&tradeType=&...&page=<page>` with headers `{authorization: self._token, user-agent, referer: https://new.land.naver.com/, accept, cookie: self._cookies}`. 200 아니면 예외. raw JSON dict 반환.
- `async close()`: httpx client 정리.
**얇은 프록시**: tradeType 비움(전체 유형) + realEstateType 설정값(기본 `APT:ABYG:JGC:PRE`). deal_types 필터는 NAS 담당(targets의 deal_types는 워커가 그대로 두고 NAS가 매칭 시 사용). 추후 네이버 쿼리 최적화 여지.
---
## 5. heartbeat (§4.4 — 전용 스키마)
`worker:naver-fetch:heartbeat` EX45. **trade-monitor의 `_shared.build_payload`(ISO ts + jobs_done/jobs_failed)와 스키마가 달라 전용 emit 작성**:
```json
{"name":"naver-fetch","kind":"fetcher","state":"idle|fetching",
"ts":<epoch int>,"last_fetch_at":<epoch int|null>,"listings_pushed":<int>,"errors":<int>}
```
- 15초 독립 태스크(주기 fetch가 2~3h라 TTL45 유지 위해 필수). `FetchState`를 읽어 발신.
- BE가 node_monitor.WORKER_REGISTRY에 `naver-fetch`(kind=fetcher)로 등재 완료 → 발신 시 `/nodes`·`/infra`에서 alive.
---
## 6. 스케줄 (fetch_loop)
- env `FETCH_INTERVAL_HOURS`(기본 2.5), `DAYTIME_START`/`DAYTIME_END`(기본 08:00/20:00 KST).
- 루프: ~5분마다 판정 — **주간창 안 && (now - last_run) ≥ interval** 이면 `run_cycle` 실행 후 `last_run=now`. 창 밖이면 idle(state=idle, fetch 안 함).
- 최상위 try/except로 루프 불사(어떤 예외도 죽이지 않음).
---
## 7. 설정 (env)
| env | 기본값 | 용도 |
|-----|--------|------|
| `NAS_BASE_URL` (compose: `NAVER_NAS_BASE_URL`) | `http://192.168.45.54:18800` | **realestate-lab 직결** targets/ingest (stock :18500 아님) |
| `INTERNAL_API_KEY` | (필수) | `X-Internal-Key` (render 워커 공유키) |
| `REDIS_URL` | `redis://192.168.45.54:6379` | heartbeat |
| `FETCH_INTERVAL_HOURS` | `2.5` | fetch 주기 |
| `DAYTIME_START` / `DAYTIME_END` | `08:00` / `20:00` | 주간창(KST) |
| `NAVER_REAL_ESTATE_TYPE` | `APT:ABYG:JGC:PRE` | 네이버 realEstateType |
| `NAVER_TOKEN_URL` | (기본 상수) | Playwright 토큰 획득용 네이버 URL |
| `NAVER_HEADLESS` | `1` | Playwright headless |
---
## 8. 에러 처리
- **토큰 획득 실패**: 사이클 skip, `errors++`, state=idle, 다음 주기 재시도.
- **targets 실패**: 사이클 skip, `errors++`.
- **동별 fetch 실패**: 해당 동 skip(warning), 나머지 동 계속.
- **ingest 실패**: 로그 error, `errors++`(다음 주기 재fetch).
- 루프 최상위 try/except.
---
## 9. 테스트 (pytest, 시스템 Python)
| 파일 | 검증 |
|------|------|
| `test_config.py` | env 기본/override |
| `test_nas_client.py` | respx — targets 파싱, ingest 페이로드/응답, X-Internal-Key, retry |
| `test_naver_client.py` | respx — fetch_articles URL·헤더(authorization/cookie)·articleList 파싱·비200 예외. (Playwright acquire_token은 실브라우저 통합이라 단위 mock: 토큰 주입 후 fetch 경로만) |
| `test_fetch_worker.py` | mock: 주간창 게이트(창 밖 skip), run_cycle 동별 조립·batches, 동 fetch 실패 격리, listings_pushed/last_fetch_at 갱신, heartbeat 필드(epoch ts/kind=fetcher) |
---
## 10. 배포
- `Dockerfile`: Playwright 베이스(`mcr.microsoft.com/playwright/python` 또는 insta-render 관례) + chromium, `COPY _shared /app/_shared`(빌드 컨텍스트 `.`), `COPY naver-fetch/. /app/`, `PYTHONPATH=/app`, uvicorn `:8000`.
- `services/docker-compose.yml`: `naver-fetch` 서비스, 포트 **18716**, `TZ=Asia/Seoul`, INTERNAL_API_KEY/REDIS/NAS env, healthcheck `/health`.
- 엔드포인트는 nginx `/api/internal/realestate/`(IP 화이트리스트, render 워커 IP 재사용).
---
## 11. 미해결 플래그 / 후속
1. **`INTERNAL_API_KEY` vs `REALESTATE_INTERNAL_KEY`** — 스펙 §config는 후자, BE 메시지는 전자(render 공유키). **BE 메시지 우선(INTERNAL_API_KEY)**, 배포 전 co-gahusb로 1줄 확인.
2. **네이버 헤드리스 탐지** — 현재 spike는 통과. 차단 시 stealth 헤더/headed 필요할 수 있음(첫 운영 모니터링).
3. **`isMoreData`/페이지 상한** — page_limit로 제한, 네이버 rate limit 재유입 시 동 간 지연 고려(첫 운영).
4. **deal_types 미사용** — 워커는 전체 유형 fetch, NAS가 필터. 필요 시 tradeType 매핑으로 최적화.
5. **토큰 재사용 실패(만료) 대비** — 사이클 중 fetch가 401 반환 시 acquire_token 1회 재시도(구현 시 반영 검토).

View File

@@ -0,0 +1,25 @@
FROM python:3.12-slim-bookworm
ENV PYTHONUNBUFFERED=1
WORKDIR /app
# Chromium 런타임 deps (Debian 12 / bookworm) — insta-render 관례
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates tzdata fonts-noto-cjk \
libnss3 libnspr4 libdbus-1-3 libatk1.0-0 libatk-bridge2.0-0 \
libcups2 libdrm2 libxkbcommon0 libxcomposite1 libxdamage1 \
libxfixes3 libxrandr2 libgbm1 libxshmfence1 libpango-1.0-0 \
libcairo2 libasound2 libatspi2.0-0 \
&& rm -rf /var/lib/apt/lists/*
COPY naver-fetch/requirements.txt /app/
RUN pip install --no-cache-dir --timeout 600 --retries 5 -r requirements.txt
RUN playwright install chromium
# 공통 모듈 (services/_shared) — 빌드 컨텍스트 . 필요 (task-watcher 함정)
COPY _shared /app/_shared
COPY naver-fetch/. /app/
ENV PYTHONPATH=/app
EXPOSE 8000
CMD ["python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "1"]

View File

@@ -0,0 +1,979 @@
# naver-fetch 워커 구현 계획
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** 네이버 매물을 가정 IP에서 fetch해 NAS로 전달하는 `naver-fetch` 워커를 구현한다 — Playwright로 Bearer 토큰 획득 → httpx로 동별 article fetch → NAS `listings-ingest`로 raw 전달 + heartbeat.
**Architecture:** FastAPI + asyncio 워커(trade-monitor·insta-render 관례). `naver_client`가 Playwright(토큰)+httpx(fetch)를 캡슐화, `fetch_worker`가 2~3h 주간 스케줄로 오케스트레이션. 얇은 프록시(파싱은 NAS). heartbeat는 fetcher 전용 스키마(epoch ts).
**Tech Stack:** Python 3.12, FastAPI, playwright==1.48.0(async, chromium), httpx(async), redis.asyncio, pytest + pytest-asyncio + respx. WSL2 Docker.
**설계 원본:** `services/naver-fetch/DESIGN.md` · 권위 계약: web-backend `realestate-lab/docs/superpowers/specs/2026-07-09-naver-listing-worker-design.md`.
## Global Constraints
- Python `3.12`, `playwright==1.48.0` (async API, chromium).
- **얇은 프록시** — raw article 그대로 전달, 파싱/매칭/알림/dedup은 NAS.
- 인증: `.env` `INTERNAL_API_KEY``X-Internal-Key` 헤더(render 워커 공유키). (스펙 §config의 `REALESTATE_INTERNAL_KEY`와 상충 — BE 메시지 우선, 배포 전 확인.)
- heartbeat 키 `worker:naver-fetch:heartbeat`, TTL **EX45**, kind `fetcher`, **epoch ts**: `{name,kind,state:idle|fetching,ts,last_fetch_at,listings_pushed,errors}`.
- ingest `fetched_at`**ISO UTC 문자열**(`YYYY-MM-DDTHH:MM:SSZ`).
- 스케줄: 주간창(기본 08:00~20:00 KST) 안에서 `FETCH_INTERVAL_HOURS`(기본 2.5) 간격.
- 테스트 실행: `C:\Users\jaeoh\AppData\Local\Programs\Python\Python312\python.exe -m pytest services/naver-fetch/tests -q` (web-ai 루트). 이하 `python` = 이 경로.
- 커밋: **web-ai repo에서만**. Docker 포트 **18716**.
- Spike 검증됨(2026-07-10): Playwright 토큰 인터셉트 + httpx `/api/articles` 200 + articleList.
---
## File Structure
| 파일 | 책임 |
|------|------|
| `services/naver-fetch/config.py` | `Settings` + `load_settings()` |
| `services/naver-fetch/nas_client.py` | `NASClient`: get_targets / post_ingest (X-Internal-Key) |
| `services/naver-fetch/naver_client.py` | `NaverClient`: acquire_token(Playwright) + fetch_articles(httpx) |
| `services/naver-fetch/fetch_worker.py` | `FetchState`, `in_daytime`, `run_cycle`, `fetch_loop`, `build_heartbeat`, `heartbeat_loop` |
| `services/naver-fetch/main.py` | FastAPI lifespan + `/health` |
| `services/naver-fetch/conftest.py` | services/ 루트 sys.path |
| `services/naver-fetch/tests/*` | 단위 테스트 |
| `services/naver-fetch/Dockerfile` | Playwright 베이스(insta-render 관례) + `_shared` |
| `services/naver-fetch/requirements.txt`, `.env.example`, `pytest.ini` | |
| `services/docker-compose.yml` (수정) | `naver-fetch` 서비스(18716) |
---
### Task 1: 스캐폴딩 + config
**Files:**
- Create: `services/naver-fetch/requirements.txt`, `conftest.py`, `pytest.ini`, `tests/__init__.py`, `config.py`
- Test: `services/naver-fetch/tests/test_config.py`
**Interfaces:**
- Produces: `Settings` dataclass fields `nas_base_url:str, internal_api_key:str, redis_url:str, fetch_interval_hours:float, daytime_start:str, daytime_end:str, naver_real_estate_type:str, naver_token_url:str, naver_headless:bool`. `load_settings() -> Settings`.
- [ ] **Step 1: 스캐폴딩**
`services/naver-fetch/requirements.txt`:
```
fastapi==0.115.6
uvicorn[standard]==0.34.0
playwright==1.48.0
redis>=5.0
httpx>=0.27
pytest>=8.0
pytest-asyncio>=0.24
respx>=0.21
```
`services/naver-fetch/conftest.py`:
```python
"""services/ 루트를 sys.path에 추가 — from _shared... import 가능하게."""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
```
`services/naver-fetch/pytest.ini`:
```
[pytest]
asyncio_mode = auto
```
`services/naver-fetch/tests/__init__.py`: (빈 파일)
- [ ] **Step 2: 실패 테스트**`services/naver-fetch/tests/test_config.py`
```python
"""Settings env 로드."""
from config import load_settings
def test_defaults(monkeypatch):
for k in ("NAS_BASE_URL", "INTERNAL_API_KEY", "REDIS_URL", "FETCH_INTERVAL_HOURS",
"DAYTIME_START", "DAYTIME_END", "NAVER_REAL_ESTATE_TYPE",
"NAVER_TOKEN_URL", "NAVER_HEADLESS"):
monkeypatch.delenv(k, raising=False)
s = load_settings()
assert s.nas_base_url == "http://192.168.45.54:18500"
assert s.fetch_interval_hours == 2.5
assert s.daytime_start == "08:00" and s.daytime_end == "20:00"
assert s.naver_headless is True
assert "new.land.naver.com" in s.naver_token_url
def test_override(monkeypatch):
monkeypatch.setenv("INTERNAL_API_KEY", "sekret")
monkeypatch.setenv("FETCH_INTERVAL_HOURS", "3")
monkeypatch.setenv("NAVER_HEADLESS", "0")
s = load_settings()
assert s.internal_api_key == "sekret"
assert s.fetch_interval_hours == 3.0
assert s.naver_headless is False
```
- [ ] **Step 3: 실패 확인**
Run: `python -m pytest services/naver-fetch/tests/test_config.py -q`
Expected: FAIL — `ModuleNotFoundError: No module named 'config'`
- [ ] **Step 4: config.py 구현**
```python
"""Settings — 환경변수 로드."""
from __future__ import annotations
import os
from dataclasses import dataclass
_DEFAULT_TOKEN_URL = (
"https://new.land.naver.com/houses?ms=37.489,126.921,16&a=APT:ABYG:JGC:PRE&e=RETAIL"
)
@dataclass
class Settings:
nas_base_url: str
internal_api_key: str
redis_url: str
fetch_interval_hours: float
daytime_start: str
daytime_end: str
naver_real_estate_type: str
naver_token_url: str
naver_headless: bool
def load_settings() -> Settings:
return Settings(
nas_base_url=os.getenv("NAS_BASE_URL", "http://192.168.45.54:18500"),
internal_api_key=os.getenv("INTERNAL_API_KEY", ""),
redis_url=os.getenv("REDIS_URL", "redis://192.168.45.54:6379"),
fetch_interval_hours=float(os.getenv("FETCH_INTERVAL_HOURS", "2.5")),
daytime_start=os.getenv("DAYTIME_START", "08:00"),
daytime_end=os.getenv("DAYTIME_END", "20:00"),
naver_real_estate_type=os.getenv("NAVER_REAL_ESTATE_TYPE", "APT:ABYG:JGC:PRE"),
naver_token_url=os.getenv("NAVER_TOKEN_URL", _DEFAULT_TOKEN_URL),
naver_headless=os.getenv("NAVER_HEADLESS", "1") == "1",
)
```
- [ ] **Step 5: 통과 확인**
Run: `python -m pytest services/naver-fetch/tests/test_config.py -q`
Expected: PASS (2 passed)
- [ ] **Step 6: 커밋**
```bash
git add services/naver-fetch/requirements.txt services/naver-fetch/conftest.py services/naver-fetch/pytest.ini services/naver-fetch/tests/__init__.py services/naver-fetch/config.py services/naver-fetch/tests/test_config.py services/naver-fetch/DESIGN.md services/naver-fetch/PLAN.md
git commit -m "feat(naver-fetch): 스캐폴딩 + config"
```
---
### Task 2: nas_client
**Files:**
- Create: `services/naver-fetch/nas_client.py`
- Test: `services/naver-fetch/tests/test_nas_client.py`
**Interfaces:**
- Produces `NASClient`:
- `__init__(base_url, api_key, timeout=15.0)`
- `async get_targets() -> dict` (GET `/api/internal/realestate/targets`)
- `async post_ingest(fetched_at: str, batches: list[dict]) -> dict` (POST `/api/internal/realestate/listings-ingest`)
- `async close()`
- [ ] **Step 1: 실패 테스트**`services/naver-fetch/tests/test_nas_client.py`
```python
"""NASClient — targets/ingest + X-Internal-Key (respx)."""
import json as _json
import httpx
import respx
from nas_client import NASClient
BASE = "http://nas.test"
@respx.mock
async def test_get_targets_sends_key():
route = respx.get(f"{BASE}/api/internal/realestate/targets").mock(
return_value=httpx.Response(200, json={
"dongs": [{"dong": "신대방동", "cortar_no": "1159010100"}],
"deal_types": ["전세", "매매"], "page_limit": 2}))
c = NASClient(BASE, "KEY")
tg = await c.get_targets()
assert tg["page_limit"] == 2
assert tg["dongs"][0]["cortar_no"] == "1159010100"
assert route.calls.last.request.headers["X-Internal-Key"] == "KEY"
await c.close()
@respx.mock
async def test_post_ingest_payload():
captured = {}
def _resp(request):
captured.update(_json.loads(request.content))
return httpx.Response(200, json={"received": 3, "new": 2, "matched": 1})
respx.post(f"{BASE}/api/internal/realestate/listings-ingest").mock(side_effect=_resp)
c = NASClient(BASE, "KEY")
batches = [{"dong": "신대방동", "articles": [{"articleNo": "1"}, {"articleNo": "2"}]}]
out = await c.post_ingest("2026-07-10T05:00:00Z", batches)
assert out == {"received": 3, "new": 2, "matched": 1}
assert captured["fetched_at"] == "2026-07-10T05:00:00Z"
assert captured["batches"] == batches
await c.close()
```
- [ ] **Step 2: 실패 확인**
Run: `python -m pytest services/naver-fetch/tests/test_nas_client.py -q`
Expected: FAIL — `ModuleNotFoundError: No module named 'nas_client'`
- [ ] **Step 3: nas_client.py 구현**
```python
"""NAS realestate 내부 계약 — X-Internal-Key + retry."""
from __future__ import annotations
import asyncio
import logging
import httpx
logger = logging.getLogger(__name__)
_MAX_ATTEMPTS = 3
_RETRY_STATUSES = {429, 500, 502, 503, 504}
class NASClient:
def __init__(self, base_url: str, api_key: str, timeout: float = 15.0):
self._base_url = base_url.rstrip("/")
self._api_key = api_key
self._client = httpx.AsyncClient(timeout=timeout)
async def close(self) -> None:
await self._client.aclose()
async def get_targets(self) -> dict:
return await self._request("GET", "/api/internal/realestate/targets")
async def post_ingest(self, fetched_at: str, batches: list[dict]) -> dict:
return await self._request(
"POST", "/api/internal/realestate/listings-ingest",
json={"fetched_at": fetched_at, "batches": batches})
async def _request(self, method: str, path: str, **kwargs) -> dict:
url = f"{self._base_url}{path}"
headers = {"X-Internal-Key": self._api_key}
for attempt in range(_MAX_ATTEMPTS):
try:
resp = await self._client.request(method, url, headers=headers, **kwargs)
if resp.status_code in _RETRY_STATUSES and attempt < _MAX_ATTEMPTS - 1:
await asyncio.sleep(2 ** attempt)
continue
resp.raise_for_status()
return resp.json()
except httpx.TimeoutException:
if attempt < _MAX_ATTEMPTS - 1:
await asyncio.sleep(2 ** attempt)
continue
raise
raise RuntimeError("retry exhausted")
```
- [ ] **Step 4: 통과 확인**
Run: `python -m pytest services/naver-fetch/tests/test_nas_client.py -q`
Expected: PASS (2 passed)
- [ ] **Step 5: 커밋**
```bash
git add services/naver-fetch/nas_client.py services/naver-fetch/tests/test_nas_client.py
git commit -m "feat(naver-fetch): NAS realestate 클라이언트 (targets/ingest)"
```
---
### Task 3: naver_client (Playwright 토큰 + httpx fetch)
**Files:**
- Create: `services/naver-fetch/naver_client.py`
- Test: `services/naver-fetch/tests/test_naver_client.py`
**Interfaces:**
- Produces `NaverClient`:
- `__init__(real_estate_type: str, token_url: str, headless: bool = True, timeout: float = 20.0)`
- `async acquire_token() -> None` — Playwright로 `self._token`(Bearer str), `self._cookies`(str) 설정. 실패 시 `RuntimeError`. (실브라우저 — 단위테스트 제외, spike/첫운영 검증)
- `async fetch_articles(cortar_no: str, page: int = 1) -> dict` — httpx GET, raw JSON dict. 토큰 없으면 `RuntimeError`.
- `async close()`
- [ ] **Step 1: 실패 테스트**`services/naver-fetch/tests/test_naver_client.py`
```python
"""NaverClient.fetch_articles — httpx 경로 (respx). acquire_token은 실브라우저라 제외."""
import httpx
import pytest
import respx
from naver_client import NaverClient
API = "https://new.land.naver.com/api/articles"
def _client():
return NaverClient(real_estate_type="APT:ABYG", token_url="https://x", headless=True)
async def test_fetch_requires_token():
c = _client()
with pytest.raises(RuntimeError):
await c.fetch_articles("1159010100", 1)
await c.close()
@respx.mock
async def test_fetch_articles_request_and_parse():
captured = {}
def _resp(request):
captured["auth"] = request.headers.get("authorization")
captured["cookie"] = request.headers.get("cookie")
captured["url"] = str(request.url)
return httpx.Response(200, json={
"isMoreData": True,
"articleList": [{"articleNo": "111", "articleName": "테스트아파트"}]})
respx.get(API).mock(side_effect=_resp)
c = _client()
c._token = "Bearer TESTJWT" # acquire_token 우회(단위테스트)
c._cookies = "NNB=abc; page_uid=xyz"
out = await c.fetch_articles("1159010100", page=2)
assert out["articleList"][0]["articleNo"] == "111"
assert captured["auth"] == "Bearer TESTJWT"
assert captured["cookie"] == "NNB=abc; page_uid=xyz"
assert "cortarNo=1159010100" in captured["url"]
assert "page=2" in captured["url"]
assert "realEstateType=APT" in captured["url"]
await c.close()
@respx.mock
async def test_fetch_non_200_raises():
respx.get(API).mock(return_value=httpx.Response(429, json={"code": "TOO_MANY_REQUESTS"}))
c = _client()
c._token = "Bearer X"
with pytest.raises(httpx.HTTPStatusError):
await c.fetch_articles("1159010100", 1)
await c.close()
```
- [ ] **Step 2: 실패 확인**
Run: `python -m pytest services/naver-fetch/tests/test_naver_client.py -q`
Expected: FAIL — `ModuleNotFoundError: No module named 'naver_client'`
- [ ] **Step 3: naver_client.py 구현**
```python
"""Naver land client — Playwright 토큰 획득 + httpx article fetch."""
from __future__ import annotations
import logging
import httpx
logger = logging.getLogger(__name__)
_UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36")
_API = "https://new.land.naver.com/api/articles"
class NaverClient:
def __init__(self, real_estate_type: str, token_url: str,
headless: bool = True, timeout: float = 20.0):
self._ret = real_estate_type
self._token_url = token_url
self._headless = headless
self._client = httpx.AsyncClient(timeout=timeout)
self._token: str | None = None
self._cookies: str = ""
async def close(self) -> None:
await self._client.aclose()
async def acquire_token(self) -> None:
"""new.land.naver.com 로드 → XHR Authorization 헤더 인터셉트."""
from playwright.async_api import async_playwright
captured: dict[str, str] = {}
async with async_playwright() as p:
browser = await p.chromium.launch(headless=self._headless)
try:
ctx = await browser.new_context(user_agent=_UA, locale="ko-KR")
page = await ctx.new_page()
def on_request(req):
auth = req.headers.get("authorization")
if auth and "bearer" in auth.lower() and "auth" not in captured:
captured["auth"] = auth
page.on("request", on_request)
try:
await page.goto(self._token_url, wait_until="networkidle", timeout=45000)
except Exception:
logger.warning("naver token goto 지연/부분 실패 — 캡처 계속")
await page.wait_for_timeout(4000)
cookies = await ctx.cookies()
self._cookies = "; ".join(f"{c['name']}={c['value']}" for c in cookies)
finally:
await browser.close()
token = captured.get("auth")
if not token:
raise RuntimeError("naver Bearer token 획득 실패")
self._token = token
logger.info("naver token 획득 (len=%d)", len(token))
async def fetch_articles(self, cortar_no: str, page: int = 1) -> dict:
if not self._token:
raise RuntimeError("token 없음 — acquire_token 먼저 호출")
params = {
"cortarNo": cortar_no, "order": "rank", "realEstateType": self._ret,
"tradeType": "", "rentPriceMin": 0, "rentPriceMax": 900000000,
"priceMin": 0, "priceMax": 900000000, "areaMin": 0, "areaMax": 900000000,
"showArticle": "false", "sameAddressGroup": "false",
"priceType": "RETAIL", "page": page,
}
headers = {
"authorization": self._token, "user-agent": _UA,
"referer": "https://new.land.naver.com/",
"accept": "application/json, text/plain, */*",
"accept-language": "ko-KR,ko;q=0.9", "cookie": self._cookies,
}
resp = await self._client.get(_API, params=params, headers=headers)
resp.raise_for_status()
return resp.json()
```
- [ ] **Step 4: 통과 확인**
Run: `python -m pytest services/naver-fetch/tests/test_naver_client.py -q`
Expected: PASS (3 passed)
- [ ] **Step 5: 커밋**
```bash
git add services/naver-fetch/naver_client.py services/naver-fetch/tests/test_naver_client.py
git commit -m "feat(naver-fetch): NaverClient (Playwright 토큰 + httpx fetch)"
```
---
### Task 4: fetch_worker (오케스트레이션 + heartbeat)
**Files:**
- Create: `services/naver-fetch/fetch_worker.py`
- Test: `services/naver-fetch/tests/test_fetch_worker.py`
**Interfaces:**
- Consumes: `NASClient`, `NaverClient`, `config.Settings`.
- Produces:
- `class FetchState``.state="idle"`, `.last_fetch_at=None`, `.listings_pushed=0`, `.errors=0`
- `in_daytime(now: datetime, start: str, end: str) -> bool`
- `async run_cycle(nas, naver, state, settings) -> None`
- `async fetch_loop(nas, naver, state, settings) -> None`
- `build_heartbeat(state) -> str` (JSON)
- `async heartbeat_loop(redis, state, interval=15, ttl=45) -> None`
- [ ] **Step 1: 실패 테스트**`services/naver-fetch/tests/test_fetch_worker.py`
```python
"""fetch_worker — 주간창/조립/격리/heartbeat."""
import datetime as dt
import json
from zoneinfo import ZoneInfo
from fetch_worker import FetchState, in_daytime, run_cycle, build_heartbeat
from config import load_settings
KST = ZoneInfo("Asia/Seoul")
def _kst(hh, mm):
return dt.datetime(2026, 7, 10, hh, mm, tzinfo=KST)
def test_in_daytime_boundaries():
assert in_daytime(_kst(8, 0), "08:00", "20:00") is True
assert in_daytime(_kst(19, 59), "08:00", "20:00") is True
assert in_daytime(_kst(20, 0), "08:00", "20:00") is False
assert in_daytime(_kst(7, 59), "08:00", "20:00") is False
class _FakeNAS:
def __init__(self, targets):
self._targets = targets
self.ingested = None
async def get_targets(self):
return self._targets
async def post_ingest(self, fetched_at, batches):
self.ingested = {"fetched_at": fetched_at, "batches": batches}
return {"received": sum(len(b["articles"]) for b in batches), "new": 1, "matched": 1}
class _FakeNaver:
def __init__(self, fail_dongs=None):
self._fail = fail_dongs or set()
self.token_acquired = False
async def acquire_token(self):
self.token_acquired = True
async def fetch_articles(self, cortar_no, page=1):
if cortar_no in self._fail:
raise RuntimeError("naver 429")
return {"isMoreData": False,
"articleList": [{"articleNo": f"{cortar_no}-{page}"}]}
async def test_run_cycle_assembles_batches_and_ingests():
nas = _FakeNAS({"dongs": [{"dong": "신대방동", "cortar_no": "1159010100"},
{"dong": "봉천동", "cortar_no": "1162010100"}],
"deal_types": ["매매"], "page_limit": 1})
naver = _FakeNaver()
state = FetchState()
await run_cycle(nas, naver, state, load_settings())
assert naver.token_acquired is True
assert nas.ingested is not None
dongs = [b["dong"] for b in nas.ingested["batches"]]
assert dongs == ["신대방동", "봉천동"]
assert state.listings_pushed == 2
assert state.last_fetch_at is not None
assert state.state == "idle"
assert nas.ingested["fetched_at"].endswith("Z")
async def test_run_cycle_dong_failure_isolated():
nas = _FakeNAS({"dongs": [{"dong": "A", "cortar_no": "1159010100"},
{"dong": "B", "cortar_no": "1162010100"}],
"page_limit": 1})
naver = _FakeNaver(fail_dongs={"1159010100"})
state = FetchState()
await run_cycle(nas, naver, state, load_settings())
assert nas.ingested is not None # 실패해도 ingest 진행
assert state.errors >= 1
batches = {b["dong"]: b["articles"] for b in nas.ingested["batches"]}
assert batches["A"] == [] # 실패 동 빈 배열
assert len(batches["B"]) == 1 # 성공 동
async def test_run_cycle_token_failure_skips_ingest():
class _BadNaver(_FakeNaver):
async def acquire_token(self):
raise RuntimeError("token 실패")
nas = _FakeNAS({"dongs": [], "page_limit": 1})
state = FetchState()
await run_cycle(nas, _BadNaver(), state, load_settings())
assert nas.ingested is None
assert state.errors == 1
assert state.state == "idle"
def test_build_heartbeat_schema():
state = FetchState()
state.state = "fetching"
state.listings_pushed = 7
d = json.loads(build_heartbeat(state))
assert d["name"] == "naver-fetch"
assert d["kind"] == "fetcher"
assert d["state"] == "fetching"
assert isinstance(d["ts"], int)
assert d["last_fetch_at"] is None
assert d["listings_pushed"] == 7
assert d["errors"] == 0
```
- [ ] **Step 2: 실패 확인**
Run: `python -m pytest services/naver-fetch/tests/test_fetch_worker.py -q`
Expected: FAIL — `ModuleNotFoundError: No module named 'fetch_worker'`
- [ ] **Step 3: fetch_worker.py 구현**
```python
"""오케스트레이션 — 주간 스케줄 fetch + fetcher heartbeat."""
from __future__ import annotations
import asyncio
import datetime as dt
import json
import logging
from zoneinfo import ZoneInfo
logger = logging.getLogger(__name__)
KST = ZoneInfo("Asia/Seoul")
HEARTBEAT_KEY = "worker:naver-fetch:heartbeat"
class FetchState:
"""fetch_loop가 갱신, heartbeat_loop가 읽는 공유 상태."""
def __init__(self):
self.state = "idle" # idle | fetching
self.last_fetch_at: int | None = None
self.listings_pushed = 0
self.errors = 0
def _parse_hhmm(s: str) -> dt.time:
hh, mm = s.split(":")
return dt.time(int(hh), int(mm))
def in_daytime(now: dt.datetime, start: str, end: str) -> bool:
t = now.timetz().replace(tzinfo=None)
return _parse_hhmm(start) <= t < _parse_hhmm(end)
def _epoch_now() -> int:
return int(dt.datetime.now(dt.timezone.utc).timestamp())
def _iso_utc_now() -> str:
return dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
async def run_cycle(nas, naver, state, settings) -> None:
state.state = "fetching"
try:
await naver.acquire_token()
except Exception:
logger.exception("naver 토큰 획득 실패")
state.errors += 1
state.state = "idle"
return
try:
tg = await nas.get_targets()
except Exception:
logger.exception("targets 조회 실패")
state.errors += 1
state.state = "idle"
return
dongs = tg.get("dongs", [])
page_limit = int(tg.get("page_limit", 1))
batches: list[dict] = []
for d in dongs:
cortar = d.get("cortar_no")
dong = d.get("dong")
articles: list[dict] = []
try:
for page in range(1, page_limit + 1):
resp = await naver.fetch_articles(cortar, page)
articles += resp.get("articleList", [])
if not resp.get("isMoreData"):
break
except Exception:
logger.exception("naver fetch 실패 dong=%s", dong)
state.errors += 1
batches.append({"dong": dong, "articles": articles})
fetched_at = _iso_utc_now()
try:
r = await nas.post_ingest(fetched_at, batches)
logger.info("ingest received=%s new=%s matched=%s",
r.get("received"), r.get("new"), r.get("matched"))
except Exception:
logger.exception("listings-ingest 실패")
state.errors += 1
state.state = "idle"
return
state.listings_pushed += sum(len(b["articles"]) for b in batches)
state.last_fetch_at = _epoch_now()
state.state = "idle"
async def fetch_loop(nas, naver, state, settings) -> None:
interval = dt.timedelta(hours=settings.fetch_interval_hours)
last_run: dt.datetime | None = None
logger.info("naver-fetch loop 시작 interval=%.1fh 주간창 %s-%s",
settings.fetch_interval_hours, settings.daytime_start, settings.daytime_end)
while True:
try:
now = dt.datetime.now(KST)
if in_daytime(now, settings.daytime_start, settings.daytime_end) and (
last_run is None or (now - last_run) >= interval):
await run_cycle(nas, naver, state, settings)
last_run = now
except asyncio.CancelledError:
logger.info("fetch_loop cancelled")
raise
except Exception:
logger.exception("fetch_loop iteration 실패")
await asyncio.sleep(300)
def build_heartbeat(state) -> str:
return json.dumps({
"name": "naver-fetch", "kind": "fetcher", "state": state.state,
"ts": _epoch_now(), "last_fetch_at": state.last_fetch_at,
"listings_pushed": state.listings_pushed, "errors": state.errors,
})
async def heartbeat_loop(redis, state, interval: int = 15, ttl: int = 45) -> None:
logger.info("naver-fetch heartbeat 시작 ttl=%ds", ttl)
while True:
try:
await redis.set(HEARTBEAT_KEY, build_heartbeat(state), ex=ttl)
except asyncio.CancelledError:
raise
except Exception:
logger.exception("heartbeat 발신 실패")
await asyncio.sleep(interval)
```
- [ ] **Step 4: 통과 확인**
Run: `python -m pytest services/naver-fetch/tests/test_fetch_worker.py -q`
Expected: PASS (5 passed)
- [ ] **Step 5: 커밋**
```bash
git add services/naver-fetch/fetch_worker.py services/naver-fetch/tests/test_fetch_worker.py
git commit -m "feat(naver-fetch): fetch_worker (run_cycle/loop/heartbeat)"
```
---
### Task 5: main.py (FastAPI lifespan + /health)
**Files:**
- Create: `services/naver-fetch/main.py`
- Test: `services/naver-fetch/tests/test_health.py`
**Interfaces:**
- Consumes: `config.load_settings`, `NASClient`, `NaverClient`, `fetch_worker.*`, `redis.asyncio`.
- Produces: FastAPI `app` + `health()``{"ok": True, "service": "naver-fetch"}`.
- [ ] **Step 1: 실패 테스트**`services/naver-fetch/tests/test_health.py`
```python
"""/health — 핸들러 직접 검증."""
from main import health
def test_health():
assert health() == {"ok": True, "service": "naver-fetch"}
```
- [ ] **Step 2: 실패 확인**
Run: `python -m pytest services/naver-fetch/tests/test_health.py -q`
Expected: FAIL — `ModuleNotFoundError: No module named 'main'`
- [ ] **Step 3: main.py 구현**
```python
"""naver-fetch FastAPI entry — lifespan(fetch_loop + heartbeat_loop) + /health."""
from __future__ import annotations
import asyncio
import logging
from contextlib import asynccontextmanager
import redis.asyncio as aioredis
from fastapi import FastAPI
import fetch_worker
from config import load_settings
from nas_client import NASClient
from naver_client import NaverClient
logging.basicConfig(level=logging.INFO,
format="%(asctime)s %(name)s %(levelname)s %(message)s")
logger = logging.getLogger(__name__)
@asynccontextmanager
async def lifespan(app: FastAPI):
settings = load_settings()
nas = NASClient(settings.nas_base_url, settings.internal_api_key)
naver = NaverClient(settings.naver_real_estate_type, settings.naver_token_url,
settings.naver_headless)
state = fetch_worker.FetchState()
redis = aioredis.from_url(settings.redis_url, decode_responses=False)
ft = asyncio.create_task(fetch_worker.fetch_loop(nas, naver, state, settings))
hb = asyncio.create_task(fetch_worker.heartbeat_loop(redis, state))
logger.info("naver-fetch lifespan 시작")
try:
yield
finally:
for t in (ft, hb):
t.cancel()
try:
await t
except asyncio.CancelledError:
pass
await naver.close()
await nas.close()
await redis.aclose()
logger.info("naver-fetch lifespan 종료")
app = FastAPI(lifespan=lifespan)
@app.get("/health")
def health():
return {"ok": True, "service": "naver-fetch"}
```
- [ ] **Step 4: 통과 확인 + 전체 스위트**
Run: `python -m pytest services/naver-fetch/tests -q`
Expected: PASS (config 2 + nas 2 + naver 3 + fetch_worker 5 + health 1 = 13)
- [ ] **Step 5: 커밋**
```bash
git add services/naver-fetch/main.py services/naver-fetch/tests/test_health.py
git commit -m "feat(naver-fetch): FastAPI lifespan + heartbeat 배선 + /health"
```
---
### Task 6: 배포 (Dockerfile + compose + .env.example)
**Files:**
- Create: `services/naver-fetch/Dockerfile`, `services/naver-fetch/.env.example`
- Modify: `services/docker-compose.yml`
**Interfaces:** 없음(배포). 검증 `docker compose config`.
- [ ] **Step 1: Dockerfile**`services/naver-fetch/Dockerfile` (insta-render Chromium 관례 복제)
```dockerfile
FROM python:3.12-slim-bookworm
ENV PYTHONUNBUFFERED=1
WORKDIR /app
# Chromium 런타임 deps (Debian 12 / bookworm) — insta-render 관례
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates tzdata fonts-noto-cjk \
libnss3 libnspr4 libdbus-1-3 libatk1.0-0 libatk-bridge2.0-0 \
libcups2 libdrm2 libxkbcommon0 libxcomposite1 libxdamage1 \
libxfixes3 libxrandr2 libgbm1 libxshmfence1 libpango-1.0-0 \
libcairo2 libasound2 libatspi2.0-0 \
&& rm -rf /var/lib/apt/lists/*
COPY naver-fetch/requirements.txt /app/
RUN pip install --no-cache-dir --timeout 600 --retries 5 -r requirements.txt
RUN playwright install chromium
# 공통 heartbeat 등 모듈 (services/_shared) — 빌드 컨텍스트 . 필요
COPY _shared /app/_shared
COPY naver-fetch/. /app/
ENV PYTHONPATH=/app
EXPOSE 8000
CMD ["python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "1"]
```
- [ ] **Step 2: .env.example**`services/naver-fetch/.env.example`
```
# naver-fetch — realestate 매물 fetch 워커
REDIS_URL=redis://192.168.45.54:6379
NAS_BASE_URL=http://192.168.45.54:18500
# render 워커 공유키 (X-Internal-Key)
INTERNAL_API_KEY=
# 스케줄 (KST)
FETCH_INTERVAL_HOURS=2.5
DAYTIME_START=08:00
DAYTIME_END=20:00
# 네이버
NAVER_REAL_ESTATE_TYPE=APT:ABYG:JGC:PRE
NAVER_HEADLESS=1
# NAVER_TOKEN_URL= # 기본값(new.land.naver.com houses) 사용
```
- [ ] **Step 3: docker-compose.yml에 서비스 추가**
`services/docker-compose.yml``trade-monitor` 블록 뒤(파일 끝, 들여쓰기 2칸)에 추가:
```yaml
naver-fetch:
build:
context: .
dockerfile: naver-fetch/Dockerfile
container_name: naver-fetch
restart: unless-stopped
ports:
- "18716:8000"
environment:
- TZ=Asia/Seoul
- REDIS_URL=${REDIS_URL:-redis://192.168.45.54:6379}
- NAS_BASE_URL=${NAS_BASE_URL:-http://192.168.45.54:18500}
- INTERNAL_API_KEY=${INTERNAL_API_KEY:-}
- FETCH_INTERVAL_HOURS=${FETCH_INTERVAL_HOURS:-2.5}
- DAYTIME_START=${DAYTIME_START:-08:00}
- DAYTIME_END=${DAYTIME_END:-20:00}
- NAVER_REAL_ESTATE_TYPE=${NAVER_REAL_ESTATE_TYPE:-APT:ABYG:JGC:PRE}
- NAVER_HEADLESS=${NAVER_HEADLESS:-1}
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
interval: 60s
timeout: 5s
retries: 3
```
- [ ] **Step 4: compose 검증**
Run: `cd services && docker compose config` (실패 시 YAML 들여쓰기 육안 확인)
Expected: `naver-fetch` 서비스 파싱, 포트 18716.
- [ ] **Step 5: 커밋**
```bash
git add services/naver-fetch/Dockerfile services/naver-fetch/.env.example services/docker-compose.yml
git commit -m "feat(naver-fetch): Dockerfile(Playwright) + compose 서비스(18716) + .env.example"
```
---
## Self-Review
**1. Spec coverage:**
- §4 `GET /targets` → Task 2 `get_targets` + Task 4 사용. ✅
- §4 `POST /listings-ingest`(fetched_at ISO, batches) → Task 2 `post_ingest` + Task 4 조립. ✅
- §4.4 heartbeat(EX45, kind=fetcher, epoch ts, 필드) → Task 4 `build_heartbeat`/`heartbeat_loop`. ✅
- 워커 루프(targets→네이버 fetch→ingest) → Task 4 `run_cycle`. ✅
- 네이버 Bearer 토큰(Playwright) + fetch → Task 3. ✅
- 스케줄(주간 2~3h) → Task 4 `in_daytime` + `fetch_loop`. ✅
- 얇은 프록시(raw 전달) → Task 4(파싱 없음, articleList 그대로). ✅
- 배포(_shared COPY, Playwright, 18716) → Task 6. ✅
**2. Placeholder scan:** 실제 코드/명령/기대출력 기재. TODO 없음. ✅
**3. Type consistency:** `FetchState`(state/last_fetch_at/listings_pushed/errors)가 Task 4 생성 ↔ `build_heartbeat` 소비 일치. `NaverClient`(acquire_token/fetch_articles) Task 3 정의 ↔ Task 4 `run_cycle` 호출 일치. `NASClient`(get_targets/post_ingest) Task 2 ↔ Task 4 일치. fetched_at ISO 문자열/heartbeat ts epoch 구분 유지. ✅
**미해결 플래그(DESIGN §11):** INTERNAL_API_KEY vs REALESTATE_INTERNAL_KEY(배포 전 BE 확인), 헤드리스 탐지, deal_types 미사용(NAS 필터), 토큰 만료 재획득.

View File

@@ -0,0 +1,36 @@
"""Settings — 환경변수 로드."""
from __future__ import annotations
import os
from dataclasses import dataclass
_DEFAULT_TOKEN_URL = (
"https://new.land.naver.com/houses?ms=37.489,126.921,16&a=APT:ABYG:JGC:PRE&e=RETAIL"
)
@dataclass
class Settings:
nas_base_url: str
internal_api_key: str
redis_url: str
fetch_interval_hours: float
daytime_start: str
daytime_end: str
naver_real_estate_type: str
naver_token_url: str
naver_headless: bool
def load_settings() -> Settings:
return Settings(
nas_base_url=os.getenv("NAS_BASE_URL", "http://192.168.45.54:18800"),
internal_api_key=os.getenv("INTERNAL_API_KEY", ""),
redis_url=os.getenv("REDIS_URL", "redis://192.168.45.54:6379"),
fetch_interval_hours=float(os.getenv("FETCH_INTERVAL_HOURS", "2.5")),
daytime_start=os.getenv("DAYTIME_START", "08:00"),
daytime_end=os.getenv("DAYTIME_END", "20:00"),
naver_real_estate_type=os.getenv("NAVER_REAL_ESTATE_TYPE", "APT:ABYG:JGC:PRE"),
naver_token_url=os.getenv("NAVER_TOKEN_URL", _DEFAULT_TOKEN_URL),
naver_headless=os.getenv("NAVER_HEADLESS", "1") == "1",
)

View File

@@ -0,0 +1,5 @@
"""services/ 루트를 sys.path에 추가 — from _shared... import 가능하게."""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))

View File

@@ -0,0 +1,129 @@
"""오케스트레이션 — 주간 스케줄 fetch + fetcher heartbeat."""
from __future__ import annotations
import asyncio
import datetime as dt
import json
import logging
from zoneinfo import ZoneInfo
logger = logging.getLogger(__name__)
KST = ZoneInfo("Asia/Seoul")
HEARTBEAT_KEY = "worker:naver-fetch:heartbeat"
class FetchState:
"""fetch_loop가 갱신, heartbeat_loop가 읽는 공유 상태."""
def __init__(self):
self.state = "idle" # idle | fetching
self.last_fetch_at: int | None = None
self.listings_pushed = 0
self.errors = 0
def _parse_hhmm(s: str) -> dt.time:
hh, mm = s.split(":")
return dt.time(int(hh), int(mm))
def in_daytime(now: dt.datetime, start: str, end: str) -> bool:
t = now.timetz().replace(tzinfo=None)
return _parse_hhmm(start) <= t < _parse_hhmm(end)
def _epoch_now() -> int:
return int(dt.datetime.now(dt.timezone.utc).timestamp())
def _iso_utc_now() -> str:
return dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
async def run_cycle(nas, naver, state, settings) -> None:
state.state = "fetching"
try:
await naver.acquire_token()
except Exception:
logger.exception("naver 토큰 획득 실패")
state.errors += 1
state.state = "idle"
return
try:
tg = await nas.get_targets()
except Exception:
logger.exception("targets 조회 실패")
state.errors += 1
state.state = "idle"
return
dongs = tg.get("dongs", [])
page_limit = int(tg.get("page_limit", 1))
fetched_at = _iso_utc_now()
# 동별로 fetch → 즉시 ingest. 전체 batch를 한 POST로 보내면 NAS 동기 처리(매물 수 비례)가
# >180s로 타임아웃하므로 동 단위로 쪼갬(각 ~수십건 → 빠름, 부분 실패 격리). batches 리스트 계약 호환.
for d in dongs:
cortar = d.get("cortar_no")
dong = d.get("dong")
try:
articles: list[dict] = []
for page in range(1, page_limit + 1):
resp = await naver.fetch_articles(cortar, page)
articles += resp.get("articleList", [])
if not resp.get("isMoreData"):
break
except Exception:
logger.exception("naver fetch 실패 dong=%s", dong)
state.errors += 1
continue
try:
r = await nas.post_ingest(fetched_at, [{"dong": dong, "articles": articles}])
# BE ingest async화(main 8e28ce9): 응답 {received,new,queued} (matched는 백그라운드라 제거됨)
logger.info("ingest dong=%s received=%s new=%s queued=%s",
dong, r.get("received"), r.get("new"), r.get("queued"))
state.listings_pushed += len(articles)
except Exception:
logger.exception("listings-ingest 실패 dong=%s", dong)
state.errors += 1
state.last_fetch_at = _epoch_now()
state.state = "idle"
async def fetch_loop(nas, naver, state, settings) -> None:
interval = dt.timedelta(hours=settings.fetch_interval_hours)
last_run: dt.datetime | None = None
logger.info("naver-fetch loop 시작 interval=%.1fh 주간창 %s-%s",
settings.fetch_interval_hours, settings.daytime_start, settings.daytime_end)
while True:
try:
now = dt.datetime.now(KST)
if in_daytime(now, settings.daytime_start, settings.daytime_end) and (
last_run is None or (now - last_run) >= interval):
await run_cycle(nas, naver, state, settings)
last_run = now
except asyncio.CancelledError:
logger.info("fetch_loop cancelled")
raise
except Exception:
logger.exception("fetch_loop iteration 실패")
await asyncio.sleep(300)
def build_heartbeat(state) -> str:
return json.dumps({
"name": "naver-fetch", "kind": "fetcher", "state": state.state,
"ts": _epoch_now(), "last_fetch_at": state.last_fetch_at,
"listings_pushed": state.listings_pushed, "errors": state.errors,
})
async def heartbeat_loop(redis, state, interval: int = 15, ttl: int = 45) -> None:
logger.info("naver-fetch heartbeat 시작 ttl=%ds", ttl)
while True:
try:
await redis.set(HEARTBEAT_KEY, build_heartbeat(state), ex=ttl)
except asyncio.CancelledError:
raise
except Exception:
logger.exception("heartbeat 발신 실패")
await asyncio.sleep(interval)

View File

@@ -0,0 +1,53 @@
"""naver-fetch FastAPI entry — lifespan(fetch_loop + heartbeat_loop) + /health."""
from __future__ import annotations
import asyncio
import logging
from contextlib import asynccontextmanager
import redis.asyncio as aioredis
from fastapi import FastAPI
import fetch_worker
from config import load_settings
from nas_client import NASClient
from naver_client import NaverClient
logging.basicConfig(level=logging.INFO,
format="%(asctime)s %(name)s %(levelname)s %(message)s")
logger = logging.getLogger(__name__)
@asynccontextmanager
async def lifespan(app: FastAPI):
settings = load_settings()
nas = NASClient(settings.nas_base_url, settings.internal_api_key)
naver = NaverClient(settings.naver_real_estate_type, settings.naver_token_url,
settings.naver_headless)
state = fetch_worker.FetchState()
redis = aioredis.from_url(settings.redis_url, decode_responses=False)
ft = asyncio.create_task(fetch_worker.fetch_loop(nas, naver, state, settings))
hb = asyncio.create_task(fetch_worker.heartbeat_loop(redis, state))
logger.info("naver-fetch lifespan 시작")
try:
yield
finally:
for t in (ft, hb):
t.cancel()
try:
await t
except asyncio.CancelledError:
pass
await naver.close()
await nas.close()
await redis.aclose()
logger.info("naver-fetch lifespan 종료")
app = FastAPI(lifespan=lifespan)
@app.get("/health")
def health():
return {"ok": True, "service": "naver-fetch"}

View File

@@ -0,0 +1,55 @@
"""NAS realestate 내부 계약 — X-Internal-Key + retry."""
from __future__ import annotations
import asyncio
import logging
import httpx
logger = logging.getLogger(__name__)
_RETRY_STATUSES = {429, 500, 502, 503, 504}
_TARGETS_TIMEOUT = 15.0
_INGEST_TIMEOUT = 180.0 # 첫 실행 bulk upsert/매칭/알림이 느림 (초 단위 수십)
class NASClient:
def __init__(self, base_url: str, api_key: str):
self._base_url = base_url.rstrip("/")
self._api_key = api_key
self._client = httpx.AsyncClient()
async def close(self) -> None:
await self._client.aclose()
async def get_targets(self) -> dict:
return await self._request("GET", "/api/internal/realestate/targets",
timeout=_TARGETS_TIMEOUT, max_attempts=3)
async def post_ingest(self, fetched_at: str, batches: list[dict]) -> dict:
# ingest는 재시도 안 함(max_attempts=1) — 타임아웃 후 재전송 시
# NAS가 같은 batch를 재처리/재알림할 위험. 실패 시 다음 사이클에 재시도.
return await self._request(
"POST", "/api/internal/realestate/listings-ingest",
json={"fetched_at": fetched_at, "batches": batches},
timeout=_INGEST_TIMEOUT, max_attempts=1)
async def _request(self, method: str, path: str, *, timeout: float,
max_attempts: int, **kwargs) -> dict:
url = f"{self._base_url}{path}"
headers = {"X-Internal-Key": self._api_key}
for attempt in range(max_attempts):
try:
resp = await self._client.request(method, url, headers=headers,
timeout=timeout, **kwargs)
if resp.status_code in _RETRY_STATUSES and attempt < max_attempts - 1:
await asyncio.sleep(2 ** attempt)
continue
resp.raise_for_status()
return resp.json()
except httpx.TimeoutException:
if attempt < max_attempts - 1:
await asyncio.sleep(2 ** attempt)
continue
raise
raise RuntimeError("retry exhausted")

View File

@@ -0,0 +1,79 @@
"""Naver land client — Playwright 토큰 획득 + httpx article fetch."""
from __future__ import annotations
import logging
import httpx
logger = logging.getLogger(__name__)
_UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36")
_API = "https://new.land.naver.com/api/articles"
class NaverClient:
def __init__(self, real_estate_type: str, token_url: str,
headless: bool = True, timeout: float = 20.0):
self._ret = real_estate_type
self._token_url = token_url
self._headless = headless
self._client = httpx.AsyncClient(timeout=timeout)
self._token: str | None = None
self._cookies: str = ""
async def close(self) -> None:
await self._client.aclose()
async def acquire_token(self) -> None:
"""new.land.naver.com 로드 → XHR Authorization 헤더 인터셉트."""
from playwright.async_api import async_playwright
captured: dict[str, str] = {}
async with async_playwright() as p:
browser = await p.chromium.launch(headless=self._headless)
try:
ctx = await browser.new_context(user_agent=_UA, locale="ko-KR")
page = await ctx.new_page()
def on_request(req):
auth = req.headers.get("authorization")
if auth and "bearer" in auth.lower() and "auth" not in captured:
captured["auth"] = auth
page.on("request", on_request)
try:
await page.goto(self._token_url, wait_until="networkidle", timeout=45000)
except Exception:
logger.warning("naver token goto 지연/부분 실패 — 캡처 계속")
await page.wait_for_timeout(4000)
cookies = await ctx.cookies()
self._cookies = "; ".join(f"{c['name']}={c['value']}" for c in cookies)
finally:
await browser.close()
token = captured.get("auth")
if not token:
raise RuntimeError("naver Bearer token 획득 실패")
self._token = token
logger.info("naver token 획득 (len=%d)", len(token))
async def fetch_articles(self, cortar_no: str, page: int = 1) -> dict:
if not self._token:
raise RuntimeError("token 없음 — acquire_token 먼저 호출")
params = {
"cortarNo": cortar_no, "order": "rank", "realEstateType": self._ret,
"tradeType": "", "rentPriceMin": 0, "rentPriceMax": 900000000,
"priceMin": 0, "priceMax": 900000000, "areaMin": 0, "areaMax": 900000000,
"showArticle": "false", "sameAddressGroup": "false",
"priceType": "RETAIL", "page": page,
}
headers = {
"authorization": self._token, "user-agent": _UA,
"referer": "https://new.land.naver.com/",
"accept": "application/json, text/plain, */*",
"accept-language": "ko-KR,ko;q=0.9", "cookie": self._cookies,
}
resp = await self._client.get(_API, params=params, headers=headers)
resp.raise_for_status()
return resp.json()

View File

@@ -0,0 +1,2 @@
[pytest]
asyncio_mode = auto

View File

@@ -0,0 +1,8 @@
fastapi==0.115.6
uvicorn[standard]==0.34.0
playwright==1.48.0
redis>=5.0
httpx>=0.27
pytest>=8.0
pytest-asyncio>=0.24
respx>=0.21

View File

View File

@@ -0,0 +1,25 @@
"""Settings env 로드."""
from config import load_settings
def test_defaults(monkeypatch):
for k in ("NAS_BASE_URL", "INTERNAL_API_KEY", "REDIS_URL", "FETCH_INTERVAL_HOURS",
"DAYTIME_START", "DAYTIME_END", "NAVER_REAL_ESTATE_TYPE",
"NAVER_TOKEN_URL", "NAVER_HEADLESS"):
monkeypatch.delenv(k, raising=False)
s = load_settings()
assert s.nas_base_url == "http://192.168.45.54:18800"
assert s.fetch_interval_hours == 2.5
assert s.daytime_start == "08:00" and s.daytime_end == "20:00"
assert s.naver_headless is True
assert "new.land.naver.com" in s.naver_token_url
def test_override(monkeypatch):
monkeypatch.setenv("INTERNAL_API_KEY", "sekret")
monkeypatch.setenv("FETCH_INTERVAL_HOURS", "3")
monkeypatch.setenv("NAVER_HEADLESS", "0")
s = load_settings()
assert s.internal_api_key == "sekret"
assert s.fetch_interval_hours == 3.0
assert s.naver_headless is False

View File

@@ -0,0 +1,107 @@
"""fetch_worker — 주간창/조립/격리/heartbeat."""
import datetime as dt
import json
from zoneinfo import ZoneInfo
from fetch_worker import FetchState, in_daytime, run_cycle, build_heartbeat
from config import load_settings
KST = ZoneInfo("Asia/Seoul")
def _kst(hh, mm):
return dt.datetime(2026, 7, 10, hh, mm, tzinfo=KST)
def test_in_daytime_boundaries():
assert in_daytime(_kst(8, 0), "08:00", "20:00") is True
assert in_daytime(_kst(19, 59), "08:00", "20:00") is True
assert in_daytime(_kst(20, 0), "08:00", "20:00") is False
assert in_daytime(_kst(7, 59), "08:00", "20:00") is False
class _FakeNAS:
def __init__(self, targets):
self._targets = targets
self.ingests = [] # 동별 post_ingest 호출 기록
async def get_targets(self):
return self._targets
async def post_ingest(self, fetched_at, batches):
self.ingests.append({"fetched_at": fetched_at, "batches": batches})
n = sum(len(b["articles"]) for b in batches)
return {"received": n, "new": n, "queued": True}
class _FakeNaver:
def __init__(self, fail_dongs=None):
self._fail = fail_dongs or set()
self.token_acquired = False
async def acquire_token(self):
self.token_acquired = True
async def fetch_articles(self, cortar_no, page=1):
if cortar_no in self._fail:
raise RuntimeError("naver 429")
return {"isMoreData": False,
"articleList": [{"articleNo": f"{cortar_no}-{page}"}]}
async def test_run_cycle_ingests_per_dong():
nas = _FakeNAS({"dongs": [{"dong": "신대방동", "cortar_no": "1159010100"},
{"dong": "봉천동", "cortar_no": "1162010100"}],
"deal_types": ["매매"], "page_limit": 1})
naver = _FakeNaver()
state = FetchState()
await run_cycle(nas, naver, state, load_settings())
assert naver.token_acquired is True
assert len(nas.ingests) == 2 # 동별 1 POST씩
dongs = [ing["batches"][0]["dong"] for ing in nas.ingests]
assert dongs == ["신대방동", "봉천동"]
assert all(len(ing["batches"]) == 1 for ing in nas.ingests)
assert state.listings_pushed == 2
assert state.last_fetch_at is not None
assert state.state == "idle"
assert nas.ingests[0]["fetched_at"].endswith("Z")
async def test_run_cycle_dong_failure_isolated():
nas = _FakeNAS({"dongs": [{"dong": "A", "cortar_no": "1159010100"},
{"dong": "B", "cortar_no": "1162010100"}],
"page_limit": 1})
naver = _FakeNaver(fail_dongs={"1159010100"})
state = FetchState()
await run_cycle(nas, naver, state, load_settings())
# A는 fetch 실패 → ingest skip, B만 ingest (부분 실패 격리)
assert len(nas.ingests) == 1
assert nas.ingests[0]["batches"][0]["dong"] == "B"
assert state.errors >= 1
async def test_run_cycle_token_failure_skips_ingest():
class _BadNaver(_FakeNaver):
async def acquire_token(self):
raise RuntimeError("token 실패")
nas = _FakeNAS({"dongs": [], "page_limit": 1})
state = FetchState()
await run_cycle(nas, _BadNaver(), state, load_settings())
assert nas.ingests == []
assert state.errors == 1
assert state.state == "idle"
def test_build_heartbeat_schema():
state = FetchState()
state.state = "fetching"
state.listings_pushed = 7
d = json.loads(build_heartbeat(state))
assert d["name"] == "naver-fetch"
assert d["kind"] == "fetcher"
assert d["state"] == "fetching"
assert isinstance(d["ts"], int)
assert d["last_fetch_at"] is None
assert d["listings_pushed"] == 7
assert d["errors"] == 0

View File

@@ -0,0 +1,6 @@
"""/health — 핸들러 직접 검증."""
from main import health
def test_health():
assert health() == {"ok": True, "service": "naver-fetch"}

View File

@@ -0,0 +1,41 @@
"""NASClient — targets/ingest + X-Internal-Key (respx)."""
import json as _json
import httpx
import respx
from nas_client import NASClient
BASE = "http://nas.test"
@respx.mock
async def test_get_targets_sends_key():
route = respx.get(f"{BASE}/api/internal/realestate/targets").mock(
return_value=httpx.Response(200, json={
"dongs": [{"dong": "신대방동", "cortar_no": "1159010100"}],
"deal_types": ["전세", "매매"], "page_limit": 2}))
c = NASClient(BASE, "KEY")
tg = await c.get_targets()
assert tg["page_limit"] == 2
assert tg["dongs"][0]["cortar_no"] == "1159010100"
assert route.calls.last.request.headers["X-Internal-Key"] == "KEY"
await c.close()
@respx.mock
async def test_post_ingest_payload():
captured = {}
def _resp(request):
captured.update(_json.loads(request.content))
return httpx.Response(200, json={"received": 3, "new": 2, "matched": 1})
respx.post(f"{BASE}/api/internal/realestate/listings-ingest").mock(side_effect=_resp)
c = NASClient(BASE, "KEY")
batches = [{"dong": "신대방동", "articles": [{"articleNo": "1"}, {"articleNo": "2"}]}]
out = await c.post_ingest("2026-07-10T05:00:00Z", batches)
assert out == {"received": 3, "new": 2, "matched": 1}
assert captured["fetched_at"] == "2026-07-10T05:00:00Z"
assert captured["batches"] == batches
await c.close()

View File

@@ -0,0 +1,55 @@
"""NaverClient.fetch_articles — httpx 경로 (respx). acquire_token은 실브라우저라 제외."""
import httpx
import pytest
import respx
from naver_client import NaverClient
API = "https://new.land.naver.com/api/articles"
def _client():
return NaverClient(real_estate_type="APT:ABYG", token_url="https://x", headless=True)
async def test_fetch_requires_token():
c = _client()
with pytest.raises(RuntimeError):
await c.fetch_articles("1159010100", 1)
await c.close()
@respx.mock
async def test_fetch_articles_request_and_parse():
captured = {}
def _resp(request):
captured["auth"] = request.headers.get("authorization")
captured["cookie"] = request.headers.get("cookie")
captured["url"] = str(request.url)
return httpx.Response(200, json={
"isMoreData": True,
"articleList": [{"articleNo": "111", "articleName": "테스트아파트"}]})
respx.get(API).mock(side_effect=_resp)
c = _client()
c._token = "Bearer TESTJWT" # acquire_token 우회(단위테스트)
c._cookies = "NNB=abc; page_uid=xyz"
out = await c.fetch_articles("1159010100", page=2)
assert out["articleList"][0]["articleNo"] == "111"
assert captured["auth"] == "Bearer TESTJWT"
assert captured["cookie"] == "NNB=abc; page_uid=xyz"
assert "cortarNo=1159010100" in captured["url"]
assert "page=2" in captured["url"]
assert "realEstateType=APT" in captured["url"]
await c.close()
@respx.mock
async def test_fetch_non_200_raises():
respx.get(API).mock(return_value=httpx.Response(429, json={"code": "TOO_MANY_REQUESTS"}))
c = _client()
c._token = "Bearer X"
with pytest.raises(httpx.HTTPStatusError):
await c.fetch_articles("1159010100", 1)
await c.close()

View File

@@ -7,10 +7,13 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates tzdata \ ca-certificates tzdata \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
COPY requirements.txt . COPY task-watcher/requirements.txt /app/
RUN pip install --no-cache-dir --timeout 600 --retries 5 -r requirements.txt RUN pip install --no-cache-dir --timeout 600 --retries 5 -r requirements.txt
COPY . . # 공통 heartbeat 모듈 (services/_shared) — watcher.py가 from _shared.heartbeat import
COPY _shared /app/_shared
COPY task-watcher/. /app/
ENV PYTHONPATH=/app
EXPOSE 8000 EXPOSE 8000
CMD ["python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "1"] CMD ["python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "1"]

View File

@@ -0,0 +1,5 @@
"""Make services/ root importable so `from _shared.heartbeat import ...` works during tests."""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))

View File

@@ -0,0 +1,16 @@
"""task-watcher heartbeat payload — state=mode + mode 필드 검증."""
import json
from _shared.heartbeat import build_payload, WorkerStats
def test_watcher_heartbeat_payload_carries_mode():
payload = json.loads(
build_payload(
"task-watcher", "watcher", "trading",
WorkerStats(), extra={"mode": "trading"},
)
)
assert payload["kind"] == "watcher"
assert payload["state"] == "trading"
assert payload["mode"] == "trading"

View File

@@ -15,6 +15,7 @@ from zoneinfo import ZoneInfo
import redis.asyncio as aioredis import redis.asyncio as aioredis
from mode import current_mode, fetch_holidays, KST from mode import current_mode, fetch_holidays, KST
from _shared.heartbeat import build_payload, WorkerStats
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -23,6 +24,10 @@ PAUSED_KEY = "queue:paused"
LOOP_INTERVAL = 30 # 초 LOOP_INTERVAL = 30 # 초
HOLIDAYS_REFRESH = 3600 # 1시간 HOLIDAYS_REFRESH = 3600 # 1시간
PAUSED_TTL = 600 # 10분 (watcher 죽어도 자동 해제) PAUSED_TTL = 600 # 10분 (watcher 죽어도 자동 해제)
HEARTBEAT_KEY = "worker:task-watcher:heartbeat"
HEARTBEAT_TTL = 45 # LOOP_INTERVAL 30s < TTL 45s → 만료 전 갱신
_HB_STATS = WorkerStats()
async def watcher_loop(): async def watcher_loop():
@@ -46,6 +51,13 @@ async def watcher_loop():
else: else:
await redis.delete(PAUSED_KEY) await redis.delete(PAUSED_KEY)
# heartbeat (LOOP_INTERVAL=30s < TTL 45s → 만료 전 갱신)
await redis.set(
HEARTBEAT_KEY,
build_payload("task-watcher", "watcher", mode, _HB_STATS, extra={"mode": mode}),
ex=HEARTBEAT_TTL,
)
if mode != last_mode: if mode != last_mode:
logger.info("mode 전환: %s%s (paused=%s)", last_mode, mode, mode == "trading") logger.info("mode 전환: %s%s (paused=%s)", last_mode, mode, mode == "trading")
last_mode = mode last_mode = mode

View File

@@ -0,0 +1,18 @@
# Plan-realtime-trade-alerts — trade-monitor
# NAS Redis (heartbeat)
REDIS_URL=redis://192.168.45.54:6379
# NAS stock 백엔드 (monitor-set / report)
NAS_BASE_URL=http://192.168.45.54:18500
WEBAI_API_KEY=
# KIS 자체 토큰 (ai_trade와 분리된 전용 app_key)
TM_KIS_APP_KEY=
TM_KIS_APP_SECRET=
TM_KIS_ACCOUNT=
TM_KIS_IS_VIRTUAL=0
# 루프 주기(초) / sell_climax 거래량 배수 임계
TM_LOOP_INTERVAL=60
TM_CLIMAX_VOL_MULT=3.0

View File

@@ -0,0 +1,166 @@
# trade-monitor 워커 — 구현 설계
> **2026-07-03 · web-ai 소유.** 실시간 매매 알람 파이프라인의 Windows-side 워커.
> **권위 계약(원본 스펙)**: web-backend repo `docs/superpowers/specs/2026-07-02-realtime-trade-alerts-design.md` §5(계약)·§6(조건).
> 이 문서는 그 계약을 Windows Docker 워커로 구현하기 위한 **구현 설계**(모듈 분해·조건 해석·배포)만 다룬다.
---
## 1. 역할 · 경계
`services/trade-monitor/` 는 형제 워커(`image-render`, `task-watcher`)와 동일한 관례를 따르는 **FastAPI + asyncio 루프** 워커다. WSL2 Docker(`services/docker-compose.yml`)에서 구동.
**책임**
1. 60초 루프로 NAS `monitor-set` 조회 → 세션 게이트.
2. 비-KRX(알파벳) 티커 skip.
3. KIS 실시간 현재가 + 일봉 OHLCV 조회 → TA 지표 계산.
4. 매수/매도 조건 평가 → 발화집합 F 구성.
5. `POST report`로 F 전체 전송(무상태 — dedup은 NAS 영속).
6. Redis heartbeat 발신(`worker:trade-monitor:heartbeat` EX45).
7. KIS 오류는 사이클/종목 단위 격리(다음 분 재시도).
**경계 밖(안 함)**: dedup 상태 보관, 텔레그램 전송(NAS 담당), KST 세션/휴장 캘린더 재구현(NAS가 `session` 판정), 주문 실행.
---
## 2. 모듈 분해
| 파일 | 책임 | 인터페이스(순수/부작용) |
|------|------|------|
| `main.py` | FastAPI app + lifespan. `monitor_loop` + `heartbeat_loop` 스폰, `/health` | 부작용(태스크 스폰) |
| `monitor.py` | 오케스트레이션 루프. monitor-set→게이트→종목순회→firing→report. 공유 `MonitorState` 갱신 | 부작용 |
| `nas_client.py` | `get_monitor_set()` / `post_report(as_of, firing)``X-WebAI-Key` + retry | 부작용(HTTP) |
| `kis_client.py` | KIS REST: `_issue_token()`(OAuth 자체 발급, 24h 캐시) + `get_quote()` + `get_daily_ohlcv()` + 0.5s throttle | 부작용(HTTP) |
| `indicators.py` | `sma`, `rsi`, `avg_volume`, `highest_high` | **순수** |
| `conditions.py` | `evaluate_buy(ctx, buy_params)` / `evaluate_sell(ctx, exit_params)``list[firing]` | **순수** |
| `config.py` | `Settings` — env 로드 | 순수 |
순수 모듈(`indicators`, `conditions`)에 조건 로직을 격리해 테이블 기반 단위 테스트로 검증 가능하게 한다. HTTP·시간·Redis는 경계 모듈에만.
---
## 3. 데이터 흐름 (monitor_loop, 60초)
```
매 사이클:
ms = nas.get_monitor_set() # §5.1
state.session = ms.session
if ms.session == "closed":
state.hb = "market_closed"; sleep; continue # KIS 호출 0
targets = filter_krx(ms.buy_targets, ms.sell_targets) # 알파벳 티커 skip
firing = []
for t in targets: # 종목 단위 try/except 격리
quote = kis.get_quote(t.ticker) # 현재가 + 당일 누적 거래량
daily = kis.get_daily_ohlcv(t.ticker, 250) # MA200·52주 고점용
ctx = build_ctx(t, quote, daily)
if t is buy_target: firing += evaluate_buy(ctx, ms.buy_params)
if t is sell_target: firing += evaluate_sell(ctx, ms.exit_params)
if firing: state.last_alert_at = now
nas.post_report(as_of=now_kst_iso, firing=firing) # §5.2 — 빈 배열도 전송(edge clear 위해)
state.hb = "market_open"
stats.jobs_done += 1
```
`heartbeat_loop`(별도 15초 태스크)이 `state`를 읽어 §5.4 페이로드 발신.
---
## 4. 조건 로직 해석 (§6)
지표는 **일봉 시계열**(최신 last, 오름차순) + **실시간 현재가**(`price`) 기준. 데이터 부족(예: MA200용 200봉 미만)이면 해당 조건은 **미발화**(graceful skip). 각 firing은 `{ticker, kind, condition, price, detail}`.
### 매수 (`buy_targets`, `buy_params={rsi_oversold, breakout_vol_mult, pullback_pct}`)
| condition | 발화 규칙(해석) | detail |
|-----------|----------------|--------|
| `buy_ma20_pullback` | `ma20>ma50>ma200`(정배열) **AND** 최근 3봉 최저가가 `ma20*(1+pullback_pct)` 이하로 접근 **AND** `price>ma20`(반등 복귀) | `ma20, ma50, ma200, recent_low` |
| `buy_breakout` | `price > 직전 20봉 최고가`(당일 제외) **AND** `today_volume > breakout_vol_mult × avg_volume(20)` | `prior_high_20, vol_mult, avg_vol_20` |
| `buy_rsi_bounce` | RSI(14) 시계열에서 `min(rsi[-3:]) < rsi_oversold` **AND** `rsi[-1] > rsi_oversold` **AND** `rsi[-1] > rsi[-2]`(반등). 사이클마다 재계산·무상태 | `rsi, rsi_prev, rsi_oversold` |
종합: 각 조건 독립 발화(신뢰도 가중합은 NAS/텔레그램 단계 책임 아님 — 워커는 조건 발화만).
### 매도 (`sell_targets`, `exit_params={stop_pct, take_pct, trailing_pct}`, target에 `avg_price, qty, holding_high`)
| condition | 발화 규칙 | detail |
|-----------|----------|--------|
| `sell_stop_loss` | `(price-avg)/avg ≤ -stop_pct` | `avg_price, pnl_pct, stop_pct` |
| `sell_take_profit` | `(price-avg)/avg ≥ take_pct` | `avg_price, pnl_pct, take_pct` |
| `sell_trailing_stop` | `price ≤ holding_high × (1-trailing_pct)` (기본 0.10) | `holding_high, trailing_pct, drawdown_pct` |
| `sell_ma_break` | `price < ma50` (추가 `price<ma200`이면 detail.severity="high") | `ma50, ma200, severity` |
| `sell_climax` | **holdings_intel 정합**: `today_volume ≥ climax_vol_x × avg_volume(20)` **AND** `price < day_high × climax_close_pct`(윗꼬리) | `vol_mult, day_high, climax_close_pct` |
`climax_vol_x`(기본 3.0)·`climax_close_pct`(기본 0.97)는 monitor-set `exit_params`에서 읽음(BE 중앙화, main ed17193). 없으면 env `TM_CLIMAX_VOL_MULT` fallback. `day_high`는 KIS quote `stck_hgpr`(당일 세션 누적 고가).
---
## 5. KIS 클라이언트 (자체 토큰)
- `_issue_token()`: `POST {base}/oauth2/tokenP {grant_type, appkey, appsecret}``access_token`(만료 24h). 메모리 캐시, 만료 10분 전 재발급. **ai_trade와 분리된 `TM_KIS_APP_KEY/SECRET`** 사용(같은 app_key 공유 시 토큰 상호 무효화 + EGW00201).
- `get_quote(ticker)`: `inquire-price`(FHKST01010100) → `stck_prpr`(현재가), `acml_vol`(당일 누적 거래량), `stck_oprc`(당일 시가).
- `get_daily_ohlcv(ticker, days=250)`: `inquire-daily-itemchartprice`(FHKST03010100) — ai_trade `kis_client.py` 로직 복제, 오름차순.
- throttle 0.5s(초당 2회) + `_throttle_lock` 직렬화 + 429/timeout 지수 backoff(ai_trade 패턴 재사용).
> ⚠️ **운영 함정**: ai_trade와 KIS를 동시 호출하면 전용 app_key라도 KIS 계정 전체 rate limit을 공유할 수 있음. 별도 app_key로 무효화는 회피되나, 운영 시 동시 부하 모니터링 필요(Phase 7 백로그 연계).
---
## 6. heartbeat (§5.4)
`_shared.heartbeat.heartbeat_loop(redis, "trade-monitor", "trader", stats, interval=15, ttl=45, state_fn=...)`.
- `state_fn``MonitorState`를 읽어 `state ∈ {market_open, market_closed, idle}` + `extra={"last_alert_at": ...}` 반환.
- **디커플링 이유**: 루프 60초 > TTL 45초 → 인라인 발신 시 만료 갭. 15초 독립 태스크로 해소(형제 워커와 동일 구조). §5.4 필수 필드(name/kind/state/ts/last_alert_at) 충족, `jobs_done/jobs_failed`는 형제 워커처럼 superset 유지.
- 초기 상태 `idle`(첫 monitor-set 조회 전).
---
## 7. 설정 (env) — `TM_` 접두사로 ai_trade와 분리
| env | 기본값 | 용도 |
|-----|--------|------|
| `NAS_BASE_URL` | `http://192.168.45.54:18500` | stock 백엔드 |
| `WEBAI_API_KEY` | (필수) | `X-WebAI-Key` |
| `REDIS_URL` | `redis://192.168.45.54:6379` | heartbeat |
| `TM_KIS_APP_KEY` / `TM_KIS_APP_SECRET` | (필수) | KIS 자체 토큰 |
| `TM_KIS_ACCOUNT` | (필수) | KIS 계좌 |
| `TM_KIS_IS_VIRTUAL` | `0` | 실전/모의 |
| `TM_LOOP_INTERVAL` | `60` | 루프 주기(초) |
| `TM_CLIMAX_VOL_MULT` | `3.0` | sell_climax 임계 |
---
## 8. 에러 처리
- **monitor-set 실패**: 사이클 skip(report 안 함), heartbeat=`idle`, 다음 분 재시도.
- **KIS 종목 실패**: 해당 종목만 skip(로그 warning), 나머지 종목 계속.
- **report 실패**: 로그 error, 다음 사이클 신선 firing 재전송(무상태라 손실 허용).
- 루프 최상위 `try/except` — 어떤 예외도 루프를 죽이지 않음(task-watcher 패턴).
---
## 9. 테스트 전략 (pytest, 시스템 Python)
| 파일 | 검증 |
|------|------|
| `test_indicators.py` | sma/rsi/avg_volume/highest_high 수치(알려진 시계열), 데이터 부족 시 None |
| `test_conditions.py` | 8개 조건 테이블 기반(발화/미발화 경계), detail 필드 |
| `test_nas_client.py` | respx — monitor-set 파싱, report 페이로드, X-WebAI-Key 헤더, retry |
| `test_kis_client.py` | respx — 토큰 발급/캐시, quote/daily 파싱, throttle |
| `test_monitor.py` | 루프 1회(mock): closed skip, 비-KRX skip, firing 조립, last_alert_at 갱신, 종목 실패 격리 |
---
## 10. 배포
- `services/trade-monitor/Dockerfile`: task-watcher 관례 복제 — `COPY _shared /app/_shared` **필수**(빌드 컨텍스트 `.` 에서), `COPY trade-monitor/. /app/`, `PYTHONPATH=/app`, uvicorn `:8000`.
- `services/docker-compose.yml`: `trade-monitor` 서비스 추가, 포트 **18715**(image-render 18714 다음), `TZ=Asia/Seoul`, KIS/WEBAI/REDIS env, healthcheck `/health`.
- `services/.env`(비커밋): `TM_KIS_*`, `WEBAI_API_KEY` 실값. `.env.example`에 키만 기재.
---
## 11. 미해결 플래그 / 후속
1. **sell_climax** — ✅ 2026-07-03 holdings_intel 정합 완료(`price < day_high × climax_close_pct` + `exit_params` 파라미터화). BE 회신 기준.
2. **KIS 지표 필드 실검증** — quote의 `acml_vol`/`stck_oprc`, daily TR 응답 필드는 첫 운영 raw 캡처로 대조.
3. **`buy_ma20_pullback`·`buy_rsi_bounce` 해석** — "current candle series" 문구를 일봉 시계열로 해석. 첫 운영 4주 IC 검증 시 재조정 가능.
4. **KIS rate limit 공존** — ai_trade와 동시 부하. 전용 app_key로 토큰 무효화는 회피, 초당 호출 총량은 운영 모니터링.
5. **after 세션 시간외 시세**`inquire-price`가 시간외 단일가를 반영하는지 첫 운영 대조.

View File

@@ -0,0 +1,19 @@
FROM python:3.12-slim-bookworm
ENV PYTHONUNBUFFERED=1
WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates tzdata \
&& rm -rf /var/lib/apt/lists/*
COPY trade-monitor/requirements.txt /app/
RUN pip install --no-cache-dir --timeout 600 --retries 5 -r requirements.txt
# 공통 heartbeat 모듈 (services/_shared) — main.py가 from _shared.heartbeat import
COPY _shared /app/_shared
COPY trade-monitor/. /app/
ENV PYTHONPATH=/app
EXPOSE 8000
CMD ["python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "1"]

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,99 @@
"""§6 조건 로직 (순수). ctx + params → firing 리스트."""
from __future__ import annotations
from indicators import sma, rsi_series, highest_high
def _fire(ctx: dict, kind: str, condition: str, price: float, detail: dict) -> dict:
return {
"ticker": ctx["ticker"], "kind": kind,
"condition": condition, "price": price, "detail": detail,
}
def evaluate_buy(ctx: dict, params: dict) -> list[dict]:
price = ctx["price"]
closes, highs, lows, vols = ctx["closes"], ctx["highs"], ctx["lows"], ctx["volumes"]
rsi_os = params.get("rsi_oversold", 30)
vol_mult = params.get("breakout_vol_mult", 1.5)
pullback = params.get("pullback_pct", 0.02)
firing: list[dict] = []
# buy_ma20_pullback — 정배열 + ma20 근접 저가 + 반등 복귀
ma20, ma50, ma200 = sma(closes, 20), sma(closes, 50), sma(closes, 200)
if ma20 and ma50 and ma200 and ma20 > ma50 > ma200 and len(lows) >= 3:
recent_low = min(lows[-3:])
if recent_low <= ma20 * (1 + pullback) and price > ma20:
firing.append(_fire(ctx, "buy", "buy_ma20_pullback", price, {
"ma20": round(ma20, 1), "ma50": round(ma50, 1),
"ma200": round(ma200, 1), "recent_low": recent_low,
}))
# buy_breakout — 직전 20봉 고점 돌파 + 거래량 배수
prior_high20 = highest_high(highs, 20)
avg_vol20 = sma(vols, 20)
if prior_high20 and avg_vol20 and price > prior_high20 \
and ctx["today_volume"] > vol_mult * avg_vol20:
firing.append(_fire(ctx, "buy", "buy_breakout", price, {
"prior_high_20": prior_high20,
"vol_mult": round(ctx["today_volume"] / avg_vol20, 2),
"avg_vol_20": round(avg_vol20, 0),
}))
# buy_rsi_bounce — RSI 과매도 후 반등 (무상태 재계산)
rs = rsi_series(closes, 14)
if len(rs) >= 3 and min(rs[-3:]) < rsi_os and rs[-1] > rsi_os and rs[-1] > rs[-2]:
firing.append(_fire(ctx, "buy", "buy_rsi_bounce", price, {
"rsi": round(rs[-1], 1), "rsi_prev": round(rs[-2], 1),
"rsi_oversold": rsi_os,
}))
return firing
def evaluate_sell(ctx: dict, params: dict) -> list[dict]:
price = ctx["price"]
avg = ctx.get("avg_price")
hh = ctx.get("holding_high")
closes, vols = ctx["closes"], ctx["volumes"]
stop = params.get("stop_pct", 0.08)
take = params.get("take_pct", 0.25)
trail = params.get("trailing_pct", 0.10)
firing: list[dict] = []
if avg:
pnl = (price - avg) / avg
if pnl <= -stop:
firing.append(_fire(ctx, "sell", "sell_stop_loss", price, {
"avg_price": avg, "pnl_pct": round(pnl, 4), "stop_pct": stop}))
if pnl >= take:
firing.append(_fire(ctx, "sell", "sell_take_profit", price, {
"avg_price": avg, "pnl_pct": round(pnl, 4), "take_pct": take}))
if hh and price <= hh * (1 - trail):
firing.append(_fire(ctx, "sell", "sell_trailing_stop", price, {
"holding_high": hh, "trailing_pct": trail,
"drawdown_pct": round((price - hh) / hh, 4)}))
ma50, ma200 = sma(closes, 50), sma(closes, 200)
if ma50 and price < ma50:
severity = "high" if (ma200 and price < ma200) else "normal"
firing.append(_fire(ctx, "sell", "sell_ma_break", price, {
"ma50": round(ma50, 1),
"ma200": round(ma200, 1) if ma200 else None,
"severity": severity}))
# sell_climax — holdings_intel 정합(stock/app/holdings_intel.py:109-118):
# 거래량 ≥ 20일평균 × climax_vol_x AND 종가 < 당일고가 × climax_close_pct (윗꼬리)
# 실시간이므로 day_high = 당일 세션 누적 고가(최신 1분봉 고가 아님).
climax_vol_x = params.get("climax_vol_x", ctx.get("climax_vol_mult", 3.0))
climax_close_pct = params.get("climax_close_pct", 0.97)
avg_vol20 = sma(vols, 20)
day_high = ctx.get("day_high")
if avg_vol20 and day_high and ctx["today_volume"] >= climax_vol_x * avg_vol20 \
and price < day_high * climax_close_pct:
firing.append(_fire(ctx, "sell", "sell_climax", price, {
"vol_mult": round(ctx["today_volume"] / avg_vol20, 2),
"day_high": day_high, "climax_close_pct": climax_close_pct}))
return firing

View File

@@ -0,0 +1,32 @@
"""Settings — 환경변수 로드. TM_ 접두사로 ai_trade와 분리."""
from __future__ import annotations
import os
from dataclasses import dataclass
@dataclass
class Settings:
nas_base_url: str
webai_api_key: str
redis_url: str
kis_app_key: str
kis_app_secret: str
kis_account: str
kis_is_virtual: bool
loop_interval: int
climax_vol_mult: float
def load_settings() -> Settings:
return Settings(
nas_base_url=os.getenv("NAS_BASE_URL", "http://192.168.45.54:18500"),
webai_api_key=os.getenv("WEBAI_API_KEY", ""),
redis_url=os.getenv("REDIS_URL", "redis://192.168.45.54:6379"),
kis_app_key=os.getenv("TM_KIS_APP_KEY", ""),
kis_app_secret=os.getenv("TM_KIS_APP_SECRET", ""),
kis_account=os.getenv("TM_KIS_ACCOUNT", ""),
kis_is_virtual=os.getenv("TM_KIS_IS_VIRTUAL", "0") == "1",
loop_interval=int(os.getenv("TM_LOOP_INTERVAL", "60")),
climax_vol_mult=float(os.getenv("TM_CLIMAX_VOL_MULT", "3.0")),
)

View File

@@ -0,0 +1,5 @@
"""services/ 루트를 sys.path에 추가 — from _shared.heartbeat import 가능하게."""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))

View File

@@ -0,0 +1,38 @@
"""순수 TA 지표 — sma / rsi_series / highest_high."""
from __future__ import annotations
def sma(values: list[float], period: int) -> float | None:
if period <= 0 or len(values) < period:
return None
return sum(values[-period:]) / period
def highest_high(highs: list[float], period: int) -> float | None:
if period <= 0 or len(highs) < period:
return None
return max(highs[-period:])
def rsi_series(closes: list[float], period: int = 14) -> list[float]:
"""Wilder RSI. 반환 리스트는 closes[period:]에 1:1 정렬. 부족하면 []."""
if len(closes) <= period:
return []
deltas = [closes[i] - closes[i - 1] for i in range(1, len(closes))]
gains = [d if d > 0 else 0.0 for d in deltas]
losses = [-d if d < 0 else 0.0 for d in deltas]
def _rsi(ag: float, al: float) -> float:
if al == 0:
return 100.0
rs = ag / al
return 100.0 - 100.0 / (1.0 + rs)
avg_gain = sum(gains[:period]) / period
avg_loss = sum(losses[:period]) / period
out = [_rsi(avg_gain, avg_loss)]
for i in range(period, len(deltas)):
avg_gain = (avg_gain * (period - 1) + gains[i]) / period
avg_loss = (avg_loss * (period - 1) + losses[i]) / period
out.append(_rsi(avg_gain, avg_loss))
return out

View File

@@ -0,0 +1,124 @@
"""KIS REST client — 자체 OAuth 토큰(TM_KIS_*) + quote + 일봉 + throttle."""
from __future__ import annotations
import asyncio
import logging
import time
from datetime import datetime, timedelta
from zoneinfo import ZoneInfo
import httpx
logger = logging.getLogger(__name__)
KST = ZoneInfo("Asia/Seoul")
_MAX_ATTEMPTS = 3
_THROTTLE_INTERVAL = 0.5 # 초당 2회
_TOKEN_MARGIN = 600 # 만료 10분 전 재발급
class KISClient:
def __init__(self, app_key, app_secret, account, is_virtual, timeout: float = 10.0):
self._app_key = app_key
self._app_secret = app_secret
self._account = account
self._base_url = (
"https://openapivts.koreainvestment.com:29443" if is_virtual
else "https://openapi.koreainvestment.com:9443"
)
self._client = httpx.AsyncClient(timeout=timeout)
self._token: str | None = None
self._token_exp: float = 0.0
self._last_throttle_at = 0.0
self._throttle_lock = asyncio.Lock()
self._token_lock = asyncio.Lock()
async def close(self) -> None:
await self._client.aclose()
async def _issue_token(self) -> str:
async with self._token_lock:
now = time.time()
if self._token and now < self._token_exp - _TOKEN_MARGIN:
return self._token
r = await self._client.post(
f"{self._base_url}/oauth2/tokenP",
json={"grant_type": "client_credentials",
"appkey": self._app_key, "appsecret": self._app_secret},
)
r.raise_for_status()
data = r.json()
self._token = data["access_token"]
self._token_exp = now + int(data.get("expires_in", 86400))
return self._token
async def _throttle(self) -> None:
async with self._throttle_lock:
elapsed = time.monotonic() - self._last_throttle_at
if elapsed < _THROTTLE_INTERVAL:
await asyncio.sleep(_THROTTLE_INTERVAL - elapsed)
self._last_throttle_at = time.monotonic()
async def _request(self, method: str, path: str, tr_id: str, **kwargs) -> dict:
token = await self._issue_token()
headers = {
"authorization": f"Bearer {token}",
"appkey": self._app_key, "appsecret": self._app_secret,
"tr_id": tr_id, "custtype": "P",
}
url = f"{self._base_url}{path}"
for attempt in range(_MAX_ATTEMPTS):
await self._throttle()
try:
resp = await self._client.request(method, url, headers=headers, **kwargs)
if resp.status_code == 429 and attempt < _MAX_ATTEMPTS - 1:
await asyncio.sleep(2 ** attempt)
continue
resp.raise_for_status()
return resp.json()
except httpx.TimeoutException:
if attempt < _MAX_ATTEMPTS - 1:
await asyncio.sleep(2 ** attempt)
continue
raise
raise RuntimeError("retry exhausted")
async def get_quote(self, ticker: str) -> dict:
raw = await self._request(
"GET", "/uapi/domestic-stock/v1/quotations/inquire-price",
tr_id="FHKST01010100",
params={"FID_COND_MRKT_DIV_CODE": "J", "FID_INPUT_ISCD": ticker},
)
o = raw.get("output", {})
return {
"price": int(o["stck_prpr"]),
"day_open": int(o["stck_oprc"]),
"day_high": int(o["stck_hgpr"]),
"today_volume": int(o["acml_vol"]),
"as_of": datetime.now(KST).isoformat(),
}
async def get_daily_ohlcv(self, ticker: str, days: int = 250) -> list[dict]:
today = datetime.now(KST).strftime("%Y%m%d")
start = (datetime.now(KST) - timedelta(days=days * 2)).strftime("%Y%m%d")
raw = await self._request(
"GET", "/uapi/domestic-stock/v1/quotations/inquire-daily-itemchartprice",
tr_id="FHKST03010100",
params={"FID_COND_MRKT_DIV_CODE": "J", "FID_INPUT_ISCD": ticker,
"FID_INPUT_DATE_1": start, "FID_INPUT_DATE_2": today,
"FID_PERIOD_DIV_CODE": "D", "FID_ORG_ADJ_PRC": "1"},
)
bars = []
for row in raw.get("output2", []):
try:
d = row["stck_bsop_date"]
bars.append({
"datetime": f"{d[:4]}-{d[4:6]}-{d[6:]}",
"open": int(row["stck_oprc"]), "high": int(row["stck_hgpr"]),
"low": int(row["stck_lwpr"]), "close": int(row["stck_clpr"]),
"volume": int(row["acml_vol"]),
})
except (KeyError, ValueError):
continue
bars.reverse()
return bars[-days:]

View File

@@ -0,0 +1,62 @@
"""trade-monitor FastAPI entry — lifespan(monitor_loop + heartbeat_loop) + /health."""
from __future__ import annotations
import asyncio
import logging
from contextlib import asynccontextmanager
import redis.asyncio as aioredis
from fastapi import FastAPI
import monitor
from config import load_settings
from kis_client import KISClient
from nas_client import NASClient
from _shared.heartbeat import heartbeat_loop, WorkerStats
logging.basicConfig(level=logging.INFO,
format="%(asctime)s %(name)s %(levelname)s %(message)s")
logger = logging.getLogger(__name__)
HEARTBEAT_INTERVAL = 15 # 60초 루프 > TTL 45초 → 독립 15초 발신으로 만료갭 해소
HEARTBEAT_TTL = 45
@asynccontextmanager
async def lifespan(app: FastAPI):
settings = load_settings()
nas = NASClient(settings.nas_base_url, settings.webai_api_key)
kis = KISClient(settings.kis_app_key, settings.kis_app_secret,
settings.kis_account, settings.kis_is_virtual)
state = monitor.MonitorState()
stats = WorkerStats()
redis = aioredis.from_url(settings.redis_url, decode_responses=False)
mon_task = asyncio.create_task(
monitor.monitor_loop(nas, kis, state, stats, settings))
hb_task = asyncio.create_task(heartbeat_loop(
redis, "trade-monitor", "trader", stats,
interval=HEARTBEAT_INTERVAL, ttl=HEARTBEAT_TTL,
state_fn=monitor.make_state_fn(state)))
logger.info("trade-monitor lifespan 시작")
try:
yield
finally:
for t in (mon_task, hb_task):
t.cancel()
try:
await t
except asyncio.CancelledError:
pass
await kis.close()
await nas.close()
await redis.aclose()
logger.info("trade-monitor lifespan 종료")
app = FastAPI(lifespan=lifespan)
@app.get("/health")
def health():
return {"ok": True, "service": "trade-monitor"}

View File

@@ -0,0 +1,114 @@
"""오케스트레이션 — monitor-set 조회 → 조건 평가 → report + heartbeat state."""
from __future__ import annotations
import asyncio
import logging
from datetime import datetime
from zoneinfo import ZoneInfo
from conditions import evaluate_buy, evaluate_sell
logger = logging.getLogger(__name__)
KST = ZoneInfo("Asia/Seoul")
class MonitorState:
"""monitor_loop가 갱신, heartbeat state_fn이 읽는 공유 상태."""
def __init__(self):
self.session_state = "idle" # market_open | market_closed | idle
self.last_alert_at: str | None = None
def filter_krx(targets: list[dict]) -> list[dict]:
"""6자리 숫자 티커(KRX)만. 알파벳 티커 skip."""
out = []
for t in targets:
tk = str(t.get("ticker", ""))
if tk.isdigit() and len(tk) == 6:
out.append(t)
return out
async def _build_ctx(kis, target: dict, settings) -> dict:
ticker = target["ticker"]
quote = await kis.get_quote(ticker)
daily = await kis.get_daily_ohlcv(ticker, 250)
return {
"ticker": ticker, "name": target.get("name", ""),
"price": quote["price"], "day_open": quote["day_open"],
"day_high": quote["day_high"],
"today_volume": quote["today_volume"],
"closes": [b["close"] for b in daily],
"highs": [b["high"] for b in daily],
"lows": [b["low"] for b in daily],
"volumes": [b["volume"] for b in daily],
"avg_price": target.get("avg_price"),
"qty": target.get("qty"),
"holding_high": target.get("holding_high"),
"climax_vol_mult": settings.climax_vol_mult,
}
async def run_cycle(nas, kis, state, stats, settings) -> None:
try:
ms = await nas.get_monitor_set()
except Exception:
logger.exception("monitor-set 조회 실패")
state.session_state = "idle"
stats.jobs_failed += 1
return
session = ms.get("session", "closed")
if session == "closed":
state.session_state = "market_closed"
return
buy_targets = filter_krx(ms.get("buy_targets", []))
sell_targets = filter_krx(ms.get("sell_targets", []))
buy_params = ms.get("buy_params", {})
exit_params = ms.get("exit_params", {})
firing: list[dict] = []
for t in buy_targets:
try:
firing += evaluate_buy(await _build_ctx(kis, t, settings), buy_params)
except Exception:
logger.exception("buy 평가 실패 %s", t.get("ticker"))
for t in sell_targets:
try:
firing += evaluate_sell(await _build_ctx(kis, t, settings), exit_params)
except Exception:
logger.exception("sell 평가 실패 %s", t.get("ticker"))
as_of = datetime.now(KST).isoformat(timespec="seconds")
if firing:
state.last_alert_at = as_of
logger.info("firing %d개: %s", len(firing),
[f"{f['ticker']}:{f['condition']}" for f in firing])
try:
await nas.post_report(as_of, firing) # 빈 배열도 전송(edge clear)
except Exception:
logger.exception("report 전송 실패")
state.session_state = "market_open"
stats.jobs_done += 1
stats.last_job_at = as_of
async def monitor_loop(nas, kis, state, stats, settings) -> None:
logger.info("trade-monitor loop 시작 interval=%ds", settings.loop_interval)
while True:
try:
await run_cycle(nas, kis, state, stats, settings)
except asyncio.CancelledError:
logger.info("monitor_loop cancelled")
raise
except Exception:
logger.exception("monitor_loop iteration 실패")
await asyncio.sleep(settings.loop_interval)
def make_state_fn(state):
async def state_fn(redis, stats):
return state.session_state, {"last_alert_at": state.last_alert_at}
return state_fn

View File

@@ -0,0 +1,48 @@
"""NAS stock 백엔드 trade-alert 계약 — X-WebAI-Key + retry."""
from __future__ import annotations
import asyncio
import logging
import httpx
logger = logging.getLogger(__name__)
_MAX_ATTEMPTS = 3
_RETRY_STATUSES = {429, 500, 502, 503, 504}
class NASClient:
def __init__(self, base_url: str, api_key: str, timeout: float = 10.0):
self._base_url = base_url.rstrip("/")
self._api_key = api_key
self._client = httpx.AsyncClient(timeout=timeout)
async def close(self) -> None:
await self._client.aclose()
async def get_monitor_set(self) -> dict:
return await self._request("GET", "/api/webai/trade-alert/monitor-set")
async def post_report(self, as_of: str, firing: list[dict]) -> dict:
return await self._request(
"POST", "/api/webai/trade-alert/report",
json={"as_of": as_of, "firing": firing})
async def _request(self, method: str, path: str, **kwargs) -> dict:
url = f"{self._base_url}{path}"
headers = {"X-WebAI-Key": self._api_key}
for attempt in range(_MAX_ATTEMPTS):
try:
resp = await self._client.request(method, url, headers=headers, **kwargs)
if resp.status_code in _RETRY_STATUSES and attempt < _MAX_ATTEMPTS - 1:
await asyncio.sleep(2 ** attempt)
continue
resp.raise_for_status()
return resp.json()
except httpx.TimeoutException:
if attempt < _MAX_ATTEMPTS - 1:
await asyncio.sleep(2 ** attempt)
continue
raise
raise RuntimeError("retry exhausted")

View File

@@ -0,0 +1,2 @@
[pytest]
asyncio_mode = auto

View File

@@ -0,0 +1,7 @@
fastapi==0.115.6
uvicorn[standard]==0.34.0
redis>=5.0
httpx>=0.27
pytest>=8.0
pytest-asyncio>=0.24
respx>=0.21

View File

View File

@@ -0,0 +1,66 @@
"""evaluate_buy — 3개 매수 조건 경계."""
from conditions import evaluate_buy
BUY_PARAMS = {"rsi_oversold": 30, "breakout_vol_mult": 1.5, "pullback_pct": 0.02}
def _ctx(**over):
base = dict(
ticker="005930", name="삼성전자", price=100.0, day_open=99.0,
today_volume=1000.0, closes=[], highs=[], lows=[], volumes=[],
avg_price=None, qty=None, holding_high=None, climax_vol_mult=3.0,
)
base.update(over)
return base
def _conditions(firing):
return {f["condition"] for f in firing}
def test_ma20_pullback_fires():
# 정배열(ma20>ma50>ma200), 최근 저가가 ma20 근처, price가 ma20 위로 반등
closes = [90.0] * 200 + [100.0] * 20 # ma20=100, ma50/ma200 낮음 → 정배열
lows = [90.0] * 217 + [100.5, 100.4, 100.3] # 최근 3봉 저가 ~ma20*(1.02)=102 이하
ctx = _ctx(price=101.0, closes=closes, highs=closes, lows=lows,
volumes=[1.0] * len(closes))
assert "buy_ma20_pullback" in _conditions(evaluate_buy(ctx, BUY_PARAMS))
def test_ma20_pullback_skips_when_not_aligned():
closes = [100.0] * 200 + [90.0] * 20 # 역배열
ctx = _ctx(price=91.0, closes=closes, highs=closes, lows=closes,
volumes=[1.0] * len(closes))
assert "buy_ma20_pullback" not in _conditions(evaluate_buy(ctx, BUY_PARAMS))
def test_breakout_fires():
closes = [50.0] * 25
highs = [60.0] * 25 # 직전 20봉 최고 60
vols = [100.0] * 25 # avg20=100
ctx = _ctx(price=61.0, today_volume=200.0, closes=closes, highs=highs,
lows=closes, volumes=vols) # 61>60, 200>1.5*100
assert "buy_breakout" in _conditions(evaluate_buy(ctx, BUY_PARAMS))
def test_breakout_skips_on_low_volume():
highs = [60.0] * 25
ctx = _ctx(price=61.0, today_volume=120.0, closes=[50.0] * 25, highs=highs,
lows=[50.0] * 25, volumes=[100.0] * 25) # 120 < 1.5*100=150
assert "buy_breakout" not in _conditions(evaluate_buy(ctx, BUY_PARAMS))
def test_rsi_bounce_fires():
# 14봉 급락으로 RSI<30 찍고 5봉 반등하여 30 위로 복귀
closes = [100.0]
for _ in range(14):
closes.append(closes[-1] * 0.97) # 하락 → RSI 저하
for _ in range(5):
closes.append(closes[-1] * 1.05) # 반등 → RSI 30 위로
ctx = _ctx(price=closes[-1], closes=closes, highs=closes, lows=closes,
volumes=[1.0] * len(closes))
assert "buy_rsi_bounce" in _conditions(evaluate_buy(ctx, BUY_PARAMS))
def test_empty_series_no_fire():
assert evaluate_buy(_ctx(), BUY_PARAMS) == []

View File

@@ -0,0 +1,89 @@
"""evaluate_sell — 5개 매도 조건 경계."""
from conditions import evaluate_sell
EXIT = {"stop_pct": 0.08, "take_pct": 0.25, "trailing_pct": 0.10,
"climax_vol_x": 3.0, "climax_close_pct": 0.97}
def _ctx(**over):
base = dict(
ticker="000660", name="SK하이닉스", price=100.0, day_open=100.0,
day_high=100.0, today_volume=100.0, closes=[100.0] * 60,
highs=[100.0] * 60, lows=[100.0] * 60, volumes=[100.0] * 60,
avg_price=100.0, qty=10, holding_high=100.0, climax_vol_mult=3.0,
)
base.update(over)
return base
def _c(firing):
return {f["condition"] for f in firing}
def test_stop_loss_fires():
ctx = _ctx(price=90.0, avg_price=100.0) # -10% <= -8%
assert "sell_stop_loss" in _c(evaluate_sell(ctx, EXIT))
def test_stop_loss_skips_above_threshold():
ctx = _ctx(price=95.0, avg_price=100.0) # -5% > -8%
assert "sell_stop_loss" not in _c(evaluate_sell(ctx, EXIT))
def test_take_profit_fires():
ctx = _ctx(price=130.0, avg_price=100.0) # +30% >= 25%
assert "sell_take_profit" in _c(evaluate_sell(ctx, EXIT))
def test_trailing_stop_fires():
ctx = _ctx(price=89.0, holding_high=100.0) # 89 <= 100*0.9=90
assert "sell_trailing_stop" in _c(evaluate_sell(ctx, EXIT))
def test_ma_break_severity_high():
# price가 ma50/ma200 아래 → severity high (ma200 계산 위해 200봉 필요)
closes = [200.0] * 200
ctx = _ctx(price=100.0, closes=closes, avg_price=100.0, holding_high=100.0)
firing = evaluate_sell(ctx, EXIT)
mb = [f for f in firing if f["condition"] == "sell_ma_break"]
assert mb and mb[0]["detail"]["severity"] == "high"
def test_climax_fires():
# holdings_intel 정합: 거래량 3배 이상 + 종가 < 당일고가×0.97 (윗꼬리)
ctx = _ctx(price=96.0, day_high=100.0, today_volume=400.0,
volumes=[100.0] * 60) # 400>=3*100, 96 < 100*0.97=97
assert "sell_climax" in _c(evaluate_sell(ctx, EXIT))
def test_climax_skips_when_not_reversal():
# 종가가 당일고가의 97% 이상 → 윗꼬리 아님
ctx = _ctx(price=99.0, day_high=100.0, today_volume=400.0,
volumes=[100.0] * 60) # 99 >= 100*0.97=97 → 반전 아님
assert "sell_climax" not in _c(evaluate_sell(ctx, EXIT))
def test_climax_uses_exit_params_vol_x():
# exit_params.climax_vol_x=5.0 → 400 < 5*100=500 → 미발화
exit5 = {**EXIT, "climax_vol_x": 5.0}
ctx = _ctx(price=96.0, day_high=100.0, today_volume=400.0,
volumes=[100.0] * 60)
assert "sell_climax" not in _c(evaluate_sell(ctx, exit5))
def test_climax_uses_exit_params_close_pct():
# climax_close_pct=0.90 → 임계 90, price=95 → 95<90? No → 미발화
exit90 = {**EXIT, "climax_close_pct": 0.90}
ctx = _ctx(price=95.0, day_high=100.0, today_volume=400.0,
volumes=[100.0] * 60)
assert "sell_climax" not in _c(evaluate_sell(ctx, exit90))
# 기본 0.97이면 95 < 97 → 발화
assert "sell_climax" in _c(evaluate_sell(ctx, EXIT))
def test_no_avg_no_pnl_conditions():
# avg_price None(보유정보 없음) → stop/take 미발화
ctx = _ctx(price=50.0, avg_price=None, holding_high=None,
closes=[100.0] * 60)
conds = _c(evaluate_sell(ctx, EXIT))
assert "sell_stop_loss" not in conds and "sell_take_profit" not in conds

View File

@@ -0,0 +1,27 @@
"""Settings env 로드 — 기본값 + override."""
from config import load_settings
def test_defaults(monkeypatch):
for k in ("NAS_BASE_URL", "WEBAI_API_KEY", "REDIS_URL", "TM_KIS_APP_KEY",
"TM_KIS_APP_SECRET", "TM_KIS_ACCOUNT", "TM_KIS_IS_VIRTUAL",
"TM_LOOP_INTERVAL", "TM_CLIMAX_VOL_MULT"):
monkeypatch.delenv(k, raising=False)
s = load_settings()
assert s.nas_base_url == "http://192.168.45.54:18500"
assert s.redis_url == "redis://192.168.45.54:6379"
assert s.kis_is_virtual is False
assert s.loop_interval == 60
assert s.climax_vol_mult == 3.0
def test_override(monkeypatch):
monkeypatch.setenv("TM_KIS_IS_VIRTUAL", "1")
monkeypatch.setenv("TM_LOOP_INTERVAL", "30")
monkeypatch.setenv("TM_CLIMAX_VOL_MULT", "2.5")
monkeypatch.setenv("WEBAI_API_KEY", "secret")
s = load_settings()
assert s.kis_is_virtual is True
assert s.loop_interval == 30
assert s.climax_vol_mult == 2.5
assert s.webai_api_key == "secret"

View File

@@ -0,0 +1,6 @@
"""/health — 라우트 핸들러 직접 검증."""
from main import health
def test_health():
assert health() == {"ok": True, "service": "trade-monitor"}

View File

@@ -0,0 +1,39 @@
"""indicators — 순수 수치 검증."""
from indicators import sma, rsi_series, highest_high
def test_sma_basic():
assert sma([1, 2, 3, 4, 5], 5) == 3.0
assert sma([1, 2, 3, 4, 5], 2) == 4.5
def test_sma_insufficient():
assert sma([1, 2], 5) is None
assert sma([], 3) is None
def test_highest_high():
assert highest_high([1, 9, 3, 4], 3) == 9
assert highest_high([1, 2, 3], 3) == 3
assert highest_high([1, 2], 3) is None
def test_rsi_all_gains_is_100():
# 단조 증가 → 손실 0 → RSI 100
closes = [float(i) for i in range(1, 20)]
rs = rsi_series(closes, 14)
assert rs, "series should not be empty"
assert rs[-1] == 100.0
def test_rsi_insufficient():
assert rsi_series([1, 2, 3], 14) == []
def test_rsi_known_range():
# 등락 섞인 시계열 → RSI는 0~100 사이
closes = [10, 11, 10.5, 11.5, 11, 12, 11.8, 12.5, 12, 13,
12.7, 13.2, 12.9, 13.5, 13.1, 13.8]
rs = rsi_series(closes, 14)
assert len(rs) == len(closes) - 14
assert all(0.0 <= v <= 100.0 for v in rs)

View File

@@ -0,0 +1,56 @@
"""KISClient — 토큰 발급/캐시 + quote/daily 파싱 (respx)."""
import httpx
import respx
from kis_client import KISClient
BASE = "https://openapi.koreainvestment.com:9443"
def _client():
return KISClient("APPKEY", "APPSECRET", "12345678-01", is_virtual=False)
@respx.mock
async def test_issue_token_cached():
route = respx.post(f"{BASE}/oauth2/tokenP").mock(
return_value=httpx.Response(200, json={"access_token": "TKN", "expires_in": 86400}))
c = _client()
t1 = await c._issue_token()
t2 = await c._issue_token()
assert t1 == "TKN" and t2 == "TKN"
assert route.call_count == 1 # 캐시 → 1회만 발급
await c.close()
@respx.mock
async def test_get_quote_parses():
respx.post(f"{BASE}/oauth2/tokenP").mock(
return_value=httpx.Response(200, json={"access_token": "TKN", "expires_in": 86400}))
respx.get(f"{BASE}/uapi/domestic-stock/v1/quotations/inquire-price").mock(
return_value=httpx.Response(200, json={"output": {
"stck_prpr": "71500", "stck_oprc": "71000", "stck_hgpr": "72000",
"acml_vol": "1234567"}}))
c = _client()
q = await c.get_quote("005930")
assert q["price"] == 71500 and q["day_open"] == 71000 and q["today_volume"] == 1234567
assert q["day_high"] == 72000
await c.close()
@respx.mock
async def test_get_daily_ascending():
respx.post(f"{BASE}/oauth2/tokenP").mock(
return_value=httpx.Response(200, json={"access_token": "TKN", "expires_in": 86400}))
# KIS는 내림차순 반환 → 오름차순으로 뒤집혀야 함
respx.get(f"{BASE}/uapi/domestic-stock/v1/quotations/inquire-daily-itemchartprice").mock(
return_value=httpx.Response(200, json={"output2": [
{"stck_bsop_date": "20260702", "stck_oprc": "100", "stck_hgpr": "110",
"stck_lwpr": "90", "stck_clpr": "105", "acml_vol": "5"},
{"stck_bsop_date": "20260701", "stck_oprc": "95", "stck_hgpr": "102",
"stck_lwpr": "94", "stck_clpr": "100", "acml_vol": "4"}]}))
c = _client()
bars = await c.get_daily_ohlcv("005930", days=250)
assert bars[0]["datetime"] == "2026-07-01"
assert bars[-1]["close"] == 105
await c.close()

View File

@@ -0,0 +1,95 @@
"""monitor.run_cycle — 게이트/필터/조립/격리."""
from monitor import MonitorState, filter_krx, run_cycle
from config import load_settings
from _shared.heartbeat import WorkerStats
def test_filter_krx_keeps_only_numeric6():
targets = [{"ticker": "005930"}, {"ticker": "AAPL"}, {"ticker": "00660"},
{"ticker": "000660"}, {"ticker": "0059301"}]
kept = {t["ticker"] for t in filter_krx(targets)}
assert kept == {"005930", "000660"}
class _FakeNAS:
def __init__(self, ms):
self._ms = ms
self.reported = None
async def get_monitor_set(self):
return self._ms
async def post_report(self, as_of, firing):
self.reported = {"as_of": as_of, "firing": firing}
return {"new_alerts": len(firing), "cleared": 0}
class _FakeKIS:
def __init__(self, price=100, fail_on=None):
self._price = price
self._fail_on = fail_on or set()
async def get_quote(self, ticker):
if ticker in self._fail_on:
raise RuntimeError("KIS down")
return {"price": self._price, "day_open": 99, "day_high": 100,
"today_volume": 1000, "as_of": "x"}
async def get_daily_ohlcv(self, ticker, days=250):
# 정배열 + 저가 근접 → ma20_pullback 발화 유도
return [{"open": 90, "high": 90, "low": 90, "close": 90, "volume": 1}] * 200 \
+ [{"open": 100, "high": 100, "low": 100, "close": 100, "volume": 1}] * 20
async def test_closed_session_skips_kis():
nas = _FakeNAS({"session": "closed"})
state, stats = MonitorState(), WorkerStats()
await run_cycle(nas, _FakeKIS(), state, stats, load_settings())
assert state.session_state == "market_closed"
assert nas.reported is None # report도 안 함
async def test_non_krx_skipped_and_report_sent():
nas = _FakeNAS({"session": "regular",
"buy_targets": [{"ticker": "AAPL", "name": "Apple"}],
"sell_targets": [], "buy_params": {}, "exit_params": {}})
state, stats = MonitorState(), WorkerStats()
await run_cycle(nas, _FakeKIS(), state, stats, load_settings())
assert state.session_state == "market_open"
assert nas.reported is not None
assert nas.reported["firing"] == [] # 알파벳 티커 skip → 빈 발화
async def test_firing_assembled_and_last_alert_set():
nas = _FakeNAS({"session": "regular",
"buy_targets": [{"ticker": "005930", "name": "삼성전자"}],
"sell_targets": [], "buy_params": {"pullback_pct": 0.02},
"exit_params": {}})
state, stats = MonitorState(), WorkerStats()
await run_cycle(nas, _FakeKIS(price=101), state, stats, load_settings())
conds = {f["condition"] for f in nas.reported["firing"]}
assert "buy_ma20_pullback" in conds
assert state.last_alert_at is not None
async def test_per_ticker_failure_isolated():
nas = _FakeNAS({"session": "regular",
"buy_targets": [{"ticker": "005930"}, {"ticker": "000660"}],
"sell_targets": [], "buy_params": {}, "exit_params": {}})
state, stats = MonitorState(), WorkerStats()
# 005930은 실패, 000660은 성공 → 루프가 죽지 않고 report 전송
await run_cycle(nas, _FakeKIS(fail_on={"005930"}), state, stats, load_settings())
assert nas.reported is not None
assert state.session_state == "market_open"
async def test_monitor_set_failure_sets_idle():
class _BadNAS(_FakeNAS):
async def get_monitor_set(self):
raise RuntimeError("NAS down")
nas = _BadNAS({})
state, stats = MonitorState(), WorkerStats()
await run_cycle(nas, _FakeKIS(), state, stats, load_settings())
assert state.session_state == "idle"
assert nas.reported is None

View File

@@ -0,0 +1,39 @@
"""NASClient — monitor-set/report + X-WebAI-Key (respx)."""
import json as _json
import httpx
import respx
from nas_client import NASClient
BASE = "http://nas.test"
@respx.mock
async def test_get_monitor_set_sends_key():
route = respx.get(f"{BASE}/api/webai/trade-alert/monitor-set").mock(
return_value=httpx.Response(200, json={"session": "regular", "buy_targets": []}))
c = NASClient(BASE, "KEY")
ms = await c.get_monitor_set()
assert ms["session"] == "regular"
assert route.calls.last.request.headers["X-WebAI-Key"] == "KEY"
await c.close()
@respx.mock
async def test_post_report_payload():
captured = {}
def _resp(request):
captured.update(_json.loads(request.content))
return httpx.Response(200, json={"new_alerts": 1, "cleared": 0})
respx.post(f"{BASE}/api/webai/trade-alert/report").mock(side_effect=_resp)
c = NASClient(BASE, "KEY")
firing = [{"ticker": "005930", "kind": "buy", "condition": "buy_breakout",
"price": 71500, "detail": {}}]
out = await c.post_report("2026-07-02T09:01:00+09:00", firing)
assert out["new_alerts"] == 1
assert captured["as_of"] == "2026-07-02T09:01:00+09:00"
assert captured["firing"] == firing
await c.close()

View File

@@ -3,11 +3,14 @@ from __future__ import annotations
import asyncio import asyncio
import logging import logging
import os
from contextlib import asynccontextmanager from contextlib import asynccontextmanager
import redis.asyncio as aioredis
from fastapi import FastAPI from fastapi import FastAPI
import worker import worker
from _shared.heartbeat import heartbeat_loop
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(name)s %(levelname)s %(message)s") logging.basicConfig(level=logging.INFO, format="%(asctime)s %(name)s %(levelname)s %(message)s")
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -16,15 +19,19 @@ logger = logging.getLogger(__name__)
@asynccontextmanager @asynccontextmanager
async def lifespan(app: FastAPI): async def lifespan(app: FastAPI):
worker_task = asyncio.create_task(worker.worker_loop()) worker_task = asyncio.create_task(worker.worker_loop())
hb_redis = aioredis.from_url(os.getenv("REDIS_URL", "redis://192.168.45.54:6379"), decode_responses=False)
hb_task = asyncio.create_task(heartbeat_loop(hb_redis, "video-render", "render", worker.stats))
logger.info("video-render lifespan 시작") logger.info("video-render lifespan 시작")
try: try:
yield yield
finally: finally:
worker_task.cancel() for t in (worker_task, hb_task):
try: t.cancel()
await worker_task try:
except asyncio.CancelledError: await t
pass except asyncio.CancelledError:
pass
await hb_redis.aclose()
logger.info("video-render lifespan 종료") logger.info("video-render lifespan 종료")

View File

@@ -94,3 +94,25 @@ async def test_poll_once_returns_false_on_timeout(monkeypatch):
assert handled is False assert handled is False
fake_queue.ack.assert_not_awaited() fake_queue.ack.assert_not_awaited()
fake_queue.fail.assert_not_awaited() fake_queue.fail.assert_not_awaited()
# ----- heartbeat stats 카운터 -----
class _OneJobQueue:
def __init__(self): self.acked = False
async def dequeue(self, timeout=5):
if self.acked: return None
return ({"job_type": "sora_generation", "task_id": "t1", "params": {}}, b"raw")
async def ack(self, raw): self.acked = True
async def fail(self, raw, payload): pass
@pytest.mark.asyncio
async def test_poll_once_increments_jobs_done(monkeypatch):
worker.stats.jobs_done = 0
monkeypatch.setattr(worker, "run_sora_generation", lambda task_id, params: None)
handled = await worker.poll_once(_OneJobQueue())
assert handled is True
assert worker.stats.jobs_done == 1
assert worker.stats.busy is False
assert worker.stats.last_job_at is not None

View File

@@ -19,6 +19,7 @@ from providers.veo import run_veo_generation
from providers.kling import run_kling_generation from providers.kling import run_kling_generation
from providers.seedance import run_seedance_generation from providers.seedance import run_seedance_generation
from _shared.reliable_queue import ReliableQueue from _shared.reliable_queue import ReliableQueue
from _shared.heartbeat import WorkerStats, utc_now_iso
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -26,6 +27,8 @@ REDIS_URL = os.getenv("REDIS_URL", "redis://192.168.45.54:6379")
QUEUE_KEY = "queue:video-render" QUEUE_KEY = "queue:video-render"
PAUSED_KEY = "queue:paused" PAUSED_KEY = "queue:paused"
stats = WorkerStats()
# string names so `unittest.mock.patch` on `worker.<name>` is correctly intercepted # string names so `unittest.mock.patch` on `worker.<name>` is correctly intercepted
_DISPATCH_TABLE = { _DISPATCH_TABLE = {
"sora_generation": "run_sora_generation", "sora_generation": "run_sora_generation",
@@ -60,14 +63,21 @@ async def poll_once(queue: ReliableQueue) -> bool:
if result is None: if result is None:
return False return False
payload, raw = result payload, raw = result
stats.busy = True
try: try:
await asyncio.to_thread(_dispatch, payload) await asyncio.to_thread(_dispatch, payload)
except Exception: except Exception:
logger.exception("dispatch unhandled exception task_id=%s", logger.exception("dispatch unhandled exception task_id=%s",
payload.get("task_id")) payload.get("task_id"))
await queue.fail(raw, payload) await queue.fail(raw, payload)
stats.jobs_failed += 1
stats.last_job_at = utc_now_iso()
stats.busy = False
return True return True
await queue.ack(raw) await queue.ack(raw)
stats.jobs_done += 1
stats.last_job_at = utc_now_iso()
stats.busy = False
return True return True