diff --git a/.cursor/rules/platform.mdc b/.cursor/rules/platform.mdc new file mode 100644 index 0000000..3ca205c --- /dev/null +++ b/.cursor/rules/platform.mdc @@ -0,0 +1,49 @@ +--- +description: MapMil platform architecture, boundaries, and agent workflow +alwaysApply: true +--- + +# MapMil Platform + +Monorepo: **ЦА** (analytics) + **ЦП** (parsing) + **contracts** + **ПИ** (distribution API). + +## Data flow + +``` +CA admin → Redis cp:jobs:{family} → CP adapter → POST /internal/ingest → CA DB → map / ПИ +``` + +| Center | Path | Role | +|--------|------|------| +| ЦА API | `centers/analytics/api/` | FastAPI, ingest, scheduler, admin | +| ЦА UI | `centers/analytics/frontend/` | Vue + Leaflet admin | +| ЦП | `centers/parsing/workers/` | Adapter registry, batch + Telegram listener | +| Contracts | `contracts/` | Shared schemas — change first | + +## Hard boundaries + +- ЦП **never** touches CA database — only `POST /internal/ingest` and `PATCH /internal/jobs/{id}` +- `source_type` events must match adapter type; stable `source_url` required (CA dedup) +- `source_config` validated via `contracts/sources.py` on admin CRUD +- Telegram session: `data/telegram.session` — local only, **never commit** +- `.env` and secrets — **never commit** + +## CP adapter model + +Each `source_type` implements `SourceAdapter.run(job_id, source_config, *, ctx) -> tuple[list[dict], str | None]`. + +Output dicts are `IngestEventItem`-shaped. Routing: `contracts/queues.py` (`SOURCE_FAMILY` → `cp:jobs:{telegram|web|nlp}`). + +Worker images: `cp-workers` (telegram), `cp-workers-web` (crawl4ai), `cp-workers-nlp` (viina). + +## Agent workflow + +1. **Contracts first** if data shape or routing changes +2. **One zone per task** — do not mix contracts + UI + infra in one unfocused change +3. **Follow existing adapters** — copy `viina.py` or `telegram.py`, not greenfield +4. **Do not touch** Telegram listener unless explicitly asked +5. **Verify**: `docker compose config`, then targeted service rebuild + +Docs: `README.md`, `centers/parsing/ARCHITECTURE.md`, `AGENTS.md`. + +For new `source_type`: use skill `add-parser-adapter`. diff --git a/.cursor/skills/add-parser-adapter/SKILL.md b/.cursor/skills/add-parser-adapter/SKILL.md new file mode 100644 index 0000000..58898f4 --- /dev/null +++ b/.cursor/skills/add-parser-adapter/SKILL.md @@ -0,0 +1,142 @@ +--- +name: add-parser-adapter +description: >- + Adds a new CP source_type end-to-end in MapMil: contracts schema and queue + family, SourceAdapter implementation, registry, optional Docker worker image, + and ParsersView.vue form fields. Use when adding a parser source, adapter, + source_type, or new ingest channel to the parsing center. +--- + +# Add Parser Adapter + +End-to-end checklist for a new `source_type` in ЦП. Work in order; do not skip contracts. + +## Inputs to clarify + +- `source_type` slug (lowercase, e.g. `rss`) +- Worker **family** (`telegram` | `web` | `nlp`) — pick by deps: + - `telegram` — Telethon, shared session + - `web` — browser/HTTP heavy (Crawl4AI, Playwright) + - `nlp` — lightweight HTTP + text (VIINA-style) +- `source_config` fields for admin UI +- Whether existing worker image can host it or needs new `requirements-*.txt` service + +## Checklist + +### 1. Contracts + +**`contracts/sources.py`** + +- Add `{Name}SourceConfig(BaseModel)` with validators +- Register in `CONFIG_MODELS["{source_type}"]` + +**`contracts/queues.py`** + +- Add `SOURCE_FAMILY["{source_type}"] = "{family}"` + +No CA code changes needed for routing — `enqueue_job` uses `queue_key_for_source`. + +### 2. Adapter + +**`centers/parsing/workers/workers/adapters/{name}.py`** + +Copy pattern from `viina.py` (text/HTTP) or `telegram.py` (Telethon): + +```python +class MyAdapter: + source_type = "{source_type}" + + async def run(self, job_id, source_config, *, ctx: WorkerContext) -> tuple[list[dict], str | None]: + cfg = MySourceConfig.model_validate(source_config or {}) + # fetch / parse → list of IngestEventItem-shaped dicts + return events, None # or ([], "error message") +``` + +Rules: +- Validate config with Pydantic model from `contracts/sources` +- Every event: stable `source_url`, `source_type="{source_type}"` +- Optional fields: `title`, `description`, `locality`, `latitude`, `longitude`, `event_date`, `region`, `topic`, `tags`, `metadata` +- Return `([], error)` on fatal config errors; partial fetch errors can go in metadata + +**`workers/adapters/registry.py`** + +- Add lazy factory in `_factories()` +- Factory name key must match `source_type` + +### 3. Dependencies & Docker + +If adapter fits an existing family, only update `ENABLED_ADAPTERS` in compose (usually not needed — one adapter per service). + +If new deps are heavy or conflict: + +1. `centers/parsing/workers/requirements-{family}.txt` (or new file) +2. New or updated service in `docker-compose.yml`: + +```yaml +cp-workers-{family}: + build: + context: . + dockerfile: centers/parsing/workers/Dockerfile + args: + REQUIREMENTS_FILE: requirements-{family}.txt + INSTALL_PLAYWRIGHT: "0" # "1" only for browser adapters + environment: + ENABLED_ADAPTERS: {source_type} + WORKER_FAMILIES: {family} + CA_API_URL: http://ca-api:8000 + REDIS_URL: redis://redis:6379/0 + INTERNAL_TOKEN: dev-internal-token + depends_on: [redis, ca-api] +``` + +### 4. Admin UI + +**`centers/analytics/frontend/src/views/ParsersView.vue`** + +- Add option to source type `