Add Cursor agent guidance for MapMil platform development.
Introduce AGENTS.md, platform rule, and add-parser-adapter skill for contracts-first adapter workflow. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Cursor
@@ -0,0 +1,49 @@
|
||||
---
|
||||
description: MapMil platform architecture, boundaries, and agent workflow
|
||||
alwaysApply: true
|
||||
---
|
||||
|
||||
# MapMil Platform
|
||||
|
||||
Monorepo: **ЦА** (analytics) + **ЦП** (parsing) + **contracts** + **ПИ** (distribution API).
|
||||
|
||||
## Data flow
|
||||
|
||||
```
|
||||
CA admin → Redis cp:jobs:{family} → CP adapter → POST /internal/ingest → CA DB → map / ПИ
|
||||
```
|
||||
|
||||
| Center | Path | Role |
|
||||
|--------|------|------|
|
||||
| ЦА API | `centers/analytics/api/` | FastAPI, ingest, scheduler, admin |
|
||||
| ЦА UI | `centers/analytics/frontend/` | Vue + Leaflet admin |
|
||||
| ЦП | `centers/parsing/workers/` | Adapter registry, batch + Telegram listener |
|
||||
| Contracts | `contracts/` | Shared schemas — change first |
|
||||
|
||||
## Hard boundaries
|
||||
|
||||
- ЦП **never** touches CA database — only `POST /internal/ingest` and `PATCH /internal/jobs/{id}`
|
||||
- `source_type` events must match adapter type; stable `source_url` required (CA dedup)
|
||||
- `source_config` validated via `contracts/sources.py` on admin CRUD
|
||||
- Telegram session: `data/telegram.session` — local only, **never commit**
|
||||
- `.env` and secrets — **never commit**
|
||||
|
||||
## CP adapter model
|
||||
|
||||
Each `source_type` implements `SourceAdapter.run(job_id, source_config, *, ctx) -> tuple[list[dict], str | None]`.
|
||||
|
||||
Output dicts are `IngestEventItem`-shaped. Routing: `contracts/queues.py` (`SOURCE_FAMILY` → `cp:jobs:{telegram|web|nlp}`).
|
||||
|
||||
Worker images: `cp-workers` (telegram), `cp-workers-web` (crawl4ai), `cp-workers-nlp` (viina).
|
||||
|
||||
## Agent workflow
|
||||
|
||||
1. **Contracts first** if data shape or routing changes
|
||||
2. **One zone per task** — do not mix contracts + UI + infra in one unfocused change
|
||||
3. **Follow existing adapters** — copy `viina.py` or `telegram.py`, not greenfield
|
||||
4. **Do not touch** Telegram listener unless explicitly asked
|
||||
5. **Verify**: `docker compose config`, then targeted service rebuild
|
||||
|
||||
Docs: `README.md`, `centers/parsing/ARCHITECTURE.md`, `AGENTS.md`.
|
||||
|
||||
For new `source_type`: use skill `add-parser-adapter`.
|
||||
@@ -0,0 +1,142 @@
|
||||
---
|
||||
name: add-parser-adapter
|
||||
description: >-
|
||||
Adds a new CP source_type end-to-end in MapMil: contracts schema and queue
|
||||
family, SourceAdapter implementation, registry, optional Docker worker image,
|
||||
and ParsersView.vue form fields. Use when adding a parser source, adapter,
|
||||
source_type, or new ingest channel to the parsing center.
|
||||
---
|
||||
|
||||
# Add Parser Adapter
|
||||
|
||||
End-to-end checklist for a new `source_type` in ЦП. Work in order; do not skip contracts.
|
||||
|
||||
## Inputs to clarify
|
||||
|
||||
- `source_type` slug (lowercase, e.g. `rss`)
|
||||
- Worker **family** (`telegram` | `web` | `nlp`) — pick by deps:
|
||||
- `telegram` — Telethon, shared session
|
||||
- `web` — browser/HTTP heavy (Crawl4AI, Playwright)
|
||||
- `nlp` — lightweight HTTP + text (VIINA-style)
|
||||
- `source_config` fields for admin UI
|
||||
- Whether existing worker image can host it or needs new `requirements-*.txt` service
|
||||
|
||||
## Checklist
|
||||
|
||||
### 1. Contracts
|
||||
|
||||
**`contracts/sources.py`**
|
||||
|
||||
- Add `{Name}SourceConfig(BaseModel)` with validators
|
||||
- Register in `CONFIG_MODELS["{source_type}"]`
|
||||
|
||||
**`contracts/queues.py`**
|
||||
|
||||
- Add `SOURCE_FAMILY["{source_type}"] = "{family}"`
|
||||
|
||||
No CA code changes needed for routing — `enqueue_job` uses `queue_key_for_source`.
|
||||
|
||||
### 2. Adapter
|
||||
|
||||
**`centers/parsing/workers/workers/adapters/{name}.py`**
|
||||
|
||||
Copy pattern from `viina.py` (text/HTTP) or `telegram.py` (Telethon):
|
||||
|
||||
```python
|
||||
class MyAdapter:
|
||||
source_type = "{source_type}"
|
||||
|
||||
async def run(self, job_id, source_config, *, ctx: WorkerContext) -> tuple[list[dict], str | None]:
|
||||
cfg = MySourceConfig.model_validate(source_config or {})
|
||||
# fetch / parse → list of IngestEventItem-shaped dicts
|
||||
return events, None # or ([], "error message")
|
||||
```
|
||||
|
||||
Rules:
|
||||
- Validate config with Pydantic model from `contracts/sources`
|
||||
- Every event: stable `source_url`, `source_type="{source_type}"`
|
||||
- Optional fields: `title`, `description`, `locality`, `latitude`, `longitude`, `event_date`, `region`, `topic`, `tags`, `metadata`
|
||||
- Return `([], error)` on fatal config errors; partial fetch errors can go in metadata
|
||||
|
||||
**`workers/adapters/registry.py`**
|
||||
|
||||
- Add lazy factory in `_factories()`
|
||||
- Factory name key must match `source_type`
|
||||
|
||||
### 3. Dependencies & Docker
|
||||
|
||||
If adapter fits an existing family, only update `ENABLED_ADAPTERS` in compose (usually not needed — one adapter per service).
|
||||
|
||||
If new deps are heavy or conflict:
|
||||
|
||||
1. `centers/parsing/workers/requirements-{family}.txt` (or new file)
|
||||
2. New or updated service in `docker-compose.yml`:
|
||||
|
||||
```yaml
|
||||
cp-workers-{family}:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: centers/parsing/workers/Dockerfile
|
||||
args:
|
||||
REQUIREMENTS_FILE: requirements-{family}.txt
|
||||
INSTALL_PLAYWRIGHT: "0" # "1" only for browser adapters
|
||||
environment:
|
||||
ENABLED_ADAPTERS: {source_type}
|
||||
WORKER_FAMILIES: {family}
|
||||
CA_API_URL: http://ca-api:8000
|
||||
REDIS_URL: redis://redis:6379/0
|
||||
INTERNAL_TOKEN: dev-internal-token
|
||||
depends_on: [redis, ca-api]
|
||||
```
|
||||
|
||||
### 4. Admin UI
|
||||
|
||||
**`centers/analytics/frontend/src/views/ParsersView.vue`**
|
||||
|
||||
- Add option to source type `<select>`
|
||||
- Extend form state + `buildSourceConfig()` for new fields
|
||||
- Add create/edit `<template>` blocks (mirror `crawl4ai` or `viina`)
|
||||
- Extend validation in submit handler
|
||||
|
||||
TypeScript unions: update `"telegram" | "crawl4ai" | "viina"` wherever used.
|
||||
|
||||
No CA router changes if `source_type` is registered in `CONFIG_MODELS` — `_validated_source_config` picks it up automatically.
|
||||
|
||||
### 5. Documentation
|
||||
|
||||
- Add row to adapter table in `centers/parsing/ARCHITECTURE.md`
|
||||
- One-line mention in `README.md` if user-facing
|
||||
|
||||
### 6. Verify
|
||||
|
||||
```bash
|
||||
docker compose config
|
||||
docker compose up --build cp-workers-{family} # or relevant service
|
||||
curl -X POST http://localhost:8080/admin/jobs \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"source_type":"{source_type}","source_config":{...},"interval_seconds":3600}'
|
||||
```
|
||||
|
||||
Check CA logs for ingest and `/admin/events` for new rows.
|
||||
|
||||
## Reference files
|
||||
|
||||
| Concern | File |
|
||||
|---------|------|
|
||||
| Config schema | `contracts/sources.py` |
|
||||
| Queue routing | `contracts/queues.py` |
|
||||
| Ingest shape | `contracts/ingest.py` |
|
||||
| Adapter protocol | `workers/adapters/base.py` |
|
||||
| Simple adapter | `workers/adapters/viina.py` |
|
||||
| LLM adapter | `workers/adapters/crawl4ai_adapter.py` |
|
||||
| Telegram + listener | `workers/adapters/telegram.py` |
|
||||
| Enqueue | `centers/analytics/api/app/services/jobs.py` |
|
||||
| Admin validation | `centers/analytics/api/app/routers/admin.py` |
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- Implementing adapter before `CONFIG_MODELS` entry
|
||||
- Hardcoding Redis queue names — use `queue_key_for_source`
|
||||
- Skipping `source_url` stability (breaks dedup)
|
||||
- Importing heavy deps at module top in `registry.py` — use lazy factories
|
||||
- Editing Telegram listener when adding unrelated sources
|
||||
Reference in New Issue
Block a user