Add Cursor agent guidance for MapMil platform development.
Introduce AGENTS.md, platform rule, and add-parser-adapter skill for contracts-first adapter workflow. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Cursor
@@ -0,0 +1,49 @@
|
||||
---
|
||||
description: MapMil platform architecture, boundaries, and agent workflow
|
||||
alwaysApply: true
|
||||
---
|
||||
|
||||
# MapMil Platform
|
||||
|
||||
Monorepo: **ЦА** (analytics) + **ЦП** (parsing) + **contracts** + **ПИ** (distribution API).
|
||||
|
||||
## Data flow
|
||||
|
||||
```
|
||||
CA admin → Redis cp:jobs:{family} → CP adapter → POST /internal/ingest → CA DB → map / ПИ
|
||||
```
|
||||
|
||||
| Center | Path | Role |
|
||||
|--------|------|------|
|
||||
| ЦА API | `centers/analytics/api/` | FastAPI, ingest, scheduler, admin |
|
||||
| ЦА UI | `centers/analytics/frontend/` | Vue + Leaflet admin |
|
||||
| ЦП | `centers/parsing/workers/` | Adapter registry, batch + Telegram listener |
|
||||
| Contracts | `contracts/` | Shared schemas — change first |
|
||||
|
||||
## Hard boundaries
|
||||
|
||||
- ЦП **never** touches CA database — only `POST /internal/ingest` and `PATCH /internal/jobs/{id}`
|
||||
- `source_type` events must match adapter type; stable `source_url` required (CA dedup)
|
||||
- `source_config` validated via `contracts/sources.py` on admin CRUD
|
||||
- Telegram session: `data/telegram.session` — local only, **never commit**
|
||||
- `.env` and secrets — **never commit**
|
||||
|
||||
## CP adapter model
|
||||
|
||||
Each `source_type` implements `SourceAdapter.run(job_id, source_config, *, ctx) -> tuple[list[dict], str | None]`.
|
||||
|
||||
Output dicts are `IngestEventItem`-shaped. Routing: `contracts/queues.py` (`SOURCE_FAMILY` → `cp:jobs:{telegram|web|nlp}`).
|
||||
|
||||
Worker images: `cp-workers` (telegram), `cp-workers-web` (crawl4ai), `cp-workers-nlp` (viina).
|
||||
|
||||
## Agent workflow
|
||||
|
||||
1. **Contracts first** if data shape or routing changes
|
||||
2. **One zone per task** — do not mix contracts + UI + infra in one unfocused change
|
||||
3. **Follow existing adapters** — copy `viina.py` or `telegram.py`, not greenfield
|
||||
4. **Do not touch** Telegram listener unless explicitly asked
|
||||
5. **Verify**: `docker compose config`, then targeted service rebuild
|
||||
|
||||
Docs: `README.md`, `centers/parsing/ARCHITECTURE.md`, `AGENTS.md`.
|
||||
|
||||
For new `source_type`: use skill `add-parser-adapter`.
|
||||
@@ -0,0 +1,142 @@
|
||||
---
|
||||
name: add-parser-adapter
|
||||
description: >-
|
||||
Adds a new CP source_type end-to-end in MapMil: contracts schema and queue
|
||||
family, SourceAdapter implementation, registry, optional Docker worker image,
|
||||
and ParsersView.vue form fields. Use when adding a parser source, adapter,
|
||||
source_type, or new ingest channel to the parsing center.
|
||||
---
|
||||
|
||||
# Add Parser Adapter
|
||||
|
||||
End-to-end checklist for a new `source_type` in ЦП. Work in order; do not skip contracts.
|
||||
|
||||
## Inputs to clarify
|
||||
|
||||
- `source_type` slug (lowercase, e.g. `rss`)
|
||||
- Worker **family** (`telegram` | `web` | `nlp`) — pick by deps:
|
||||
- `telegram` — Telethon, shared session
|
||||
- `web` — browser/HTTP heavy (Crawl4AI, Playwright)
|
||||
- `nlp` — lightweight HTTP + text (VIINA-style)
|
||||
- `source_config` fields for admin UI
|
||||
- Whether existing worker image can host it or needs new `requirements-*.txt` service
|
||||
|
||||
## Checklist
|
||||
|
||||
### 1. Contracts
|
||||
|
||||
**`contracts/sources.py`**
|
||||
|
||||
- Add `{Name}SourceConfig(BaseModel)` with validators
|
||||
- Register in `CONFIG_MODELS["{source_type}"]`
|
||||
|
||||
**`contracts/queues.py`**
|
||||
|
||||
- Add `SOURCE_FAMILY["{source_type}"] = "{family}"`
|
||||
|
||||
No CA code changes needed for routing — `enqueue_job` uses `queue_key_for_source`.
|
||||
|
||||
### 2. Adapter
|
||||
|
||||
**`centers/parsing/workers/workers/adapters/{name}.py`**
|
||||
|
||||
Copy pattern from `viina.py` (text/HTTP) or `telegram.py` (Telethon):
|
||||
|
||||
```python
|
||||
class MyAdapter:
|
||||
source_type = "{source_type}"
|
||||
|
||||
async def run(self, job_id, source_config, *, ctx: WorkerContext) -> tuple[list[dict], str | None]:
|
||||
cfg = MySourceConfig.model_validate(source_config or {})
|
||||
# fetch / parse → list of IngestEventItem-shaped dicts
|
||||
return events, None # or ([], "error message")
|
||||
```
|
||||
|
||||
Rules:
|
||||
- Validate config with Pydantic model from `contracts/sources`
|
||||
- Every event: stable `source_url`, `source_type="{source_type}"`
|
||||
- Optional fields: `title`, `description`, `locality`, `latitude`, `longitude`, `event_date`, `region`, `topic`, `tags`, `metadata`
|
||||
- Return `([], error)` on fatal config errors; partial fetch errors can go in metadata
|
||||
|
||||
**`workers/adapters/registry.py`**
|
||||
|
||||
- Add lazy factory in `_factories()`
|
||||
- Factory name key must match `source_type`
|
||||
|
||||
### 3. Dependencies & Docker
|
||||
|
||||
If adapter fits an existing family, only update `ENABLED_ADAPTERS` in compose (usually not needed — one adapter per service).
|
||||
|
||||
If new deps are heavy or conflict:
|
||||
|
||||
1. `centers/parsing/workers/requirements-{family}.txt` (or new file)
|
||||
2. New or updated service in `docker-compose.yml`:
|
||||
|
||||
```yaml
|
||||
cp-workers-{family}:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: centers/parsing/workers/Dockerfile
|
||||
args:
|
||||
REQUIREMENTS_FILE: requirements-{family}.txt
|
||||
INSTALL_PLAYWRIGHT: "0" # "1" only for browser adapters
|
||||
environment:
|
||||
ENABLED_ADAPTERS: {source_type}
|
||||
WORKER_FAMILIES: {family}
|
||||
CA_API_URL: http://ca-api:8000
|
||||
REDIS_URL: redis://redis:6379/0
|
||||
INTERNAL_TOKEN: dev-internal-token
|
||||
depends_on: [redis, ca-api]
|
||||
```
|
||||
|
||||
### 4. Admin UI
|
||||
|
||||
**`centers/analytics/frontend/src/views/ParsersView.vue`**
|
||||
|
||||
- Add option to source type `<select>`
|
||||
- Extend form state + `buildSourceConfig()` for new fields
|
||||
- Add create/edit `<template>` blocks (mirror `crawl4ai` or `viina`)
|
||||
- Extend validation in submit handler
|
||||
|
||||
TypeScript unions: update `"telegram" | "crawl4ai" | "viina"` wherever used.
|
||||
|
||||
No CA router changes if `source_type` is registered in `CONFIG_MODELS` — `_validated_source_config` picks it up automatically.
|
||||
|
||||
### 5. Documentation
|
||||
|
||||
- Add row to adapter table in `centers/parsing/ARCHITECTURE.md`
|
||||
- One-line mention in `README.md` if user-facing
|
||||
|
||||
### 6. Verify
|
||||
|
||||
```bash
|
||||
docker compose config
|
||||
docker compose up --build cp-workers-{family} # or relevant service
|
||||
curl -X POST http://localhost:8080/admin/jobs \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"source_type":"{source_type}","source_config":{...},"interval_seconds":3600}'
|
||||
```
|
||||
|
||||
Check CA logs for ingest and `/admin/events` for new rows.
|
||||
|
||||
## Reference files
|
||||
|
||||
| Concern | File |
|
||||
|---------|------|
|
||||
| Config schema | `contracts/sources.py` |
|
||||
| Queue routing | `contracts/queues.py` |
|
||||
| Ingest shape | `contracts/ingest.py` |
|
||||
| Adapter protocol | `workers/adapters/base.py` |
|
||||
| Simple adapter | `workers/adapters/viina.py` |
|
||||
| LLM adapter | `workers/adapters/crawl4ai_adapter.py` |
|
||||
| Telegram + listener | `workers/adapters/telegram.py` |
|
||||
| Enqueue | `centers/analytics/api/app/services/jobs.py` |
|
||||
| Admin validation | `centers/analytics/api/app/routers/admin.py` |
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- Implementing adapter before `CONFIG_MODELS` entry
|
||||
- Hardcoding Redis queue names — use `queue_key_for_source`
|
||||
- Skipping `source_url` stability (breaks dedup)
|
||||
- Importing heavy deps at module top in `registry.py` — use lazy factories
|
||||
- Editing Telegram listener when adding unrelated sources
|
||||
@@ -0,0 +1,100 @@
|
||||
# MapMil — guide for AI agents
|
||||
|
||||
Quick orientation for Cursor and other coding agents working in this repository.
|
||||
|
||||
## What this is
|
||||
|
||||
**MapMil** is a monorepo platform: parsing center (ЦП) collects events from external sources, analytics center (ЦА) stores them and serves a map + admin UI, external consumers (ПИ) read via distribution API.
|
||||
|
||||
```
|
||||
ЦА (centers/analytics) ←→ Redis ←→ ЦП (centers/parsing)
|
||||
↓ ingest ↑ adapters
|
||||
PostgreSQL + map UI Telegram / Crawl4AI / VIINA
|
||||
↓
|
||||
GET /api/v1/events → ПИ
|
||||
```
|
||||
|
||||
## Repository map
|
||||
|
||||
| Path | Stack | Purpose |
|
||||
|------|-------|---------|
|
||||
| `contracts/` | Pydantic | `ingest`, `jobs`, `sources`, `queues` — shared truth |
|
||||
| `centers/analytics/api/` | FastAPI, SQLAlchemy | Admin API, ingest, scheduler, map queries |
|
||||
| `centers/analytics/frontend/` | Vue 3, Leaflet | Map, parsers, events, analytics, consumers |
|
||||
| `centers/parsing/workers/` | Python async | CP workers, adapter registry, LLM extract |
|
||||
| `docker-compose.yml` | Docker | `ca-*`, `cp-workers*`, `redis` |
|
||||
| `data/` | Telethon session | **Gitignored** — do not commit |
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
cp .env.example .env # then set TELEGRAM_API_ID/HASH, optional DEEPSEEK_API_KEY
|
||||
docker compose up --build # full stack → http://localhost:8080
|
||||
docker compose config # validate compose
|
||||
docker compose up --build cp-workers-web # single worker rebuild
|
||||
```
|
||||
|
||||
Health: `curl http://localhost:8080/api/health`
|
||||
|
||||
Create Telegram parser:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8080/admin/jobs \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"source_type":"telegram","source_config":{"channel":"example","limit":50}}'
|
||||
```
|
||||
|
||||
## Conventions
|
||||
|
||||
### Contracts-first
|
||||
|
||||
Any change to event shape, `source_config`, or queue routing starts in `contracts/`, then CA/CP/UI.
|
||||
|
||||
### CP adapters
|
||||
|
||||
- Protocol: `workers/adapters/base.py` (`SourceAdapter`)
|
||||
- Registry: `workers/adapters/registry.py` (`ENABLED_ADAPTERS`)
|
||||
- Queue family: `contracts/queues.py` (`SOURCE_FAMILY`)
|
||||
- Config schema: `contracts/sources.py` (`CONFIG_MODELS`)
|
||||
|
||||
### Ingest output
|
||||
|
||||
Adapters return dicts matching `IngestEventItem` (`contracts/ingest.py`). Required: `source_url`. Dedup in CA by `source_url`.
|
||||
|
||||
### LLM extract (runtime, not dev)
|
||||
|
||||
`extract_mode: llm` uses DeepSeek via `workers/llm_extract.py`. Key: `DEEPSEEK_API_KEY` in `.env`. Do not confuse with Cursor dev agents.
|
||||
|
||||
## Common tasks
|
||||
|
||||
| Task | Start here | Skill |
|
||||
|------|------------|-------|
|
||||
| New parser source | `contracts/sources.py` | `add-parser-adapter` |
|
||||
| Admin API change | `centers/analytics/api/app/routers/` | — |
|
||||
| Map / UI | `centers/analytics/frontend/src/` | — |
|
||||
| New worker image | `docker-compose.yml` + `requirements-*.txt` | `add-parser-adapter` |
|
||||
|
||||
## Do not
|
||||
|
||||
- Commit `.env`, `data/`, or API keys
|
||||
- Add direct DB access from CP workers
|
||||
- Change `IngestEventItem` without checking ingest + map + ПИ impact
|
||||
- Modify Telegram listener/session logic unless the task requires it
|
||||
- Reintroduce legacy root `backend/` or `frontend/` (removed; use `centers/`)
|
||||
|
||||
## Cursor setup
|
||||
|
||||
- Rule: `.cursor/rules/platform.mdc` (always on)
|
||||
- Skill: `.cursor/skills/add-parser-adapter/` for end-to-end new sources
|
||||
- Deep docs: `README.md`, `centers/parsing/ARCHITECTURE.md`
|
||||
|
||||
## Commit style
|
||||
|
||||
Follow existing history — imperative, concise:
|
||||
|
||||
```
|
||||
Add CP source adapter registry with multi-worker queues and LLM extract.
|
||||
Add Telegram listener, scheduler, and extended parsers admin UI.
|
||||
```
|
||||
|
||||
One logical change per commit; do not bundle unrelated zones.
|
||||
Reference in New Issue
Block a user