Add Profile/Channel entities with CRUD and remove legacy parser builder.

Introduce reusable ParserProfile and ParseChannel, pair jobs with enqueue flatten, Events admin CRUD, and drop inline/legacy parser-builder UI and aliases.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
2026-08-16 14:51:03 +03:00
co-authored by Cursor
parent e196de7320
commit 71f8cf169e
30 changed files with 2157 additions and 954 deletions
+1 -1
View File
@@ -14,7 +14,7 @@ TEST_PI_API_KEY=test-pi-api-key-change-me
# DeepSeek LLM: # DeepSeek LLM:
# - extract_mode=llm on CP workers (Telegram unstructured + Crawl4AI) # - extract_mode=llm on CP workers (Telegram unstructured + Crawl4AI)
# - parser-builder Generate on ca-api (one-shot profile; runtime stays rule-based) # - Generate профиля парсера на ca-api (один раз; runtime — только правила)
# DEEPSEEK_API_KEY=sk-... # DEEPSEEK_API_KEY=sk-...
# DEEPSEEK_BASE_URL=https://api.deepseek.com # DEEPSEEK_BASE_URL=https://api.deepseek.com
# DEEPSEEK_MODEL=deepseek-chat # DEEPSEEK_MODEL=deepseek-chat
+29 -17
View File
@@ -58,12 +58,13 @@ centers/analytics/
│ ├── routers/ │ ├── routers/
│ │ ├── objects.py # /api/health, /api/objects, media │ │ ├── objects.py # /api/health, /api/objects, media
│ │ ├── map.py # /api/map/* │ │ ├── map.py # /api/map/*
│ │ ├── admin.py # /admin/* │ │ ├── admin.py # /admin/* (jobs, events, consumers)
│ │ ├── parser_builder.py # /admin/parser-builder/* │ │ ├── parser_profiles.py # /admin/parser-profiles/*
│ │ ├── parse_channels.py # /admin/parse-channels/*
│ │ ├── internal.py # /internal/* (только ЦП) │ │ ├── internal.py # /internal/* (только ЦП)
│ │ └── v1.py # /api/v1/* (ПИ) │ │ └── v1.py # /api/v1/* (ПИ)
│ └── services/ │ └── services/
│ ├── jobs.py # Redis RPUSH │ ├── jobs.py # Redis RPUSH + flatten Profile/Channel
│ ├── scheduler.py # периодический re-queue │ ├── scheduler.py # периодический re-queue
│ ├── ingest.py # дедуп + map sync │ ├── ingest.py # дедуп + map sync
│ ├── parser_builder.py # DeepSeek → HeuristicProfile (один раз) │ ├── parser_builder.py # DeepSeek → HeuristicProfile (один раз)
@@ -76,7 +77,7 @@ centers/analytics/
├── Dockerfile # Vite build + nginx ├── Dockerfile # Vite build + nginx
├── nginx.conf # proxy /api /admin /internal → ca-api ├── nginx.conf # proxy /api /admin /internal → ca-api
└── src/ └── src/
├── views/ # Map, Parsers, ParserBuilder, Events, … ├── views/ # Map, Channels, Profiles, Parsers, …
├── components/ # карта, CRUD объектов ├── components/ # карта, CRUD объектов
├── api/ # HTTP-клиенты ├── api/ # HTTP-клиенты
└── router/index.ts └── router/index.ts
@@ -87,7 +88,9 @@ centers/analytics/
| Модель | Таблица | Назначение | | Модель | Таблица | Назначение |
|--------|--------|------------| |--------|--------|------------|
| `Event` | `events` | Нормализованное событие; UK `source_url` | | `Event` | `events` | Нормализованное событие; UK `source_url` |
| `ParseJob` | `parse_jobs` | Конфиг парсера, интервал, статус | | `ParserProfile` | `parser_profiles` | Статичный heuristic-профиль (JSON-правила) |
| `ParseChannel` | `parse_channels` | Канал (Telegram handle) |
| `ParseJob` | `parse_jobs` | Связка канал+профиль, интервал, статус; FK `profile_id` + `channel_id` |
| `MapObject` | `map_objects` | Точка на карте (event или ручная) | | `MapObject` | `map_objects` | Точка на карте (event или ручная) |
| `ObjectMedia` | `object_media` | Файлы к объектам | | `ObjectMedia` | `object_media` | Файлы к объектам |
| `Consumer` | `consumers` | Подписчик ПИ (hash ключа) | | `Consumer` | `consumers` | Подписчик ПИ (hash ключа) |
@@ -98,26 +101,34 @@ centers/analytics/
| Prefix | Кто вызывает | Содержание | | Prefix | Кто вызывает | Содержание |
|--------|--------------|------------| |--------|--------------|------------|
| `/api/*` | UI, публичный health | Карта, объекты, медиа | | `/api/*` | UI, публичный health | Карта, объекты, медиа |
| `/admin/*` | UI admin | Jobs, events, analytics, consumers, parser-builder | | `/admin/*` | UI admin | Jobs, channels, profiles, events, analytics, consumers |
| `/internal/*` | Только ЦП | ingest, job status, listener subscriptions | | `/internal/*` | Только ЦП | ingest, job status, listener subscriptions |
| `/api/v1/*` | Внешние клиенты | Events с Bearer-ключом | | `/api/v1/*` | Внешние клиенты | Events с Bearer-ключом |
Internal защищён заголовком `X-Internal-Token` (`INTERNAL_TOKEN`). Internal защищён заголовком `X-Internal-Token` (`INTERNAL_TOKEN`).
### Конструктор парсера ### Профили, каналы и пары
Раздел UI `/parser-builder` → `POST /admin/parser-builder/generate|preview|jobs`: 1. **Канал** (`CRUD /admin/parse-channels`) — handle Telegram + метаданные.
2. **Профиль** (`CRUD /admin/parser-profiles`) — Generate (DeepSeek один раз по образцу) → Preview → Save `heuristic_profile`.
3. **Пара** (`POST /admin/jobs` с `channel_id` + `profile_id`) → `ParseJob`; при enqueue ЦА **разворачивает** пару в плоский `source_config`:
1. Менеджер вставляет образец поста. ```json
2. DeepSeek (`DEEPSEEK_API_KEY` на **ca-api**) один раз возвращает `HeuristicProfile` (regex/line/marker). {
3. Preview и сохранение job с `extract_mode=profile` + JSON профиля в `source_config`. "channel": "<ParseChannel.channel>",
4. ЦП применяет профиль статически (batch + listener); LLM на ingest не вызывается. "limit": 100,
"extract_mode": "profile",
"heuristic_profile": { "...из ParserProfile..." }
}
```
Roadmap: кастомные пользовательские таблицы подменяют только список `target-fields` при генерации. ЦП по-прежнему получает только Redis payload — без доступа к таблицам Profile/Channel.
Roadmap: кастомные пользовательские таблицы подменяют список `target-fields` при генерации; профиль будет ссылаться на `schema_id`.
### Jobs и scheduler ### Jobs и scheduler
1. `POST /admin/jobs` / retry → запись `ParseJob` + `enqueue_job` (`services/jobs.py`). 1. `POST /admin/jobs` / retry → запись `ParseJob` + `enqueue_parse_job` (flatten + RPUSH).
2. Очередь: `cp:jobs:{family}` из `contracts/queues.py`. 2. Очередь: `cp:jobs:{family}` из `contracts/queues.py`.
3. `services/scheduler.py` — тик ~30 с, повторная постановка активных jobs по `interval_seconds`. 3. `services/scheduler.py` — тик ~30 с, повторная постановка активных jobs по `interval_seconds`.
@@ -135,8 +146,9 @@ Roadmap: кастомные пользовательские таблицы по
| Path | View | | Path | View |
|------|------| |------|------|
| `/` | `MapViewPage.vue` | | `/` | `MapViewPage.vue` |
| `/parsers` | `ParsersView.vue` | | `/channels` | `ChannelsView.vue` |
| `/parser-builder` | `ParserBuilderView.vue` | | `/parser-profiles` | `ParserProfilesView.vue` |
| `/parsers` | `ParsersView.vue` (связки канал+профиль) |
| `/events` | `EventsView.vue` | | `/events` | `EventsView.vue` |
| `/analytics` | `AnalyticsView.vue` | | `/analytics` | `AnalyticsView.vue` |
| `/consumers` | `ConsumersView.vue` | | `/consumers` | `ConsumersView.vue` |
@@ -156,7 +168,7 @@ Roadmap: кастомные пользовательские таблицы по
| `INTERNAL_TOKEN` | Auth ЦП ↔ ЦА | | `INTERNAL_TOKEN` | Auth ЦП ↔ ЦА |
| `TEST_PI_API_KEY` | Seed consumer `test-pi` | | `TEST_PI_API_KEY` | Seed consumer `test-pi` |
| `UPLOAD_DIR` | Медиа (по умолчанию `/data/uploads`) | | `UPLOAD_DIR` | Медиа (по умолчанию `/data/uploads`) |
| `DEEPSEEK_API_KEY` | Конструктор парсера (Generate); не нужен для preview/runtime profile | | `DEEPSEEK_API_KEY` | Generate профиля на ca-api; не нужен для preview/runtime |
| `DEEPSEEK_BASE_URL` / `DEEPSEEK_MODEL` | Опционально | | `DEEPSEEK_BASE_URL` / `DEEPSEEK_MODEL` | Опционально |
### Типовые точки входа в код ### Типовые точки входа в код
+3 -2
View File
@@ -13,7 +13,7 @@ for _candidate in (_HERE.parent, *_HERE.parents):
break break
from .database import Base, engine, get_db from .database import Base, engine, get_db
from .routers import admin, internal, map, objects, parser_builder, v1 from .routers import admin, internal, map, objects, parse_channels, parser_profiles, v1
from .seed import seed_objects, seed_test_consumer from .seed import seed_objects, seed_test_consumer
from .services.migrations import migrate_schema from .services.migrations import migrate_schema
from .services.scheduler import start_scheduler from .services.scheduler import start_scheduler
@@ -50,5 +50,6 @@ app.include_router(objects.router)
app.include_router(map.router) app.include_router(map.router)
app.include_router(internal.router) app.include_router(internal.router)
app.include_router(admin.router) app.include_router(admin.router)
app.include_router(parser_builder.router) app.include_router(parser_profiles.router)
app.include_router(parse_channels.router)
app.include_router(v1.router) app.include_router(v1.router)
+45
View File
@@ -93,12 +93,54 @@ class Event(Base):
) )
class ParserProfile(Base):
__tablename__ = "parser_profiles"
id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)
name: Mapped[str] = mapped_column(String(255), nullable=False)
sample_post: Mapped[str] = mapped_column(Text, default="")
heuristic_profile: Mapped[dict | None] = mapped_column(JSON, nullable=True)
status: Mapped[str] = mapped_column(String(50), default="draft", index=True)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=lambda: datetime.now(timezone.utc),
)
jobs: Mapped[list["ParseJob"]] = relationship(back_populates="profile")
class ParseChannel(Base):
__tablename__ = "parse_channels"
id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)
name: Mapped[str] = mapped_column(String(255), nullable=False)
source_type: Mapped[str] = mapped_column(String(50), nullable=False, default="telegram")
channel: Mapped[str] = mapped_column(String(255), nullable=False)
is_active: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=lambda: datetime.now(timezone.utc),
)
jobs: Mapped[list["ParseJob"]] = relationship(back_populates="channel")
class ParseJob(Base): class ParseJob(Base):
__tablename__ = "parse_jobs" __tablename__ = "parse_jobs"
id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True) id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)
source_type: Mapped[str] = mapped_column(String(50), nullable=False) source_type: Mapped[str] = mapped_column(String(50), nullable=False)
source_config: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict) source_config: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
profile_id: Mapped[int | None] = mapped_column(
ForeignKey("parser_profiles.id", ondelete="SET NULL"),
nullable=True,
index=True,
)
channel_id: Mapped[int | None] = mapped_column(
ForeignKey("parse_channels.id", ondelete="SET NULL"),
nullable=True,
index=True,
)
schedule: Mapped[str | None] = mapped_column(String(100), nullable=True) schedule: Mapped[str | None] = mapped_column(String(100), nullable=True)
interval_seconds: Mapped[int] = mapped_column(Integer, default=3600, nullable=False) interval_seconds: Mapped[int] = mapped_column(Integer, default=3600, nullable=False)
is_active: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False) is_active: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False)
@@ -110,6 +152,9 @@ class ParseJob(Base):
default=lambda: datetime.now(timezone.utc), default=lambda: datetime.now(timezone.utc),
) )
profile: Mapped["ParserProfile | None"] = relationship(back_populates="jobs")
channel: Mapped["ParseChannel | None"] = relationship(back_populates="jobs")
class Consumer(Base): class Consumer(Base):
__tablename__ = "consumers" __tablename__ = "consumers"
+217 -19
View File
@@ -4,14 +4,16 @@ from fastapi import APIRouter, Depends, HTTPException, Query
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from ..database import get_db from ..database import get_db
from ..models import Consumer, Event, ParseJob from ..models import Consumer, Event, MapObject, ParseChannel, ParseJob, ParserProfile
from ..schemas import ( from ..schemas import (
AnalyticsSummary, AnalyticsSummary,
ConsumerCreate, ConsumerCreate,
ConsumerRead, ConsumerRead,
ConsumerUpdate, ConsumerUpdate,
EventCreate,
EventListResponse, EventListResponse,
EventRead, EventRead,
EventUpdate,
ParseJobCreate, ParseJobCreate,
ParseJobRead, ParseJobRead,
ParseJobUpdate, ParseJobUpdate,
@@ -30,9 +32,10 @@ from ..services.filtering import (
create_consumer, create_consumer,
list_consumers, list_consumers,
rotate_consumer_key, rotate_consumer_key,
sync_event_to_map_object,
update_consumer, update_consumer,
) )
from ..services.jobs import enqueue_job from ..services.jobs import enqueue_parse_job, flatten_pair_config
router = APIRouter(prefix="/admin", tags=["admin"]) router = APIRouter(prefix="/admin", tags=["admin"])
@@ -48,28 +51,85 @@ def _validated_source_config(source_type: str, source_config: dict) -> dict:
raise HTTPException(status_code=400, detail=str(exc)) from exc raise HTTPException(status_code=400, detail=str(exc)) from exc
def _job_to_read(job: ParseJob) -> ParseJobRead:
return ParseJobRead(
id=job.id,
source_type=job.source_type,
source_config=job.source_config or {},
profile_id=job.profile_id,
channel_id=job.channel_id,
schedule=job.schedule,
interval_seconds=job.interval_seconds,
is_active=job.is_active,
status=job.status,
last_run_at=job.last_run_at,
last_error=job.last_error,
created_at=job.created_at,
profile_name=job.profile.name if job.profile else None,
channel_name=job.channel.name if job.channel else None,
channel_handle=job.channel.channel if job.channel else None,
)
@router.post("/jobs", response_model=ParseJobRead, status_code=201) @router.post("/jobs", response_model=ParseJobRead, status_code=201)
def create_parse_job(payload: ParseJobCreate, db: Session = Depends(get_db)): def create_parse_job(payload: ParseJobCreate, db: Session = Depends(get_db)):
source_config = _validated_source_config(payload.source_type, payload.source_config) pair_mode = payload.profile_id is not None or payload.channel_id is not None
job = ParseJob( if pair_mode:
source_type=payload.source_type, if payload.profile_id is None or payload.channel_id is None:
source_config=source_config, raise HTTPException(
schedule=payload.schedule, status_code=400,
interval_seconds=payload.interval_seconds, detail="Both profile_id and channel_id are required to create a pair",
is_active=payload.is_active, )
status="queued", profile = db.query(ParserProfile).filter(ParserProfile.id == payload.profile_id).first()
) channel = db.query(ParseChannel).filter(ParseChannel.id == payload.channel_id).first()
if not profile:
raise HTTPException(status_code=404, detail="Profile not found")
if not channel:
raise HTTPException(status_code=404, detail="Channel not found")
if not profile.heuristic_profile:
raise HTTPException(status_code=400, detail="Profile has no heuristic_profile")
if not channel.is_active:
raise HTTPException(status_code=400, detail="Channel is inactive")
try:
source_config = flatten_pair_config(channel, profile, limit=payload.limit)
except Exception as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
job = ParseJob(
source_type=channel.source_type or "telegram",
source_config=source_config,
profile_id=profile.id,
channel_id=channel.id,
schedule=payload.schedule,
interval_seconds=payload.interval_seconds,
is_active=payload.is_active,
status="queued",
)
else:
source_config = _validated_source_config(payload.source_type, payload.source_config)
job = ParseJob(
source_type=payload.source_type,
source_config=source_config,
schedule=payload.schedule,
interval_seconds=payload.interval_seconds,
is_active=payload.is_active,
status="queued",
)
db.add(job) db.add(job)
db.commit() db.commit()
db.refresh(job) db.refresh(job)
enqueue_job(job.id, job.source_type, job.source_config) enqueue_parse_job(db, job)
return job db.refresh(job)
return _job_to_read(job)
@router.get("/jobs", response_model=list[ParseJobRead]) @router.get("/jobs", response_model=list[ParseJobRead])
def list_parse_jobs(db: Session = Depends(get_db)): def list_parse_jobs(db: Session = Depends(get_db)):
return db.query(ParseJob).order_by(ParseJob.id.desc()).all() jobs = db.query(ParseJob).order_by(ParseJob.id.desc()).all()
return [_job_to_read(job) for job in jobs]
@router.get("/jobs/{job_id}", response_model=ParseJobRead) @router.get("/jobs/{job_id}", response_model=ParseJobRead)
@@ -77,7 +137,7 @@ def get_parse_job(job_id: int, db: Session = Depends(get_db)):
job = db.query(ParseJob).filter(ParseJob.id == job_id).first() job = db.query(ParseJob).filter(ParseJob.id == job_id).first()
if not job: if not job:
raise HTTPException(status_code=404, detail="Job not found") raise HTTPException(status_code=404, detail="Job not found")
return job return _job_to_read(job)
@router.post("/jobs/{job_id}/retry", response_model=ParseJobRead) @router.post("/jobs/{job_id}/retry", response_model=ParseJobRead)
@@ -93,8 +153,12 @@ def retry_parse_job(job_id: int, db: Session = Depends(get_db)):
db.commit() db.commit()
db.refresh(job) db.refresh(job)
enqueue_job(job.id, job.source_type, job.source_config) try:
return job enqueue_parse_job(db, job)
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
db.refresh(job)
return _job_to_read(job)
@router.patch("/jobs/{job_id}", response_model=ParseJobRead) @router.patch("/jobs/{job_id}", response_model=ParseJobRead)
@@ -107,8 +171,36 @@ def update_parse_job(
if not job: if not job:
raise HTTPException(status_code=404, detail="Job not found") raise HTTPException(status_code=404, detail="Job not found")
if payload.source_config is not None: if payload.profile_id is not None:
profile = db.query(ParserProfile).filter(ParserProfile.id == payload.profile_id).first()
if not profile:
raise HTTPException(status_code=404, detail="Profile not found")
job.profile_id = profile.id
if payload.channel_id is not None:
channel = db.query(ParseChannel).filter(ParseChannel.id == payload.channel_id).first()
if not channel:
raise HTTPException(status_code=404, detail="Channel not found")
job.channel_id = channel.id
if job.profile_id and job.channel_id:
profile = db.query(ParserProfile).filter(ParserProfile.id == job.profile_id).first()
channel = db.query(ParseChannel).filter(ParseChannel.id == job.channel_id).first()
if profile and channel:
existing_limit = (job.source_config or {}).get("limit", 100)
limit = payload.limit if payload.limit is not None else existing_limit
if not isinstance(limit, int):
limit = 100
try:
job.source_config = flatten_pair_config(channel, profile, limit=limit)
except Exception as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
elif payload.source_config is not None:
job.source_config = _validated_source_config(job.source_type, payload.source_config) job.source_config = _validated_source_config(job.source_type, payload.source_config)
elif payload.limit is not None and isinstance(job.source_config, dict):
cfg = dict(job.source_config)
cfg["limit"] = payload.limit
job.source_config = _validated_source_config(job.source_type, cfg)
if payload.interval_seconds is not None: if payload.interval_seconds is not None:
job.interval_seconds = payload.interval_seconds job.interval_seconds = payload.interval_seconds
if payload.is_active is not None: if payload.is_active is not None:
@@ -116,7 +208,7 @@ def update_parse_job(
db.commit() db.commit()
db.refresh(job) db.refresh(job)
return job return _job_to_read(job)
@router.delete("/jobs/{job_id}", status_code=204) @router.delete("/jobs/{job_id}", status_code=204)
@@ -159,6 +251,112 @@ def list_events(
return EventListResponse(items=items, total=total) return EventListResponse(items=items, total=total)
def _ensure_unique_source_url(db: Session, source_url: str, *, exclude_id: int | None = None) -> None:
query = db.query(Event).filter(Event.source_url == source_url)
if exclude_id is not None:
query = query.filter(Event.id != exclude_id)
if query.first():
raise HTTPException(status_code=409, detail="Event with this source_url already exists")
def _sync_or_clear_map_object(db: Session, event: Event) -> None:
if event.latitude is not None and event.longitude is not None:
sync_event_to_map_object(db, event)
return
linked = db.query(MapObject).filter(MapObject.event_id == event.id).first()
if linked:
db.delete(linked)
@router.post("/events", response_model=EventRead, status_code=201)
def create_event(payload: EventCreate, db: Session = Depends(get_db)):
source_url = payload.source_url.strip()
if not source_url:
raise HTTPException(status_code=400, detail="source_url is required")
_ensure_unique_source_url(db, source_url)
if (payload.latitude is None) != (payload.longitude is None):
raise HTTPException(status_code=400, detail="latitude and longitude must be set together")
event = Event(
source_type=payload.source_type.strip() or "manual",
source_url=source_url,
raw_text=payload.raw_text or "",
title=payload.title or "",
description=payload.description or "",
locality=payload.locality or "",
latitude=payload.latitude,
longitude=payload.longitude,
event_date=payload.event_date,
region=payload.region,
topic=payload.topic,
tags=payload.tags,
metadata_=payload.metadata,
)
db.add(event)
db.flush()
_sync_or_clear_map_object(db, event)
db.commit()
db.refresh(event)
return event
@router.get("/events/{event_id}", response_model=EventRead)
def get_event(event_id: int, db: Session = Depends(get_db)):
event = db.query(Event).filter(Event.id == event_id).first()
if not event:
raise HTTPException(status_code=404, detail="Event not found")
return event
@router.patch("/events/{event_id}", response_model=EventRead)
def update_event(event_id: int, payload: EventUpdate, db: Session = Depends(get_db)):
event = db.query(Event).filter(Event.id == event_id).first()
if not event:
raise HTTPException(status_code=404, detail="Event not found")
data = payload.model_dump(exclude_unset=True)
if "source_url" in data and data["source_url"] is not None:
source_url = data["source_url"].strip()
if not source_url:
raise HTTPException(status_code=400, detail="source_url cannot be empty")
_ensure_unique_source_url(db, source_url, exclude_id=event.id)
event.source_url = source_url
data.pop("source_url")
if "metadata" in data:
event.metadata_ = data.pop("metadata")
if "source_type" in data and data["source_type"] is not None:
event.source_type = data.pop("source_type").strip() or event.source_type
for field, value in data.items():
setattr(event, field, value)
lat = event.latitude
lon = event.longitude
if (lat is None) != (lon is None):
raise HTTPException(status_code=400, detail="latitude and longitude must be set together")
_sync_or_clear_map_object(db, event)
db.commit()
db.refresh(event)
return event
@router.delete("/events/{event_id}", status_code=204)
def delete_event(event_id: int, db: Session = Depends(get_db)):
event = db.query(Event).filter(Event.id == event_id).first()
if not event:
raise HTTPException(status_code=404, detail="Event not found")
linked = db.query(MapObject).filter(MapObject.event_id == event.id).all()
for obj in linked:
db.delete(obj)
db.delete(event)
db.commit()
@router.get("/analytics/summary", response_model=AnalyticsSummary) @router.get("/analytics/summary", response_model=AnalyticsSummary)
def analytics_summary(db: Session = Depends(get_db)): def analytics_summary(db: Session = Depends(get_db)):
return get_analytics_summary(db) return get_analytics_summary(db)
@@ -8,6 +8,7 @@ from ..deps import verify_internal_token
from ..models import ParseJob from ..models import ParseJob
from ..schemas import IngestRequest, IngestResponse, ListenerSubscription from ..schemas import IngestRequest, IngestResponse, ListenerSubscription
from ..services.ingest import ingest_events from ..services.ingest import ingest_events
from ..services.jobs import resolve_job_source_config
router = APIRouter(prefix="/internal", tags=["internal"]) router = APIRouter(prefix="/internal", tags=["internal"])
@@ -68,14 +69,19 @@ def listener_subscriptions(
) )
result: list[ListenerSubscription] = [] result: list[ListenerSubscription] = []
for job in jobs: for job in jobs:
channel = job.source_config.get("channel") if job.source_config else None try:
source_config = resolve_job_source_config(db, job)
except ValueError:
continue
channel = source_config.get("channel")
if not channel: if not channel:
continue continue
result.append( result.append(
ListenerSubscription( ListenerSubscription(
job_id=job.id, job_id=job.id,
channel=str(channel), channel=str(channel),
source_config=dict(job.source_config or {}), source_config=dict(source_config),
) )
) )
db.commit()
return result return result
@@ -0,0 +1,98 @@
"""Admin CRUD for parse channels (Telegram handles in MVP)."""
from __future__ import annotations
from fastapi import APIRouter, Depends, HTTPException
from sqlalchemy.orm import Session
from ..database import get_db
from ..models import ParseChannel, ParseJob
from ..schemas import ParseChannelCreate, ParseChannelRead, ParseChannelUpdate
router = APIRouter(prefix="/admin/parse-channels", tags=["parse-channels"])
ALLOWED_SOURCE_TYPES = {"telegram"}
@router.get("", response_model=list[ParseChannelRead])
def list_channels(db: Session = Depends(get_db)):
return db.query(ParseChannel).order_by(ParseChannel.id.desc()).all()
@router.post("", response_model=ParseChannelRead, status_code=201)
def create_channel(payload: ParseChannelCreate, db: Session = Depends(get_db)):
source_type = (payload.source_type or "telegram").strip()
if source_type not in ALLOWED_SOURCE_TYPES:
raise HTTPException(
status_code=400,
detail=f"source_type must be one of: {', '.join(sorted(ALLOWED_SOURCE_TYPES))}",
)
channel = ParseChannel(
name=payload.name.strip(),
source_type=source_type,
channel=payload.channel.strip().lstrip("@"),
is_active=payload.is_active,
)
db.add(channel)
db.commit()
db.refresh(channel)
return channel
@router.get("/{channel_id}", response_model=ParseChannelRead)
def get_channel(channel_id: int, db: Session = Depends(get_db)):
channel = db.query(ParseChannel).filter(ParseChannel.id == channel_id).first()
if not channel:
raise HTTPException(status_code=404, detail="Channel not found")
return channel
@router.patch("/{channel_id}", response_model=ParseChannelRead)
def update_channel(
channel_id: int,
payload: ParseChannelUpdate,
db: Session = Depends(get_db),
):
channel = db.query(ParseChannel).filter(ParseChannel.id == channel_id).first()
if not channel:
raise HTTPException(status_code=404, detail="Channel not found")
if payload.name is not None:
channel.name = payload.name.strip()
if payload.source_type is not None:
source_type = payload.source_type.strip()
if source_type not in ALLOWED_SOURCE_TYPES:
raise HTTPException(
status_code=400,
detail=f"source_type must be one of: {', '.join(sorted(ALLOWED_SOURCE_TYPES))}",
)
channel.source_type = source_type
if payload.channel is not None:
channel.channel = payload.channel.strip().lstrip("@")
if payload.is_active is not None:
channel.is_active = payload.is_active
db.commit()
db.refresh(channel)
return channel
@router.delete("/{channel_id}", status_code=204)
def delete_channel(channel_id: int, db: Session = Depends(get_db)):
channel = db.query(ParseChannel).filter(ParseChannel.id == channel_id).first()
if not channel:
raise HTTPException(status_code=404, detail="Channel not found")
linked = (
db.query(ParseJob)
.filter(ParseJob.channel_id == channel_id, ParseJob.status.in_(("queued", "running")))
.count()
)
if linked:
raise HTTPException(
status_code=409,
detail="Cannot delete channel used by queued/running jobs",
)
db.delete(channel)
db.commit()
@@ -1,108 +0,0 @@
"""Admin API: construct static telegram parsers from a sample post."""
from __future__ import annotations
from typing import Any
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel, Field
from sqlalchemy.orm import Session
from contracts.heuristic_profile import HeuristicProfile
from contracts.sources import TelegramSourceConfig
from ..database import get_db
from ..models import ParseJob
from ..schemas import ParseJobRead
from ..services.jobs import enqueue_job
from ..services import parser_builder as builder
router = APIRouter(prefix="/admin/parser-builder", tags=["parser-builder"])
class GenerateRequest(BaseModel):
sample_post: str = Field(min_length=1)
class GenerateResponse(BaseModel):
profile: dict[str, Any]
class PreviewRequest(BaseModel):
sample_post: str = Field(min_length=1)
profile: dict[str, Any]
class PreviewResponse(BaseModel):
fields: dict[str, str]
class CreateProfileJobRequest(BaseModel):
channel: str = Field(min_length=1)
profile: dict[str, Any]
sample_post: str | None = None
limit: int = Field(default=100, ge=1, le=1000)
interval_seconds: int = Field(default=3600, ge=60, le=604800)
is_active: bool = True
@router.get("/target-fields")
def target_fields():
return {"fields": builder.get_target_fields()}
@router.post("/generate", response_model=GenerateResponse)
async def generate_profile(payload: GenerateRequest):
if not builder.deepseek_enabled():
raise HTTPException(
status_code=503,
detail="DEEPSEEK_API_KEY is not set on ca-api. Add it to .env for parser generation.",
)
try:
profile = await builder.generate_profile(payload.sample_post)
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
except Exception as exc:
raise HTTPException(
status_code=502,
detail=f"DeepSeek generate failed: {exc}",
) from exc
return GenerateResponse(profile=profile.model_dump())
@router.post("/preview", response_model=PreviewResponse)
def preview_profile(payload: PreviewRequest):
try:
HeuristicProfile.model_validate(payload.profile)
fields = builder.preview_with_profile(payload.sample_post, payload.profile)
except Exception as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
return PreviewResponse(fields=fields)
@router.post("/jobs", response_model=ParseJobRead, status_code=201)
def create_profile_job(payload: CreateProfileJobRequest, db: Session = Depends(get_db)):
try:
cfg = TelegramSourceConfig(
channel=payload.channel.strip(),
limit=payload.limit,
extract_mode="profile",
heuristic_profile=payload.profile,
sample_post=payload.sample_post,
)
except Exception as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
job = ParseJob(
source_type="telegram",
source_config=cfg.model_dump(),
interval_seconds=payload.interval_seconds,
is_active=payload.is_active,
status="queued",
)
db.add(job)
db.commit()
db.refresh(job)
enqueue_job(job.id, job.source_type, job.source_config)
return job
@@ -0,0 +1,164 @@
"""Admin CRUD for reusable parser profiles + generate/preview."""
from __future__ import annotations
from typing import Any
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel, Field
from sqlalchemy.orm import Session
from contracts.heuristic_profile import HeuristicProfile
from ..database import get_db
from ..models import ParseJob, ParserProfile
from ..schemas import ParserProfileCreate, ParserProfileRead, ParserProfileUpdate
from ..services import parser_builder as builder
router = APIRouter(prefix="/admin/parser-profiles", tags=["parser-profiles"])
class GenerateRequest(BaseModel):
sample_post: str = Field(min_length=1)
class GenerateResponse(BaseModel):
profile: dict[str, Any]
class PreviewRequest(BaseModel):
sample_post: str = Field(min_length=1)
heuristic_profile: dict[str, Any] | None = None
# Accept alias used by older builder UI
profile: dict[str, Any] | None = None
class PreviewResponse(BaseModel):
fields: dict[str, str]
def _validate_heuristic_profile(raw: dict[str, Any] | None) -> dict[str, Any] | None:
if raw is None:
return None
try:
return HeuristicProfile.model_validate(raw).model_dump()
except Exception as exc:
raise HTTPException(status_code=400, detail=f"Invalid heuristic_profile: {exc}") from exc
def _profile_status(heuristic_profile: dict | None, explicit: str | None = None) -> str:
if explicit:
return explicit
return "ready" if heuristic_profile else "draft"
@router.get("/target-fields")
def target_fields():
return {"fields": builder.get_target_fields()}
@router.post("/generate", response_model=GenerateResponse)
async def generate_profile(payload: GenerateRequest):
if not builder.deepseek_enabled():
raise HTTPException(
status_code=503,
detail="DEEPSEEK_API_KEY is not set on ca-api. Add it to .env for parser generation.",
)
try:
profile = await builder.generate_profile(payload.sample_post)
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
except Exception as exc:
raise HTTPException(
status_code=502,
detail=f"DeepSeek generate failed: {exc}",
) from exc
return GenerateResponse(profile=profile.model_dump())
@router.post("/preview", response_model=PreviewResponse)
def preview_profile(payload: PreviewRequest):
raw = payload.heuristic_profile if payload.heuristic_profile is not None else payload.profile
if raw is None:
raise HTTPException(status_code=400, detail="heuristic_profile is required")
try:
HeuristicProfile.model_validate(raw)
fields = builder.preview_with_profile(payload.sample_post, raw)
except Exception as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
return PreviewResponse(fields=fields)
@router.get("", response_model=list[ParserProfileRead])
def list_profiles(db: Session = Depends(get_db)):
return db.query(ParserProfile).order_by(ParserProfile.id.desc()).all()
@router.post("", response_model=ParserProfileRead, status_code=201)
def create_profile(payload: ParserProfileCreate, db: Session = Depends(get_db)):
heuristic = _validate_heuristic_profile(payload.heuristic_profile)
profile = ParserProfile(
name=payload.name.strip(),
sample_post=payload.sample_post or "",
heuristic_profile=heuristic,
status=_profile_status(heuristic, payload.status),
)
db.add(profile)
db.commit()
db.refresh(profile)
return profile
@router.get("/{profile_id}", response_model=ParserProfileRead)
def get_profile(profile_id: int, db: Session = Depends(get_db)):
profile = db.query(ParserProfile).filter(ParserProfile.id == profile_id).first()
if not profile:
raise HTTPException(status_code=404, detail="Profile not found")
return profile
@router.patch("/{profile_id}", response_model=ParserProfileRead)
def update_profile(
profile_id: int,
payload: ParserProfileUpdate,
db: Session = Depends(get_db),
):
profile = db.query(ParserProfile).filter(ParserProfile.id == profile_id).first()
if not profile:
raise HTTPException(status_code=404, detail="Profile not found")
if payload.name is not None:
profile.name = payload.name.strip()
if payload.sample_post is not None:
profile.sample_post = payload.sample_post
if "heuristic_profile" in payload.model_fields_set:
profile.heuristic_profile = _validate_heuristic_profile(payload.heuristic_profile)
if payload.status is not None:
profile.status = payload.status
elif "heuristic_profile" in payload.model_fields_set:
profile.status = _profile_status(profile.heuristic_profile)
db.commit()
db.refresh(profile)
return profile
@router.delete("/{profile_id}", status_code=204)
def delete_profile(profile_id: int, db: Session = Depends(get_db)):
profile = db.query(ParserProfile).filter(ParserProfile.id == profile_id).first()
if not profile:
raise HTTPException(status_code=404, detail="Profile not found")
linked = (
db.query(ParseJob)
.filter(ParseJob.profile_id == profile_id, ParseJob.status.in_(("queued", "running")))
.count()
)
if linked:
raise HTTPException(
status_code=409,
detail="Cannot delete profile used by queued/running jobs",
)
db.delete(profile)
db.commit()
+93
View File
@@ -92,6 +92,38 @@ class EventRead(BaseModel):
metadata: dict[str, Any] | None = Field(validation_alias="metadata_") metadata: dict[str, Any] | None = Field(validation_alias="metadata_")
class EventCreate(BaseModel):
source_type: str = Field(default="manual", min_length=1, max_length=50)
source_url: str = Field(min_length=1, max_length=512)
raw_text: str = ""
title: str = ""
description: str = ""
locality: str = ""
latitude: float | None = Field(default=None, ge=-90, le=90)
longitude: float | None = Field(default=None, ge=-180, le=180)
event_date: datetime | None = None
region: str | None = None
topic: str | None = None
tags: list[str] | None = None
metadata: dict[str, Any] | None = None
class EventUpdate(BaseModel):
source_type: str | None = Field(default=None, min_length=1, max_length=50)
source_url: str | None = Field(default=None, min_length=1, max_length=512)
raw_text: str | None = None
title: str | None = None
description: str | None = None
locality: str | None = None
latitude: float | None = Field(default=None, ge=-90, le=90)
longitude: float | None = Field(default=None, ge=-180, le=180)
event_date: datetime | None = None
region: str | None = None
topic: str | None = None
tags: list[str] | None = None
metadata: dict[str, Any] | None = None
class IngestEventItem(BaseModel): class IngestEventItem(BaseModel):
source_type: str = "telegram" source_type: str = "telegram"
source_url: str source_url: str
@@ -127,9 +159,62 @@ class IngestResponse(BaseModel):
map_objects_synced: int map_objects_synced: int
class ParserProfileCreate(BaseModel):
name: str = Field(min_length=1, max_length=255)
sample_post: str = ""
heuristic_profile: dict[str, Any] | None = None
status: str | None = None
class ParserProfileUpdate(BaseModel):
name: str | None = Field(default=None, min_length=1, max_length=255)
sample_post: str | None = None
heuristic_profile: dict[str, Any] | None = None
status: str | None = None
class ParserProfileRead(BaseModel):
model_config = ConfigDict(from_attributes=True)
id: int
name: str
sample_post: str
heuristic_profile: dict[str, Any] | None
status: str
created_at: datetime
class ParseChannelCreate(BaseModel):
name: str = Field(min_length=1, max_length=255)
source_type: str = "telegram"
channel: str = Field(min_length=1, max_length=255)
is_active: bool = True
class ParseChannelUpdate(BaseModel):
name: str | None = Field(default=None, min_length=1, max_length=255)
source_type: str | None = None
channel: str | None = Field(default=None, min_length=1, max_length=255)
is_active: bool | None = None
class ParseChannelRead(BaseModel):
model_config = ConfigDict(from_attributes=True)
id: int
name: str
source_type: str
channel: str
is_active: bool
created_at: datetime
class ParseJobCreate(BaseModel): class ParseJobCreate(BaseModel):
source_type: str = "telegram" source_type: str = "telegram"
source_config: dict[str, Any] = Field(default_factory=dict) source_config: dict[str, Any] = Field(default_factory=dict)
profile_id: int | None = None
channel_id: int | None = None
limit: int = Field(default=100, ge=1, le=1000)
schedule: str | None = None schedule: str | None = None
interval_seconds: int = Field(default=3600, ge=60, le=604800) interval_seconds: int = Field(default=3600, ge=60, le=604800)
is_active: bool = True is_active: bool = True
@@ -137,6 +222,9 @@ class ParseJobCreate(BaseModel):
class ParseJobUpdate(BaseModel): class ParseJobUpdate(BaseModel):
source_config: dict[str, Any] | None = None source_config: dict[str, Any] | None = None
profile_id: int | None = None
channel_id: int | None = None
limit: int | None = Field(default=None, ge=1, le=1000)
interval_seconds: int | None = Field(default=None, ge=60, le=604800) interval_seconds: int | None = Field(default=None, ge=60, le=604800)
is_active: bool | None = None is_active: bool | None = None
@@ -147,6 +235,8 @@ class ParseJobRead(BaseModel):
id: int id: int
source_type: str source_type: str
source_config: dict[str, Any] source_config: dict[str, Any]
profile_id: int | None = None
channel_id: int | None = None
schedule: str | None schedule: str | None
interval_seconds: int interval_seconds: int
is_active: bool is_active: bool
@@ -154,6 +244,9 @@ class ParseJobRead(BaseModel):
last_run_at: datetime | None last_run_at: datetime | None
last_error: str | None last_error: str | None
created_at: datetime created_at: datetime
profile_name: str | None = None
channel_name: str | None = None
channel_handle: str | None = None
class ConsumerCreate(BaseModel): class ConsumerCreate(BaseModel):
+52 -1
View File
@@ -4,6 +4,7 @@ import sys
from pathlib import Path from pathlib import Path
import redis import redis
from sqlalchemy.orm import Session
# Allow importing shared contracts from monorepo root in local runs # Allow importing shared contracts from monorepo root in local runs
_HERE = Path(__file__).resolve() _HERE = Path(__file__).resolve()
@@ -14,7 +15,9 @@ for _candidate in (_HERE.parent, *_HERE.parents):
break break
from contracts.queues import queue_key_for_source # noqa: E402 from contracts.queues import queue_key_for_source # noqa: E402
from contracts.sources import parse_source_config # noqa: E402 from contracts.sources import TelegramSourceConfig, parse_source_config # noqa: E402
from ..models import ParseChannel, ParseJob, ParserProfile # noqa: E402
REDIS_URL = os.getenv("REDIS_URL", "redis://redis:6379/0") REDIS_URL = os.getenv("REDIS_URL", "redis://redis:6379/0")
@@ -23,6 +26,47 @@ def get_redis() -> redis.Redis:
return redis.from_url(REDIS_URL, decode_responses=True) return redis.from_url(REDIS_URL, decode_responses=True)
def flatten_pair_config(
channel: ParseChannel,
profile: ParserProfile,
*,
limit: int = 100,
) -> dict:
"""Expand Profile + Channel into Redis/CP source_config."""
if not profile.heuristic_profile:
raise ValueError("ParserProfile.heuristic_profile is empty")
cfg = TelegramSourceConfig(
channel=channel.channel.strip(),
limit=limit,
extract_mode="profile",
heuristic_profile=profile.heuristic_profile,
sample_post=profile.sample_post or None,
)
return cfg.model_dump()
def resolve_job_source_config(db: Session, job: ParseJob) -> dict:
"""
For pair jobs, rebuild flat source_config from current Profile/Channel.
Legacy jobs keep stored source_config.
"""
if job.profile_id and job.channel_id:
profile = db.query(ParserProfile).filter(ParserProfile.id == job.profile_id).first()
channel = db.query(ParseChannel).filter(ParseChannel.id == job.channel_id).first()
if not profile:
raise ValueError(f"ParserProfile {job.profile_id} not found")
if not channel:
raise ValueError(f"ParseChannel {job.channel_id} not found")
existing = job.source_config or {}
limit = existing.get("limit", 100)
if not isinstance(limit, int) or limit < 1:
limit = 100
flattened = flatten_pair_config(channel, profile, limit=limit)
job.source_config = flattened
return flattened
return dict(job.source_config or {})
def enqueue_job(job_id: int, source_type: str, source_config: dict) -> None: def enqueue_job(job_id: int, source_type: str, source_config: dict) -> None:
# Validate known configs early; unknown types still raise from queue_key_for_source # Validate known configs early; unknown types still raise from queue_key_for_source
try: try:
@@ -38,3 +82,10 @@ def enqueue_job(job_id: int, source_type: str, source_config: dict) -> None:
} }
key = queue_key_for_source(source_type) key = queue_key_for_source(source_type)
get_redis().rpush(key, json.dumps(payload)) get_redis().rpush(key, json.dumps(payload))
def enqueue_parse_job(db: Session, job: ParseJob) -> None:
"""Resolve (flatten if pair) then push to Redis."""
source_config = resolve_job_source_config(db, job)
db.commit()
enqueue_job(job.id, job.source_type, source_config)
@@ -5,20 +5,29 @@ from sqlalchemy.engine import Engine
def migrate_schema(engine: Engine) -> None: def migrate_schema(engine: Engine) -> None:
"""Apply lightweight schema updates for existing deployments.""" """Apply lightweight schema updates for existing deployments."""
inspector = inspect(engine) inspector = inspect(engine)
if "parse_jobs" not in inspector.get_table_names(): tables = set(inspector.get_table_names())
return
columns = {col["name"] for col in inspector.get_columns("parse_jobs")}
statements: list[str] = [] statements: list[str] = []
if "interval_seconds" not in columns: if "parse_jobs" in tables:
statements.append( columns = {col["name"] for col in inspector.get_columns("parse_jobs")}
"ALTER TABLE parse_jobs ADD COLUMN interval_seconds INTEGER NOT NULL DEFAULT 3600" if "interval_seconds" not in columns:
) statements.append(
if "is_active" not in columns: "ALTER TABLE parse_jobs ADD COLUMN interval_seconds INTEGER NOT NULL DEFAULT 3600"
statements.append( )
"ALTER TABLE parse_jobs ADD COLUMN is_active BOOLEAN NOT NULL DEFAULT TRUE" if "is_active" not in columns:
) statements.append(
"ALTER TABLE parse_jobs ADD COLUMN is_active BOOLEAN NOT NULL DEFAULT TRUE"
)
if "profile_id" not in columns:
statements.append("ALTER TABLE parse_jobs ADD COLUMN profile_id INTEGER")
if "channel_id" not in columns:
statements.append("ALTER TABLE parse_jobs ADD COLUMN channel_id INTEGER")
# create_all handles new tables; FKs on existing DBs may need indexes
if "parse_jobs" in tables:
# Re-inspect after potential adds is not needed for FK constraints here —
# create_all + nullable FKs are enough for MVP; optional constraints below.
pass
if not statements: if not statements:
return return
@@ -5,7 +5,7 @@ from datetime import datetime, timezone
from ..database import SessionLocal from ..database import SessionLocal
from ..models import ParseJob from ..models import ParseJob
from .jobs import enqueue_job from .jobs import enqueue_parse_job
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -36,7 +36,14 @@ def run_scheduler_tick() -> None:
job.status = "queued" job.status = "queued"
job.last_error = None job.last_error = None
db.commit() db.commit()
enqueue_job(job.id, job.source_type, job.source_config) try:
enqueue_parse_job(db, job)
except ValueError as exc:
job.status = "failed"
job.last_error = str(exc)
db.commit()
logger.warning("Skip re-queue job %s: %s", job.id, exc)
continue
logger.info("Re-queued recurring job %s (interval %ss)", job.id, job.interval_seconds) logger.info("Re-queued recurring job %s (interval %ss)", job.id, job.interval_seconds)
except Exception: except Exception:
logger.exception("Scheduler tick failed") logger.exception("Scheduler tick failed")
@@ -4,9 +4,11 @@ import type {
Consumer, Consumer,
ConsumerCreate, ConsumerCreate,
ConsumerUpdate, ConsumerUpdate,
EventCreate,
EventFilters, EventFilters,
EventListResponse, EventListResponse,
EventRecord, EventRecord,
EventUpdate,
ParseJob, ParseJob,
ParseJobCreate, ParseJobCreate,
ParseJobUpdate, ParseJobUpdate,
@@ -69,6 +71,26 @@ export function fetchEvents(filters: EventFilters = {}): Promise<EventListRespon
); );
} }
export function createEvent(payload: EventCreate): Promise<EventRecord> {
return request<EventRecord>(
"/events",
{ method: "POST", body: JSON.stringify(payload) },
ADMIN_BASE,
);
}
export function updateEvent(id: number, payload: EventUpdate): Promise<EventRecord> {
return request<EventRecord>(
`/events/${id}`,
{ method: "PATCH", body: JSON.stringify(payload) },
ADMIN_BASE,
);
}
export function deleteEvent(id: number): Promise<void> {
return request<void>(`/events/${id}`, { method: "DELETE" }, ADMIN_BASE);
}
export function fetchConsumers(): Promise<Consumer[]> { export function fetchConsumers(): Promise<Consumer[]> {
return request<Consumer[]>("/consumers", undefined, ADMIN_BASE); return request<Consumer[]>("/consumers", undefined, ADMIN_BASE);
} }
@@ -0,0 +1,84 @@
import { ADMIN_BASE, request } from "./client";
import type {
HeuristicProfile,
ParseChannel,
ParseChannelCreate,
ParseChannelUpdate,
ParserProfile,
ParserProfileCreate,
ParserProfileUpdate,
TargetField,
} from "../types/admin";
export type { HeuristicProfile, TargetField } from "../types/admin";
export function fetchChannels(): Promise<ParseChannel[]> {
return request<ParseChannel[]>("/parse-channels", undefined, ADMIN_BASE);
}
export function createChannel(payload: ParseChannelCreate): Promise<ParseChannel> {
return request<ParseChannel>(
"/parse-channels",
{ method: "POST", body: JSON.stringify(payload) },
ADMIN_BASE,
);
}
export function updateChannel(id: number, payload: ParseChannelUpdate): Promise<ParseChannel> {
return request<ParseChannel>(
`/parse-channels/${id}`,
{ method: "PATCH", body: JSON.stringify(payload) },
ADMIN_BASE,
);
}
export function deleteChannel(id: number): Promise<void> {
return request<void>(`/parse-channels/${id}`, { method: "DELETE" }, ADMIN_BASE);
}
export function fetchProfiles(): Promise<ParserProfile[]> {
return request<ParserProfile[]>("/parser-profiles", undefined, ADMIN_BASE);
}
export function createProfile(payload: ParserProfileCreate): Promise<ParserProfile> {
return request<ParserProfile>(
"/parser-profiles",
{ method: "POST", body: JSON.stringify(payload) },
ADMIN_BASE,
);
}
export function updateProfile(id: number, payload: ParserProfileUpdate): Promise<ParserProfile> {
return request<ParserProfile>(
`/parser-profiles/${id}`,
{ method: "PATCH", body: JSON.stringify(payload) },
ADMIN_BASE,
);
}
export function deleteProfile(id: number): Promise<void> {
return request<void>(`/parser-profiles/${id}`, { method: "DELETE" }, ADMIN_BASE);
}
export function fetchTargetFields(): Promise<{ fields: TargetField[] }> {
return request<{ fields: TargetField[] }>("/parser-profiles/target-fields", undefined, ADMIN_BASE);
}
export function generateParserProfile(sample_post: string): Promise<{ profile: HeuristicProfile }> {
return request<{ profile: HeuristicProfile }>(
"/parser-profiles/generate",
{ method: "POST", body: JSON.stringify({ sample_post }) },
ADMIN_BASE,
);
}
export function previewParserProfile(
sample_post: string,
heuristic_profile: HeuristicProfile | Record<string, unknown>,
): Promise<{ fields: Record<string, string> }> {
return request<{ fields: Record<string, string> }>(
"/parser-profiles/preview",
{ method: "POST", body: JSON.stringify({ sample_post, heuristic_profile }) },
ADMIN_BASE,
);
}
@@ -1,52 +0,0 @@
import { ADMIN_BASE, request } from "./client";
import type { ParseJob } from "../types/admin";
export type TargetField = {
name: string;
type: string;
description: string;
};
export type HeuristicProfile = {
version: number;
fields: Record<string, Record<string, unknown>>;
notes?: string;
};
export function fetchTargetFields(): Promise<{ fields: TargetField[] }> {
return request<{ fields: TargetField[] }>("/parser-builder/target-fields", undefined, ADMIN_BASE);
}
export function generateParserProfile(sample_post: string): Promise<{ profile: HeuristicProfile }> {
return request<{ profile: HeuristicProfile }>(
"/parser-builder/generate",
{ method: "POST", body: JSON.stringify({ sample_post }) },
ADMIN_BASE,
);
}
export function previewParserProfile(
sample_post: string,
profile: HeuristicProfile | Record<string, unknown>,
): Promise<{ fields: Record<string, string> }> {
return request<{ fields: Record<string, string> }>(
"/parser-builder/preview",
{ method: "POST", body: JSON.stringify({ sample_post, profile }) },
ADMIN_BASE,
);
}
export function createProfileJob(payload: {
channel: string;
profile: HeuristicProfile | Record<string, unknown>;
sample_post?: string;
limit?: number;
interval_seconds?: number;
is_active?: boolean;
}): Promise<ParseJob> {
return request<ParseJob>(
"/parser-builder/jobs",
{ method: "POST", body: JSON.stringify(payload) },
ADMIN_BASE,
);
}
@@ -6,8 +6,9 @@ const route = useRoute();
const navItems = [ const navItems = [
{ to: "/", label: "Карта", exact: true }, { to: "/", label: "Карта", exact: true },
{ to: "/channels", label: "Каналы" },
{ to: "/parser-profiles", label: "Профили" },
{ to: "/parsers", label: "Парсеры" }, { to: "/parsers", label: "Парсеры" },
{ to: "/parser-builder", label: "Конструктор" },
{ to: "/events", label: "События" }, { to: "/events", label: "События" },
{ to: "/analytics", label: "Аналитика" }, { to: "/analytics", label: "Аналитика" },
{ to: "/consumers", label: "ПИ" }, { to: "/consumers", label: "ПИ" },
@@ -2,10 +2,11 @@ import { createRouter, createWebHistory } from "vue-router";
import AppLayout from "../layouts/AppLayout.vue"; import AppLayout from "../layouts/AppLayout.vue";
import AnalyticsView from "../views/AnalyticsView.vue"; import AnalyticsView from "../views/AnalyticsView.vue";
import ChannelsView from "../views/ChannelsView.vue";
import ConsumersView from "../views/ConsumersView.vue"; import ConsumersView from "../views/ConsumersView.vue";
import EventsView from "../views/EventsView.vue"; import EventsView from "../views/EventsView.vue";
import MapViewPage from "../views/MapViewPage.vue"; import MapViewPage from "../views/MapViewPage.vue";
import ParserBuilderView from "../views/ParserBuilderView.vue"; import ParserProfilesView from "../views/ParserProfilesView.vue";
import ParsersView from "../views/ParsersView.vue"; import ParsersView from "../views/ParsersView.vue";
const router = createRouter({ const router = createRouter({
@@ -16,8 +17,9 @@ const router = createRouter({
component: AppLayout, component: AppLayout,
children: [ children: [
{ path: "", name: "map", component: MapViewPage }, { path: "", name: "map", component: MapViewPage },
{ path: "channels", name: "channels", component: ChannelsView },
{ path: "parser-profiles", name: "parser-profiles", component: ParserProfilesView },
{ path: "parsers", name: "parsers", component: ParsersView }, { path: "parsers", name: "parsers", component: ParsersView },
{ path: "parser-builder", name: "parser-builder", component: ParserBuilderView },
{ path: "events", name: "events", component: EventsView }, { path: "events", name: "events", component: EventsView },
{ path: "analytics", name: "analytics", component: AnalyticsView }, { path: "analytics", name: "analytics", component: AnalyticsView },
{ path: "consumers", name: "consumers", component: ConsumersView }, { path: "consumers", name: "consumers", component: ConsumersView },
+9
View File
@@ -125,6 +125,15 @@ textarea {
font-size: 0.8rem; font-size: 0.8rem;
} }
.btn-danger {
color: #b71c1c;
border-color: #ef9a9a;
}
.btn-danger:hover {
background: #ffebee;
}
.link-btn { .link-btn {
background: none; background: none;
border: none; border: none;
+103 -2
View File
@@ -2,6 +2,8 @@ export interface ParseJob {
id: number; id: number;
source_type: string; source_type: string;
source_config: Record<string, unknown>; source_config: Record<string, unknown>;
profile_id?: number | null;
channel_id?: number | null;
schedule: string | null; schedule: string | null;
interval_seconds: number; interval_seconds: number;
is_active: boolean; is_active: boolean;
@@ -9,11 +11,17 @@ export interface ParseJob {
last_run_at: string | null; last_run_at: string | null;
last_error: string | null; last_error: string | null;
created_at: string; created_at: string;
profile_name?: string | null;
channel_name?: string | null;
channel_handle?: string | null;
} }
export interface ParseJobCreate { export interface ParseJobCreate {
source_type: string; source_type?: string;
source_config: Record<string, unknown>; source_config?: Record<string, unknown>;
profile_id?: number;
channel_id?: number;
limit?: number;
schedule?: string | null; schedule?: string | null;
interval_seconds?: number; interval_seconds?: number;
is_active?: boolean; is_active?: boolean;
@@ -21,10 +29,71 @@ export interface ParseJobCreate {
export interface ParseJobUpdate { export interface ParseJobUpdate {
source_config?: Record<string, unknown>; source_config?: Record<string, unknown>;
profile_id?: number;
channel_id?: number;
limit?: number;
interval_seconds?: number; interval_seconds?: number;
is_active?: boolean; is_active?: boolean;
} }
export interface ParserProfile {
id: number;
name: string;
sample_post: string;
heuristic_profile: Record<string, unknown> | null;
status: string;
created_at: string;
}
export interface ParserProfileCreate {
name: string;
sample_post?: string;
heuristic_profile?: Record<string, unknown> | null;
status?: string;
}
export interface ParserProfileUpdate {
name?: string;
sample_post?: string;
heuristic_profile?: Record<string, unknown> | null;
status?: string;
}
export interface ParseChannel {
id: number;
name: string;
source_type: string;
channel: string;
is_active: boolean;
created_at: string;
}
export interface ParseChannelCreate {
name: string;
source_type?: string;
channel: string;
is_active?: boolean;
}
export interface ParseChannelUpdate {
name?: string;
source_type?: string;
channel?: string;
is_active?: boolean;
}
export type TargetField = {
name: string;
type: string;
description: string;
};
export type HeuristicProfile = {
version: number;
fields: Record<string, Record<string, unknown>>;
notes?: string;
};
export interface EventRecord { export interface EventRecord {
id: number; id: number;
source_type: string; source_type: string;
@@ -43,6 +112,38 @@ export interface EventRecord {
metadata: Record<string, unknown> | null; metadata: Record<string, unknown> | null;
} }
export interface EventCreate {
source_type?: string;
source_url: string;
raw_text?: string;
title?: string;
description?: string;
locality?: string;
latitude?: number | null;
longitude?: number | null;
event_date?: string | null;
region?: string | null;
topic?: string | null;
tags?: string[] | null;
metadata?: Record<string, unknown> | null;
}
export interface EventUpdate {
source_type?: string;
source_url?: string;
raw_text?: string;
title?: string;
description?: string;
locality?: string;
latitude?: number | null;
longitude?: number | null;
event_date?: string | null;
region?: string | null;
topic?: string | null;
tags?: string[] | null;
metadata?: Record<string, unknown> | null;
}
export interface EventListResponse { export interface EventListResponse {
items: EventRecord[]; items: EventRecord[];
total: number; total: number;
@@ -0,0 +1,273 @@
<script setup lang="ts">
import { onMounted, ref } from "vue";
import {
createChannel,
deleteChannel,
fetchChannels,
updateChannel,
} from "../api/entities";
import type { ParseChannel } from "../types/admin";
const channels = ref<ParseChannel[]>([]);
const loading = ref(true);
const error = ref("");
const submitting = ref(false);
const form = ref({
name: "",
channel: "",
is_active: true,
});
const editModal = ref<{
open: boolean;
channel: ParseChannel | null;
name: string;
handle: string;
is_active: boolean;
}>({
open: false,
channel: null,
name: "",
handle: "",
is_active: true,
});
async function loadChannels() {
loading.value = true;
error.value = "";
try {
channels.value = await fetchChannels();
} catch (err) {
error.value = err instanceof Error ? err.message : "Не удалось загрузить каналы";
} finally {
loading.value = false;
}
}
async function handleCreate() {
if (!form.value.name.trim() || !form.value.channel.trim()) {
error.value = "Укажите название и handle канала";
return;
}
submitting.value = true;
error.value = "";
try {
await createChannel({
name: form.value.name.trim(),
channel: form.value.channel.trim(),
source_type: "telegram",
is_active: form.value.is_active,
});
form.value = { name: "", channel: "", is_active: true };
await loadChannels();
} catch (err) {
error.value = err instanceof Error ? err.message : "Не удалось создать канал";
} finally {
submitting.value = false;
}
}
function openEdit(ch: ParseChannel) {
editModal.value = {
open: true,
channel: ch,
name: ch.name,
handle: ch.channel,
is_active: ch.is_active,
};
}
function closeEdit() {
editModal.value.open = false;
editModal.value.channel = null;
}
async function handleSaveEdit() {
if (!editModal.value.channel) return;
if (!editModal.value.name.trim() || !editModal.value.handle.trim()) {
error.value = "Укажите название и handle";
return;
}
submitting.value = true;
error.value = "";
try {
await updateChannel(editModal.value.channel.id, {
name: editModal.value.name.trim(),
channel: editModal.value.handle.trim(),
is_active: editModal.value.is_active,
});
closeEdit();
await loadChannels();
} catch (err) {
error.value = err instanceof Error ? err.message : "Не удалось сохранить";
} finally {
submitting.value = false;
}
}
async function handleDelete(ch: ParseChannel) {
if (!window.confirm(`Удалить канал «${ch.name}» (@${ch.channel})?`)) return;
error.value = "";
try {
await deleteChannel(ch.id);
await loadChannels();
} catch (err) {
error.value = err instanceof Error ? err.message : "Не удалось удалить";
}
}
function formatDate(value: string): string {
return new Date(value).toLocaleString("ru-RU");
}
onMounted(loadChannels);
</script>
<template>
<div class="page">
<h2 class="page-heading">Каналы</h2>
<p class="intro">
Каналы для парсинга (Telegram). Связка канал + профиль создаётся на странице
<router-link to="/parsers">Парсеры</router-link>.
</p>
<section class="card">
<h3>Новый канал</h3>
<form class="admin-form" @submit.prevent="handleCreate">
<div class="form-row">
<label>
Название
<input v-model="form.name" type="text" placeholder="Сводки фронта" required />
</label>
<label>
Handle / id
<input v-model="form.channel" type="text" placeholder="example_channel" required />
</label>
<label class="checkbox-label">
<input v-model="form.is_active" type="checkbox" />
Активен
</label>
</div>
<div class="form-actions">
<button type="submit" class="btn btn-primary" :disabled="submitting">
{{ submitting ? "Создание..." : "Создать канал" }}
</button>
</div>
</form>
<p v-if="error" class="form-error">{{ error }}</p>
</section>
<section class="card">
<h3>Список <span v-if="loading" class="muted">(загрузка...)</span></h3>
<div class="table-wrap">
<table class="admin-table">
<thead>
<tr>
<th>ID</th>
<th>Название</th>
<th>Канал</th>
<th>Тип</th>
<th>Активен</th>
<th>Создан</th>
<th></th>
</tr>
</thead>
<tbody>
<tr v-for="ch in channels" :key="ch.id">
<td>{{ ch.id }}</td>
<td>{{ ch.name }}</td>
<td><code>@{{ ch.channel }}</code></td>
<td>{{ ch.source_type }}</td>
<td>{{ ch.is_active ? "да" : "нет" }}</td>
<td>{{ formatDate(ch.created_at) }}</td>
<td class="actions">
<button type="button" class="btn btn-sm" @click="openEdit(ch)">Изменить</button>
<button type="button" class="btn btn-sm btn-danger" @click="handleDelete(ch)">
Удалить
</button>
</td>
</tr>
<tr v-if="!loading && !channels.length">
<td colspan="7" class="muted">Пока нет каналов</td>
</tr>
</tbody>
</table>
</div>
</section>
<div v-if="editModal.open" class="modal-backdrop" @click.self="closeEdit">
<div class="modal">
<h3>Редактировать канал #{{ editModal.channel?.id }}</h3>
<form class="admin-form" @submit.prevent="handleSaveEdit">
<label>
Название
<input v-model="editModal.name" type="text" required />
</label>
<label>
Handle / id
<input v-model="editModal.handle" type="text" required />
</label>
<label class="checkbox-label">
<input v-model="editModal.is_active" type="checkbox" />
Активен
</label>
<div class="form-actions">
<button type="button" class="btn" @click="closeEdit">Отмена</button>
<button type="submit" class="btn btn-primary" :disabled="submitting">Сохранить</button>
</div>
</form>
</div>
</div>
</div>
</template>
<style scoped>
.intro {
margin: 0 0 1rem;
font-size: 0.9rem;
color: #555;
line-height: 1.5;
}
.checkbox-label {
display: flex;
align-items: center;
gap: 0.5rem;
margin-top: 1.5rem;
}
.actions {
display: flex;
gap: 0.35rem;
flex-wrap: wrap;
}
.modal-backdrop {
position: fixed;
inset: 0;
background: rgba(0, 0, 0, 0.35);
display: flex;
align-items: center;
justify-content: center;
z-index: 50;
}
.modal {
background: #fff;
border-radius: 8px;
padding: 1.25rem 1.5rem;
width: min(420px, 92vw);
box-shadow: 0 8px 24px rgba(0, 0, 0, 0.15);
}
.modal h3 {
margin: 0 0 1rem;
font-size: 1.05rem;
}
.modal .admin-form label {
display: block;
margin-bottom: 0.75rem;
}
</style>
@@ -1,7 +1,7 @@
<script setup lang="ts"> <script setup lang="ts">
import { computed, onMounted, ref, watch } from "vue"; import { computed, onMounted, ref, watch } from "vue";
import { useRouter } from "vue-router"; import { useRouter } from "vue-router";
import { fetchEvents } from "../api/admin"; import { createEvent, deleteEvent, fetchEvents, updateEvent } from "../api/admin";
import type { EventFilters, EventRecord } from "../types/admin"; import type { EventFilters, EventRecord } from "../types/admin";
const router = useRouter(); const router = useRouter();
@@ -10,6 +10,7 @@ const items = ref<EventRecord[]>([]);
const total = ref(0); const total = ref(0);
const loading = ref(true); const loading = ref(true);
const error = ref(""); const error = ref("");
const submitting = ref(false);
const filters = ref({ const filters = ref({
search: "", search: "",
@@ -24,8 +25,44 @@ const filters = ref({
const page = ref(1); const page = ref(1);
const pageSize = 25; const pageSize = 25;
const selectedEvent = ref<EventRecord | null>(null); type EventForm = {
const modalOpen = ref(false); source_type: string;
source_url: string;
title: string;
description: string;
locality: string;
region: string;
topic: string;
raw_text: string;
latitude: string;
longitude: string;
event_date: string;
tags: string;
};
function emptyForm(): EventForm {
return {
source_type: "manual",
source_url: "",
title: "",
description: "",
locality: "",
region: "",
topic: "",
raw_text: "",
latitude: "",
longitude: "",
event_date: "",
tags: "",
};
}
const createForm = ref(emptyForm());
const editModal = ref<{ open: boolean; event: EventRecord | null; form: EventForm }>({
open: false,
event: null,
form: emptyForm(),
});
const totalPages = computed(() => Math.max(1, Math.ceil(total.value / pageSize))); const totalPages = computed(() => Math.max(1, Math.ceil(total.value / pageSize)));
@@ -76,14 +113,128 @@ function resetFilters() {
loadEvents(); loadEvents();
} }
function openModal(event: EventRecord) { function parseOptionalFloat(value: string): number | null {
selectedEvent.value = event; const trimmed = value.trim();
modalOpen.value = true; if (!trimmed) return null;
const num = Number(trimmed);
if (Number.isNaN(num)) throw new Error("Координаты должны быть числами");
return num;
} }
function closeModal() { function parseTags(value: string): string[] | null {
modalOpen.value = false; const tags = value
selectedEvent.value = null; .split(",")
.map((t) => t.trim())
.filter(Boolean);
return tags.length ? tags : null;
}
function toDateTimeLocal(value: string | null): string {
if (!value) return "";
const d = new Date(value);
if (Number.isNaN(d.getTime())) return "";
const pad = (n: number) => String(n).padStart(2, "0");
return `${d.getFullYear()}-${pad(d.getMonth() + 1)}-${pad(d.getDate())}T${pad(d.getHours())}:${pad(d.getMinutes())}`;
}
function formToPayload(form: EventForm) {
const latitude = parseOptionalFloat(form.latitude);
const longitude = parseOptionalFloat(form.longitude);
if ((latitude == null) !== (longitude == null)) {
throw new Error("Укажите обе координаты или ни одной");
}
const source_url = form.source_url.trim();
if (!source_url) throw new Error("Укажите source_url");
return {
source_type: form.source_type.trim() || "manual",
source_url,
title: form.title.trim(),
description: form.description.trim(),
locality: form.locality.trim(),
region: form.region.trim() || null,
topic: form.topic.trim() || null,
raw_text: form.raw_text,
latitude,
longitude,
event_date: form.event_date ? new Date(form.event_date).toISOString() : null,
tags: parseTags(form.tags),
};
}
function eventToForm(event: EventRecord): EventForm {
return {
source_type: event.source_type || "manual",
source_url: event.source_url,
title: event.title || "",
description: event.description || "",
locality: event.locality || "",
region: event.region || "",
topic: event.topic || "",
raw_text: event.raw_text || "",
latitude: event.latitude != null ? String(event.latitude) : "",
longitude: event.longitude != null ? String(event.longitude) : "",
event_date: toDateTimeLocal(event.event_date),
tags: event.tags?.join(", ") ?? "",
};
}
async function handleCreate() {
submitting.value = true;
error.value = "";
try {
await createEvent(formToPayload(createForm.value));
createForm.value = emptyForm();
createForm.value.source_url = `manual://${crypto.randomUUID()}`;
await loadEvents();
} catch (err) {
error.value = err instanceof Error ? err.message : "Не удалось создать событие";
} finally {
submitting.value = false;
}
}
function openEdit(event: EventRecord) {
editModal.value = {
open: true,
event,
form: eventToForm(event),
};
}
function closeEdit() {
editModal.value.open = false;
editModal.value.event = null;
}
async function handleSaveEdit() {
if (!editModal.value.event) return;
submitting.value = true;
error.value = "";
try {
await updateEvent(editModal.value.event.id, formToPayload(editModal.value.form));
closeEdit();
await loadEvents();
} catch (err) {
error.value = err instanceof Error ? err.message : "Не удалось сохранить событие";
} finally {
submitting.value = false;
}
}
async function handleDelete(event: EventRecord) {
const label = event.title || event.source_url || `#${event.id}`;
if (!window.confirm(`Удалить событие #${event.id} («${label}»)? Связанная точка на карте тоже будет удалена.`)) {
return;
}
error.value = "";
try {
await deleteEvent(event.id);
if (editModal.value.event?.id === event.id) closeEdit();
await loadEvents();
} catch (err) {
error.value = err instanceof Error ? err.message : "Не удалось удалить событие";
}
} }
function goToMap(event: EventRecord) { function goToMap(event: EventRecord) {
@@ -103,13 +254,77 @@ function truncate(text: string, max = 80): string {
watch(page, loadEvents); watch(page, loadEvents);
onMounted(loadEvents); onMounted(() => {
createForm.value.source_url = `manual://${crypto.randomUUID()}`;
loadEvents();
});
</script> </script>
<template> <template>
<div class="page"> <div class="page">
<h2 class="page-heading">События</h2> <h2 class="page-heading">События</h2>
<section class="card">
<h3>Новое событие</h3>
<form class="admin-form" @submit.prevent="handleCreate">
<div class="form-row">
<label>
Заголовок
<input v-model="createForm.title" type="text" placeholder="Краткий заголовок" />
</label>
<label>
Источник (type)
<input v-model="createForm.source_type" type="text" placeholder="manual" />
</label>
<label class="span-2">
source_url
<input v-model="createForm.source_url" type="text" required />
</label>
<label class="span-2">
Описание
<textarea v-model="createForm.description" rows="2" />
</label>
<label>
Регион
<input v-model="createForm.region" type="text" />
</label>
<label>
Тема
<input v-model="createForm.topic" type="text" />
</label>
<label>
Населённый пункт
<input v-model="createForm.locality" type="text" />
</label>
<label>
Дата события
<input v-model="createForm.event_date" type="datetime-local" />
</label>
<label>
Широта
<input v-model="createForm.latitude" type="text" placeholder="48.1234" />
</label>
<label>
Долгота
<input v-model="createForm.longitude" type="text" placeholder="37.6543" />
</label>
<label>
Теги (через запятую)
<input v-model="createForm.tags" type="text" placeholder="обстрел, нп" />
</label>
<label class="span-2">
Сырой текст
<textarea v-model="createForm.raw_text" rows="3" />
</label>
</div>
<div class="form-actions">
<button type="submit" class="btn btn-primary" :disabled="submitting">
{{ submitting ? "Создание..." : "Создать событие" }}
</button>
</div>
</form>
</section>
<section class="card"> <section class="card">
<form class="admin-form filters-form" @submit.prevent="applyFilters"> <form class="admin-form filters-form" @submit.prevent="applyFilters">
<div class="form-row"> <div class="form-row">
@@ -177,7 +392,7 @@ onMounted(loadEvents);
<tr v-for="event in items" :key="event.id"> <tr v-for="event in items" :key="event.id">
<td>{{ event.id }}</td> <td>{{ event.id }}</td>
<td> <td>
<button class="link-btn" @click="openModal(event)"> <button class="link-btn" type="button" @click="openEdit(event)">
{{ truncate(event.title || event.description || "—") }} {{ truncate(event.title || event.description || "—") }}
</button> </button>
</td> </td>
@@ -191,14 +406,19 @@ onMounted(loadEvents);
</template> </template>
<template v-else>—</template> <template v-else>—</template>
</td> </td>
<td> <td class="actions">
<button class="btn btn-sm" type="button" @click="openEdit(event)">Изменить</button>
<button <button
v-if="event.latitude != null && event.longitude != null" v-if="event.latitude != null && event.longitude != null"
class="btn btn-sm" class="btn btn-sm"
type="button"
@click="goToMap(event)" @click="goToMap(event)"
> >
На карте На карте
</button> </button>
<button class="btn btn-sm btn-danger" type="button" @click="handleDelete(event)">
Удалить
</button>
</td> </td>
</tr> </tr>
</tbody> </tbody>
@@ -206,56 +426,129 @@ onMounted(loadEvents);
</div> </div>
<div class="pagination"> <div class="pagination">
<button class="btn btn-sm" :disabled="page <= 1" @click="page--">← Назад</button> <button class="btn btn-sm" type="button" :disabled="page <= 1" @click="page--">← Назад</button>
<span>Стр. {{ page }} / {{ totalPages }}</span> <span>Стр. {{ page }} / {{ totalPages }}</span>
<button class="btn btn-sm" :disabled="page >= totalPages" @click="page++">Вперёд →</button> <button class="btn btn-sm" type="button" :disabled="page >= totalPages" @click="page++">
Вперёд →
</button>
</div> </div>
</section> </section>
<div v-if="modalOpen && selectedEvent" class="modal-overlay" @click.self="closeModal"> <div v-if="editModal.open && editModal.event" class="modal-overlay" @click.self="closeEdit">
<div class="modal"> <div class="modal modal-wide">
<header class="modal-header"> <header class="modal-header">
<h3>Событие #{{ selectedEvent.id }}</h3> <h3>Событие #{{ editModal.event.id }}</h3>
<button class="btn btn-sm" @click="closeModal">✕</button> <button class="btn btn-sm" type="button" @click="closeEdit">✕</button>
</header> </header>
<div class="modal-body"> <form class="modal-body admin-form" @submit.prevent="handleSaveEdit">
<dl class="detail-list"> <div class="form-row">
<dt>Заголовок</dt> <label>
<dd>{{ selectedEvent.title || "—" }}</dd> Заголовок
<dt>Описание</dt> <input v-model="editModal.form.title" type="text" />
<dd>{{ selectedEvent.description || "—" }}</dd> </label>
<dt>Источник</dt> <label>
<dd>{{ selectedEvent.source_type }} — {{ selectedEvent.source_url }}</dd> Источник (type)
<dt>Регион / Тема</dt> <input v-model="editModal.form.source_type" type="text" />
<dd>{{ selectedEvent.region ?? "—" }} / {{ selectedEvent.topic ?? "—" }}</dd> </label>
<dt>Населённый пункт</dt> <label class="span-2">
<dd>{{ selectedEvent.locality || "—" }}</dd> source_url
<dt>Дата события</dt> <input v-model="editModal.form.source_url" type="text" required />
<dd>{{ formatDate(selectedEvent.event_date) }}</dd> </label>
<dt>Ingested</dt> <label class="span-2">
<dd>{{ formatDate(selectedEvent.ingested_at) }}</dd> Описание
<dt>Координаты</dt> <textarea v-model="editModal.form.description" rows="2" />
<dd> </label>
<template v-if="selectedEvent.latitude != null"> <label>
{{ selectedEvent.latitude }}, {{ selectedEvent.longitude }} Регион
</template> <input v-model="editModal.form.region" type="text" />
<template v-else>—</template> </label>
</dd> <label>
<dt>Текст</dt> Тема
<dd class="raw-text">{{ selectedEvent.raw_text || "—" }}</dd> <input v-model="editModal.form.topic" type="text" />
</dl> </label>
</div> <label>
<footer class="modal-footer"> Населённый пункт
<button <input v-model="editModal.form.locality" type="text" />
v-if="selectedEvent.latitude != null" </label>
class="btn btn-primary" <label>
@click="goToMap(selectedEvent)" Дата события
> <input v-model="editModal.form.event_date" type="datetime-local" />
Показать на карте </label>
</button> <label>
<button class="btn" @click="closeModal">Закрыть</button> Широта
</footer> <input v-model="editModal.form.latitude" type="text" />
</label>
<label>
Долгота
<input v-model="editModal.form.longitude" type="text" />
</label>
<label>
Теги
<input v-model="editModal.form.tags" type="text" />
</label>
<label class="span-2">
Сырой текст
<textarea v-model="editModal.form.raw_text" rows="4" />
</label>
<p class="hint span-2">
Ingested: {{ formatDate(editModal.event.ingested_at) }}. При координатах создаётся/обновляется
точка на карте; без координат связанная точка удаляется.
</p>
</div>
<div class="form-actions">
<button type="button" class="btn" @click="closeEdit">Отмена</button>
<button
v-if="editModal.event.latitude != null"
type="button"
class="btn"
@click="goToMap(editModal.event)"
>
На карте
</button>
<button type="submit" class="btn btn-primary" :disabled="submitting">
{{ submitting ? "Сохранение..." : "Сохранить" }}
</button>
</div>
</form>
</div> </div>
</div> </div>
</div> </div>
</template> </template>
<style scoped>
.form-row .span-2 {
grid-column: span 2;
}
.form-row textarea {
width: 100%;
font: inherit;
padding: 0.4rem 0.5rem;
border: 1px solid #d1d5db;
border-radius: 4px;
resize: vertical;
}
.actions {
display: flex;
gap: 0.35rem;
flex-wrap: wrap;
white-space: nowrap;
}
.hint {
margin: 0;
font-size: 0.85rem;
color: #666;
line-height: 1.4;
}
.modal-wide {
width: min(720px, 94vw);
}
.mono {
font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
font-size: 0.8rem;
}
</style>
@@ -1,320 +0,0 @@
<script setup lang="ts">
import { onMounted, ref } from "vue";
import { useRouter } from "vue-router";
import {
createProfileJob,
fetchTargetFields,
generateParserProfile,
previewParserProfile,
type HeuristicProfile,
type TargetField,
} from "../api/parserBuilder";
const INTERVAL_OPTIONS = [
{ value: 900, label: "15 мин" },
{ value: 1800, label: "30 мин" },
{ value: 3600, label: "1 ч" },
{ value: 10800, label: "3 ч" },
{ value: 21600, label: "6 ч" },
{ value: 86400, label: "24 ч" },
];
const router = useRouter();
const samplePost = ref("");
const targetFields = ref<TargetField[]>([]);
const profile = ref<HeuristicProfile | null>(null);
const profileJson = ref("");
const previewFields = ref<Record<string, string> | null>(null);
const channel = ref("");
const limit = ref(100);
const intervalSeconds = ref(3600);
const loadingFields = ref(true);
const generating = ref(false);
const previewing = ref(false);
const saving = ref(false);
const error = ref("");
const success = ref("");
async function loadTargetFields() {
loadingFields.value = true;
try {
const data = await fetchTargetFields();
targetFields.value = data.fields;
error.value = "";
} catch (err) {
error.value = err instanceof Error ? err.message : "Не удалось загрузить целевые поля";
} finally {
loadingFields.value = false;
}
}
function syncProfileFromJson(): HeuristicProfile | null {
if (!profileJson.value.trim()) return null;
try {
const parsed = JSON.parse(profileJson.value) as HeuristicProfile;
profile.value = parsed;
error.value = "";
return parsed;
} catch {
error.value = "Некорректный JSON профиля";
return null;
}
}
async function handleGenerate() {
if (!samplePost.value.trim()) {
error.value = "Вставьте образец поста";
return;
}
generating.value = true;
error.value = "";
success.value = "";
previewFields.value = null;
try {
const data = await generateParserProfile(samplePost.value);
profile.value = data.profile;
profileJson.value = JSON.stringify(data.profile, null, 2);
success.value = "Профиль сгенерирован. Preview и runtime идут только по правилам — без LLM.";
} catch (err) {
error.value = err instanceof Error ? err.message : "Ошибка генерации";
} finally {
generating.value = false;
}
}
async function handlePreview() {
const current = syncProfileFromJson();
if (!current) {
if (!error.value) error.value = "Сначала сгенерируйте профиль";
return;
}
if (!samplePost.value.trim()) {
error.value = "Нужен образец поста для preview";
return;
}
previewing.value = true;
error.value = "";
try {
const data = await previewParserProfile(samplePost.value, current);
previewFields.value = data.fields;
} catch (err) {
error.value = err instanceof Error ? err.message : "Ошибка preview";
} finally {
previewing.value = false;
}
}
async function handleCreate() {
const current = syncProfileFromJson();
if (!current) {
if (!error.value) error.value = "Нужен профиль";
return;
}
if (!channel.value.trim()) {
error.value = "Укажите канал Telegram";
return;
}
saving.value = true;
error.value = "";
success.value = "";
try {
const job = await createProfileJob({
channel: channel.value.trim(),
profile: current,
sample_post: samplePost.value.trim() || undefined,
limit: limit.value,
interval_seconds: intervalSeconds.value,
is_active: true,
});
success.value = `Парсер #${job.id} создан (extract_mode=profile).`;
await router.push({ name: "parsers" });
} catch (err) {
error.value = err instanceof Error ? err.message : "Не удалось создать парсер";
} finally {
saving.value = false;
}
}
onMounted(loadTargetFields);
</script>
<template>
<div class="page">
<h2 class="page-heading">Конструктор парсера</h2>
<p class="intro">
Вставьте образец структурированного поста → DeepSeek один раз построит статичный профиль правил.
Дальше preview и парсинг канала идут только по профилю, без LLM на каждый пост.
</p>
<section class="card">
<h3>Образец поста</h3>
<textarea
v-model="samplePost"
class="sample-area"
rows="12"
placeholder="15.03.2024 Населённый пункт&#10;&#10;Описание события…&#10;&#10;48.123456, 37.654321&#10;#тема"
/>
<div class="form-actions">
<button
type="button"
class="btn btn-primary"
:disabled="generating || !samplePost.trim()"
@click="handleGenerate"
>
{{ generating ? "Генерация..." : "Сгенерировать парсер" }}
</button>
<button
type="button"
class="btn"
:disabled="previewing || !profileJson.trim()"
@click="handlePreview"
>
{{ previewing ? "Preview..." : "Preview" }}
</button>
</div>
</section>
<div class="grid-2">
<section class="card">
<h3>Целевая таблица <span class="muted">(Event / ingest)</span></h3>
<p v-if="loadingFields" class="muted">Загрузка…</p>
<div v-else class="table-wrap">
<table class="admin-table">
<thead>
<tr>
<th>Поле</th>
<th>Тип</th>
<th>Описание</th>
</tr>
</thead>
<tbody>
<tr v-for="field in targetFields" :key="field.name">
<td><code>{{ field.name }}</code></td>
<td>{{ field.type }}</td>
<td>{{ field.description }}</td>
</tr>
</tbody>
</table>
</div>
</section>
<section class="card">
<h3>Профиль (JSON)</h3>
<p v-if="!profileJson" class="muted">Появится после генерации. Можно слегка поправить правила.</p>
<textarea
v-model="profileJson"
class="profile-area"
rows="16"
:disabled="!profileJson && !profile"
spellcheck="false"
/>
</section>
</div>
<section v-if="previewFields" class="card">
<h3>Preview строки</h3>
<div class="table-wrap">
<table class="admin-table">
<thead>
<tr>
<th v-for="key in Object.keys(previewFields)" :key="key">{{ key }}</th>
</tr>
</thead>
<tbody>
<tr>
<td v-for="(val, key) in previewFields" :key="key" class="preview-cell">
{{ val || "—" }}
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section class="card">
<h3>Создать Telegram-парсер</h3>
<form class="admin-form" @submit.prevent="handleCreate">
<div class="form-row">
<label>
Канал
<input v-model="channel" type="text" placeholder="example_channel" required />
</label>
<label>
Лимит batch
<input v-model.number="limit" type="number" min="1" max="1000" />
</label>
<label>
Интервал
<select v-model.number="intervalSeconds">
<option v-for="opt in INTERVAL_OPTIONS" :key="opt.value" :value="opt.value">
{{ opt.label }}
</option>
</select>
</label>
</div>
<div class="form-actions">
<button
type="submit"
class="btn btn-primary"
:disabled="saving || !profileJson.trim() || !channel.trim()"
>
{{ saving ? "Создание..." : "Создать парсер" }}
</button>
</div>
</form>
</section>
<p v-if="error" class="form-error">{{ error }}</p>
<p v-if="success" class="form-success">{{ success }}</p>
</div>
</template>
<style scoped>
.intro {
margin: 0 0 1rem;
font-size: 0.9rem;
color: #555;
line-height: 1.5;
max-width: 72ch;
}
.sample-area,
.profile-area {
width: 100%;
font: inherit;
font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
font-size: 0.8rem;
padding: 0.6rem 0.75rem;
border: 1px solid #d1d5db;
border-radius: 6px;
resize: vertical;
line-height: 1.45;
}
.grid-2 {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 1rem;
}
@media (max-width: 960px) {
.grid-2 {
grid-template-columns: 1fr;
}
}
.preview-cell {
max-width: 180px;
white-space: pre-wrap;
word-break: break-word;
font-size: 0.8rem;
}
.form-success {
color: #2e7d32;
font-size: 0.875rem;
margin-top: 0.5rem;
}
</style>
@@ -0,0 +1,422 @@
<script setup lang="ts">
import { onMounted, ref } from "vue";
import {
createProfile,
deleteProfile,
fetchProfiles,
fetchTargetFields,
generateParserProfile,
previewParserProfile,
updateProfile,
type HeuristicProfile,
type TargetField,
} from "../api/entities";
import type { ParserProfile } from "../types/admin";
const profiles = ref<ParserProfile[]>([]);
const loading = ref(true);
const error = ref("");
const success = ref("");
const submitting = ref(false);
const samplePost = ref("");
const profileName = ref("");
const targetFields = ref<TargetField[]>([]);
const profileJson = ref("");
const previewFields = ref<Record<string, string> | null>(null);
const generating = ref(false);
const previewing = ref(false);
const loadingFields = ref(true);
const editingId = ref<number | null>(null);
const profileStatus = ref<"draft" | "ready">("draft");
async function loadProfiles() {
loading.value = true;
try {
profiles.value = await fetchProfiles();
error.value = "";
} catch (err) {
error.value = err instanceof Error ? err.message : "Не удалось загрузить профили";
} finally {
loading.value = false;
}
}
async function loadTargetFields() {
loadingFields.value = true;
try {
const data = await fetchTargetFields();
targetFields.value = data.fields;
} catch (err) {
error.value = err instanceof Error ? err.message : "Не удалось загрузить целевые поля";
} finally {
loadingFields.value = false;
}
}
function syncProfileFromJson(): HeuristicProfile | null {
if (!profileJson.value.trim()) return null;
try {
return JSON.parse(profileJson.value) as HeuristicProfile;
} catch {
error.value = "Некорректный JSON профиля";
return null;
}
}
async function handleGenerate() {
if (!samplePost.value.trim()) {
error.value = "Вставьте образец поста";
return;
}
generating.value = true;
error.value = "";
success.value = "";
previewFields.value = null;
try {
const data = await generateParserProfile(samplePost.value);
profileJson.value = JSON.stringify(data.profile, null, 2);
success.value = "Профиль сгенерирован. Preview и runtime — только по правилам, без LLM.";
} catch (err) {
error.value = err instanceof Error ? err.message : "Ошибка генерации";
} finally {
generating.value = false;
}
}
async function handlePreview() {
const current = syncProfileFromJson();
if (!current) {
if (!error.value) error.value = "Сначала сгенерируйте или вставьте профиль";
return;
}
if (!samplePost.value.trim()) {
error.value = "Нужен образец поста для preview";
return;
}
previewing.value = true;
error.value = "";
try {
const data = await previewParserProfile(samplePost.value, current);
previewFields.value = data.fields;
} catch (err) {
error.value = err instanceof Error ? err.message : "Ошибка preview";
} finally {
previewing.value = false;
}
}
function resetEditor() {
editingId.value = null;
profileName.value = "";
samplePost.value = "";
profileJson.value = "";
profileStatus.value = "draft";
previewFields.value = null;
success.value = "";
}
function openEdit(p: ParserProfile) {
editingId.value = p.id;
profileName.value = p.name;
samplePost.value = p.sample_post || "";
profileJson.value = p.heuristic_profile
? JSON.stringify(p.heuristic_profile, null, 2)
: "";
profileStatus.value = p.status === "ready" ? "ready" : "draft";
previewFields.value = null;
success.value = "";
error.value = "";
window.scrollTo({ top: 0, behavior: "smooth" });
}
async function handleSave() {
if (!profileName.value.trim()) {
error.value = "Укажите название профиля";
return;
}
const current = syncProfileFromJson();
if (!current) {
if (!error.value) error.value = "Нужен JSON профиля (Generate или вручную)";
return;
}
submitting.value = true;
error.value = "";
success.value = "";
try {
if (editingId.value != null) {
await updateProfile(editingId.value, {
name: profileName.value.trim(),
sample_post: samplePost.value,
heuristic_profile: current,
status: profileStatus.value,
});
success.value = `Профиль #${editingId.value} обновлён`;
} else {
const created = await createProfile({
name: profileName.value.trim(),
sample_post: samplePost.value,
heuristic_profile: current,
status: profileStatus.value === "draft" ? "draft" : "ready",
});
success.value = `Профиль #${created.id} сохранён`;
editingId.value = created.id;
profileStatus.value = created.status === "ready" ? "ready" : "draft";
}
await loadProfiles();
} catch (err) {
error.value = err instanceof Error ? err.message : "Не удалось сохранить профиль";
} finally {
submitting.value = false;
}
}
async function handleDelete(p: ParserProfile) {
if (!window.confirm(`Удалить профиль «${p.name}»?`)) return;
error.value = "";
try {
await deleteProfile(p.id);
if (editingId.value === p.id) resetEditor();
await loadProfiles();
} catch (err) {
error.value = err instanceof Error ? err.message : "Не удалось удалить";
}
}
function formatDate(value: string): string {
return new Date(value).toLocaleString("ru-RU");
}
onMounted(() => {
loadProfiles();
loadTargetFields();
});
</script>
<template>
<div class="page">
<h2 class="page-heading">Профили парсера</h2>
<p class="intro">
Статичный эвристический профиль: Generate (DeepSeek один раз) → Preview → Сохранить.
Дальше профиль подключается к каналу на странице
<router-link to="/parsers">Парсеры</router-link>.
</p>
<section class="card">
<div class="card-head">
<h3>{{ editingId != null ? `Редактирование #${editingId}` : "Новый профиль" }}</h3>
<button v-if="editingId != null" type="button" class="btn btn-sm" @click="resetEditor">
Сбросить
</button>
</div>
<div class="form-row">
<label>
Название
<input v-model="profileName" type="text" placeholder="Сводка: дата + НП + coords" />
</label>
<label>
Статус
<select v-model="profileStatus">
<option value="draft">draft</option>
<option value="ready">ready</option>
</select>
</label>
</div>
<label>
Образец поста
<textarea
v-model="samplePost"
class="sample-area"
rows="10"
placeholder="15.03.2024 Населённый пункт&#10;&#10;Описание…&#10;&#10;48.123456, 37.654321"
/>
</label>
<div class="form-actions">
<button
type="button"
class="btn btn-primary"
:disabled="generating || !samplePost.trim()"
@click="handleGenerate"
>
{{ generating ? "Генерация..." : "Generate" }}
</button>
<button
type="button"
class="btn"
:disabled="previewing || !profileJson.trim()"
@click="handlePreview"
>
{{ previewing ? "Preview..." : "Preview" }}
</button>
<button
type="button"
class="btn btn-primary"
:disabled="submitting || !profileJson.trim() || !profileName.trim()"
@click="handleSave"
>
{{ submitting ? "Сохранение..." : editingId != null ? "Обновить" : "Сохранить профиль" }}
</button>
</div>
</section>
<div class="grid-2">
<section class="card">
<h3>Целевые поля <span class="muted">(Event)</span></h3>
<p v-if="loadingFields" class="muted">Загрузка…</p>
<div v-else class="table-wrap">
<table class="admin-table">
<thead>
<tr>
<th>Поле</th>
<th>Тип</th>
<th>Описание</th>
</tr>
</thead>
<tbody>
<tr v-for="field in targetFields" :key="field.name">
<td><code>{{ field.name }}</code></td>
<td>{{ field.type }}</td>
<td>{{ field.description }}</td>
</tr>
</tbody>
</table>
</div>
</section>
<section class="card">
<h3>Профиль (JSON)</h3>
<textarea
v-model="profileJson"
class="profile-area"
rows="16"
spellcheck="false"
placeholder="Появится после Generate"
/>
</section>
</div>
<section v-if="previewFields" class="card">
<h3>Preview строки</h3>
<div class="table-wrap">
<table class="admin-table">
<thead>
<tr>
<th v-for="key in Object.keys(previewFields)" :key="key">{{ key }}</th>
</tr>
</thead>
<tbody>
<tr>
<td v-for="(val, key) in previewFields" :key="key" class="preview-cell">
{{ val || "—" }}
</td>
</tr>
</tbody>
</table>
</div>
</section>
<p v-if="error" class="form-error">{{ error }}</p>
<p v-if="success" class="form-success">{{ success }}</p>
<section class="card">
<h3>Сохранённые профили <span v-if="loading" class="muted">(загрузка...)</span></h3>
<div class="table-wrap">
<table class="admin-table">
<thead>
<tr>
<th>ID</th>
<th>Название</th>
<th>Статус</th>
<th>Создан</th>
<th></th>
</tr>
</thead>
<tbody>
<tr v-for="p in profiles" :key="p.id">
<td>{{ p.id }}</td>
<td>{{ p.name }}</td>
<td>{{ p.status }}</td>
<td>{{ formatDate(p.created_at) }}</td>
<td class="actions">
<button type="button" class="btn btn-sm" @click="openEdit(p)">Изменить</button>
<button type="button" class="btn btn-sm btn-danger" @click="handleDelete(p)">
Удалить
</button>
</td>
</tr>
<tr v-if="!loading && !profiles.length">
<td colspan="5" class="muted">Пока нет профилей</td>
</tr>
</tbody>
</table>
</div>
</section>
</div>
</template>
<style scoped>
.intro {
margin: 0 0 1rem;
font-size: 0.9rem;
color: #555;
line-height: 1.5;
max-width: 72ch;
}
.card-head {
display: flex;
align-items: center;
justify-content: space-between;
gap: 0.75rem;
margin-bottom: 0.5rem;
}
.card-head h3 {
margin: 0;
}
.sample-area,
.profile-area {
width: 100%;
font: inherit;
font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
font-size: 0.8rem;
padding: 0.6rem 0.75rem;
border: 1px solid #d1d5db;
border-radius: 6px;
resize: vertical;
line-height: 1.45;
margin-top: 0.35rem;
}
.grid-2 {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 1rem;
}
@media (max-width: 960px) {
.grid-2 {
grid-template-columns: 1fr;
}
}
.preview-cell {
max-width: 180px;
white-space: pre-wrap;
word-break: break-word;
font-size: 0.8rem;
}
.form-success {
color: #2e7d32;
font-size: 0.875rem;
margin-top: 0.5rem;
}
.actions {
display: flex;
gap: 0.35rem;
flex-wrap: wrap;
}
</style>
@@ -1,7 +1,8 @@
<script setup lang="ts"> <script setup lang="ts">
import { computed, onMounted, onUnmounted, ref } from "vue"; import { computed, onMounted, onUnmounted, ref } from "vue";
import { createJob, deleteJob, fetchJobs, retryJob, updateJob } from "../api/admin"; import { createJob, deleteJob, fetchJobs, retryJob, updateJob } from "../api/admin";
import type { ParseJob } from "../types/admin"; import { fetchChannels, fetchProfiles } from "../api/entities";
import type { ParseChannel, ParseJob, ParserProfile } from "../types/admin";
const INTERVAL_OPTIONS = [ const INTERVAL_OPTIONS = [
{ value: 900, label: "15 мин" }, { value: 900, label: "15 мин" },
@@ -12,105 +13,65 @@ const INTERVAL_OPTIONS = [
{ value: 86400, label: "24 ч" }, { value: 86400, label: "24 ч" },
]; ];
const SOURCE_TYPES = [
{ value: "telegram", label: "Telegram" },
{ value: "crawl4ai", label: "Crawl4AI (web)" },
{ value: "viina", label: "VIINA (news NLP)" },
] as const;
const jobs = ref<ParseJob[]>([]); const jobs = ref<ParseJob[]>([]);
const channels = ref<ParseChannel[]>([]);
const profiles = ref<ParserProfile[]>([]);
const loading = ref(true); const loading = ref(true);
const error = ref(""); const error = ref("");
const submitting = ref(false); const submitting = ref(false);
const forcingJobId = ref<number | null>(null); const forcingJobId = ref<number | null>(null);
const editingJob = ref<ParseJob | null>(null); const editingJob = ref<ParseJob | null>(null);
const pairForm = ref({
channel_id: null as number | null,
profile_id: null as number | null,
limit: 100,
interval_seconds: 3600,
});
const editForm = ref({ const editForm = ref({
channel: "", channel_id: null as number | null,
limit: 50, profile_id: null as number | null,
urlsText: "", limit: 100,
textsText: "",
domain_profile: "generic_news",
input_mode: "urls" as "urls" | "texts" | "mixed",
extract_mode: "heuristic" as "heuristic" | "llm",
interval_seconds: 3600, interval_seconds: 3600,
is_active: true, is_active: true,
}); });
const form = ref({ const readyProfiles = computed(() =>
source_type: "telegram" as "telegram" | "crawl4ai" | "viina", profiles.value.filter((p) => !!p.heuristic_profile),
channel: "", );
limit: 50, const activeChannels = computed(() => channels.value.filter((c) => c.is_active));
urlsText: "",
textsText: "", const editChannelOptions = computed(() => channels.value);
domain_profile: "generic_news",
input_mode: "urls" as "urls" | "texts" | "mixed", const editProfileOptions = computed(() => {
extract_mode: "heuristic" as "heuristic" | "llm", const list = [...readyProfiles.value];
interval_seconds: 3600, const id = editForm.value.profile_id;
if (id != null && !list.some((p) => p.id === id)) {
const found = profiles.value.find((p) => p.id === id);
if (found) list.unshift(found);
}
return list;
}); });
let pollTimer: ReturnType<typeof setInterval> | null = null; let pollTimer: ReturnType<typeof setInterval> | null = null;
function channelFromConfig(config: Record<string, unknown>): string { function isPairJob(job: ParseJob | null): boolean {
return typeof config.channel === "string" ? config.channel : ""; return !!job && job.profile_id != null && job.channel_id != null;
} }
function limitFromConfig(config: Record<string, unknown>): number { function limitFromConfig(config: Record<string, unknown>): number {
const limit = config.limit; const limit = config.limit;
return typeof limit === "number" ? limit : 50; return typeof limit === "number" ? limit : 100;
}
function urlsFromConfig(config: Record<string, unknown>): string[] {
return Array.isArray(config.urls)
? config.urls.filter((u): u is string => typeof u === "string")
: [];
}
function textsFromConfig(config: Record<string, unknown>): string[] {
return Array.isArray(config.texts)
? config.texts.filter((t): t is string => typeof t === "string")
: [];
}
function parseLines(text: string): string[] {
return text
.split(/\r?\n/)
.map((line) => line.trim())
.filter(Boolean);
}
function extractModeLabel(cfg: Record<string, unknown>): string {
if (cfg.extract_mode === "llm") return " [LLM]";
if (cfg.extract_mode === "profile") return " [profile]";
return "";
} }
function sourceSummary(job: ParseJob): string { function sourceSummary(job: ParseJob): string {
const cfg = job.source_config; if (isPairJob(job)) {
const mode = extractModeLabel(cfg); const ch = job.channel_name || job.channel_handle || `#${job.channel_id}`;
if (job.source_type === "telegram") { const pr = job.profile_name || `#${job.profile_id}`;
return (channelFromConfig(cfg) || "—") + mode; return `${ch} × ${pr}`;
}
const urls = urlsFromConfig(cfg);
if (urls.length) return (urls.length === 1 ? urls[0] : `${urls.length} URL`) + mode;
const texts = textsFromConfig(cfg);
if (texts.length) return `${texts.length} текст(ов)`;
return "—";
}
function isProfileJob(job: ParseJob | null): boolean {
return !!job && job.source_config.extract_mode === "profile";
}
function profileJson(job: ParseJob | null): string {
if (!job) return "";
const profile = job.source_config.heuristic_profile;
if (!profile || typeof profile !== "object") return "";
try {
return JSON.stringify(profile, null, 2);
} catch {
return "";
} }
return `${job.source_type} (без связки)`;
} }
function formatInterval(seconds: number): string { function formatInterval(seconds: number): string {
@@ -121,64 +82,6 @@ function formatInterval(seconds: number): string {
return `${Math.round(seconds / 86400)} д`; return `${Math.round(seconds / 86400)} д`;
} }
function buildSourceConfig(
sourceType: string,
data: {
channel: string;
limit: number;
urlsText: string;
textsText: string;
domain_profile: string;
input_mode: "urls" | "texts" | "mixed";
extract_mode: "heuristic" | "llm";
},
existing?: Record<string, unknown>,
): Record<string, unknown> {
if (sourceType === "telegram") {
// Preserve builder-created profile jobs (edit form has no profile editor)
if (existing?.extract_mode === "profile" && existing.heuristic_profile) {
return {
channel: data.channel.trim(),
limit: data.limit,
extract_mode: "profile",
heuristic_profile: existing.heuristic_profile,
sample_post: existing.sample_post ?? null,
};
}
return {
channel: data.channel.trim(),
limit: data.limit,
extract_mode: data.extract_mode,
};
}
if (sourceType === "crawl4ai") {
return {
urls: parseLines(data.urlsText),
domain_profile: data.domain_profile.trim() || "generic_news",
extract_mode: data.extract_mode,
};
}
return {
urls: parseLines(data.urlsText),
texts: parseLines(data.textsText),
input_mode: data.input_mode,
};
}
const formHint = computed(() => {
if (form.value.source_type === "telegram") {
return form.value.extract_mode === "llm"
? "LLM (DeepSeek): неструктурированные посты канала разбираются в locality/date/coords/description."
: "Активные Telegram-парсеры подхватываются real-time listener; batch забирает последние N постов (эвристики).";
}
if (form.value.source_type === "crawl4ai") {
return form.value.extract_mode === "llm"
? "Crawl4AI + LLM (DeepSeek): страница → structured event. Нужен DEEPSEEK_API_KEY и cp-workers-web."
: "Crawl4AI обходит URL и нормализует страницы эвристиками. Обрабатывает cp-workers-web.";
}
return "VIINA-адаптер извлекает инциденты из новостных URL/текстов. Обрабатывает cp-workers-nlp.";
});
async function loadJobs() { async function loadJobs() {
try { try {
jobs.value = await fetchJobs(); jobs.value = await fetchJobs();
@@ -190,68 +93,51 @@ async function loadJobs() {
} }
} }
async function handleSubmit() { async function loadEntities() {
if (form.value.source_type === "telegram" && !form.value.channel.trim()) { try {
error.value = "Укажите канал Telegram"; const [chs, prs] = await Promise.all([fetchChannels(), fetchProfiles()]);
return; channels.value = chs;
} profiles.value = prs;
if (form.value.source_type === "crawl4ai" && !parseLines(form.value.urlsText).length) { } catch {
error.value = "Укажите хотя бы один URL"; /* selects stay empty */
return;
}
if (form.value.source_type === "viina") {
const urls = parseLines(form.value.urlsText);
const texts = parseLines(form.value.textsText);
if (form.value.input_mode === "urls" && !urls.length) {
error.value = "Укажите URL для VIINA";
return;
}
if (form.value.input_mode === "texts" && !texts.length) {
error.value = "Укажите тексты для VIINA";
return;
}
if (form.value.input_mode === "mixed" && !urls.length && !texts.length) {
error.value = "Укажите URL или тексты";
return;
}
} }
}
async function handleCreatePair() {
if (!pairForm.value.channel_id || !pairForm.value.profile_id) {
error.value = "Выберите канал и профиль";
return;
}
submitting.value = true; submitting.value = true;
error.value = ""; error.value = "";
try { try {
await createJob({ await createJob({
source_type: form.value.source_type, channel_id: pairForm.value.channel_id,
source_config: buildSourceConfig(form.value.source_type, form.value), profile_id: pairForm.value.profile_id,
interval_seconds: form.value.interval_seconds, limit: pairForm.value.limit,
interval_seconds: pairForm.value.interval_seconds,
is_active: true, is_active: true,
}); });
form.value.channel = ""; pairForm.value.channel_id = null;
form.value.urlsText = ""; pairForm.value.profile_id = null;
form.value.textsText = "";
await loadJobs(); await loadJobs();
} catch (err) { } catch (err) {
error.value = err instanceof Error ? err.message : "Не удалось создать задание"; error.value = err instanceof Error ? err.message : "Не удалось создать связку";
} finally { } finally {
submitting.value = false; submitting.value = false;
} }
} }
function openEdit(job: ParseJob) { function openEdit(job: ParseJob) {
if (!isPairJob(job)) {
error.value = "Редактируются только связки канал+профиль. Остальные задания можно удалить.";
return;
}
editingJob.value = job; editingJob.value = job;
editForm.value = { editForm.value = {
channel: channelFromConfig(job.source_config), channel_id: job.channel_id ?? null,
profile_id: job.profile_id ?? null,
limit: limitFromConfig(job.source_config), limit: limitFromConfig(job.source_config),
urlsText: urlsFromConfig(job.source_config).join("\n"),
textsText: textsFromConfig(job.source_config).join("\n"),
domain_profile:
typeof job.source_config.domain_profile === "string"
? job.source_config.domain_profile
: "generic_news",
input_mode:
job.source_config.input_mode === "texts" || job.source_config.input_mode === "mixed"
? job.source_config.input_mode
: "urls",
extract_mode: job.source_config.extract_mode === "llm" ? "llm" : "heuristic",
interval_seconds: job.interval_seconds, interval_seconds: job.interval_seconds,
is_active: job.is_active, is_active: job.is_active,
}; };
@@ -262,19 +148,18 @@ function closeEdit() {
} }
async function handleSaveEdit() { async function handleSaveEdit() {
if (!editingJob.value) return; if (!editingJob.value || !isPairJob(editingJob.value)) return;
const sourceType = editingJob.value.source_type; if (!editForm.value.channel_id || !editForm.value.profile_id) {
error.value = "Выберите канал и профиль";
if (sourceType === "telegram" && !editForm.value.channel.trim()) {
error.value = "Укажите канал Telegram";
return; return;
} }
submitting.value = true; submitting.value = true;
error.value = ""; error.value = "";
try { try {
await updateJob(editingJob.value.id, { await updateJob(editingJob.value.id, {
source_config: buildSourceConfig(sourceType, editForm.value, editingJob.value.source_config), channel_id: editForm.value.channel_id,
profile_id: editForm.value.profile_id,
limit: editForm.value.limit,
interval_seconds: editForm.value.interval_seconds, interval_seconds: editForm.value.interval_seconds,
is_active: editForm.value.is_active, is_active: editForm.value.is_active,
}); });
@@ -333,6 +218,7 @@ function formatDate(value: string | null): string {
onMounted(() => { onMounted(() => {
loadJobs(); loadJobs();
loadEntities();
pollTimer = setInterval(loadJobs, 5000); pollTimer = setInterval(loadJobs, 5000);
}); });
@@ -346,82 +232,42 @@ onUnmounted(() => {
<h2 class="page-heading">Парсеры</h2> <h2 class="page-heading">Парсеры</h2>
<section class="card"> <section class="card">
<h3>Новый парсер</h3> <h3>Новая связка (канал + профиль)</h3>
<p class="hint"> <p class="hint">
{{ formHint }} Создайте
Дубликаты по <code>source_url</code> не записываются. <router-link to="/channels">канал</router-link>
Статичный парсер по образцу поста — и
<router-link to="/parser-builder">Конструктор</router-link>. <router-link to="/parser-profiles">профиль</router-link>,
затем свяжите их здесь. В Redis уходит плоский
<code>extract_mode=profile</code> + <code>heuristic_profile</code>.
</p> </p>
<form class="admin-form" @submit.prevent="handleSubmit"> <form class="admin-form" @submit.prevent="handleCreatePair">
<div class="form-row"> <div class="form-row">
<label> <label>
Тип источника Канал
<select v-model="form.source_type"> <select v-model.number="pairForm.channel_id" required>
<option v-for="opt in SOURCE_TYPES" :key="opt.value" :value="opt.value"> <option :value="null" disabled>Выберите…</option>
{{ opt.label }} <option v-for="ch in activeChannels" :key="ch.id" :value="ch.id">
{{ ch.name }} (@{{ ch.channel }})
</option> </option>
</select> </select>
</label> </label>
<label>
<template v-if="form.source_type === 'telegram'"> Профиль
<label> <select v-model.number="pairForm.profile_id" required>
Канал <option :value="null" disabled>Выберите…</option>
<input v-model="form.channel" type="text" placeholder="creamy_caprice" required /> <option v-for="p in readyProfiles" :key="p.id" :value="p.id">
</label> {{ p.name }} (#{{ p.id }})
<label> </option>
Лимит </select>
<input v-model.number="form.limit" type="number" min="1" max="1000" /> </label>
</label> <label>
<label> Лимит batch
Извлечение <input v-model.number="pairForm.limit" type="number" min="1" max="1000" />
<select v-model="form.extract_mode"> </label>
<option value="heuristic">Эвристики (структурированные посты)</option>
<option value="llm">LLM DeepSeek (неструктурированные)</option>
</select>
</label>
</template>
<template v-else-if="form.source_type === 'crawl4ai'">
<label class="span-2">
URL (по одному в строке)
<textarea v-model="form.urlsText" rows="3" placeholder="https://example.com/news/…" required />
</label>
<label>
Domain profile
<input v-model="form.domain_profile" type="text" placeholder="generic_news" />
</label>
<label>
Извлечение
<select v-model="form.extract_mode">
<option value="heuristic">Эвристики (regex)</option>
<option value="llm">LLM DeepSeek (Crawl4AI)</option>
</select>
</label>
</template>
<template v-else>
<label>
Режим ввода
<select v-model="form.input_mode">
<option value="urls">URLs</option>
<option value="texts">Texts</option>
<option value="mixed">Mixed</option>
</select>
</label>
<label v-if="form.input_mode !== 'texts'" class="span-2">
URL (по одному в строке)
<textarea v-model="form.urlsText" rows="3" placeholder="https://example.com/article…" />
</label>
<label v-if="form.input_mode !== 'urls'" class="span-2">
Тексты (по одному блоку в строке)
<textarea v-model="form.textsText" rows="3" placeholder="Текст статьи…" />
</label>
</template>
<label> <label>
Интервал Интервал
<select v-model.number="form.interval_seconds"> <select v-model.number="pairForm.interval_seconds">
<option v-for="opt in INTERVAL_OPTIONS" :key="opt.value" :value="opt.value"> <option v-for="opt in INTERVAL_OPTIONS" :key="opt.value" :value="opt.value">
{{ opt.label }} {{ opt.label }}
</option> </option>
@@ -429,16 +275,24 @@ onUnmounted(() => {
</label> </label>
</div> </div>
<div class="form-actions"> <div class="form-actions">
<button type="submit" class="btn btn-primary" :disabled="submitting"> <button
{{ submitting ? "Создание..." : "Создать парсер" }} type="submit"
class="btn btn-primary"
:disabled="submitting || !pairForm.channel_id || !pairForm.profile_id"
>
{{ submitting ? "Создание..." : "Создать связку" }}
</button> </button>
</div> </div>
</form> </form>
<p v-if="!activeChannels.length || !readyProfiles.length" class="hint">
Нет активных каналов или готовых профилей —
сначала добавьте их на страницах Каналы / Профили.
</p>
<p v-if="error" class="form-error">{{ error }}</p> <p v-if="error" class="form-error">{{ error }}</p>
</section> </section>
<section class="card"> <section class="card">
<h3>Парсеры <span v-if="loading" class="muted">(загрузка...)</span></h3> <h3>Связки <span v-if="loading" class="muted">(загрузка...)</span></h3>
<div class="table-wrap"> <div class="table-wrap">
<table class="admin-table"> <table class="admin-table">
<thead> <thead>
@@ -470,16 +324,18 @@ onUnmounted(() => {
<td>{{ formatDate(job.last_run_at) }}</td> <td>{{ formatDate(job.last_run_at) }}</td>
<td class="error-cell">{{ job.last_error ?? "—" }}</td> <td class="error-cell">{{ job.last_error ?? "—" }}</td>
<td class="actions-cell"> <td class="actions-cell">
<button class="btn btn-sm" type="button" @click="openEdit(job)">Изменить</button> <button
v-if="isPairJob(job)"
class="btn btn-sm"
type="button"
@click="openEdit(job)"
>
Изменить
</button>
<button <button
class="btn btn-sm" class="btn btn-sm"
type="button" type="button"
:disabled="!canForceRun(job) || forcingJobId === job.id" :disabled="!canForceRun(job) || forcingJobId === job.id"
:title="
canForceRun(job)
? 'Принудительно выполнить запрос постов'
: 'Парсер уже в очереди или выполняется'
"
@click="handleForceRun(job.id)" @click="handleForceRun(job.id)"
> >
{{ forcingJobId === job.id ? "Запуск..." : "Запустить" }} {{ forcingJobId === job.id ? "Запуск..." : "Запустить" }}
@@ -501,69 +357,29 @@ onUnmounted(() => {
<div v-if="editingJob" class="modal-backdrop" @click.self="closeEdit"> <div v-if="editingJob" class="modal-backdrop" @click.self="closeEdit">
<div class="modal card"> <div class="modal card">
<h3>Редактировать парсер #{{ editingJob.id }} ({{ editingJob.source_type }})</h3> <h3>Редактировать связку #{{ editingJob.id }}</h3>
<form class="admin-form" @submit.prevent="handleSaveEdit"> <form class="admin-form" @submit.prevent="handleSaveEdit">
<div class="form-row"> <div class="form-row">
<template v-if="editingJob.source_type === 'telegram'"> <label>
<label> Канал
Канал <select v-model.number="editForm.channel_id" required>
<input v-model="editForm.channel" type="text" required /> <option v-for="ch in editChannelOptions" :key="ch.id" :value="ch.id">
</label> {{ ch.name }} (@{{ ch.channel }}){{ ch.is_active ? "" : " — выкл." }}
<label> </option>
Лимит </select>
<input v-model.number="editForm.limit" type="number" min="1" max="1000" /> </label>
</label> <label>
<label v-if="!isProfileJob(editingJob)"> Профиль
Извлечение <select v-model.number="editForm.profile_id" required>
<select v-model="editForm.extract_mode"> <option v-for="p in editProfileOptions" :key="p.id" :value="p.id">
<option value="heuristic">Эвристики</option> {{ p.name }} (#{{ p.id }})
<option value="llm">LLM DeepSeek</option> </option>
</select> </select>
</label> </label>
<p v-else class="hint profile-hint"> <label>
Режим: <code>profile</code> (статичный профиль из конструктора). Лимит batch
Сменить правила — через повторную генерацию в «Конструктор». <input v-model.number="editForm.limit" type="number" min="1" max="1000" />
</p> </label>
<label v-if="isProfileJob(editingJob)" class="span-2">
Профиль (только просмотр)
<textarea class="profile-ro" :value="profileJson(editingJob)" rows="8" readonly />
</label>
</template>
<template v-else-if="editingJob.source_type === 'crawl4ai'">
<label class="span-2">
URL (по одному в строке)
<textarea v-model="editForm.urlsText" rows="3" required />
</label>
<label>
Domain profile
<input v-model="editForm.domain_profile" type="text" />
</label>
<label>
Извлечение
<select v-model="editForm.extract_mode">
<option value="heuristic">Эвристики</option>
<option value="llm">LLM DeepSeek</option>
</select>
</label>
</template>
<template v-else>
<label>
Режим ввода
<select v-model="editForm.input_mode">
<option value="urls">URLs</option>
<option value="texts">Texts</option>
<option value="mixed">Mixed</option>
</select>
</label>
<label v-if="editForm.input_mode !== 'texts'" class="span-2">
URL
<textarea v-model="editForm.urlsText" rows="3" />
</label>
<label v-if="editForm.input_mode !== 'urls'" class="span-2">
Тексты
<textarea v-model="editForm.textsText" rows="3" />
</label>
</template>
<label> <label>
Интервал Интервал
<select v-model.number="editForm.interval_seconds"> <select v-model.number="editForm.interval_seconds">
@@ -597,19 +413,6 @@ onUnmounted(() => {
line-height: 1.5; line-height: 1.5;
} }
.form-row .span-2 {
grid-column: span 2;
}
.form-row textarea {
width: 100%;
font: inherit;
padding: 0.4rem 0.5rem;
border: 1px solid #d1d5db;
border-radius: 4px;
resize: vertical;
}
.source-cell { .source-cell {
max-width: 280px; max-width: 280px;
overflow: hidden; overflow: hidden;
@@ -625,11 +428,6 @@ onUnmounted(() => {
margin-right: 0.25rem; margin-right: 0.25rem;
} }
.btn-danger {
color: #b91c1c;
border-color: #fecaca;
}
.modal-backdrop { .modal-backdrop {
position: fixed; position: fixed;
inset: 0; inset: 0;
@@ -643,8 +441,6 @@ onUnmounted(() => {
.modal { .modal {
width: min(560px, 92vw); width: min(560px, 92vw);
margin: 0; margin: 0;
max-height: 90vh;
overflow: auto;
} }
.checkbox-row { .checkbox-row {
@@ -654,16 +450,12 @@ onUnmounted(() => {
margin-top: 1.5rem; margin-top: 1.5rem;
} }
.profile-hint { .error-cell {
flex: 1 1 100%; max-width: 200px;
margin: 0; overflow: hidden;
} text-overflow: ellipsis;
white-space: nowrap;
.profile-ro { font-size: 0.8rem;
width: 100%; color: #b91c1c;
font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
font-size: 0.75rem;
background: #f8f8f8;
color: #444;
} }
</style> </style>
+1 -1
View File
@@ -70,7 +70,7 @@ async def run(job_id, source_config, *, ctx) -> tuple[list[dict], str | None]
- `heuristic_profile` в `source_config` (схема `contracts/heuristic_profile.py`); - `heuristic_profile` в `source_config` (схема `contracts/heuristic_profile.py`);
- интерпретатор: `workers/heuristic_profile.py` (те же правила, что preview в ЦА); - интерпретатор: `workers/heuristic_profile.py` (те же правила, что preview в ЦА);
- Telegram batch + listener применяют профиль без DeepSeek; - Telegram batch + listener применяют профиль без DeepSeek;
- генерация профиля — только в админке (`/admin/parser-builder/generate`). - генерация профиля — только в админке (`/admin/parser-profiles/generate`).
```mermaid ```mermaid
flowchart LR flowchart LR
+1 -1
View File
@@ -56,7 +56,7 @@ class HeuristicProfile(BaseModel):
def target_field_specs() -> list[dict[str, str]]: def target_field_specs() -> list[dict[str, str]]:
"""Fixed target table for parser-builder UI (roadmap: custom tables later).""" """Fixed target table for profile UI (roadmap: custom tables later)."""
return [ return [
{"name": "title", "type": "string", "description": "Short event title"}, {"name": "title", "type": "string", "description": "Short event title"},
{"name": "description", "type": "string", "description": "Event summary / body"}, {"name": "description", "type": "string", "description": "Event summary / body"},
+1 -1
View File
@@ -184,7 +184,7 @@ async def run(job_id, source_config, *, ctx) -> tuple[list[dict], str | None]
Опционально: `extract_mode: llm` (DeepSeek) для telegram/crawl4ai в batch. Опционально: `extract_mode: llm` (DeepSeek) для telegram/crawl4ai в batch.
**Конструктор парсера** (`/parser-builder`): менеджер вставляет образец поста → DeepSeek один раз строит `HeuristicProfile` (статичные regex/правила) → job с `extract_mode=profile`. Runtime (batch + listener) применяет только профиль, без LLM. Кастомные пользовательские таблицы — roadmap (подмена `target-fields`). **Профиль + канал:** UI `/parser-profiles` (Generate/Preview/CRUD) и `/channels`; связка на `/parsers` → `ParseJob` с FK. При enqueue ЦА flatten в `extract_mode=profile` + `heuristic_profile`. Runtime (batch + listener) применяет только профиль, без LLM. Кастомные пользовательские таблицы — roadmap.
Подробности: [centers/parsing/ARCHITECTURE.md](../centers/parsing/ARCHITECTURE.md), [centers/analytics/ARCHITECTURE.md](../centers/analytics/ARCHITECTURE.md). Подробности: [centers/parsing/ARCHITECTURE.md](../centers/parsing/ARCHITECTURE.md), [centers/analytics/ARCHITECTURE.md](../centers/analytics/ARCHITECTURE.md).
+1 -1
View File
@@ -63,7 +63,7 @@ Listener добавляет флаг `listener: true` на стороне CA API
Реестр: `CONFIG_MODELS` + `parse_source_config(source_type, raw)`. Реестр: `CONFIG_MODELS` + `parse_source_config(source_type, raw)`.
Профиль конструктора: `contracts/heuristic_profile.py` (`HeuristicProfile`, `apply_profile`). Профиль парсера: `contracts/heuristic_profile.py` (`HeuristicProfile`, `apply_profile`). Сущности Profile/Channel живут в БД ЦА; в Redis уходит плоский `heuristic_profile`.
--- ---
+2 -2
View File
@@ -5,7 +5,7 @@
- Docker + Docker Compose - Docker + Docker Compose
- Файл `.env` (из `.env.example`) - Файл `.env` (из `.env.example`)
- Для Telegram: `data/telegram.session` + `TELEGRAM_API_ID` / `TELEGRAM_API_HASH` - Для Telegram: `data/telegram.session` + `TELEGRAM_API_ID` / `TELEGRAM_API_HASH`
- Опционально: `DEEPSEEK_API_KEY` для `extract_mode: llm` (воркеры) и конструктора парсера на `ca-api` - Опционально: `DEEPSEEK_API_KEY` для `extract_mode: llm` (воркеры) и Generate профиля на `ca-api`
## Быстрый старт ## Быстрый старт
@@ -109,7 +109,7 @@ docker compose logs -f cp-workers-web
| Frontend 000 / нет контейнеров | `docker compose up -d` (без `--build`, если registry недоступен, но образы уже есть) | | Frontend 000 / нет контейнеров | `docker compose up -d` (без `--build`, если registry недоступен, но образы уже есть) |
| Job не берётся | Смотреть `ENABLED_ADAPTERS` / `WORKER_FAMILIES` нужного сервиса | | Job не берётся | Смотреть `ENABLED_ADAPTERS` / `WORKER_FAMILIES` нужного сервиса |
| LLM не работает | `DEEPSEEK_API_KEY` в `.env`, перезапуск `cp-workers` / `cp-workers-web` | | LLM не работает | `DEEPSEEK_API_KEY` в `.env`, перезапуск `cp-workers` / `cp-workers-web` |
| Generate в конструкторе 503 | `DEEPSEEK_API_KEY` в `.env` + `env_file` у `ca-api`, перезапуск `ca-api` | | Generate профиля 503 | `DEEPSEEK_API_KEY` в `.env` + `env_file` у `ca-api`, перезапуск `ca-api` |
| Изменения UI не видны | Пересобрать `ca-frontend` | | Изменения UI не видны | Пересобрать `ca-frontend` |
## Границы при разработке ## Границы при разработке