Add Profile/Channel entities with CRUD and remove legacy parser builder.
Introduce reusable ParserProfile and ParseChannel, pair jobs with enqueue flatten, Events admin CRUD, and drop inline/legacy parser-builder UI and aliases. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
+1
-1
@@ -14,7 +14,7 @@ TEST_PI_API_KEY=test-pi-api-key-change-me
|
|||||||
|
|
||||||
# DeepSeek LLM:
|
# DeepSeek LLM:
|
||||||
# - extract_mode=llm on CP workers (Telegram unstructured + Crawl4AI)
|
# - extract_mode=llm on CP workers (Telegram unstructured + Crawl4AI)
|
||||||
# - parser-builder Generate on ca-api (one-shot profile; runtime stays rule-based)
|
# - Generate профиля парсера на ca-api (один раз; runtime — только правила)
|
||||||
# DEEPSEEK_API_KEY=sk-...
|
# DEEPSEEK_API_KEY=sk-...
|
||||||
# DEEPSEEK_BASE_URL=https://api.deepseek.com
|
# DEEPSEEK_BASE_URL=https://api.deepseek.com
|
||||||
# DEEPSEEK_MODEL=deepseek-chat
|
# DEEPSEEK_MODEL=deepseek-chat
|
||||||
|
|||||||
@@ -58,12 +58,13 @@ centers/analytics/
|
|||||||
│ ├── routers/
|
│ ├── routers/
|
||||||
│ │ ├── objects.py # /api/health, /api/objects, media
|
│ │ ├── objects.py # /api/health, /api/objects, media
|
||||||
│ │ ├── map.py # /api/map/*
|
│ │ ├── map.py # /api/map/*
|
||||||
│ │ ├── admin.py # /admin/*
|
│ │ ├── admin.py # /admin/* (jobs, events, consumers)
|
||||||
│ │ ├── parser_builder.py # /admin/parser-builder/*
|
│ │ ├── parser_profiles.py # /admin/parser-profiles/*
|
||||||
|
│ │ ├── parse_channels.py # /admin/parse-channels/*
|
||||||
│ │ ├── internal.py # /internal/* (только ЦП)
|
│ │ ├── internal.py # /internal/* (только ЦП)
|
||||||
│ │ └── v1.py # /api/v1/* (ПИ)
|
│ │ └── v1.py # /api/v1/* (ПИ)
|
||||||
│ └── services/
|
│ └── services/
|
||||||
│ ├── jobs.py # Redis RPUSH
|
│ ├── jobs.py # Redis RPUSH + flatten Profile/Channel
|
||||||
│ ├── scheduler.py # периодический re-queue
|
│ ├── scheduler.py # периодический re-queue
|
||||||
│ ├── ingest.py # дедуп + map sync
|
│ ├── ingest.py # дедуп + map sync
|
||||||
│ ├── parser_builder.py # DeepSeek → HeuristicProfile (один раз)
|
│ ├── parser_builder.py # DeepSeek → HeuristicProfile (один раз)
|
||||||
@@ -76,7 +77,7 @@ centers/analytics/
|
|||||||
├── Dockerfile # Vite build + nginx
|
├── Dockerfile # Vite build + nginx
|
||||||
├── nginx.conf # proxy /api /admin /internal → ca-api
|
├── nginx.conf # proxy /api /admin /internal → ca-api
|
||||||
└── src/
|
└── src/
|
||||||
├── views/ # Map, Parsers, ParserBuilder, Events, …
|
├── views/ # Map, Channels, Profiles, Parsers, …
|
||||||
├── components/ # карта, CRUD объектов
|
├── components/ # карта, CRUD объектов
|
||||||
├── api/ # HTTP-клиенты
|
├── api/ # HTTP-клиенты
|
||||||
└── router/index.ts
|
└── router/index.ts
|
||||||
@@ -87,7 +88,9 @@ centers/analytics/
|
|||||||
| Модель | Таблица | Назначение |
|
| Модель | Таблица | Назначение |
|
||||||
|--------|--------|------------|
|
|--------|--------|------------|
|
||||||
| `Event` | `events` | Нормализованное событие; UK `source_url` |
|
| `Event` | `events` | Нормализованное событие; UK `source_url` |
|
||||||
| `ParseJob` | `parse_jobs` | Конфиг парсера, интервал, статус |
|
| `ParserProfile` | `parser_profiles` | Статичный heuristic-профиль (JSON-правила) |
|
||||||
|
| `ParseChannel` | `parse_channels` | Канал (Telegram handle) |
|
||||||
|
| `ParseJob` | `parse_jobs` | Связка канал+профиль, интервал, статус; FK `profile_id` + `channel_id` |
|
||||||
| `MapObject` | `map_objects` | Точка на карте (event или ручная) |
|
| `MapObject` | `map_objects` | Точка на карте (event или ручная) |
|
||||||
| `ObjectMedia` | `object_media` | Файлы к объектам |
|
| `ObjectMedia` | `object_media` | Файлы к объектам |
|
||||||
| `Consumer` | `consumers` | Подписчик ПИ (hash ключа) |
|
| `Consumer` | `consumers` | Подписчик ПИ (hash ключа) |
|
||||||
@@ -98,26 +101,34 @@ centers/analytics/
|
|||||||
| Prefix | Кто вызывает | Содержание |
|
| Prefix | Кто вызывает | Содержание |
|
||||||
|--------|--------------|------------|
|
|--------|--------------|------------|
|
||||||
| `/api/*` | UI, публичный health | Карта, объекты, медиа |
|
| `/api/*` | UI, публичный health | Карта, объекты, медиа |
|
||||||
| `/admin/*` | UI admin | Jobs, events, analytics, consumers, parser-builder |
|
| `/admin/*` | UI admin | Jobs, channels, profiles, events, analytics, consumers |
|
||||||
| `/internal/*` | Только ЦП | ingest, job status, listener subscriptions |
|
| `/internal/*` | Только ЦП | ingest, job status, listener subscriptions |
|
||||||
| `/api/v1/*` | Внешние клиенты | Events с Bearer-ключом |
|
| `/api/v1/*` | Внешние клиенты | Events с Bearer-ключом |
|
||||||
|
|
||||||
Internal защищён заголовком `X-Internal-Token` (`INTERNAL_TOKEN`).
|
Internal защищён заголовком `X-Internal-Token` (`INTERNAL_TOKEN`).
|
||||||
|
|
||||||
### Конструктор парсера
|
### Профили, каналы и пары
|
||||||
|
|
||||||
Раздел UI `/parser-builder` → `POST /admin/parser-builder/generate|preview|jobs`:
|
1. **Канал** (`CRUD /admin/parse-channels`) — handle Telegram + метаданные.
|
||||||
|
2. **Профиль** (`CRUD /admin/parser-profiles`) — Generate (DeepSeek один раз по образцу) → Preview → Save `heuristic_profile`.
|
||||||
|
3. **Пара** (`POST /admin/jobs` с `channel_id` + `profile_id`) → `ParseJob`; при enqueue ЦА **разворачивает** пару в плоский `source_config`:
|
||||||
|
|
||||||
1. Менеджер вставляет образец поста.
|
```json
|
||||||
2. DeepSeek (`DEEPSEEK_API_KEY` на **ca-api**) один раз возвращает `HeuristicProfile` (regex/line/marker).
|
{
|
||||||
3. Preview и сохранение job с `extract_mode=profile` + JSON профиля в `source_config`.
|
"channel": "<ParseChannel.channel>",
|
||||||
4. ЦП применяет профиль статически (batch + listener); LLM на ingest не вызывается.
|
"limit": 100,
|
||||||
|
"extract_mode": "profile",
|
||||||
|
"heuristic_profile": { "...из ParserProfile..." }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
Roadmap: кастомные пользовательские таблицы подменяют только список `target-fields` при генерации.
|
ЦП по-прежнему получает только Redis payload — без доступа к таблицам Profile/Channel.
|
||||||
|
|
||||||
|
Roadmap: кастомные пользовательские таблицы подменяют список `target-fields` при генерации; профиль будет ссылаться на `schema_id`.
|
||||||
|
|
||||||
### Jobs и scheduler
|
### Jobs и scheduler
|
||||||
|
|
||||||
1. `POST /admin/jobs` / retry → запись `ParseJob` + `enqueue_job` (`services/jobs.py`).
|
1. `POST /admin/jobs` / retry → запись `ParseJob` + `enqueue_parse_job` (flatten + RPUSH).
|
||||||
2. Очередь: `cp:jobs:{family}` из `contracts/queues.py`.
|
2. Очередь: `cp:jobs:{family}` из `contracts/queues.py`.
|
||||||
3. `services/scheduler.py` — тик ~30 с, повторная постановка активных jobs по `interval_seconds`.
|
3. `services/scheduler.py` — тик ~30 с, повторная постановка активных jobs по `interval_seconds`.
|
||||||
|
|
||||||
@@ -135,8 +146,9 @@ Roadmap: кастомные пользовательские таблицы по
|
|||||||
| Path | View |
|
| Path | View |
|
||||||
|------|------|
|
|------|------|
|
||||||
| `/` | `MapViewPage.vue` |
|
| `/` | `MapViewPage.vue` |
|
||||||
| `/parsers` | `ParsersView.vue` |
|
| `/channels` | `ChannelsView.vue` |
|
||||||
| `/parser-builder` | `ParserBuilderView.vue` |
|
| `/parser-profiles` | `ParserProfilesView.vue` |
|
||||||
|
| `/parsers` | `ParsersView.vue` (связки канал+профиль) |
|
||||||
| `/events` | `EventsView.vue` |
|
| `/events` | `EventsView.vue` |
|
||||||
| `/analytics` | `AnalyticsView.vue` |
|
| `/analytics` | `AnalyticsView.vue` |
|
||||||
| `/consumers` | `ConsumersView.vue` |
|
| `/consumers` | `ConsumersView.vue` |
|
||||||
@@ -156,7 +168,7 @@ Roadmap: кастомные пользовательские таблицы по
|
|||||||
| `INTERNAL_TOKEN` | Auth ЦП ↔ ЦА |
|
| `INTERNAL_TOKEN` | Auth ЦП ↔ ЦА |
|
||||||
| `TEST_PI_API_KEY` | Seed consumer `test-pi` |
|
| `TEST_PI_API_KEY` | Seed consumer `test-pi` |
|
||||||
| `UPLOAD_DIR` | Медиа (по умолчанию `/data/uploads`) |
|
| `UPLOAD_DIR` | Медиа (по умолчанию `/data/uploads`) |
|
||||||
| `DEEPSEEK_API_KEY` | Конструктор парсера (Generate); не нужен для preview/runtime profile |
|
| `DEEPSEEK_API_KEY` | Generate профиля на ca-api; не нужен для preview/runtime |
|
||||||
| `DEEPSEEK_BASE_URL` / `DEEPSEEK_MODEL` | Опционально |
|
| `DEEPSEEK_BASE_URL` / `DEEPSEEK_MODEL` | Опционально |
|
||||||
|
|
||||||
### Типовые точки входа в код
|
### Типовые точки входа в код
|
||||||
|
|||||||
@@ -13,7 +13,7 @@ for _candidate in (_HERE.parent, *_HERE.parents):
|
|||||||
break
|
break
|
||||||
|
|
||||||
from .database import Base, engine, get_db
|
from .database import Base, engine, get_db
|
||||||
from .routers import admin, internal, map, objects, parser_builder, v1
|
from .routers import admin, internal, map, objects, parse_channels, parser_profiles, v1
|
||||||
from .seed import seed_objects, seed_test_consumer
|
from .seed import seed_objects, seed_test_consumer
|
||||||
from .services.migrations import migrate_schema
|
from .services.migrations import migrate_schema
|
||||||
from .services.scheduler import start_scheduler
|
from .services.scheduler import start_scheduler
|
||||||
@@ -50,5 +50,6 @@ app.include_router(objects.router)
|
|||||||
app.include_router(map.router)
|
app.include_router(map.router)
|
||||||
app.include_router(internal.router)
|
app.include_router(internal.router)
|
||||||
app.include_router(admin.router)
|
app.include_router(admin.router)
|
||||||
app.include_router(parser_builder.router)
|
app.include_router(parser_profiles.router)
|
||||||
|
app.include_router(parse_channels.router)
|
||||||
app.include_router(v1.router)
|
app.include_router(v1.router)
|
||||||
|
|||||||
@@ -93,12 +93,54 @@ class Event(Base):
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ParserProfile(Base):
|
||||||
|
__tablename__ = "parser_profiles"
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)
|
||||||
|
name: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||||
|
sample_post: Mapped[str] = mapped_column(Text, default="")
|
||||||
|
heuristic_profile: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||||
|
status: Mapped[str] = mapped_column(String(50), default="draft", index=True)
|
||||||
|
created_at: Mapped[datetime] = mapped_column(
|
||||||
|
DateTime(timezone=True),
|
||||||
|
default=lambda: datetime.now(timezone.utc),
|
||||||
|
)
|
||||||
|
|
||||||
|
jobs: Mapped[list["ParseJob"]] = relationship(back_populates="profile")
|
||||||
|
|
||||||
|
|
||||||
|
class ParseChannel(Base):
|
||||||
|
__tablename__ = "parse_channels"
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)
|
||||||
|
name: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||||
|
source_type: Mapped[str] = mapped_column(String(50), nullable=False, default="telegram")
|
||||||
|
channel: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||||
|
is_active: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False)
|
||||||
|
created_at: Mapped[datetime] = mapped_column(
|
||||||
|
DateTime(timezone=True),
|
||||||
|
default=lambda: datetime.now(timezone.utc),
|
||||||
|
)
|
||||||
|
|
||||||
|
jobs: Mapped[list["ParseJob"]] = relationship(back_populates="channel")
|
||||||
|
|
||||||
|
|
||||||
class ParseJob(Base):
|
class ParseJob(Base):
|
||||||
__tablename__ = "parse_jobs"
|
__tablename__ = "parse_jobs"
|
||||||
|
|
||||||
id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)
|
id: Mapped[int] = mapped_column(Integer, primary_key=True, index=True)
|
||||||
source_type: Mapped[str] = mapped_column(String(50), nullable=False)
|
source_type: Mapped[str] = mapped_column(String(50), nullable=False)
|
||||||
source_config: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
source_config: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||||
|
profile_id: Mapped[int | None] = mapped_column(
|
||||||
|
ForeignKey("parser_profiles.id", ondelete="SET NULL"),
|
||||||
|
nullable=True,
|
||||||
|
index=True,
|
||||||
|
)
|
||||||
|
channel_id: Mapped[int | None] = mapped_column(
|
||||||
|
ForeignKey("parse_channels.id", ondelete="SET NULL"),
|
||||||
|
nullable=True,
|
||||||
|
index=True,
|
||||||
|
)
|
||||||
schedule: Mapped[str | None] = mapped_column(String(100), nullable=True)
|
schedule: Mapped[str | None] = mapped_column(String(100), nullable=True)
|
||||||
interval_seconds: Mapped[int] = mapped_column(Integer, default=3600, nullable=False)
|
interval_seconds: Mapped[int] = mapped_column(Integer, default=3600, nullable=False)
|
||||||
is_active: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False)
|
is_active: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False)
|
||||||
@@ -110,6 +152,9 @@ class ParseJob(Base):
|
|||||||
default=lambda: datetime.now(timezone.utc),
|
default=lambda: datetime.now(timezone.utc),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
profile: Mapped["ParserProfile | None"] = relationship(back_populates="jobs")
|
||||||
|
channel: Mapped["ParseChannel | None"] = relationship(back_populates="jobs")
|
||||||
|
|
||||||
|
|
||||||
class Consumer(Base):
|
class Consumer(Base):
|
||||||
__tablename__ = "consumers"
|
__tablename__ = "consumers"
|
||||||
|
|||||||
@@ -4,14 +4,16 @@ from fastapi import APIRouter, Depends, HTTPException, Query
|
|||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
from ..database import get_db
|
from ..database import get_db
|
||||||
from ..models import Consumer, Event, ParseJob
|
from ..models import Consumer, Event, MapObject, ParseChannel, ParseJob, ParserProfile
|
||||||
from ..schemas import (
|
from ..schemas import (
|
||||||
AnalyticsSummary,
|
AnalyticsSummary,
|
||||||
ConsumerCreate,
|
ConsumerCreate,
|
||||||
ConsumerRead,
|
ConsumerRead,
|
||||||
ConsumerUpdate,
|
ConsumerUpdate,
|
||||||
|
EventCreate,
|
||||||
EventListResponse,
|
EventListResponse,
|
||||||
EventRead,
|
EventRead,
|
||||||
|
EventUpdate,
|
||||||
ParseJobCreate,
|
ParseJobCreate,
|
||||||
ParseJobRead,
|
ParseJobRead,
|
||||||
ParseJobUpdate,
|
ParseJobUpdate,
|
||||||
@@ -30,9 +32,10 @@ from ..services.filtering import (
|
|||||||
create_consumer,
|
create_consumer,
|
||||||
list_consumers,
|
list_consumers,
|
||||||
rotate_consumer_key,
|
rotate_consumer_key,
|
||||||
|
sync_event_to_map_object,
|
||||||
update_consumer,
|
update_consumer,
|
||||||
)
|
)
|
||||||
from ..services.jobs import enqueue_job
|
from ..services.jobs import enqueue_parse_job, flatten_pair_config
|
||||||
|
|
||||||
router = APIRouter(prefix="/admin", tags=["admin"])
|
router = APIRouter(prefix="/admin", tags=["admin"])
|
||||||
|
|
||||||
@@ -48,8 +51,62 @@ def _validated_source_config(source_type: str, source_config: dict) -> dict:
|
|||||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||||
|
|
||||||
|
|
||||||
|
def _job_to_read(job: ParseJob) -> ParseJobRead:
|
||||||
|
return ParseJobRead(
|
||||||
|
id=job.id,
|
||||||
|
source_type=job.source_type,
|
||||||
|
source_config=job.source_config or {},
|
||||||
|
profile_id=job.profile_id,
|
||||||
|
channel_id=job.channel_id,
|
||||||
|
schedule=job.schedule,
|
||||||
|
interval_seconds=job.interval_seconds,
|
||||||
|
is_active=job.is_active,
|
||||||
|
status=job.status,
|
||||||
|
last_run_at=job.last_run_at,
|
||||||
|
last_error=job.last_error,
|
||||||
|
created_at=job.created_at,
|
||||||
|
profile_name=job.profile.name if job.profile else None,
|
||||||
|
channel_name=job.channel.name if job.channel else None,
|
||||||
|
channel_handle=job.channel.channel if job.channel else None,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@router.post("/jobs", response_model=ParseJobRead, status_code=201)
|
@router.post("/jobs", response_model=ParseJobRead, status_code=201)
|
||||||
def create_parse_job(payload: ParseJobCreate, db: Session = Depends(get_db)):
|
def create_parse_job(payload: ParseJobCreate, db: Session = Depends(get_db)):
|
||||||
|
pair_mode = payload.profile_id is not None or payload.channel_id is not None
|
||||||
|
if pair_mode:
|
||||||
|
if payload.profile_id is None or payload.channel_id is None:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=400,
|
||||||
|
detail="Both profile_id and channel_id are required to create a pair",
|
||||||
|
)
|
||||||
|
profile = db.query(ParserProfile).filter(ParserProfile.id == payload.profile_id).first()
|
||||||
|
channel = db.query(ParseChannel).filter(ParseChannel.id == payload.channel_id).first()
|
||||||
|
if not profile:
|
||||||
|
raise HTTPException(status_code=404, detail="Profile not found")
|
||||||
|
if not channel:
|
||||||
|
raise HTTPException(status_code=404, detail="Channel not found")
|
||||||
|
if not profile.heuristic_profile:
|
||||||
|
raise HTTPException(status_code=400, detail="Profile has no heuristic_profile")
|
||||||
|
if not channel.is_active:
|
||||||
|
raise HTTPException(status_code=400, detail="Channel is inactive")
|
||||||
|
|
||||||
|
try:
|
||||||
|
source_config = flatten_pair_config(channel, profile, limit=payload.limit)
|
||||||
|
except Exception as exc:
|
||||||
|
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||||
|
|
||||||
|
job = ParseJob(
|
||||||
|
source_type=channel.source_type or "telegram",
|
||||||
|
source_config=source_config,
|
||||||
|
profile_id=profile.id,
|
||||||
|
channel_id=channel.id,
|
||||||
|
schedule=payload.schedule,
|
||||||
|
interval_seconds=payload.interval_seconds,
|
||||||
|
is_active=payload.is_active,
|
||||||
|
status="queued",
|
||||||
|
)
|
||||||
|
else:
|
||||||
source_config = _validated_source_config(payload.source_type, payload.source_config)
|
source_config = _validated_source_config(payload.source_type, payload.source_config)
|
||||||
job = ParseJob(
|
job = ParseJob(
|
||||||
source_type=payload.source_type,
|
source_type=payload.source_type,
|
||||||
@@ -59,17 +116,20 @@ def create_parse_job(payload: ParseJobCreate, db: Session = Depends(get_db)):
|
|||||||
is_active=payload.is_active,
|
is_active=payload.is_active,
|
||||||
status="queued",
|
status="queued",
|
||||||
)
|
)
|
||||||
|
|
||||||
db.add(job)
|
db.add(job)
|
||||||
db.commit()
|
db.commit()
|
||||||
db.refresh(job)
|
db.refresh(job)
|
||||||
|
|
||||||
enqueue_job(job.id, job.source_type, job.source_config)
|
enqueue_parse_job(db, job)
|
||||||
return job
|
db.refresh(job)
|
||||||
|
return _job_to_read(job)
|
||||||
|
|
||||||
|
|
||||||
@router.get("/jobs", response_model=list[ParseJobRead])
|
@router.get("/jobs", response_model=list[ParseJobRead])
|
||||||
def list_parse_jobs(db: Session = Depends(get_db)):
|
def list_parse_jobs(db: Session = Depends(get_db)):
|
||||||
return db.query(ParseJob).order_by(ParseJob.id.desc()).all()
|
jobs = db.query(ParseJob).order_by(ParseJob.id.desc()).all()
|
||||||
|
return [_job_to_read(job) for job in jobs]
|
||||||
|
|
||||||
|
|
||||||
@router.get("/jobs/{job_id}", response_model=ParseJobRead)
|
@router.get("/jobs/{job_id}", response_model=ParseJobRead)
|
||||||
@@ -77,7 +137,7 @@ def get_parse_job(job_id: int, db: Session = Depends(get_db)):
|
|||||||
job = db.query(ParseJob).filter(ParseJob.id == job_id).first()
|
job = db.query(ParseJob).filter(ParseJob.id == job_id).first()
|
||||||
if not job:
|
if not job:
|
||||||
raise HTTPException(status_code=404, detail="Job not found")
|
raise HTTPException(status_code=404, detail="Job not found")
|
||||||
return job
|
return _job_to_read(job)
|
||||||
|
|
||||||
|
|
||||||
@router.post("/jobs/{job_id}/retry", response_model=ParseJobRead)
|
@router.post("/jobs/{job_id}/retry", response_model=ParseJobRead)
|
||||||
@@ -93,8 +153,12 @@ def retry_parse_job(job_id: int, db: Session = Depends(get_db)):
|
|||||||
db.commit()
|
db.commit()
|
||||||
db.refresh(job)
|
db.refresh(job)
|
||||||
|
|
||||||
enqueue_job(job.id, job.source_type, job.source_config)
|
try:
|
||||||
return job
|
enqueue_parse_job(db, job)
|
||||||
|
except ValueError as exc:
|
||||||
|
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||||
|
db.refresh(job)
|
||||||
|
return _job_to_read(job)
|
||||||
|
|
||||||
|
|
||||||
@router.patch("/jobs/{job_id}", response_model=ParseJobRead)
|
@router.patch("/jobs/{job_id}", response_model=ParseJobRead)
|
||||||
@@ -107,8 +171,36 @@ def update_parse_job(
|
|||||||
if not job:
|
if not job:
|
||||||
raise HTTPException(status_code=404, detail="Job not found")
|
raise HTTPException(status_code=404, detail="Job not found")
|
||||||
|
|
||||||
if payload.source_config is not None:
|
if payload.profile_id is not None:
|
||||||
|
profile = db.query(ParserProfile).filter(ParserProfile.id == payload.profile_id).first()
|
||||||
|
if not profile:
|
||||||
|
raise HTTPException(status_code=404, detail="Profile not found")
|
||||||
|
job.profile_id = profile.id
|
||||||
|
if payload.channel_id is not None:
|
||||||
|
channel = db.query(ParseChannel).filter(ParseChannel.id == payload.channel_id).first()
|
||||||
|
if not channel:
|
||||||
|
raise HTTPException(status_code=404, detail="Channel not found")
|
||||||
|
job.channel_id = channel.id
|
||||||
|
|
||||||
|
if job.profile_id and job.channel_id:
|
||||||
|
profile = db.query(ParserProfile).filter(ParserProfile.id == job.profile_id).first()
|
||||||
|
channel = db.query(ParseChannel).filter(ParseChannel.id == job.channel_id).first()
|
||||||
|
if profile and channel:
|
||||||
|
existing_limit = (job.source_config or {}).get("limit", 100)
|
||||||
|
limit = payload.limit if payload.limit is not None else existing_limit
|
||||||
|
if not isinstance(limit, int):
|
||||||
|
limit = 100
|
||||||
|
try:
|
||||||
|
job.source_config = flatten_pair_config(channel, profile, limit=limit)
|
||||||
|
except Exception as exc:
|
||||||
|
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||||
|
elif payload.source_config is not None:
|
||||||
job.source_config = _validated_source_config(job.source_type, payload.source_config)
|
job.source_config = _validated_source_config(job.source_type, payload.source_config)
|
||||||
|
elif payload.limit is not None and isinstance(job.source_config, dict):
|
||||||
|
cfg = dict(job.source_config)
|
||||||
|
cfg["limit"] = payload.limit
|
||||||
|
job.source_config = _validated_source_config(job.source_type, cfg)
|
||||||
|
|
||||||
if payload.interval_seconds is not None:
|
if payload.interval_seconds is not None:
|
||||||
job.interval_seconds = payload.interval_seconds
|
job.interval_seconds = payload.interval_seconds
|
||||||
if payload.is_active is not None:
|
if payload.is_active is not None:
|
||||||
@@ -116,7 +208,7 @@ def update_parse_job(
|
|||||||
|
|
||||||
db.commit()
|
db.commit()
|
||||||
db.refresh(job)
|
db.refresh(job)
|
||||||
return job
|
return _job_to_read(job)
|
||||||
|
|
||||||
|
|
||||||
@router.delete("/jobs/{job_id}", status_code=204)
|
@router.delete("/jobs/{job_id}", status_code=204)
|
||||||
@@ -159,6 +251,112 @@ def list_events(
|
|||||||
return EventListResponse(items=items, total=total)
|
return EventListResponse(items=items, total=total)
|
||||||
|
|
||||||
|
|
||||||
|
def _ensure_unique_source_url(db: Session, source_url: str, *, exclude_id: int | None = None) -> None:
|
||||||
|
query = db.query(Event).filter(Event.source_url == source_url)
|
||||||
|
if exclude_id is not None:
|
||||||
|
query = query.filter(Event.id != exclude_id)
|
||||||
|
if query.first():
|
||||||
|
raise HTTPException(status_code=409, detail="Event with this source_url already exists")
|
||||||
|
|
||||||
|
|
||||||
|
def _sync_or_clear_map_object(db: Session, event: Event) -> None:
|
||||||
|
if event.latitude is not None and event.longitude is not None:
|
||||||
|
sync_event_to_map_object(db, event)
|
||||||
|
return
|
||||||
|
linked = db.query(MapObject).filter(MapObject.event_id == event.id).first()
|
||||||
|
if linked:
|
||||||
|
db.delete(linked)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/events", response_model=EventRead, status_code=201)
|
||||||
|
def create_event(payload: EventCreate, db: Session = Depends(get_db)):
|
||||||
|
source_url = payload.source_url.strip()
|
||||||
|
if not source_url:
|
||||||
|
raise HTTPException(status_code=400, detail="source_url is required")
|
||||||
|
_ensure_unique_source_url(db, source_url)
|
||||||
|
|
||||||
|
if (payload.latitude is None) != (payload.longitude is None):
|
||||||
|
raise HTTPException(status_code=400, detail="latitude and longitude must be set together")
|
||||||
|
|
||||||
|
event = Event(
|
||||||
|
source_type=payload.source_type.strip() or "manual",
|
||||||
|
source_url=source_url,
|
||||||
|
raw_text=payload.raw_text or "",
|
||||||
|
title=payload.title or "",
|
||||||
|
description=payload.description or "",
|
||||||
|
locality=payload.locality or "",
|
||||||
|
latitude=payload.latitude,
|
||||||
|
longitude=payload.longitude,
|
||||||
|
event_date=payload.event_date,
|
||||||
|
region=payload.region,
|
||||||
|
topic=payload.topic,
|
||||||
|
tags=payload.tags,
|
||||||
|
metadata_=payload.metadata,
|
||||||
|
)
|
||||||
|
db.add(event)
|
||||||
|
db.flush()
|
||||||
|
_sync_or_clear_map_object(db, event)
|
||||||
|
db.commit()
|
||||||
|
db.refresh(event)
|
||||||
|
return event
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/events/{event_id}", response_model=EventRead)
|
||||||
|
def get_event(event_id: int, db: Session = Depends(get_db)):
|
||||||
|
event = db.query(Event).filter(Event.id == event_id).first()
|
||||||
|
if not event:
|
||||||
|
raise HTTPException(status_code=404, detail="Event not found")
|
||||||
|
return event
|
||||||
|
|
||||||
|
|
||||||
|
@router.patch("/events/{event_id}", response_model=EventRead)
|
||||||
|
def update_event(event_id: int, payload: EventUpdate, db: Session = Depends(get_db)):
|
||||||
|
event = db.query(Event).filter(Event.id == event_id).first()
|
||||||
|
if not event:
|
||||||
|
raise HTTPException(status_code=404, detail="Event not found")
|
||||||
|
|
||||||
|
data = payload.model_dump(exclude_unset=True)
|
||||||
|
if "source_url" in data and data["source_url"] is not None:
|
||||||
|
source_url = data["source_url"].strip()
|
||||||
|
if not source_url:
|
||||||
|
raise HTTPException(status_code=400, detail="source_url cannot be empty")
|
||||||
|
_ensure_unique_source_url(db, source_url, exclude_id=event.id)
|
||||||
|
event.source_url = source_url
|
||||||
|
data.pop("source_url")
|
||||||
|
|
||||||
|
if "metadata" in data:
|
||||||
|
event.metadata_ = data.pop("metadata")
|
||||||
|
|
||||||
|
if "source_type" in data and data["source_type"] is not None:
|
||||||
|
event.source_type = data.pop("source_type").strip() or event.source_type
|
||||||
|
|
||||||
|
for field, value in data.items():
|
||||||
|
setattr(event, field, value)
|
||||||
|
|
||||||
|
lat = event.latitude
|
||||||
|
lon = event.longitude
|
||||||
|
if (lat is None) != (lon is None):
|
||||||
|
raise HTTPException(status_code=400, detail="latitude and longitude must be set together")
|
||||||
|
|
||||||
|
_sync_or_clear_map_object(db, event)
|
||||||
|
db.commit()
|
||||||
|
db.refresh(event)
|
||||||
|
return event
|
||||||
|
|
||||||
|
|
||||||
|
@router.delete("/events/{event_id}", status_code=204)
|
||||||
|
def delete_event(event_id: int, db: Session = Depends(get_db)):
|
||||||
|
event = db.query(Event).filter(Event.id == event_id).first()
|
||||||
|
if not event:
|
||||||
|
raise HTTPException(status_code=404, detail="Event not found")
|
||||||
|
|
||||||
|
linked = db.query(MapObject).filter(MapObject.event_id == event.id).all()
|
||||||
|
for obj in linked:
|
||||||
|
db.delete(obj)
|
||||||
|
db.delete(event)
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
@router.get("/analytics/summary", response_model=AnalyticsSummary)
|
@router.get("/analytics/summary", response_model=AnalyticsSummary)
|
||||||
def analytics_summary(db: Session = Depends(get_db)):
|
def analytics_summary(db: Session = Depends(get_db)):
|
||||||
return get_analytics_summary(db)
|
return get_analytics_summary(db)
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ from ..deps import verify_internal_token
|
|||||||
from ..models import ParseJob
|
from ..models import ParseJob
|
||||||
from ..schemas import IngestRequest, IngestResponse, ListenerSubscription
|
from ..schemas import IngestRequest, IngestResponse, ListenerSubscription
|
||||||
from ..services.ingest import ingest_events
|
from ..services.ingest import ingest_events
|
||||||
|
from ..services.jobs import resolve_job_source_config
|
||||||
|
|
||||||
router = APIRouter(prefix="/internal", tags=["internal"])
|
router = APIRouter(prefix="/internal", tags=["internal"])
|
||||||
|
|
||||||
@@ -68,14 +69,19 @@ def listener_subscriptions(
|
|||||||
)
|
)
|
||||||
result: list[ListenerSubscription] = []
|
result: list[ListenerSubscription] = []
|
||||||
for job in jobs:
|
for job in jobs:
|
||||||
channel = job.source_config.get("channel") if job.source_config else None
|
try:
|
||||||
|
source_config = resolve_job_source_config(db, job)
|
||||||
|
except ValueError:
|
||||||
|
continue
|
||||||
|
channel = source_config.get("channel")
|
||||||
if not channel:
|
if not channel:
|
||||||
continue
|
continue
|
||||||
result.append(
|
result.append(
|
||||||
ListenerSubscription(
|
ListenerSubscription(
|
||||||
job_id=job.id,
|
job_id=job.id,
|
||||||
channel=str(channel),
|
channel=str(channel),
|
||||||
source_config=dict(job.source_config or {}),
|
source_config=dict(source_config),
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
db.commit()
|
||||||
return result
|
return result
|
||||||
|
|||||||
@@ -0,0 +1,98 @@
|
|||||||
|
"""Admin CRUD for parse channels (Telegram handles in MVP)."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from fastapi import APIRouter, Depends, HTTPException
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from ..database import get_db
|
||||||
|
from ..models import ParseChannel, ParseJob
|
||||||
|
from ..schemas import ParseChannelCreate, ParseChannelRead, ParseChannelUpdate
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/admin/parse-channels", tags=["parse-channels"])
|
||||||
|
|
||||||
|
ALLOWED_SOURCE_TYPES = {"telegram"}
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("", response_model=list[ParseChannelRead])
|
||||||
|
def list_channels(db: Session = Depends(get_db)):
|
||||||
|
return db.query(ParseChannel).order_by(ParseChannel.id.desc()).all()
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("", response_model=ParseChannelRead, status_code=201)
|
||||||
|
def create_channel(payload: ParseChannelCreate, db: Session = Depends(get_db)):
|
||||||
|
source_type = (payload.source_type or "telegram").strip()
|
||||||
|
if source_type not in ALLOWED_SOURCE_TYPES:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=400,
|
||||||
|
detail=f"source_type must be one of: {', '.join(sorted(ALLOWED_SOURCE_TYPES))}",
|
||||||
|
)
|
||||||
|
channel = ParseChannel(
|
||||||
|
name=payload.name.strip(),
|
||||||
|
source_type=source_type,
|
||||||
|
channel=payload.channel.strip().lstrip("@"),
|
||||||
|
is_active=payload.is_active,
|
||||||
|
)
|
||||||
|
db.add(channel)
|
||||||
|
db.commit()
|
||||||
|
db.refresh(channel)
|
||||||
|
return channel
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/{channel_id}", response_model=ParseChannelRead)
|
||||||
|
def get_channel(channel_id: int, db: Session = Depends(get_db)):
|
||||||
|
channel = db.query(ParseChannel).filter(ParseChannel.id == channel_id).first()
|
||||||
|
if not channel:
|
||||||
|
raise HTTPException(status_code=404, detail="Channel not found")
|
||||||
|
return channel
|
||||||
|
|
||||||
|
|
||||||
|
@router.patch("/{channel_id}", response_model=ParseChannelRead)
|
||||||
|
def update_channel(
|
||||||
|
channel_id: int,
|
||||||
|
payload: ParseChannelUpdate,
|
||||||
|
db: Session = Depends(get_db),
|
||||||
|
):
|
||||||
|
channel = db.query(ParseChannel).filter(ParseChannel.id == channel_id).first()
|
||||||
|
if not channel:
|
||||||
|
raise HTTPException(status_code=404, detail="Channel not found")
|
||||||
|
|
||||||
|
if payload.name is not None:
|
||||||
|
channel.name = payload.name.strip()
|
||||||
|
if payload.source_type is not None:
|
||||||
|
source_type = payload.source_type.strip()
|
||||||
|
if source_type not in ALLOWED_SOURCE_TYPES:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=400,
|
||||||
|
detail=f"source_type must be one of: {', '.join(sorted(ALLOWED_SOURCE_TYPES))}",
|
||||||
|
)
|
||||||
|
channel.source_type = source_type
|
||||||
|
if payload.channel is not None:
|
||||||
|
channel.channel = payload.channel.strip().lstrip("@")
|
||||||
|
if payload.is_active is not None:
|
||||||
|
channel.is_active = payload.is_active
|
||||||
|
|
||||||
|
db.commit()
|
||||||
|
db.refresh(channel)
|
||||||
|
return channel
|
||||||
|
|
||||||
|
|
||||||
|
@router.delete("/{channel_id}", status_code=204)
|
||||||
|
def delete_channel(channel_id: int, db: Session = Depends(get_db)):
|
||||||
|
channel = db.query(ParseChannel).filter(ParseChannel.id == channel_id).first()
|
||||||
|
if not channel:
|
||||||
|
raise HTTPException(status_code=404, detail="Channel not found")
|
||||||
|
|
||||||
|
linked = (
|
||||||
|
db.query(ParseJob)
|
||||||
|
.filter(ParseJob.channel_id == channel_id, ParseJob.status.in_(("queued", "running")))
|
||||||
|
.count()
|
||||||
|
)
|
||||||
|
if linked:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=409,
|
||||||
|
detail="Cannot delete channel used by queued/running jobs",
|
||||||
|
)
|
||||||
|
|
||||||
|
db.delete(channel)
|
||||||
|
db.commit()
|
||||||
@@ -1,108 +0,0 @@
|
|||||||
"""Admin API: construct static telegram parsers from a sample post."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
from fastapi import APIRouter, Depends, HTTPException
|
|
||||||
from pydantic import BaseModel, Field
|
|
||||||
from sqlalchemy.orm import Session
|
|
||||||
|
|
||||||
from contracts.heuristic_profile import HeuristicProfile
|
|
||||||
from contracts.sources import TelegramSourceConfig
|
|
||||||
|
|
||||||
from ..database import get_db
|
|
||||||
from ..models import ParseJob
|
|
||||||
from ..schemas import ParseJobRead
|
|
||||||
from ..services.jobs import enqueue_job
|
|
||||||
from ..services import parser_builder as builder
|
|
||||||
|
|
||||||
router = APIRouter(prefix="/admin/parser-builder", tags=["parser-builder"])
|
|
||||||
|
|
||||||
|
|
||||||
class GenerateRequest(BaseModel):
|
|
||||||
sample_post: str = Field(min_length=1)
|
|
||||||
|
|
||||||
|
|
||||||
class GenerateResponse(BaseModel):
|
|
||||||
profile: dict[str, Any]
|
|
||||||
|
|
||||||
|
|
||||||
class PreviewRequest(BaseModel):
|
|
||||||
sample_post: str = Field(min_length=1)
|
|
||||||
profile: dict[str, Any]
|
|
||||||
|
|
||||||
|
|
||||||
class PreviewResponse(BaseModel):
|
|
||||||
fields: dict[str, str]
|
|
||||||
|
|
||||||
|
|
||||||
class CreateProfileJobRequest(BaseModel):
|
|
||||||
channel: str = Field(min_length=1)
|
|
||||||
profile: dict[str, Any]
|
|
||||||
sample_post: str | None = None
|
|
||||||
limit: int = Field(default=100, ge=1, le=1000)
|
|
||||||
interval_seconds: int = Field(default=3600, ge=60, le=604800)
|
|
||||||
is_active: bool = True
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/target-fields")
|
|
||||||
def target_fields():
|
|
||||||
return {"fields": builder.get_target_fields()}
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/generate", response_model=GenerateResponse)
|
|
||||||
async def generate_profile(payload: GenerateRequest):
|
|
||||||
if not builder.deepseek_enabled():
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=503,
|
|
||||||
detail="DEEPSEEK_API_KEY is not set on ca-api. Add it to .env for parser generation.",
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
profile = await builder.generate_profile(payload.sample_post)
|
|
||||||
except ValueError as exc:
|
|
||||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
|
||||||
except Exception as exc:
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=502,
|
|
||||||
detail=f"DeepSeek generate failed: {exc}",
|
|
||||||
) from exc
|
|
||||||
return GenerateResponse(profile=profile.model_dump())
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/preview", response_model=PreviewResponse)
|
|
||||||
def preview_profile(payload: PreviewRequest):
|
|
||||||
try:
|
|
||||||
HeuristicProfile.model_validate(payload.profile)
|
|
||||||
fields = builder.preview_with_profile(payload.sample_post, payload.profile)
|
|
||||||
except Exception as exc:
|
|
||||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
|
||||||
return PreviewResponse(fields=fields)
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/jobs", response_model=ParseJobRead, status_code=201)
|
|
||||||
def create_profile_job(payload: CreateProfileJobRequest, db: Session = Depends(get_db)):
|
|
||||||
try:
|
|
||||||
cfg = TelegramSourceConfig(
|
|
||||||
channel=payload.channel.strip(),
|
|
||||||
limit=payload.limit,
|
|
||||||
extract_mode="profile",
|
|
||||||
heuristic_profile=payload.profile,
|
|
||||||
sample_post=payload.sample_post,
|
|
||||||
)
|
|
||||||
except Exception as exc:
|
|
||||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
|
||||||
|
|
||||||
job = ParseJob(
|
|
||||||
source_type="telegram",
|
|
||||||
source_config=cfg.model_dump(),
|
|
||||||
interval_seconds=payload.interval_seconds,
|
|
||||||
is_active=payload.is_active,
|
|
||||||
status="queued",
|
|
||||||
)
|
|
||||||
db.add(job)
|
|
||||||
db.commit()
|
|
||||||
db.refresh(job)
|
|
||||||
|
|
||||||
enqueue_job(job.id, job.source_type, job.source_config)
|
|
||||||
return job
|
|
||||||
@@ -0,0 +1,164 @@
|
|||||||
|
"""Admin CRUD for reusable parser profiles + generate/preview."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from fastapi import APIRouter, Depends, HTTPException
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from contracts.heuristic_profile import HeuristicProfile
|
||||||
|
|
||||||
|
from ..database import get_db
|
||||||
|
from ..models import ParseJob, ParserProfile
|
||||||
|
from ..schemas import ParserProfileCreate, ParserProfileRead, ParserProfileUpdate
|
||||||
|
from ..services import parser_builder as builder
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/admin/parser-profiles", tags=["parser-profiles"])
|
||||||
|
|
||||||
|
|
||||||
|
class GenerateRequest(BaseModel):
|
||||||
|
sample_post: str = Field(min_length=1)
|
||||||
|
|
||||||
|
|
||||||
|
class GenerateResponse(BaseModel):
|
||||||
|
profile: dict[str, Any]
|
||||||
|
|
||||||
|
|
||||||
|
class PreviewRequest(BaseModel):
|
||||||
|
sample_post: str = Field(min_length=1)
|
||||||
|
heuristic_profile: dict[str, Any] | None = None
|
||||||
|
# Accept alias used by older builder UI
|
||||||
|
profile: dict[str, Any] | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class PreviewResponse(BaseModel):
|
||||||
|
fields: dict[str, str]
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_heuristic_profile(raw: dict[str, Any] | None) -> dict[str, Any] | None:
|
||||||
|
if raw is None:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
return HeuristicProfile.model_validate(raw).model_dump()
|
||||||
|
except Exception as exc:
|
||||||
|
raise HTTPException(status_code=400, detail=f"Invalid heuristic_profile: {exc}") from exc
|
||||||
|
|
||||||
|
|
||||||
|
def _profile_status(heuristic_profile: dict | None, explicit: str | None = None) -> str:
|
||||||
|
if explicit:
|
||||||
|
return explicit
|
||||||
|
return "ready" if heuristic_profile else "draft"
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/target-fields")
|
||||||
|
def target_fields():
|
||||||
|
return {"fields": builder.get_target_fields()}
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/generate", response_model=GenerateResponse)
|
||||||
|
async def generate_profile(payload: GenerateRequest):
|
||||||
|
if not builder.deepseek_enabled():
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=503,
|
||||||
|
detail="DEEPSEEK_API_KEY is not set on ca-api. Add it to .env for parser generation.",
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
profile = await builder.generate_profile(payload.sample_post)
|
||||||
|
except ValueError as exc:
|
||||||
|
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||||
|
except Exception as exc:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=502,
|
||||||
|
detail=f"DeepSeek generate failed: {exc}",
|
||||||
|
) from exc
|
||||||
|
return GenerateResponse(profile=profile.model_dump())
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/preview", response_model=PreviewResponse)
|
||||||
|
def preview_profile(payload: PreviewRequest):
|
||||||
|
raw = payload.heuristic_profile if payload.heuristic_profile is not None else payload.profile
|
||||||
|
if raw is None:
|
||||||
|
raise HTTPException(status_code=400, detail="heuristic_profile is required")
|
||||||
|
try:
|
||||||
|
HeuristicProfile.model_validate(raw)
|
||||||
|
fields = builder.preview_with_profile(payload.sample_post, raw)
|
||||||
|
except Exception as exc:
|
||||||
|
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||||
|
return PreviewResponse(fields=fields)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("", response_model=list[ParserProfileRead])
|
||||||
|
def list_profiles(db: Session = Depends(get_db)):
|
||||||
|
return db.query(ParserProfile).order_by(ParserProfile.id.desc()).all()
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("", response_model=ParserProfileRead, status_code=201)
|
||||||
|
def create_profile(payload: ParserProfileCreate, db: Session = Depends(get_db)):
|
||||||
|
heuristic = _validate_heuristic_profile(payload.heuristic_profile)
|
||||||
|
profile = ParserProfile(
|
||||||
|
name=payload.name.strip(),
|
||||||
|
sample_post=payload.sample_post or "",
|
||||||
|
heuristic_profile=heuristic,
|
||||||
|
status=_profile_status(heuristic, payload.status),
|
||||||
|
)
|
||||||
|
db.add(profile)
|
||||||
|
db.commit()
|
||||||
|
db.refresh(profile)
|
||||||
|
return profile
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/{profile_id}", response_model=ParserProfileRead)
|
||||||
|
def get_profile(profile_id: int, db: Session = Depends(get_db)):
|
||||||
|
profile = db.query(ParserProfile).filter(ParserProfile.id == profile_id).first()
|
||||||
|
if not profile:
|
||||||
|
raise HTTPException(status_code=404, detail="Profile not found")
|
||||||
|
return profile
|
||||||
|
|
||||||
|
|
||||||
|
@router.patch("/{profile_id}", response_model=ParserProfileRead)
|
||||||
|
def update_profile(
|
||||||
|
profile_id: int,
|
||||||
|
payload: ParserProfileUpdate,
|
||||||
|
db: Session = Depends(get_db),
|
||||||
|
):
|
||||||
|
profile = db.query(ParserProfile).filter(ParserProfile.id == profile_id).first()
|
||||||
|
if not profile:
|
||||||
|
raise HTTPException(status_code=404, detail="Profile not found")
|
||||||
|
|
||||||
|
if payload.name is not None:
|
||||||
|
profile.name = payload.name.strip()
|
||||||
|
if payload.sample_post is not None:
|
||||||
|
profile.sample_post = payload.sample_post
|
||||||
|
if "heuristic_profile" in payload.model_fields_set:
|
||||||
|
profile.heuristic_profile = _validate_heuristic_profile(payload.heuristic_profile)
|
||||||
|
if payload.status is not None:
|
||||||
|
profile.status = payload.status
|
||||||
|
elif "heuristic_profile" in payload.model_fields_set:
|
||||||
|
profile.status = _profile_status(profile.heuristic_profile)
|
||||||
|
|
||||||
|
db.commit()
|
||||||
|
db.refresh(profile)
|
||||||
|
return profile
|
||||||
|
|
||||||
|
|
||||||
|
@router.delete("/{profile_id}", status_code=204)
|
||||||
|
def delete_profile(profile_id: int, db: Session = Depends(get_db)):
|
||||||
|
profile = db.query(ParserProfile).filter(ParserProfile.id == profile_id).first()
|
||||||
|
if not profile:
|
||||||
|
raise HTTPException(status_code=404, detail="Profile not found")
|
||||||
|
|
||||||
|
linked = (
|
||||||
|
db.query(ParseJob)
|
||||||
|
.filter(ParseJob.profile_id == profile_id, ParseJob.status.in_(("queued", "running")))
|
||||||
|
.count()
|
||||||
|
)
|
||||||
|
if linked:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=409,
|
||||||
|
detail="Cannot delete profile used by queued/running jobs",
|
||||||
|
)
|
||||||
|
|
||||||
|
db.delete(profile)
|
||||||
|
db.commit()
|
||||||
@@ -92,6 +92,38 @@ class EventRead(BaseModel):
|
|||||||
metadata: dict[str, Any] | None = Field(validation_alias="metadata_")
|
metadata: dict[str, Any] | None = Field(validation_alias="metadata_")
|
||||||
|
|
||||||
|
|
||||||
|
class EventCreate(BaseModel):
|
||||||
|
source_type: str = Field(default="manual", min_length=1, max_length=50)
|
||||||
|
source_url: str = Field(min_length=1, max_length=512)
|
||||||
|
raw_text: str = ""
|
||||||
|
title: str = ""
|
||||||
|
description: str = ""
|
||||||
|
locality: str = ""
|
||||||
|
latitude: float | None = Field(default=None, ge=-90, le=90)
|
||||||
|
longitude: float | None = Field(default=None, ge=-180, le=180)
|
||||||
|
event_date: datetime | None = None
|
||||||
|
region: str | None = None
|
||||||
|
topic: str | None = None
|
||||||
|
tags: list[str] | None = None
|
||||||
|
metadata: dict[str, Any] | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class EventUpdate(BaseModel):
|
||||||
|
source_type: str | None = Field(default=None, min_length=1, max_length=50)
|
||||||
|
source_url: str | None = Field(default=None, min_length=1, max_length=512)
|
||||||
|
raw_text: str | None = None
|
||||||
|
title: str | None = None
|
||||||
|
description: str | None = None
|
||||||
|
locality: str | None = None
|
||||||
|
latitude: float | None = Field(default=None, ge=-90, le=90)
|
||||||
|
longitude: float | None = Field(default=None, ge=-180, le=180)
|
||||||
|
event_date: datetime | None = None
|
||||||
|
region: str | None = None
|
||||||
|
topic: str | None = None
|
||||||
|
tags: list[str] | None = None
|
||||||
|
metadata: dict[str, Any] | None = None
|
||||||
|
|
||||||
|
|
||||||
class IngestEventItem(BaseModel):
|
class IngestEventItem(BaseModel):
|
||||||
source_type: str = "telegram"
|
source_type: str = "telegram"
|
||||||
source_url: str
|
source_url: str
|
||||||
@@ -127,9 +159,62 @@ class IngestResponse(BaseModel):
|
|||||||
map_objects_synced: int
|
map_objects_synced: int
|
||||||
|
|
||||||
|
|
||||||
|
class ParserProfileCreate(BaseModel):
|
||||||
|
name: str = Field(min_length=1, max_length=255)
|
||||||
|
sample_post: str = ""
|
||||||
|
heuristic_profile: dict[str, Any] | None = None
|
||||||
|
status: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class ParserProfileUpdate(BaseModel):
|
||||||
|
name: str | None = Field(default=None, min_length=1, max_length=255)
|
||||||
|
sample_post: str | None = None
|
||||||
|
heuristic_profile: dict[str, Any] | None = None
|
||||||
|
status: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class ParserProfileRead(BaseModel):
|
||||||
|
model_config = ConfigDict(from_attributes=True)
|
||||||
|
|
||||||
|
id: int
|
||||||
|
name: str
|
||||||
|
sample_post: str
|
||||||
|
heuristic_profile: dict[str, Any] | None
|
||||||
|
status: str
|
||||||
|
created_at: datetime
|
||||||
|
|
||||||
|
|
||||||
|
class ParseChannelCreate(BaseModel):
|
||||||
|
name: str = Field(min_length=1, max_length=255)
|
||||||
|
source_type: str = "telegram"
|
||||||
|
channel: str = Field(min_length=1, max_length=255)
|
||||||
|
is_active: bool = True
|
||||||
|
|
||||||
|
|
||||||
|
class ParseChannelUpdate(BaseModel):
|
||||||
|
name: str | None = Field(default=None, min_length=1, max_length=255)
|
||||||
|
source_type: str | None = None
|
||||||
|
channel: str | None = Field(default=None, min_length=1, max_length=255)
|
||||||
|
is_active: bool | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class ParseChannelRead(BaseModel):
|
||||||
|
model_config = ConfigDict(from_attributes=True)
|
||||||
|
|
||||||
|
id: int
|
||||||
|
name: str
|
||||||
|
source_type: str
|
||||||
|
channel: str
|
||||||
|
is_active: bool
|
||||||
|
created_at: datetime
|
||||||
|
|
||||||
|
|
||||||
class ParseJobCreate(BaseModel):
|
class ParseJobCreate(BaseModel):
|
||||||
source_type: str = "telegram"
|
source_type: str = "telegram"
|
||||||
source_config: dict[str, Any] = Field(default_factory=dict)
|
source_config: dict[str, Any] = Field(default_factory=dict)
|
||||||
|
profile_id: int | None = None
|
||||||
|
channel_id: int | None = None
|
||||||
|
limit: int = Field(default=100, ge=1, le=1000)
|
||||||
schedule: str | None = None
|
schedule: str | None = None
|
||||||
interval_seconds: int = Field(default=3600, ge=60, le=604800)
|
interval_seconds: int = Field(default=3600, ge=60, le=604800)
|
||||||
is_active: bool = True
|
is_active: bool = True
|
||||||
@@ -137,6 +222,9 @@ class ParseJobCreate(BaseModel):
|
|||||||
|
|
||||||
class ParseJobUpdate(BaseModel):
|
class ParseJobUpdate(BaseModel):
|
||||||
source_config: dict[str, Any] | None = None
|
source_config: dict[str, Any] | None = None
|
||||||
|
profile_id: int | None = None
|
||||||
|
channel_id: int | None = None
|
||||||
|
limit: int | None = Field(default=None, ge=1, le=1000)
|
||||||
interval_seconds: int | None = Field(default=None, ge=60, le=604800)
|
interval_seconds: int | None = Field(default=None, ge=60, le=604800)
|
||||||
is_active: bool | None = None
|
is_active: bool | None = None
|
||||||
|
|
||||||
@@ -147,6 +235,8 @@ class ParseJobRead(BaseModel):
|
|||||||
id: int
|
id: int
|
||||||
source_type: str
|
source_type: str
|
||||||
source_config: dict[str, Any]
|
source_config: dict[str, Any]
|
||||||
|
profile_id: int | None = None
|
||||||
|
channel_id: int | None = None
|
||||||
schedule: str | None
|
schedule: str | None
|
||||||
interval_seconds: int
|
interval_seconds: int
|
||||||
is_active: bool
|
is_active: bool
|
||||||
@@ -154,6 +244,9 @@ class ParseJobRead(BaseModel):
|
|||||||
last_run_at: datetime | None
|
last_run_at: datetime | None
|
||||||
last_error: str | None
|
last_error: str | None
|
||||||
created_at: datetime
|
created_at: datetime
|
||||||
|
profile_name: str | None = None
|
||||||
|
channel_name: str | None = None
|
||||||
|
channel_handle: str | None = None
|
||||||
|
|
||||||
|
|
||||||
class ConsumerCreate(BaseModel):
|
class ConsumerCreate(BaseModel):
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import sys
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import redis
|
import redis
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
# Allow importing shared contracts from monorepo root in local runs
|
# Allow importing shared contracts from monorepo root in local runs
|
||||||
_HERE = Path(__file__).resolve()
|
_HERE = Path(__file__).resolve()
|
||||||
@@ -14,7 +15,9 @@ for _candidate in (_HERE.parent, *_HERE.parents):
|
|||||||
break
|
break
|
||||||
|
|
||||||
from contracts.queues import queue_key_for_source # noqa: E402
|
from contracts.queues import queue_key_for_source # noqa: E402
|
||||||
from contracts.sources import parse_source_config # noqa: E402
|
from contracts.sources import TelegramSourceConfig, parse_source_config # noqa: E402
|
||||||
|
|
||||||
|
from ..models import ParseChannel, ParseJob, ParserProfile # noqa: E402
|
||||||
|
|
||||||
REDIS_URL = os.getenv("REDIS_URL", "redis://redis:6379/0")
|
REDIS_URL = os.getenv("REDIS_URL", "redis://redis:6379/0")
|
||||||
|
|
||||||
@@ -23,6 +26,47 @@ def get_redis() -> redis.Redis:
|
|||||||
return redis.from_url(REDIS_URL, decode_responses=True)
|
return redis.from_url(REDIS_URL, decode_responses=True)
|
||||||
|
|
||||||
|
|
||||||
|
def flatten_pair_config(
|
||||||
|
channel: ParseChannel,
|
||||||
|
profile: ParserProfile,
|
||||||
|
*,
|
||||||
|
limit: int = 100,
|
||||||
|
) -> dict:
|
||||||
|
"""Expand Profile + Channel into Redis/CP source_config."""
|
||||||
|
if not profile.heuristic_profile:
|
||||||
|
raise ValueError("ParserProfile.heuristic_profile is empty")
|
||||||
|
cfg = TelegramSourceConfig(
|
||||||
|
channel=channel.channel.strip(),
|
||||||
|
limit=limit,
|
||||||
|
extract_mode="profile",
|
||||||
|
heuristic_profile=profile.heuristic_profile,
|
||||||
|
sample_post=profile.sample_post or None,
|
||||||
|
)
|
||||||
|
return cfg.model_dump()
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_job_source_config(db: Session, job: ParseJob) -> dict:
|
||||||
|
"""
|
||||||
|
For pair jobs, rebuild flat source_config from current Profile/Channel.
|
||||||
|
Legacy jobs keep stored source_config.
|
||||||
|
"""
|
||||||
|
if job.profile_id and job.channel_id:
|
||||||
|
profile = db.query(ParserProfile).filter(ParserProfile.id == job.profile_id).first()
|
||||||
|
channel = db.query(ParseChannel).filter(ParseChannel.id == job.channel_id).first()
|
||||||
|
if not profile:
|
||||||
|
raise ValueError(f"ParserProfile {job.profile_id} not found")
|
||||||
|
if not channel:
|
||||||
|
raise ValueError(f"ParseChannel {job.channel_id} not found")
|
||||||
|
existing = job.source_config or {}
|
||||||
|
limit = existing.get("limit", 100)
|
||||||
|
if not isinstance(limit, int) or limit < 1:
|
||||||
|
limit = 100
|
||||||
|
flattened = flatten_pair_config(channel, profile, limit=limit)
|
||||||
|
job.source_config = flattened
|
||||||
|
return flattened
|
||||||
|
return dict(job.source_config or {})
|
||||||
|
|
||||||
|
|
||||||
def enqueue_job(job_id: int, source_type: str, source_config: dict) -> None:
|
def enqueue_job(job_id: int, source_type: str, source_config: dict) -> None:
|
||||||
# Validate known configs early; unknown types still raise from queue_key_for_source
|
# Validate known configs early; unknown types still raise from queue_key_for_source
|
||||||
try:
|
try:
|
||||||
@@ -38,3 +82,10 @@ def enqueue_job(job_id: int, source_type: str, source_config: dict) -> None:
|
|||||||
}
|
}
|
||||||
key = queue_key_for_source(source_type)
|
key = queue_key_for_source(source_type)
|
||||||
get_redis().rpush(key, json.dumps(payload))
|
get_redis().rpush(key, json.dumps(payload))
|
||||||
|
|
||||||
|
|
||||||
|
def enqueue_parse_job(db: Session, job: ParseJob) -> None:
|
||||||
|
"""Resolve (flatten if pair) then push to Redis."""
|
||||||
|
source_config = resolve_job_source_config(db, job)
|
||||||
|
db.commit()
|
||||||
|
enqueue_job(job.id, job.source_type, source_config)
|
||||||
|
|||||||
@@ -5,12 +5,11 @@ from sqlalchemy.engine import Engine
|
|||||||
def migrate_schema(engine: Engine) -> None:
|
def migrate_schema(engine: Engine) -> None:
|
||||||
"""Apply lightweight schema updates for existing deployments."""
|
"""Apply lightweight schema updates for existing deployments."""
|
||||||
inspector = inspect(engine)
|
inspector = inspect(engine)
|
||||||
if "parse_jobs" not in inspector.get_table_names():
|
tables = set(inspector.get_table_names())
|
||||||
return
|
|
||||||
|
|
||||||
columns = {col["name"] for col in inspector.get_columns("parse_jobs")}
|
|
||||||
statements: list[str] = []
|
statements: list[str] = []
|
||||||
|
|
||||||
|
if "parse_jobs" in tables:
|
||||||
|
columns = {col["name"] for col in inspector.get_columns("parse_jobs")}
|
||||||
if "interval_seconds" not in columns:
|
if "interval_seconds" not in columns:
|
||||||
statements.append(
|
statements.append(
|
||||||
"ALTER TABLE parse_jobs ADD COLUMN interval_seconds INTEGER NOT NULL DEFAULT 3600"
|
"ALTER TABLE parse_jobs ADD COLUMN interval_seconds INTEGER NOT NULL DEFAULT 3600"
|
||||||
@@ -19,6 +18,16 @@ def migrate_schema(engine: Engine) -> None:
|
|||||||
statements.append(
|
statements.append(
|
||||||
"ALTER TABLE parse_jobs ADD COLUMN is_active BOOLEAN NOT NULL DEFAULT TRUE"
|
"ALTER TABLE parse_jobs ADD COLUMN is_active BOOLEAN NOT NULL DEFAULT TRUE"
|
||||||
)
|
)
|
||||||
|
if "profile_id" not in columns:
|
||||||
|
statements.append("ALTER TABLE parse_jobs ADD COLUMN profile_id INTEGER")
|
||||||
|
if "channel_id" not in columns:
|
||||||
|
statements.append("ALTER TABLE parse_jobs ADD COLUMN channel_id INTEGER")
|
||||||
|
|
||||||
|
# create_all handles new tables; FKs on existing DBs may need indexes
|
||||||
|
if "parse_jobs" in tables:
|
||||||
|
# Re-inspect after potential adds is not needed for FK constraints here —
|
||||||
|
# create_all + nullable FKs are enough for MVP; optional constraints below.
|
||||||
|
pass
|
||||||
|
|
||||||
if not statements:
|
if not statements:
|
||||||
return
|
return
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ from datetime import datetime, timezone
|
|||||||
|
|
||||||
from ..database import SessionLocal
|
from ..database import SessionLocal
|
||||||
from ..models import ParseJob
|
from ..models import ParseJob
|
||||||
from .jobs import enqueue_job
|
from .jobs import enqueue_parse_job
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -36,7 +36,14 @@ def run_scheduler_tick() -> None:
|
|||||||
job.status = "queued"
|
job.status = "queued"
|
||||||
job.last_error = None
|
job.last_error = None
|
||||||
db.commit()
|
db.commit()
|
||||||
enqueue_job(job.id, job.source_type, job.source_config)
|
try:
|
||||||
|
enqueue_parse_job(db, job)
|
||||||
|
except ValueError as exc:
|
||||||
|
job.status = "failed"
|
||||||
|
job.last_error = str(exc)
|
||||||
|
db.commit()
|
||||||
|
logger.warning("Skip re-queue job %s: %s", job.id, exc)
|
||||||
|
continue
|
||||||
logger.info("Re-queued recurring job %s (interval %ss)", job.id, job.interval_seconds)
|
logger.info("Re-queued recurring job %s (interval %ss)", job.id, job.interval_seconds)
|
||||||
except Exception:
|
except Exception:
|
||||||
logger.exception("Scheduler tick failed")
|
logger.exception("Scheduler tick failed")
|
||||||
|
|||||||
@@ -4,9 +4,11 @@ import type {
|
|||||||
Consumer,
|
Consumer,
|
||||||
ConsumerCreate,
|
ConsumerCreate,
|
||||||
ConsumerUpdate,
|
ConsumerUpdate,
|
||||||
|
EventCreate,
|
||||||
EventFilters,
|
EventFilters,
|
||||||
EventListResponse,
|
EventListResponse,
|
||||||
EventRecord,
|
EventRecord,
|
||||||
|
EventUpdate,
|
||||||
ParseJob,
|
ParseJob,
|
||||||
ParseJobCreate,
|
ParseJobCreate,
|
||||||
ParseJobUpdate,
|
ParseJobUpdate,
|
||||||
@@ -69,6 +71,26 @@ export function fetchEvents(filters: EventFilters = {}): Promise<EventListRespon
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export function createEvent(payload: EventCreate): Promise<EventRecord> {
|
||||||
|
return request<EventRecord>(
|
||||||
|
"/events",
|
||||||
|
{ method: "POST", body: JSON.stringify(payload) },
|
||||||
|
ADMIN_BASE,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function updateEvent(id: number, payload: EventUpdate): Promise<EventRecord> {
|
||||||
|
return request<EventRecord>(
|
||||||
|
`/events/${id}`,
|
||||||
|
{ method: "PATCH", body: JSON.stringify(payload) },
|
||||||
|
ADMIN_BASE,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function deleteEvent(id: number): Promise<void> {
|
||||||
|
return request<void>(`/events/${id}`, { method: "DELETE" }, ADMIN_BASE);
|
||||||
|
}
|
||||||
|
|
||||||
export function fetchConsumers(): Promise<Consumer[]> {
|
export function fetchConsumers(): Promise<Consumer[]> {
|
||||||
return request<Consumer[]>("/consumers", undefined, ADMIN_BASE);
|
return request<Consumer[]>("/consumers", undefined, ADMIN_BASE);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,84 @@
|
|||||||
|
import { ADMIN_BASE, request } from "./client";
|
||||||
|
import type {
|
||||||
|
HeuristicProfile,
|
||||||
|
ParseChannel,
|
||||||
|
ParseChannelCreate,
|
||||||
|
ParseChannelUpdate,
|
||||||
|
ParserProfile,
|
||||||
|
ParserProfileCreate,
|
||||||
|
ParserProfileUpdate,
|
||||||
|
TargetField,
|
||||||
|
} from "../types/admin";
|
||||||
|
|
||||||
|
export type { HeuristicProfile, TargetField } from "../types/admin";
|
||||||
|
|
||||||
|
export function fetchChannels(): Promise<ParseChannel[]> {
|
||||||
|
return request<ParseChannel[]>("/parse-channels", undefined, ADMIN_BASE);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createChannel(payload: ParseChannelCreate): Promise<ParseChannel> {
|
||||||
|
return request<ParseChannel>(
|
||||||
|
"/parse-channels",
|
||||||
|
{ method: "POST", body: JSON.stringify(payload) },
|
||||||
|
ADMIN_BASE,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function updateChannel(id: number, payload: ParseChannelUpdate): Promise<ParseChannel> {
|
||||||
|
return request<ParseChannel>(
|
||||||
|
`/parse-channels/${id}`,
|
||||||
|
{ method: "PATCH", body: JSON.stringify(payload) },
|
||||||
|
ADMIN_BASE,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function deleteChannel(id: number): Promise<void> {
|
||||||
|
return request<void>(`/parse-channels/${id}`, { method: "DELETE" }, ADMIN_BASE);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function fetchProfiles(): Promise<ParserProfile[]> {
|
||||||
|
return request<ParserProfile[]>("/parser-profiles", undefined, ADMIN_BASE);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createProfile(payload: ParserProfileCreate): Promise<ParserProfile> {
|
||||||
|
return request<ParserProfile>(
|
||||||
|
"/parser-profiles",
|
||||||
|
{ method: "POST", body: JSON.stringify(payload) },
|
||||||
|
ADMIN_BASE,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function updateProfile(id: number, payload: ParserProfileUpdate): Promise<ParserProfile> {
|
||||||
|
return request<ParserProfile>(
|
||||||
|
`/parser-profiles/${id}`,
|
||||||
|
{ method: "PATCH", body: JSON.stringify(payload) },
|
||||||
|
ADMIN_BASE,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function deleteProfile(id: number): Promise<void> {
|
||||||
|
return request<void>(`/parser-profiles/${id}`, { method: "DELETE" }, ADMIN_BASE);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function fetchTargetFields(): Promise<{ fields: TargetField[] }> {
|
||||||
|
return request<{ fields: TargetField[] }>("/parser-profiles/target-fields", undefined, ADMIN_BASE);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function generateParserProfile(sample_post: string): Promise<{ profile: HeuristicProfile }> {
|
||||||
|
return request<{ profile: HeuristicProfile }>(
|
||||||
|
"/parser-profiles/generate",
|
||||||
|
{ method: "POST", body: JSON.stringify({ sample_post }) },
|
||||||
|
ADMIN_BASE,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function previewParserProfile(
|
||||||
|
sample_post: string,
|
||||||
|
heuristic_profile: HeuristicProfile | Record<string, unknown>,
|
||||||
|
): Promise<{ fields: Record<string, string> }> {
|
||||||
|
return request<{ fields: Record<string, string> }>(
|
||||||
|
"/parser-profiles/preview",
|
||||||
|
{ method: "POST", body: JSON.stringify({ sample_post, heuristic_profile }) },
|
||||||
|
ADMIN_BASE,
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -1,52 +0,0 @@
|
|||||||
import { ADMIN_BASE, request } from "./client";
|
|
||||||
import type { ParseJob } from "../types/admin";
|
|
||||||
|
|
||||||
export type TargetField = {
|
|
||||||
name: string;
|
|
||||||
type: string;
|
|
||||||
description: string;
|
|
||||||
};
|
|
||||||
|
|
||||||
export type HeuristicProfile = {
|
|
||||||
version: number;
|
|
||||||
fields: Record<string, Record<string, unknown>>;
|
|
||||||
notes?: string;
|
|
||||||
};
|
|
||||||
|
|
||||||
export function fetchTargetFields(): Promise<{ fields: TargetField[] }> {
|
|
||||||
return request<{ fields: TargetField[] }>("/parser-builder/target-fields", undefined, ADMIN_BASE);
|
|
||||||
}
|
|
||||||
|
|
||||||
export function generateParserProfile(sample_post: string): Promise<{ profile: HeuristicProfile }> {
|
|
||||||
return request<{ profile: HeuristicProfile }>(
|
|
||||||
"/parser-builder/generate",
|
|
||||||
{ method: "POST", body: JSON.stringify({ sample_post }) },
|
|
||||||
ADMIN_BASE,
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
export function previewParserProfile(
|
|
||||||
sample_post: string,
|
|
||||||
profile: HeuristicProfile | Record<string, unknown>,
|
|
||||||
): Promise<{ fields: Record<string, string> }> {
|
|
||||||
return request<{ fields: Record<string, string> }>(
|
|
||||||
"/parser-builder/preview",
|
|
||||||
{ method: "POST", body: JSON.stringify({ sample_post, profile }) },
|
|
||||||
ADMIN_BASE,
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
export function createProfileJob(payload: {
|
|
||||||
channel: string;
|
|
||||||
profile: HeuristicProfile | Record<string, unknown>;
|
|
||||||
sample_post?: string;
|
|
||||||
limit?: number;
|
|
||||||
interval_seconds?: number;
|
|
||||||
is_active?: boolean;
|
|
||||||
}): Promise<ParseJob> {
|
|
||||||
return request<ParseJob>(
|
|
||||||
"/parser-builder/jobs",
|
|
||||||
{ method: "POST", body: JSON.stringify(payload) },
|
|
||||||
ADMIN_BASE,
|
|
||||||
);
|
|
||||||
}
|
|
||||||
@@ -6,8 +6,9 @@ const route = useRoute();
|
|||||||
|
|
||||||
const navItems = [
|
const navItems = [
|
||||||
{ to: "/", label: "Карта", exact: true },
|
{ to: "/", label: "Карта", exact: true },
|
||||||
|
{ to: "/channels", label: "Каналы" },
|
||||||
|
{ to: "/parser-profiles", label: "Профили" },
|
||||||
{ to: "/parsers", label: "Парсеры" },
|
{ to: "/parsers", label: "Парсеры" },
|
||||||
{ to: "/parser-builder", label: "Конструктор" },
|
|
||||||
{ to: "/events", label: "События" },
|
{ to: "/events", label: "События" },
|
||||||
{ to: "/analytics", label: "Аналитика" },
|
{ to: "/analytics", label: "Аналитика" },
|
||||||
{ to: "/consumers", label: "ПИ" },
|
{ to: "/consumers", label: "ПИ" },
|
||||||
|
|||||||
@@ -2,10 +2,11 @@ import { createRouter, createWebHistory } from "vue-router";
|
|||||||
|
|
||||||
import AppLayout from "../layouts/AppLayout.vue";
|
import AppLayout from "../layouts/AppLayout.vue";
|
||||||
import AnalyticsView from "../views/AnalyticsView.vue";
|
import AnalyticsView from "../views/AnalyticsView.vue";
|
||||||
|
import ChannelsView from "../views/ChannelsView.vue";
|
||||||
import ConsumersView from "../views/ConsumersView.vue";
|
import ConsumersView from "../views/ConsumersView.vue";
|
||||||
import EventsView from "../views/EventsView.vue";
|
import EventsView from "../views/EventsView.vue";
|
||||||
import MapViewPage from "../views/MapViewPage.vue";
|
import MapViewPage from "../views/MapViewPage.vue";
|
||||||
import ParserBuilderView from "../views/ParserBuilderView.vue";
|
import ParserProfilesView from "../views/ParserProfilesView.vue";
|
||||||
import ParsersView from "../views/ParsersView.vue";
|
import ParsersView from "../views/ParsersView.vue";
|
||||||
|
|
||||||
const router = createRouter({
|
const router = createRouter({
|
||||||
@@ -16,8 +17,9 @@ const router = createRouter({
|
|||||||
component: AppLayout,
|
component: AppLayout,
|
||||||
children: [
|
children: [
|
||||||
{ path: "", name: "map", component: MapViewPage },
|
{ path: "", name: "map", component: MapViewPage },
|
||||||
|
{ path: "channels", name: "channels", component: ChannelsView },
|
||||||
|
{ path: "parser-profiles", name: "parser-profiles", component: ParserProfilesView },
|
||||||
{ path: "parsers", name: "parsers", component: ParsersView },
|
{ path: "parsers", name: "parsers", component: ParsersView },
|
||||||
{ path: "parser-builder", name: "parser-builder", component: ParserBuilderView },
|
|
||||||
{ path: "events", name: "events", component: EventsView },
|
{ path: "events", name: "events", component: EventsView },
|
||||||
{ path: "analytics", name: "analytics", component: AnalyticsView },
|
{ path: "analytics", name: "analytics", component: AnalyticsView },
|
||||||
{ path: "consumers", name: "consumers", component: ConsumersView },
|
{ path: "consumers", name: "consumers", component: ConsumersView },
|
||||||
|
|||||||
@@ -125,6 +125,15 @@ textarea {
|
|||||||
font-size: 0.8rem;
|
font-size: 0.8rem;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.btn-danger {
|
||||||
|
color: #b71c1c;
|
||||||
|
border-color: #ef9a9a;
|
||||||
|
}
|
||||||
|
|
||||||
|
.btn-danger:hover {
|
||||||
|
background: #ffebee;
|
||||||
|
}
|
||||||
|
|
||||||
.link-btn {
|
.link-btn {
|
||||||
background: none;
|
background: none;
|
||||||
border: none;
|
border: none;
|
||||||
|
|||||||
@@ -2,6 +2,8 @@ export interface ParseJob {
|
|||||||
id: number;
|
id: number;
|
||||||
source_type: string;
|
source_type: string;
|
||||||
source_config: Record<string, unknown>;
|
source_config: Record<string, unknown>;
|
||||||
|
profile_id?: number | null;
|
||||||
|
channel_id?: number | null;
|
||||||
schedule: string | null;
|
schedule: string | null;
|
||||||
interval_seconds: number;
|
interval_seconds: number;
|
||||||
is_active: boolean;
|
is_active: boolean;
|
||||||
@@ -9,11 +11,17 @@ export interface ParseJob {
|
|||||||
last_run_at: string | null;
|
last_run_at: string | null;
|
||||||
last_error: string | null;
|
last_error: string | null;
|
||||||
created_at: string;
|
created_at: string;
|
||||||
|
profile_name?: string | null;
|
||||||
|
channel_name?: string | null;
|
||||||
|
channel_handle?: string | null;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface ParseJobCreate {
|
export interface ParseJobCreate {
|
||||||
source_type: string;
|
source_type?: string;
|
||||||
source_config: Record<string, unknown>;
|
source_config?: Record<string, unknown>;
|
||||||
|
profile_id?: number;
|
||||||
|
channel_id?: number;
|
||||||
|
limit?: number;
|
||||||
schedule?: string | null;
|
schedule?: string | null;
|
||||||
interval_seconds?: number;
|
interval_seconds?: number;
|
||||||
is_active?: boolean;
|
is_active?: boolean;
|
||||||
@@ -21,10 +29,71 @@ export interface ParseJobCreate {
|
|||||||
|
|
||||||
export interface ParseJobUpdate {
|
export interface ParseJobUpdate {
|
||||||
source_config?: Record<string, unknown>;
|
source_config?: Record<string, unknown>;
|
||||||
|
profile_id?: number;
|
||||||
|
channel_id?: number;
|
||||||
|
limit?: number;
|
||||||
interval_seconds?: number;
|
interval_seconds?: number;
|
||||||
is_active?: boolean;
|
is_active?: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface ParserProfile {
|
||||||
|
id: number;
|
||||||
|
name: string;
|
||||||
|
sample_post: string;
|
||||||
|
heuristic_profile: Record<string, unknown> | null;
|
||||||
|
status: string;
|
||||||
|
created_at: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ParserProfileCreate {
|
||||||
|
name: string;
|
||||||
|
sample_post?: string;
|
||||||
|
heuristic_profile?: Record<string, unknown> | null;
|
||||||
|
status?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ParserProfileUpdate {
|
||||||
|
name?: string;
|
||||||
|
sample_post?: string;
|
||||||
|
heuristic_profile?: Record<string, unknown> | null;
|
||||||
|
status?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ParseChannel {
|
||||||
|
id: number;
|
||||||
|
name: string;
|
||||||
|
source_type: string;
|
||||||
|
channel: string;
|
||||||
|
is_active: boolean;
|
||||||
|
created_at: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ParseChannelCreate {
|
||||||
|
name: string;
|
||||||
|
source_type?: string;
|
||||||
|
channel: string;
|
||||||
|
is_active?: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ParseChannelUpdate {
|
||||||
|
name?: string;
|
||||||
|
source_type?: string;
|
||||||
|
channel?: string;
|
||||||
|
is_active?: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export type TargetField = {
|
||||||
|
name: string;
|
||||||
|
type: string;
|
||||||
|
description: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type HeuristicProfile = {
|
||||||
|
version: number;
|
||||||
|
fields: Record<string, Record<string, unknown>>;
|
||||||
|
notes?: string;
|
||||||
|
};
|
||||||
|
|
||||||
export interface EventRecord {
|
export interface EventRecord {
|
||||||
id: number;
|
id: number;
|
||||||
source_type: string;
|
source_type: string;
|
||||||
@@ -43,6 +112,38 @@ export interface EventRecord {
|
|||||||
metadata: Record<string, unknown> | null;
|
metadata: Record<string, unknown> | null;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface EventCreate {
|
||||||
|
source_type?: string;
|
||||||
|
source_url: string;
|
||||||
|
raw_text?: string;
|
||||||
|
title?: string;
|
||||||
|
description?: string;
|
||||||
|
locality?: string;
|
||||||
|
latitude?: number | null;
|
||||||
|
longitude?: number | null;
|
||||||
|
event_date?: string | null;
|
||||||
|
region?: string | null;
|
||||||
|
topic?: string | null;
|
||||||
|
tags?: string[] | null;
|
||||||
|
metadata?: Record<string, unknown> | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface EventUpdate {
|
||||||
|
source_type?: string;
|
||||||
|
source_url?: string;
|
||||||
|
raw_text?: string;
|
||||||
|
title?: string;
|
||||||
|
description?: string;
|
||||||
|
locality?: string;
|
||||||
|
latitude?: number | null;
|
||||||
|
longitude?: number | null;
|
||||||
|
event_date?: string | null;
|
||||||
|
region?: string | null;
|
||||||
|
topic?: string | null;
|
||||||
|
tags?: string[] | null;
|
||||||
|
metadata?: Record<string, unknown> | null;
|
||||||
|
}
|
||||||
|
|
||||||
export interface EventListResponse {
|
export interface EventListResponse {
|
||||||
items: EventRecord[];
|
items: EventRecord[];
|
||||||
total: number;
|
total: number;
|
||||||
|
|||||||
@@ -0,0 +1,273 @@
|
|||||||
|
<script setup lang="ts">
|
||||||
|
import { onMounted, ref } from "vue";
|
||||||
|
import {
|
||||||
|
createChannel,
|
||||||
|
deleteChannel,
|
||||||
|
fetchChannels,
|
||||||
|
updateChannel,
|
||||||
|
} from "../api/entities";
|
||||||
|
import type { ParseChannel } from "../types/admin";
|
||||||
|
|
||||||
|
const channels = ref<ParseChannel[]>([]);
|
||||||
|
const loading = ref(true);
|
||||||
|
const error = ref("");
|
||||||
|
const submitting = ref(false);
|
||||||
|
|
||||||
|
const form = ref({
|
||||||
|
name: "",
|
||||||
|
channel: "",
|
||||||
|
is_active: true,
|
||||||
|
});
|
||||||
|
|
||||||
|
const editModal = ref<{
|
||||||
|
open: boolean;
|
||||||
|
channel: ParseChannel | null;
|
||||||
|
name: string;
|
||||||
|
handle: string;
|
||||||
|
is_active: boolean;
|
||||||
|
}>({
|
||||||
|
open: false,
|
||||||
|
channel: null,
|
||||||
|
name: "",
|
||||||
|
handle: "",
|
||||||
|
is_active: true,
|
||||||
|
});
|
||||||
|
|
||||||
|
async function loadChannels() {
|
||||||
|
loading.value = true;
|
||||||
|
error.value = "";
|
||||||
|
try {
|
||||||
|
channels.value = await fetchChannels();
|
||||||
|
} catch (err) {
|
||||||
|
error.value = err instanceof Error ? err.message : "Не удалось загрузить каналы";
|
||||||
|
} finally {
|
||||||
|
loading.value = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function handleCreate() {
|
||||||
|
if (!form.value.name.trim() || !form.value.channel.trim()) {
|
||||||
|
error.value = "Укажите название и handle канала";
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
submitting.value = true;
|
||||||
|
error.value = "";
|
||||||
|
try {
|
||||||
|
await createChannel({
|
||||||
|
name: form.value.name.trim(),
|
||||||
|
channel: form.value.channel.trim(),
|
||||||
|
source_type: "telegram",
|
||||||
|
is_active: form.value.is_active,
|
||||||
|
});
|
||||||
|
form.value = { name: "", channel: "", is_active: true };
|
||||||
|
await loadChannels();
|
||||||
|
} catch (err) {
|
||||||
|
error.value = err instanceof Error ? err.message : "Не удалось создать канал";
|
||||||
|
} finally {
|
||||||
|
submitting.value = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function openEdit(ch: ParseChannel) {
|
||||||
|
editModal.value = {
|
||||||
|
open: true,
|
||||||
|
channel: ch,
|
||||||
|
name: ch.name,
|
||||||
|
handle: ch.channel,
|
||||||
|
is_active: ch.is_active,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function closeEdit() {
|
||||||
|
editModal.value.open = false;
|
||||||
|
editModal.value.channel = null;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function handleSaveEdit() {
|
||||||
|
if (!editModal.value.channel) return;
|
||||||
|
if (!editModal.value.name.trim() || !editModal.value.handle.trim()) {
|
||||||
|
error.value = "Укажите название и handle";
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
submitting.value = true;
|
||||||
|
error.value = "";
|
||||||
|
try {
|
||||||
|
await updateChannel(editModal.value.channel.id, {
|
||||||
|
name: editModal.value.name.trim(),
|
||||||
|
channel: editModal.value.handle.trim(),
|
||||||
|
is_active: editModal.value.is_active,
|
||||||
|
});
|
||||||
|
closeEdit();
|
||||||
|
await loadChannels();
|
||||||
|
} catch (err) {
|
||||||
|
error.value = err instanceof Error ? err.message : "Не удалось сохранить";
|
||||||
|
} finally {
|
||||||
|
submitting.value = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function handleDelete(ch: ParseChannel) {
|
||||||
|
if (!window.confirm(`Удалить канал «${ch.name}» (@${ch.channel})?`)) return;
|
||||||
|
error.value = "";
|
||||||
|
try {
|
||||||
|
await deleteChannel(ch.id);
|
||||||
|
await loadChannels();
|
||||||
|
} catch (err) {
|
||||||
|
error.value = err instanceof Error ? err.message : "Не удалось удалить";
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function formatDate(value: string): string {
|
||||||
|
return new Date(value).toLocaleString("ru-RU");
|
||||||
|
}
|
||||||
|
|
||||||
|
onMounted(loadChannels);
|
||||||
|
</script>
|
||||||
|
|
||||||
|
<template>
|
||||||
|
<div class="page">
|
||||||
|
<h2 class="page-heading">Каналы</h2>
|
||||||
|
<p class="intro">
|
||||||
|
Каналы для парсинга (Telegram). Связка канал + профиль создаётся на странице
|
||||||
|
<router-link to="/parsers">Парсеры</router-link>.
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<section class="card">
|
||||||
|
<h3>Новый канал</h3>
|
||||||
|
<form class="admin-form" @submit.prevent="handleCreate">
|
||||||
|
<div class="form-row">
|
||||||
|
<label>
|
||||||
|
Название
|
||||||
|
<input v-model="form.name" type="text" placeholder="Сводки фронта" required />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
Handle / id
|
||||||
|
<input v-model="form.channel" type="text" placeholder="example_channel" required />
|
||||||
|
</label>
|
||||||
|
<label class="checkbox-label">
|
||||||
|
<input v-model="form.is_active" type="checkbox" />
|
||||||
|
Активен
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
|
<div class="form-actions">
|
||||||
|
<button type="submit" class="btn btn-primary" :disabled="submitting">
|
||||||
|
{{ submitting ? "Создание..." : "Создать канал" }}
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
|
<p v-if="error" class="form-error">{{ error }}</p>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="card">
|
||||||
|
<h3>Список <span v-if="loading" class="muted">(загрузка...)</span></h3>
|
||||||
|
<div class="table-wrap">
|
||||||
|
<table class="admin-table">
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th>ID</th>
|
||||||
|
<th>Название</th>
|
||||||
|
<th>Канал</th>
|
||||||
|
<th>Тип</th>
|
||||||
|
<th>Активен</th>
|
||||||
|
<th>Создан</th>
|
||||||
|
<th></th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr v-for="ch in channels" :key="ch.id">
|
||||||
|
<td>{{ ch.id }}</td>
|
||||||
|
<td>{{ ch.name }}</td>
|
||||||
|
<td><code>@{{ ch.channel }}</code></td>
|
||||||
|
<td>{{ ch.source_type }}</td>
|
||||||
|
<td>{{ ch.is_active ? "да" : "нет" }}</td>
|
||||||
|
<td>{{ formatDate(ch.created_at) }}</td>
|
||||||
|
<td class="actions">
|
||||||
|
<button type="button" class="btn btn-sm" @click="openEdit(ch)">Изменить</button>
|
||||||
|
<button type="button" class="btn btn-sm btn-danger" @click="handleDelete(ch)">
|
||||||
|
Удалить
|
||||||
|
</button>
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
<tr v-if="!loading && !channels.length">
|
||||||
|
<td colspan="7" class="muted">Пока нет каналов</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<div v-if="editModal.open" class="modal-backdrop" @click.self="closeEdit">
|
||||||
|
<div class="modal">
|
||||||
|
<h3>Редактировать канал #{{ editModal.channel?.id }}</h3>
|
||||||
|
<form class="admin-form" @submit.prevent="handleSaveEdit">
|
||||||
|
<label>
|
||||||
|
Название
|
||||||
|
<input v-model="editModal.name" type="text" required />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
Handle / id
|
||||||
|
<input v-model="editModal.handle" type="text" required />
|
||||||
|
</label>
|
||||||
|
<label class="checkbox-label">
|
||||||
|
<input v-model="editModal.is_active" type="checkbox" />
|
||||||
|
Активен
|
||||||
|
</label>
|
||||||
|
<div class="form-actions">
|
||||||
|
<button type="button" class="btn" @click="closeEdit">Отмена</button>
|
||||||
|
<button type="submit" class="btn btn-primary" :disabled="submitting">Сохранить</button>
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</template>
|
||||||
|
|
||||||
|
<style scoped>
|
||||||
|
.intro {
|
||||||
|
margin: 0 0 1rem;
|
||||||
|
font-size: 0.9rem;
|
||||||
|
color: #555;
|
||||||
|
line-height: 1.5;
|
||||||
|
}
|
||||||
|
|
||||||
|
.checkbox-label {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 0.5rem;
|
||||||
|
margin-top: 1.5rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.actions {
|
||||||
|
display: flex;
|
||||||
|
gap: 0.35rem;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
.modal-backdrop {
|
||||||
|
position: fixed;
|
||||||
|
inset: 0;
|
||||||
|
background: rgba(0, 0, 0, 0.35);
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
z-index: 50;
|
||||||
|
}
|
||||||
|
|
||||||
|
.modal {
|
||||||
|
background: #fff;
|
||||||
|
border-radius: 8px;
|
||||||
|
padding: 1.25rem 1.5rem;
|
||||||
|
width: min(420px, 92vw);
|
||||||
|
box-shadow: 0 8px 24px rgba(0, 0, 0, 0.15);
|
||||||
|
}
|
||||||
|
|
||||||
|
.modal h3 {
|
||||||
|
margin: 0 0 1rem;
|
||||||
|
font-size: 1.05rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.modal .admin-form label {
|
||||||
|
display: block;
|
||||||
|
margin-bottom: 0.75rem;
|
||||||
|
}
|
||||||
|
</style>
|
||||||
@@ -1,7 +1,7 @@
|
|||||||
<script setup lang="ts">
|
<script setup lang="ts">
|
||||||
import { computed, onMounted, ref, watch } from "vue";
|
import { computed, onMounted, ref, watch } from "vue";
|
||||||
import { useRouter } from "vue-router";
|
import { useRouter } from "vue-router";
|
||||||
import { fetchEvents } from "../api/admin";
|
import { createEvent, deleteEvent, fetchEvents, updateEvent } from "../api/admin";
|
||||||
import type { EventFilters, EventRecord } from "../types/admin";
|
import type { EventFilters, EventRecord } from "../types/admin";
|
||||||
|
|
||||||
const router = useRouter();
|
const router = useRouter();
|
||||||
@@ -10,6 +10,7 @@ const items = ref<EventRecord[]>([]);
|
|||||||
const total = ref(0);
|
const total = ref(0);
|
||||||
const loading = ref(true);
|
const loading = ref(true);
|
||||||
const error = ref("");
|
const error = ref("");
|
||||||
|
const submitting = ref(false);
|
||||||
|
|
||||||
const filters = ref({
|
const filters = ref({
|
||||||
search: "",
|
search: "",
|
||||||
@@ -24,8 +25,44 @@ const filters = ref({
|
|||||||
const page = ref(1);
|
const page = ref(1);
|
||||||
const pageSize = 25;
|
const pageSize = 25;
|
||||||
|
|
||||||
const selectedEvent = ref<EventRecord | null>(null);
|
type EventForm = {
|
||||||
const modalOpen = ref(false);
|
source_type: string;
|
||||||
|
source_url: string;
|
||||||
|
title: string;
|
||||||
|
description: string;
|
||||||
|
locality: string;
|
||||||
|
region: string;
|
||||||
|
topic: string;
|
||||||
|
raw_text: string;
|
||||||
|
latitude: string;
|
||||||
|
longitude: string;
|
||||||
|
event_date: string;
|
||||||
|
tags: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
function emptyForm(): EventForm {
|
||||||
|
return {
|
||||||
|
source_type: "manual",
|
||||||
|
source_url: "",
|
||||||
|
title: "",
|
||||||
|
description: "",
|
||||||
|
locality: "",
|
||||||
|
region: "",
|
||||||
|
topic: "",
|
||||||
|
raw_text: "",
|
||||||
|
latitude: "",
|
||||||
|
longitude: "",
|
||||||
|
event_date: "",
|
||||||
|
tags: "",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const createForm = ref(emptyForm());
|
||||||
|
const editModal = ref<{ open: boolean; event: EventRecord | null; form: EventForm }>({
|
||||||
|
open: false,
|
||||||
|
event: null,
|
||||||
|
form: emptyForm(),
|
||||||
|
});
|
||||||
|
|
||||||
const totalPages = computed(() => Math.max(1, Math.ceil(total.value / pageSize)));
|
const totalPages = computed(() => Math.max(1, Math.ceil(total.value / pageSize)));
|
||||||
|
|
||||||
@@ -76,14 +113,128 @@ function resetFilters() {
|
|||||||
loadEvents();
|
loadEvents();
|
||||||
}
|
}
|
||||||
|
|
||||||
function openModal(event: EventRecord) {
|
function parseOptionalFloat(value: string): number | null {
|
||||||
selectedEvent.value = event;
|
const trimmed = value.trim();
|
||||||
modalOpen.value = true;
|
if (!trimmed) return null;
|
||||||
|
const num = Number(trimmed);
|
||||||
|
if (Number.isNaN(num)) throw new Error("Координаты должны быть числами");
|
||||||
|
return num;
|
||||||
}
|
}
|
||||||
|
|
||||||
function closeModal() {
|
function parseTags(value: string): string[] | null {
|
||||||
modalOpen.value = false;
|
const tags = value
|
||||||
selectedEvent.value = null;
|
.split(",")
|
||||||
|
.map((t) => t.trim())
|
||||||
|
.filter(Boolean);
|
||||||
|
return tags.length ? tags : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
function toDateTimeLocal(value: string | null): string {
|
||||||
|
if (!value) return "";
|
||||||
|
const d = new Date(value);
|
||||||
|
if (Number.isNaN(d.getTime())) return "";
|
||||||
|
const pad = (n: number) => String(n).padStart(2, "0");
|
||||||
|
return `${d.getFullYear()}-${pad(d.getMonth() + 1)}-${pad(d.getDate())}T${pad(d.getHours())}:${pad(d.getMinutes())}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function formToPayload(form: EventForm) {
|
||||||
|
const latitude = parseOptionalFloat(form.latitude);
|
||||||
|
const longitude = parseOptionalFloat(form.longitude);
|
||||||
|
if ((latitude == null) !== (longitude == null)) {
|
||||||
|
throw new Error("Укажите обе координаты или ни одной");
|
||||||
|
}
|
||||||
|
const source_url = form.source_url.trim();
|
||||||
|
if (!source_url) throw new Error("Укажите source_url");
|
||||||
|
|
||||||
|
return {
|
||||||
|
source_type: form.source_type.trim() || "manual",
|
||||||
|
source_url,
|
||||||
|
title: form.title.trim(),
|
||||||
|
description: form.description.trim(),
|
||||||
|
locality: form.locality.trim(),
|
||||||
|
region: form.region.trim() || null,
|
||||||
|
topic: form.topic.trim() || null,
|
||||||
|
raw_text: form.raw_text,
|
||||||
|
latitude,
|
||||||
|
longitude,
|
||||||
|
event_date: form.event_date ? new Date(form.event_date).toISOString() : null,
|
||||||
|
tags: parseTags(form.tags),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function eventToForm(event: EventRecord): EventForm {
|
||||||
|
return {
|
||||||
|
source_type: event.source_type || "manual",
|
||||||
|
source_url: event.source_url,
|
||||||
|
title: event.title || "",
|
||||||
|
description: event.description || "",
|
||||||
|
locality: event.locality || "",
|
||||||
|
region: event.region || "",
|
||||||
|
topic: event.topic || "",
|
||||||
|
raw_text: event.raw_text || "",
|
||||||
|
latitude: event.latitude != null ? String(event.latitude) : "",
|
||||||
|
longitude: event.longitude != null ? String(event.longitude) : "",
|
||||||
|
event_date: toDateTimeLocal(event.event_date),
|
||||||
|
tags: event.tags?.join(", ") ?? "",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async function handleCreate() {
|
||||||
|
submitting.value = true;
|
||||||
|
error.value = "";
|
||||||
|
try {
|
||||||
|
await createEvent(formToPayload(createForm.value));
|
||||||
|
createForm.value = emptyForm();
|
||||||
|
createForm.value.source_url = `manual://${crypto.randomUUID()}`;
|
||||||
|
await loadEvents();
|
||||||
|
} catch (err) {
|
||||||
|
error.value = err instanceof Error ? err.message : "Не удалось создать событие";
|
||||||
|
} finally {
|
||||||
|
submitting.value = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function openEdit(event: EventRecord) {
|
||||||
|
editModal.value = {
|
||||||
|
open: true,
|
||||||
|
event,
|
||||||
|
form: eventToForm(event),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function closeEdit() {
|
||||||
|
editModal.value.open = false;
|
||||||
|
editModal.value.event = null;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function handleSaveEdit() {
|
||||||
|
if (!editModal.value.event) return;
|
||||||
|
submitting.value = true;
|
||||||
|
error.value = "";
|
||||||
|
try {
|
||||||
|
await updateEvent(editModal.value.event.id, formToPayload(editModal.value.form));
|
||||||
|
closeEdit();
|
||||||
|
await loadEvents();
|
||||||
|
} catch (err) {
|
||||||
|
error.value = err instanceof Error ? err.message : "Не удалось сохранить событие";
|
||||||
|
} finally {
|
||||||
|
submitting.value = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function handleDelete(event: EventRecord) {
|
||||||
|
const label = event.title || event.source_url || `#${event.id}`;
|
||||||
|
if (!window.confirm(`Удалить событие #${event.id} («${label}»)? Связанная точка на карте тоже будет удалена.`)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
error.value = "";
|
||||||
|
try {
|
||||||
|
await deleteEvent(event.id);
|
||||||
|
if (editModal.value.event?.id === event.id) closeEdit();
|
||||||
|
await loadEvents();
|
||||||
|
} catch (err) {
|
||||||
|
error.value = err instanceof Error ? err.message : "Не удалось удалить событие";
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
function goToMap(event: EventRecord) {
|
function goToMap(event: EventRecord) {
|
||||||
@@ -103,13 +254,77 @@ function truncate(text: string, max = 80): string {
|
|||||||
|
|
||||||
watch(page, loadEvents);
|
watch(page, loadEvents);
|
||||||
|
|
||||||
onMounted(loadEvents);
|
onMounted(() => {
|
||||||
|
createForm.value.source_url = `manual://${crypto.randomUUID()}`;
|
||||||
|
loadEvents();
|
||||||
|
});
|
||||||
</script>
|
</script>
|
||||||
|
|
||||||
<template>
|
<template>
|
||||||
<div class="page">
|
<div class="page">
|
||||||
<h2 class="page-heading">События</h2>
|
<h2 class="page-heading">События</h2>
|
||||||
|
|
||||||
|
<section class="card">
|
||||||
|
<h3>Новое событие</h3>
|
||||||
|
<form class="admin-form" @submit.prevent="handleCreate">
|
||||||
|
<div class="form-row">
|
||||||
|
<label>
|
||||||
|
Заголовок
|
||||||
|
<input v-model="createForm.title" type="text" placeholder="Краткий заголовок" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
Источник (type)
|
||||||
|
<input v-model="createForm.source_type" type="text" placeholder="manual" />
|
||||||
|
</label>
|
||||||
|
<label class="span-2">
|
||||||
|
source_url
|
||||||
|
<input v-model="createForm.source_url" type="text" required />
|
||||||
|
</label>
|
||||||
|
<label class="span-2">
|
||||||
|
Описание
|
||||||
|
<textarea v-model="createForm.description" rows="2" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
Регион
|
||||||
|
<input v-model="createForm.region" type="text" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
Тема
|
||||||
|
<input v-model="createForm.topic" type="text" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
Населённый пункт
|
||||||
|
<input v-model="createForm.locality" type="text" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
Дата события
|
||||||
|
<input v-model="createForm.event_date" type="datetime-local" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
Широта
|
||||||
|
<input v-model="createForm.latitude" type="text" placeholder="48.1234" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
Долгота
|
||||||
|
<input v-model="createForm.longitude" type="text" placeholder="37.6543" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
Теги (через запятую)
|
||||||
|
<input v-model="createForm.tags" type="text" placeholder="обстрел, нп" />
|
||||||
|
</label>
|
||||||
|
<label class="span-2">
|
||||||
|
Сырой текст
|
||||||
|
<textarea v-model="createForm.raw_text" rows="3" />
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
|
<div class="form-actions">
|
||||||
|
<button type="submit" class="btn btn-primary" :disabled="submitting">
|
||||||
|
{{ submitting ? "Создание..." : "Создать событие" }}
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
|
</section>
|
||||||
|
|
||||||
<section class="card">
|
<section class="card">
|
||||||
<form class="admin-form filters-form" @submit.prevent="applyFilters">
|
<form class="admin-form filters-form" @submit.prevent="applyFilters">
|
||||||
<div class="form-row">
|
<div class="form-row">
|
||||||
@@ -177,7 +392,7 @@ onMounted(loadEvents);
|
|||||||
<tr v-for="event in items" :key="event.id">
|
<tr v-for="event in items" :key="event.id">
|
||||||
<td>{{ event.id }}</td>
|
<td>{{ event.id }}</td>
|
||||||
<td>
|
<td>
|
||||||
<button class="link-btn" @click="openModal(event)">
|
<button class="link-btn" type="button" @click="openEdit(event)">
|
||||||
{{ truncate(event.title || event.description || "—") }}
|
{{ truncate(event.title || event.description || "—") }}
|
||||||
</button>
|
</button>
|
||||||
</td>
|
</td>
|
||||||
@@ -191,14 +406,19 @@ onMounted(loadEvents);
|
|||||||
</template>
|
</template>
|
||||||
<template v-else>—</template>
|
<template v-else>—</template>
|
||||||
</td>
|
</td>
|
||||||
<td>
|
<td class="actions">
|
||||||
|
<button class="btn btn-sm" type="button" @click="openEdit(event)">Изменить</button>
|
||||||
<button
|
<button
|
||||||
v-if="event.latitude != null && event.longitude != null"
|
v-if="event.latitude != null && event.longitude != null"
|
||||||
class="btn btn-sm"
|
class="btn btn-sm"
|
||||||
|
type="button"
|
||||||
@click="goToMap(event)"
|
@click="goToMap(event)"
|
||||||
>
|
>
|
||||||
На карте
|
На карте
|
||||||
</button>
|
</button>
|
||||||
|
<button class="btn btn-sm btn-danger" type="button" @click="handleDelete(event)">
|
||||||
|
Удалить
|
||||||
|
</button>
|
||||||
</td>
|
</td>
|
||||||
</tr>
|
</tr>
|
||||||
</tbody>
|
</tbody>
|
||||||
@@ -206,56 +426,129 @@ onMounted(loadEvents);
|
|||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div class="pagination">
|
<div class="pagination">
|
||||||
<button class="btn btn-sm" :disabled="page <= 1" @click="page--">← Назад</button>
|
<button class="btn btn-sm" type="button" :disabled="page <= 1" @click="page--">← Назад</button>
|
||||||
<span>Стр. {{ page }} / {{ totalPages }}</span>
|
<span>Стр. {{ page }} / {{ totalPages }}</span>
|
||||||
<button class="btn btn-sm" :disabled="page >= totalPages" @click="page++">Вперёд →</button>
|
<button class="btn btn-sm" type="button" :disabled="page >= totalPages" @click="page++">
|
||||||
|
Вперёд →
|
||||||
|
</button>
|
||||||
</div>
|
</div>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
<div v-if="modalOpen && selectedEvent" class="modal-overlay" @click.self="closeModal">
|
<div v-if="editModal.open && editModal.event" class="modal-overlay" @click.self="closeEdit">
|
||||||
<div class="modal">
|
<div class="modal modal-wide">
|
||||||
<header class="modal-header">
|
<header class="modal-header">
|
||||||
<h3>Событие #{{ selectedEvent.id }}</h3>
|
<h3>Событие #{{ editModal.event.id }}</h3>
|
||||||
<button class="btn btn-sm" @click="closeModal">✕</button>
|
<button class="btn btn-sm" type="button" @click="closeEdit">✕</button>
|
||||||
</header>
|
</header>
|
||||||
<div class="modal-body">
|
<form class="modal-body admin-form" @submit.prevent="handleSaveEdit">
|
||||||
<dl class="detail-list">
|
<div class="form-row">
|
||||||
<dt>Заголовок</dt>
|
<label>
|
||||||
<dd>{{ selectedEvent.title || "—" }}</dd>
|
Заголовок
|
||||||
<dt>Описание</dt>
|
<input v-model="editModal.form.title" type="text" />
|
||||||
<dd>{{ selectedEvent.description || "—" }}</dd>
|
</label>
|
||||||
<dt>Источник</dt>
|
<label>
|
||||||
<dd>{{ selectedEvent.source_type }} — {{ selectedEvent.source_url }}</dd>
|
Источник (type)
|
||||||
<dt>Регион / Тема</dt>
|
<input v-model="editModal.form.source_type" type="text" />
|
||||||
<dd>{{ selectedEvent.region ?? "—" }} / {{ selectedEvent.topic ?? "—" }}</dd>
|
</label>
|
||||||
<dt>Населённый пункт</dt>
|
<label class="span-2">
|
||||||
<dd>{{ selectedEvent.locality || "—" }}</dd>
|
source_url
|
||||||
<dt>Дата события</dt>
|
<input v-model="editModal.form.source_url" type="text" required />
|
||||||
<dd>{{ formatDate(selectedEvent.event_date) }}</dd>
|
</label>
|
||||||
<dt>Ingested</dt>
|
<label class="span-2">
|
||||||
<dd>{{ formatDate(selectedEvent.ingested_at) }}</dd>
|
Описание
|
||||||
<dt>Координаты</dt>
|
<textarea v-model="editModal.form.description" rows="2" />
|
||||||
<dd>
|
</label>
|
||||||
<template v-if="selectedEvent.latitude != null">
|
<label>
|
||||||
{{ selectedEvent.latitude }}, {{ selectedEvent.longitude }}
|
Регион
|
||||||
</template>
|
<input v-model="editModal.form.region" type="text" />
|
||||||
<template v-else>—</template>
|
</label>
|
||||||
</dd>
|
<label>
|
||||||
<dt>Текст</dt>
|
Тема
|
||||||
<dd class="raw-text">{{ selectedEvent.raw_text || "—" }}</dd>
|
<input v-model="editModal.form.topic" type="text" />
|
||||||
</dl>
|
</label>
|
||||||
|
<label>
|
||||||
|
Населённый пункт
|
||||||
|
<input v-model="editModal.form.locality" type="text" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
Дата события
|
||||||
|
<input v-model="editModal.form.event_date" type="datetime-local" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
Широта
|
||||||
|
<input v-model="editModal.form.latitude" type="text" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
Долгота
|
||||||
|
<input v-model="editModal.form.longitude" type="text" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
Теги
|
||||||
|
<input v-model="editModal.form.tags" type="text" />
|
||||||
|
</label>
|
||||||
|
<label class="span-2">
|
||||||
|
Сырой текст
|
||||||
|
<textarea v-model="editModal.form.raw_text" rows="4" />
|
||||||
|
</label>
|
||||||
|
<p class="hint span-2">
|
||||||
|
Ingested: {{ formatDate(editModal.event.ingested_at) }}. При координатах создаётся/обновляется
|
||||||
|
точка на карте; без координат связанная точка удаляется.
|
||||||
|
</p>
|
||||||
</div>
|
</div>
|
||||||
<footer class="modal-footer">
|
<div class="form-actions">
|
||||||
|
<button type="button" class="btn" @click="closeEdit">Отмена</button>
|
||||||
<button
|
<button
|
||||||
v-if="selectedEvent.latitude != null"
|
v-if="editModal.event.latitude != null"
|
||||||
class="btn btn-primary"
|
type="button"
|
||||||
@click="goToMap(selectedEvent)"
|
class="btn"
|
||||||
|
@click="goToMap(editModal.event)"
|
||||||
>
|
>
|
||||||
Показать на карте
|
На карте
|
||||||
</button>
|
</button>
|
||||||
<button class="btn" @click="closeModal">Закрыть</button>
|
<button type="submit" class="btn btn-primary" :disabled="submitting">
|
||||||
</footer>
|
{{ submitting ? "Сохранение..." : "Сохранить" }}
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</template>
|
</template>
|
||||||
|
|
||||||
|
<style scoped>
|
||||||
|
.form-row .span-2 {
|
||||||
|
grid-column: span 2;
|
||||||
|
}
|
||||||
|
|
||||||
|
.form-row textarea {
|
||||||
|
width: 100%;
|
||||||
|
font: inherit;
|
||||||
|
padding: 0.4rem 0.5rem;
|
||||||
|
border: 1px solid #d1d5db;
|
||||||
|
border-radius: 4px;
|
||||||
|
resize: vertical;
|
||||||
|
}
|
||||||
|
|
||||||
|
.actions {
|
||||||
|
display: flex;
|
||||||
|
gap: 0.35rem;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
.hint {
|
||||||
|
margin: 0;
|
||||||
|
font-size: 0.85rem;
|
||||||
|
color: #666;
|
||||||
|
line-height: 1.4;
|
||||||
|
}
|
||||||
|
|
||||||
|
.modal-wide {
|
||||||
|
width: min(720px, 94vw);
|
||||||
|
}
|
||||||
|
|
||||||
|
.mono {
|
||||||
|
font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
|
||||||
|
font-size: 0.8rem;
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
|||||||
@@ -1,320 +0,0 @@
|
|||||||
<script setup lang="ts">
|
|
||||||
import { onMounted, ref } from "vue";
|
|
||||||
import { useRouter } from "vue-router";
|
|
||||||
import {
|
|
||||||
createProfileJob,
|
|
||||||
fetchTargetFields,
|
|
||||||
generateParserProfile,
|
|
||||||
previewParserProfile,
|
|
||||||
type HeuristicProfile,
|
|
||||||
type TargetField,
|
|
||||||
} from "../api/parserBuilder";
|
|
||||||
|
|
||||||
const INTERVAL_OPTIONS = [
|
|
||||||
{ value: 900, label: "15 мин" },
|
|
||||||
{ value: 1800, label: "30 мин" },
|
|
||||||
{ value: 3600, label: "1 ч" },
|
|
||||||
{ value: 10800, label: "3 ч" },
|
|
||||||
{ value: 21600, label: "6 ч" },
|
|
||||||
{ value: 86400, label: "24 ч" },
|
|
||||||
];
|
|
||||||
|
|
||||||
const router = useRouter();
|
|
||||||
|
|
||||||
const samplePost = ref("");
|
|
||||||
const targetFields = ref<TargetField[]>([]);
|
|
||||||
const profile = ref<HeuristicProfile | null>(null);
|
|
||||||
const profileJson = ref("");
|
|
||||||
const previewFields = ref<Record<string, string> | null>(null);
|
|
||||||
const channel = ref("");
|
|
||||||
const limit = ref(100);
|
|
||||||
const intervalSeconds = ref(3600);
|
|
||||||
|
|
||||||
const loadingFields = ref(true);
|
|
||||||
const generating = ref(false);
|
|
||||||
const previewing = ref(false);
|
|
||||||
const saving = ref(false);
|
|
||||||
const error = ref("");
|
|
||||||
const success = ref("");
|
|
||||||
|
|
||||||
async function loadTargetFields() {
|
|
||||||
loadingFields.value = true;
|
|
||||||
try {
|
|
||||||
const data = await fetchTargetFields();
|
|
||||||
targetFields.value = data.fields;
|
|
||||||
error.value = "";
|
|
||||||
} catch (err) {
|
|
||||||
error.value = err instanceof Error ? err.message : "Не удалось загрузить целевые поля";
|
|
||||||
} finally {
|
|
||||||
loadingFields.value = false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function syncProfileFromJson(): HeuristicProfile | null {
|
|
||||||
if (!profileJson.value.trim()) return null;
|
|
||||||
try {
|
|
||||||
const parsed = JSON.parse(profileJson.value) as HeuristicProfile;
|
|
||||||
profile.value = parsed;
|
|
||||||
error.value = "";
|
|
||||||
return parsed;
|
|
||||||
} catch {
|
|
||||||
error.value = "Некорректный JSON профиля";
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async function handleGenerate() {
|
|
||||||
if (!samplePost.value.trim()) {
|
|
||||||
error.value = "Вставьте образец поста";
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
generating.value = true;
|
|
||||||
error.value = "";
|
|
||||||
success.value = "";
|
|
||||||
previewFields.value = null;
|
|
||||||
try {
|
|
||||||
const data = await generateParserProfile(samplePost.value);
|
|
||||||
profile.value = data.profile;
|
|
||||||
profileJson.value = JSON.stringify(data.profile, null, 2);
|
|
||||||
success.value = "Профиль сгенерирован. Preview и runtime идут только по правилам — без LLM.";
|
|
||||||
} catch (err) {
|
|
||||||
error.value = err instanceof Error ? err.message : "Ошибка генерации";
|
|
||||||
} finally {
|
|
||||||
generating.value = false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async function handlePreview() {
|
|
||||||
const current = syncProfileFromJson();
|
|
||||||
if (!current) {
|
|
||||||
if (!error.value) error.value = "Сначала сгенерируйте профиль";
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
if (!samplePost.value.trim()) {
|
|
||||||
error.value = "Нужен образец поста для preview";
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
previewing.value = true;
|
|
||||||
error.value = "";
|
|
||||||
try {
|
|
||||||
const data = await previewParserProfile(samplePost.value, current);
|
|
||||||
previewFields.value = data.fields;
|
|
||||||
} catch (err) {
|
|
||||||
error.value = err instanceof Error ? err.message : "Ошибка preview";
|
|
||||||
} finally {
|
|
||||||
previewing.value = false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async function handleCreate() {
|
|
||||||
const current = syncProfileFromJson();
|
|
||||||
if (!current) {
|
|
||||||
if (!error.value) error.value = "Нужен профиль";
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
if (!channel.value.trim()) {
|
|
||||||
error.value = "Укажите канал Telegram";
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
saving.value = true;
|
|
||||||
error.value = "";
|
|
||||||
success.value = "";
|
|
||||||
try {
|
|
||||||
const job = await createProfileJob({
|
|
||||||
channel: channel.value.trim(),
|
|
||||||
profile: current,
|
|
||||||
sample_post: samplePost.value.trim() || undefined,
|
|
||||||
limit: limit.value,
|
|
||||||
interval_seconds: intervalSeconds.value,
|
|
||||||
is_active: true,
|
|
||||||
});
|
|
||||||
success.value = `Парсер #${job.id} создан (extract_mode=profile).`;
|
|
||||||
await router.push({ name: "parsers" });
|
|
||||||
} catch (err) {
|
|
||||||
error.value = err instanceof Error ? err.message : "Не удалось создать парсер";
|
|
||||||
} finally {
|
|
||||||
saving.value = false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
onMounted(loadTargetFields);
|
|
||||||
</script>
|
|
||||||
|
|
||||||
<template>
|
|
||||||
<div class="page">
|
|
||||||
<h2 class="page-heading">Конструктор парсера</h2>
|
|
||||||
<p class="intro">
|
|
||||||
Вставьте образец структурированного поста → DeepSeek один раз построит статичный профиль правил.
|
|
||||||
Дальше preview и парсинг канала идут только по профилю, без LLM на каждый пост.
|
|
||||||
</p>
|
|
||||||
|
|
||||||
<section class="card">
|
|
||||||
<h3>Образец поста</h3>
|
|
||||||
<textarea
|
|
||||||
v-model="samplePost"
|
|
||||||
class="sample-area"
|
|
||||||
rows="12"
|
|
||||||
placeholder="15.03.2024 Населённый пункт Описание события… 48.123456, 37.654321 #тема"
|
|
||||||
/>
|
|
||||||
<div class="form-actions">
|
|
||||||
<button
|
|
||||||
type="button"
|
|
||||||
class="btn btn-primary"
|
|
||||||
:disabled="generating || !samplePost.trim()"
|
|
||||||
@click="handleGenerate"
|
|
||||||
>
|
|
||||||
{{ generating ? "Генерация..." : "Сгенерировать парсер" }}
|
|
||||||
</button>
|
|
||||||
<button
|
|
||||||
type="button"
|
|
||||||
class="btn"
|
|
||||||
:disabled="previewing || !profileJson.trim()"
|
|
||||||
@click="handlePreview"
|
|
||||||
>
|
|
||||||
{{ previewing ? "Preview..." : "Preview" }}
|
|
||||||
</button>
|
|
||||||
</div>
|
|
||||||
</section>
|
|
||||||
|
|
||||||
<div class="grid-2">
|
|
||||||
<section class="card">
|
|
||||||
<h3>Целевая таблица <span class="muted">(Event / ingest)</span></h3>
|
|
||||||
<p v-if="loadingFields" class="muted">Загрузка…</p>
|
|
||||||
<div v-else class="table-wrap">
|
|
||||||
<table class="admin-table">
|
|
||||||
<thead>
|
|
||||||
<tr>
|
|
||||||
<th>Поле</th>
|
|
||||||
<th>Тип</th>
|
|
||||||
<th>Описание</th>
|
|
||||||
</tr>
|
|
||||||
</thead>
|
|
||||||
<tbody>
|
|
||||||
<tr v-for="field in targetFields" :key="field.name">
|
|
||||||
<td><code>{{ field.name }}</code></td>
|
|
||||||
<td>{{ field.type }}</td>
|
|
||||||
<td>{{ field.description }}</td>
|
|
||||||
</tr>
|
|
||||||
</tbody>
|
|
||||||
</table>
|
|
||||||
</div>
|
|
||||||
</section>
|
|
||||||
|
|
||||||
<section class="card">
|
|
||||||
<h3>Профиль (JSON)</h3>
|
|
||||||
<p v-if="!profileJson" class="muted">Появится после генерации. Можно слегка поправить правила.</p>
|
|
||||||
<textarea
|
|
||||||
v-model="profileJson"
|
|
||||||
class="profile-area"
|
|
||||||
rows="16"
|
|
||||||
:disabled="!profileJson && !profile"
|
|
||||||
spellcheck="false"
|
|
||||||
/>
|
|
||||||
</section>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<section v-if="previewFields" class="card">
|
|
||||||
<h3>Preview строки</h3>
|
|
||||||
<div class="table-wrap">
|
|
||||||
<table class="admin-table">
|
|
||||||
<thead>
|
|
||||||
<tr>
|
|
||||||
<th v-for="key in Object.keys(previewFields)" :key="key">{{ key }}</th>
|
|
||||||
</tr>
|
|
||||||
</thead>
|
|
||||||
<tbody>
|
|
||||||
<tr>
|
|
||||||
<td v-for="(val, key) in previewFields" :key="key" class="preview-cell">
|
|
||||||
{{ val || "—" }}
|
|
||||||
</td>
|
|
||||||
</tr>
|
|
||||||
</tbody>
|
|
||||||
</table>
|
|
||||||
</div>
|
|
||||||
</section>
|
|
||||||
|
|
||||||
<section class="card">
|
|
||||||
<h3>Создать Telegram-парсер</h3>
|
|
||||||
<form class="admin-form" @submit.prevent="handleCreate">
|
|
||||||
<div class="form-row">
|
|
||||||
<label>
|
|
||||||
Канал
|
|
||||||
<input v-model="channel" type="text" placeholder="example_channel" required />
|
|
||||||
</label>
|
|
||||||
<label>
|
|
||||||
Лимит batch
|
|
||||||
<input v-model.number="limit" type="number" min="1" max="1000" />
|
|
||||||
</label>
|
|
||||||
<label>
|
|
||||||
Интервал
|
|
||||||
<select v-model.number="intervalSeconds">
|
|
||||||
<option v-for="opt in INTERVAL_OPTIONS" :key="opt.value" :value="opt.value">
|
|
||||||
{{ opt.label }}
|
|
||||||
</option>
|
|
||||||
</select>
|
|
||||||
</label>
|
|
||||||
</div>
|
|
||||||
<div class="form-actions">
|
|
||||||
<button
|
|
||||||
type="submit"
|
|
||||||
class="btn btn-primary"
|
|
||||||
:disabled="saving || !profileJson.trim() || !channel.trim()"
|
|
||||||
>
|
|
||||||
{{ saving ? "Создание..." : "Создать парсер" }}
|
|
||||||
</button>
|
|
||||||
</div>
|
|
||||||
</form>
|
|
||||||
</section>
|
|
||||||
|
|
||||||
<p v-if="error" class="form-error">{{ error }}</p>
|
|
||||||
<p v-if="success" class="form-success">{{ success }}</p>
|
|
||||||
</div>
|
|
||||||
</template>
|
|
||||||
|
|
||||||
<style scoped>
|
|
||||||
.intro {
|
|
||||||
margin: 0 0 1rem;
|
|
||||||
font-size: 0.9rem;
|
|
||||||
color: #555;
|
|
||||||
line-height: 1.5;
|
|
||||||
max-width: 72ch;
|
|
||||||
}
|
|
||||||
|
|
||||||
.sample-area,
|
|
||||||
.profile-area {
|
|
||||||
width: 100%;
|
|
||||||
font: inherit;
|
|
||||||
font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
|
|
||||||
font-size: 0.8rem;
|
|
||||||
padding: 0.6rem 0.75rem;
|
|
||||||
border: 1px solid #d1d5db;
|
|
||||||
border-radius: 6px;
|
|
||||||
resize: vertical;
|
|
||||||
line-height: 1.45;
|
|
||||||
}
|
|
||||||
|
|
||||||
.grid-2 {
|
|
||||||
display: grid;
|
|
||||||
grid-template-columns: 1fr 1fr;
|
|
||||||
gap: 1rem;
|
|
||||||
}
|
|
||||||
|
|
||||||
@media (max-width: 960px) {
|
|
||||||
.grid-2 {
|
|
||||||
grid-template-columns: 1fr;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
.preview-cell {
|
|
||||||
max-width: 180px;
|
|
||||||
white-space: pre-wrap;
|
|
||||||
word-break: break-word;
|
|
||||||
font-size: 0.8rem;
|
|
||||||
}
|
|
||||||
|
|
||||||
.form-success {
|
|
||||||
color: #2e7d32;
|
|
||||||
font-size: 0.875rem;
|
|
||||||
margin-top: 0.5rem;
|
|
||||||
}
|
|
||||||
</style>
|
|
||||||
@@ -0,0 +1,422 @@
|
|||||||
|
<script setup lang="ts">
|
||||||
|
import { onMounted, ref } from "vue";
|
||||||
|
import {
|
||||||
|
createProfile,
|
||||||
|
deleteProfile,
|
||||||
|
fetchProfiles,
|
||||||
|
fetchTargetFields,
|
||||||
|
generateParserProfile,
|
||||||
|
previewParserProfile,
|
||||||
|
updateProfile,
|
||||||
|
type HeuristicProfile,
|
||||||
|
type TargetField,
|
||||||
|
} from "../api/entities";
|
||||||
|
import type { ParserProfile } from "../types/admin";
|
||||||
|
|
||||||
|
const profiles = ref<ParserProfile[]>([]);
|
||||||
|
const loading = ref(true);
|
||||||
|
const error = ref("");
|
||||||
|
const success = ref("");
|
||||||
|
const submitting = ref(false);
|
||||||
|
|
||||||
|
const samplePost = ref("");
|
||||||
|
const profileName = ref("");
|
||||||
|
const targetFields = ref<TargetField[]>([]);
|
||||||
|
const profileJson = ref("");
|
||||||
|
const previewFields = ref<Record<string, string> | null>(null);
|
||||||
|
const generating = ref(false);
|
||||||
|
const previewing = ref(false);
|
||||||
|
const loadingFields = ref(true);
|
||||||
|
const editingId = ref<number | null>(null);
|
||||||
|
const profileStatus = ref<"draft" | "ready">("draft");
|
||||||
|
|
||||||
|
async function loadProfiles() {
|
||||||
|
loading.value = true;
|
||||||
|
try {
|
||||||
|
profiles.value = await fetchProfiles();
|
||||||
|
error.value = "";
|
||||||
|
} catch (err) {
|
||||||
|
error.value = err instanceof Error ? err.message : "Не удалось загрузить профили";
|
||||||
|
} finally {
|
||||||
|
loading.value = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function loadTargetFields() {
|
||||||
|
loadingFields.value = true;
|
||||||
|
try {
|
||||||
|
const data = await fetchTargetFields();
|
||||||
|
targetFields.value = data.fields;
|
||||||
|
} catch (err) {
|
||||||
|
error.value = err instanceof Error ? err.message : "Не удалось загрузить целевые поля";
|
||||||
|
} finally {
|
||||||
|
loadingFields.value = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function syncProfileFromJson(): HeuristicProfile | null {
|
||||||
|
if (!profileJson.value.trim()) return null;
|
||||||
|
try {
|
||||||
|
return JSON.parse(profileJson.value) as HeuristicProfile;
|
||||||
|
} catch {
|
||||||
|
error.value = "Некорректный JSON профиля";
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function handleGenerate() {
|
||||||
|
if (!samplePost.value.trim()) {
|
||||||
|
error.value = "Вставьте образец поста";
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
generating.value = true;
|
||||||
|
error.value = "";
|
||||||
|
success.value = "";
|
||||||
|
previewFields.value = null;
|
||||||
|
try {
|
||||||
|
const data = await generateParserProfile(samplePost.value);
|
||||||
|
profileJson.value = JSON.stringify(data.profile, null, 2);
|
||||||
|
success.value = "Профиль сгенерирован. Preview и runtime — только по правилам, без LLM.";
|
||||||
|
} catch (err) {
|
||||||
|
error.value = err instanceof Error ? err.message : "Ошибка генерации";
|
||||||
|
} finally {
|
||||||
|
generating.value = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function handlePreview() {
|
||||||
|
const current = syncProfileFromJson();
|
||||||
|
if (!current) {
|
||||||
|
if (!error.value) error.value = "Сначала сгенерируйте или вставьте профиль";
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (!samplePost.value.trim()) {
|
||||||
|
error.value = "Нужен образец поста для preview";
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
previewing.value = true;
|
||||||
|
error.value = "";
|
||||||
|
try {
|
||||||
|
const data = await previewParserProfile(samplePost.value, current);
|
||||||
|
previewFields.value = data.fields;
|
||||||
|
} catch (err) {
|
||||||
|
error.value = err instanceof Error ? err.message : "Ошибка preview";
|
||||||
|
} finally {
|
||||||
|
previewing.value = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function resetEditor() {
|
||||||
|
editingId.value = null;
|
||||||
|
profileName.value = "";
|
||||||
|
samplePost.value = "";
|
||||||
|
profileJson.value = "";
|
||||||
|
profileStatus.value = "draft";
|
||||||
|
previewFields.value = null;
|
||||||
|
success.value = "";
|
||||||
|
}
|
||||||
|
|
||||||
|
function openEdit(p: ParserProfile) {
|
||||||
|
editingId.value = p.id;
|
||||||
|
profileName.value = p.name;
|
||||||
|
samplePost.value = p.sample_post || "";
|
||||||
|
profileJson.value = p.heuristic_profile
|
||||||
|
? JSON.stringify(p.heuristic_profile, null, 2)
|
||||||
|
: "";
|
||||||
|
profileStatus.value = p.status === "ready" ? "ready" : "draft";
|
||||||
|
previewFields.value = null;
|
||||||
|
success.value = "";
|
||||||
|
error.value = "";
|
||||||
|
window.scrollTo({ top: 0, behavior: "smooth" });
|
||||||
|
}
|
||||||
|
|
||||||
|
async function handleSave() {
|
||||||
|
if (!profileName.value.trim()) {
|
||||||
|
error.value = "Укажите название профиля";
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const current = syncProfileFromJson();
|
||||||
|
if (!current) {
|
||||||
|
if (!error.value) error.value = "Нужен JSON профиля (Generate или вручную)";
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
submitting.value = true;
|
||||||
|
error.value = "";
|
||||||
|
success.value = "";
|
||||||
|
try {
|
||||||
|
if (editingId.value != null) {
|
||||||
|
await updateProfile(editingId.value, {
|
||||||
|
name: profileName.value.trim(),
|
||||||
|
sample_post: samplePost.value,
|
||||||
|
heuristic_profile: current,
|
||||||
|
status: profileStatus.value,
|
||||||
|
});
|
||||||
|
success.value = `Профиль #${editingId.value} обновлён`;
|
||||||
|
} else {
|
||||||
|
const created = await createProfile({
|
||||||
|
name: profileName.value.trim(),
|
||||||
|
sample_post: samplePost.value,
|
||||||
|
heuristic_profile: current,
|
||||||
|
status: profileStatus.value === "draft" ? "draft" : "ready",
|
||||||
|
});
|
||||||
|
success.value = `Профиль #${created.id} сохранён`;
|
||||||
|
editingId.value = created.id;
|
||||||
|
profileStatus.value = created.status === "ready" ? "ready" : "draft";
|
||||||
|
}
|
||||||
|
await loadProfiles();
|
||||||
|
} catch (err) {
|
||||||
|
error.value = err instanceof Error ? err.message : "Не удалось сохранить профиль";
|
||||||
|
} finally {
|
||||||
|
submitting.value = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function handleDelete(p: ParserProfile) {
|
||||||
|
if (!window.confirm(`Удалить профиль «${p.name}»?`)) return;
|
||||||
|
error.value = "";
|
||||||
|
try {
|
||||||
|
await deleteProfile(p.id);
|
||||||
|
if (editingId.value === p.id) resetEditor();
|
||||||
|
await loadProfiles();
|
||||||
|
} catch (err) {
|
||||||
|
error.value = err instanceof Error ? err.message : "Не удалось удалить";
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function formatDate(value: string): string {
|
||||||
|
return new Date(value).toLocaleString("ru-RU");
|
||||||
|
}
|
||||||
|
|
||||||
|
onMounted(() => {
|
||||||
|
loadProfiles();
|
||||||
|
loadTargetFields();
|
||||||
|
});
|
||||||
|
</script>
|
||||||
|
|
||||||
|
<template>
|
||||||
|
<div class="page">
|
||||||
|
<h2 class="page-heading">Профили парсера</h2>
|
||||||
|
<p class="intro">
|
||||||
|
Статичный эвристический профиль: Generate (DeepSeek один раз) → Preview → Сохранить.
|
||||||
|
Дальше профиль подключается к каналу на странице
|
||||||
|
<router-link to="/parsers">Парсеры</router-link>.
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<section class="card">
|
||||||
|
<div class="card-head">
|
||||||
|
<h3>{{ editingId != null ? `Редактирование #${editingId}` : "Новый профиль" }}</h3>
|
||||||
|
<button v-if="editingId != null" type="button" class="btn btn-sm" @click="resetEditor">
|
||||||
|
Сбросить
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
<div class="form-row">
|
||||||
|
<label>
|
||||||
|
Название
|
||||||
|
<input v-model="profileName" type="text" placeholder="Сводка: дата + НП + coords" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
Статус
|
||||||
|
<select v-model="profileStatus">
|
||||||
|
<option value="draft">draft</option>
|
||||||
|
<option value="ready">ready</option>
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
|
<label>
|
||||||
|
Образец поста
|
||||||
|
<textarea
|
||||||
|
v-model="samplePost"
|
||||||
|
class="sample-area"
|
||||||
|
rows="10"
|
||||||
|
placeholder="15.03.2024 Населённый пункт Описание… 48.123456, 37.654321"
|
||||||
|
/>
|
||||||
|
</label>
|
||||||
|
<div class="form-actions">
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
class="btn btn-primary"
|
||||||
|
:disabled="generating || !samplePost.trim()"
|
||||||
|
@click="handleGenerate"
|
||||||
|
>
|
||||||
|
{{ generating ? "Генерация..." : "Generate" }}
|
||||||
|
</button>
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
class="btn"
|
||||||
|
:disabled="previewing || !profileJson.trim()"
|
||||||
|
@click="handlePreview"
|
||||||
|
>
|
||||||
|
{{ previewing ? "Preview..." : "Preview" }}
|
||||||
|
</button>
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
class="btn btn-primary"
|
||||||
|
:disabled="submitting || !profileJson.trim() || !profileName.trim()"
|
||||||
|
@click="handleSave"
|
||||||
|
>
|
||||||
|
{{ submitting ? "Сохранение..." : editingId != null ? "Обновить" : "Сохранить профиль" }}
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<div class="grid-2">
|
||||||
|
<section class="card">
|
||||||
|
<h3>Целевые поля <span class="muted">(Event)</span></h3>
|
||||||
|
<p v-if="loadingFields" class="muted">Загрузка…</p>
|
||||||
|
<div v-else class="table-wrap">
|
||||||
|
<table class="admin-table">
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th>Поле</th>
|
||||||
|
<th>Тип</th>
|
||||||
|
<th>Описание</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr v-for="field in targetFields" :key="field.name">
|
||||||
|
<td><code>{{ field.name }}</code></td>
|
||||||
|
<td>{{ field.type }}</td>
|
||||||
|
<td>{{ field.description }}</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="card">
|
||||||
|
<h3>Профиль (JSON)</h3>
|
||||||
|
<textarea
|
||||||
|
v-model="profileJson"
|
||||||
|
class="profile-area"
|
||||||
|
rows="16"
|
||||||
|
spellcheck="false"
|
||||||
|
placeholder="Появится после Generate"
|
||||||
|
/>
|
||||||
|
</section>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<section v-if="previewFields" class="card">
|
||||||
|
<h3>Preview строки</h3>
|
||||||
|
<div class="table-wrap">
|
||||||
|
<table class="admin-table">
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th v-for="key in Object.keys(previewFields)" :key="key">{{ key }}</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr>
|
||||||
|
<td v-for="(val, key) in previewFields" :key="key" class="preview-cell">
|
||||||
|
{{ val || "—" }}
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<p v-if="error" class="form-error">{{ error }}</p>
|
||||||
|
<p v-if="success" class="form-success">{{ success }}</p>
|
||||||
|
|
||||||
|
<section class="card">
|
||||||
|
<h3>Сохранённые профили <span v-if="loading" class="muted">(загрузка...)</span></h3>
|
||||||
|
<div class="table-wrap">
|
||||||
|
<table class="admin-table">
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th>ID</th>
|
||||||
|
<th>Название</th>
|
||||||
|
<th>Статус</th>
|
||||||
|
<th>Создан</th>
|
||||||
|
<th></th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr v-for="p in profiles" :key="p.id">
|
||||||
|
<td>{{ p.id }}</td>
|
||||||
|
<td>{{ p.name }}</td>
|
||||||
|
<td>{{ p.status }}</td>
|
||||||
|
<td>{{ formatDate(p.created_at) }}</td>
|
||||||
|
<td class="actions">
|
||||||
|
<button type="button" class="btn btn-sm" @click="openEdit(p)">Изменить</button>
|
||||||
|
<button type="button" class="btn btn-sm btn-danger" @click="handleDelete(p)">
|
||||||
|
Удалить
|
||||||
|
</button>
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
<tr v-if="!loading && !profiles.length">
|
||||||
|
<td colspan="5" class="muted">Пока нет профилей</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
</div>
|
||||||
|
</template>
|
||||||
|
|
||||||
|
<style scoped>
|
||||||
|
.intro {
|
||||||
|
margin: 0 0 1rem;
|
||||||
|
font-size: 0.9rem;
|
||||||
|
color: #555;
|
||||||
|
line-height: 1.5;
|
||||||
|
max-width: 72ch;
|
||||||
|
}
|
||||||
|
|
||||||
|
.card-head {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: space-between;
|
||||||
|
gap: 0.75rem;
|
||||||
|
margin-bottom: 0.5rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.card-head h3 {
|
||||||
|
margin: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.sample-area,
|
||||||
|
.profile-area {
|
||||||
|
width: 100%;
|
||||||
|
font: inherit;
|
||||||
|
font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
|
||||||
|
font-size: 0.8rem;
|
||||||
|
padding: 0.6rem 0.75rem;
|
||||||
|
border: 1px solid #d1d5db;
|
||||||
|
border-radius: 6px;
|
||||||
|
resize: vertical;
|
||||||
|
line-height: 1.45;
|
||||||
|
margin-top: 0.35rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.grid-2 {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: 1fr 1fr;
|
||||||
|
gap: 1rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
@media (max-width: 960px) {
|
||||||
|
.grid-2 {
|
||||||
|
grid-template-columns: 1fr;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
.preview-cell {
|
||||||
|
max-width: 180px;
|
||||||
|
white-space: pre-wrap;
|
||||||
|
word-break: break-word;
|
||||||
|
font-size: 0.8rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.form-success {
|
||||||
|
color: #2e7d32;
|
||||||
|
font-size: 0.875rem;
|
||||||
|
margin-top: 0.5rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.actions {
|
||||||
|
display: flex;
|
||||||
|
gap: 0.35rem;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
}
|
||||||
|
</style>
|
||||||
@@ -1,7 +1,8 @@
|
|||||||
<script setup lang="ts">
|
<script setup lang="ts">
|
||||||
import { computed, onMounted, onUnmounted, ref } from "vue";
|
import { computed, onMounted, onUnmounted, ref } from "vue";
|
||||||
import { createJob, deleteJob, fetchJobs, retryJob, updateJob } from "../api/admin";
|
import { createJob, deleteJob, fetchJobs, retryJob, updateJob } from "../api/admin";
|
||||||
import type { ParseJob } from "../types/admin";
|
import { fetchChannels, fetchProfiles } from "../api/entities";
|
||||||
|
import type { ParseChannel, ParseJob, ParserProfile } from "../types/admin";
|
||||||
|
|
||||||
const INTERVAL_OPTIONS = [
|
const INTERVAL_OPTIONS = [
|
||||||
{ value: 900, label: "15 мин" },
|
{ value: 900, label: "15 мин" },
|
||||||
@@ -12,105 +13,65 @@ const INTERVAL_OPTIONS = [
|
|||||||
{ value: 86400, label: "24 ч" },
|
{ value: 86400, label: "24 ч" },
|
||||||
];
|
];
|
||||||
|
|
||||||
const SOURCE_TYPES = [
|
|
||||||
{ value: "telegram", label: "Telegram" },
|
|
||||||
{ value: "crawl4ai", label: "Crawl4AI (web)" },
|
|
||||||
{ value: "viina", label: "VIINA (news NLP)" },
|
|
||||||
] as const;
|
|
||||||
|
|
||||||
const jobs = ref<ParseJob[]>([]);
|
const jobs = ref<ParseJob[]>([]);
|
||||||
|
const channels = ref<ParseChannel[]>([]);
|
||||||
|
const profiles = ref<ParserProfile[]>([]);
|
||||||
const loading = ref(true);
|
const loading = ref(true);
|
||||||
const error = ref("");
|
const error = ref("");
|
||||||
const submitting = ref(false);
|
const submitting = ref(false);
|
||||||
const forcingJobId = ref<number | null>(null);
|
const forcingJobId = ref<number | null>(null);
|
||||||
const editingJob = ref<ParseJob | null>(null);
|
const editingJob = ref<ParseJob | null>(null);
|
||||||
|
|
||||||
|
const pairForm = ref({
|
||||||
|
channel_id: null as number | null,
|
||||||
|
profile_id: null as number | null,
|
||||||
|
limit: 100,
|
||||||
|
interval_seconds: 3600,
|
||||||
|
});
|
||||||
|
|
||||||
const editForm = ref({
|
const editForm = ref({
|
||||||
channel: "",
|
channel_id: null as number | null,
|
||||||
limit: 50,
|
profile_id: null as number | null,
|
||||||
urlsText: "",
|
limit: 100,
|
||||||
textsText: "",
|
|
||||||
domain_profile: "generic_news",
|
|
||||||
input_mode: "urls" as "urls" | "texts" | "mixed",
|
|
||||||
extract_mode: "heuristic" as "heuristic" | "llm",
|
|
||||||
interval_seconds: 3600,
|
interval_seconds: 3600,
|
||||||
is_active: true,
|
is_active: true,
|
||||||
});
|
});
|
||||||
|
|
||||||
const form = ref({
|
const readyProfiles = computed(() =>
|
||||||
source_type: "telegram" as "telegram" | "crawl4ai" | "viina",
|
profiles.value.filter((p) => !!p.heuristic_profile),
|
||||||
channel: "",
|
);
|
||||||
limit: 50,
|
const activeChannels = computed(() => channels.value.filter((c) => c.is_active));
|
||||||
urlsText: "",
|
|
||||||
textsText: "",
|
const editChannelOptions = computed(() => channels.value);
|
||||||
domain_profile: "generic_news",
|
|
||||||
input_mode: "urls" as "urls" | "texts" | "mixed",
|
const editProfileOptions = computed(() => {
|
||||||
extract_mode: "heuristic" as "heuristic" | "llm",
|
const list = [...readyProfiles.value];
|
||||||
interval_seconds: 3600,
|
const id = editForm.value.profile_id;
|
||||||
|
if (id != null && !list.some((p) => p.id === id)) {
|
||||||
|
const found = profiles.value.find((p) => p.id === id);
|
||||||
|
if (found) list.unshift(found);
|
||||||
|
}
|
||||||
|
return list;
|
||||||
});
|
});
|
||||||
|
|
||||||
let pollTimer: ReturnType<typeof setInterval> | null = null;
|
let pollTimer: ReturnType<typeof setInterval> | null = null;
|
||||||
|
|
||||||
function channelFromConfig(config: Record<string, unknown>): string {
|
function isPairJob(job: ParseJob | null): boolean {
|
||||||
return typeof config.channel === "string" ? config.channel : "";
|
return !!job && job.profile_id != null && job.channel_id != null;
|
||||||
}
|
}
|
||||||
|
|
||||||
function limitFromConfig(config: Record<string, unknown>): number {
|
function limitFromConfig(config: Record<string, unknown>): number {
|
||||||
const limit = config.limit;
|
const limit = config.limit;
|
||||||
return typeof limit === "number" ? limit : 50;
|
return typeof limit === "number" ? limit : 100;
|
||||||
}
|
|
||||||
|
|
||||||
function urlsFromConfig(config: Record<string, unknown>): string[] {
|
|
||||||
return Array.isArray(config.urls)
|
|
||||||
? config.urls.filter((u): u is string => typeof u === "string")
|
|
||||||
: [];
|
|
||||||
}
|
|
||||||
|
|
||||||
function textsFromConfig(config: Record<string, unknown>): string[] {
|
|
||||||
return Array.isArray(config.texts)
|
|
||||||
? config.texts.filter((t): t is string => typeof t === "string")
|
|
||||||
: [];
|
|
||||||
}
|
|
||||||
|
|
||||||
function parseLines(text: string): string[] {
|
|
||||||
return text
|
|
||||||
.split(/\r?\n/)
|
|
||||||
.map((line) => line.trim())
|
|
||||||
.filter(Boolean);
|
|
||||||
}
|
|
||||||
|
|
||||||
function extractModeLabel(cfg: Record<string, unknown>): string {
|
|
||||||
if (cfg.extract_mode === "llm") return " [LLM]";
|
|
||||||
if (cfg.extract_mode === "profile") return " [profile]";
|
|
||||||
return "";
|
|
||||||
}
|
}
|
||||||
|
|
||||||
function sourceSummary(job: ParseJob): string {
|
function sourceSummary(job: ParseJob): string {
|
||||||
const cfg = job.source_config;
|
if (isPairJob(job)) {
|
||||||
const mode = extractModeLabel(cfg);
|
const ch = job.channel_name || job.channel_handle || `#${job.channel_id}`;
|
||||||
if (job.source_type === "telegram") {
|
const pr = job.profile_name || `#${job.profile_id}`;
|
||||||
return (channelFromConfig(cfg) || "—") + mode;
|
return `${ch} × ${pr}`;
|
||||||
}
|
|
||||||
const urls = urlsFromConfig(cfg);
|
|
||||||
if (urls.length) return (urls.length === 1 ? urls[0] : `${urls.length} URL`) + mode;
|
|
||||||
const texts = textsFromConfig(cfg);
|
|
||||||
if (texts.length) return `${texts.length} текст(ов)`;
|
|
||||||
return "—";
|
|
||||||
}
|
|
||||||
|
|
||||||
function isProfileJob(job: ParseJob | null): boolean {
|
|
||||||
return !!job && job.source_config.extract_mode === "profile";
|
|
||||||
}
|
|
||||||
|
|
||||||
function profileJson(job: ParseJob | null): string {
|
|
||||||
if (!job) return "";
|
|
||||||
const profile = job.source_config.heuristic_profile;
|
|
||||||
if (!profile || typeof profile !== "object") return "";
|
|
||||||
try {
|
|
||||||
return JSON.stringify(profile, null, 2);
|
|
||||||
} catch {
|
|
||||||
return "";
|
|
||||||
}
|
}
|
||||||
|
return `${job.source_type} (без связки)`;
|
||||||
}
|
}
|
||||||
|
|
||||||
function formatInterval(seconds: number): string {
|
function formatInterval(seconds: number): string {
|
||||||
@@ -121,64 +82,6 @@ function formatInterval(seconds: number): string {
|
|||||||
return `${Math.round(seconds / 86400)} д`;
|
return `${Math.round(seconds / 86400)} д`;
|
||||||
}
|
}
|
||||||
|
|
||||||
function buildSourceConfig(
|
|
||||||
sourceType: string,
|
|
||||||
data: {
|
|
||||||
channel: string;
|
|
||||||
limit: number;
|
|
||||||
urlsText: string;
|
|
||||||
textsText: string;
|
|
||||||
domain_profile: string;
|
|
||||||
input_mode: "urls" | "texts" | "mixed";
|
|
||||||
extract_mode: "heuristic" | "llm";
|
|
||||||
},
|
|
||||||
existing?: Record<string, unknown>,
|
|
||||||
): Record<string, unknown> {
|
|
||||||
if (sourceType === "telegram") {
|
|
||||||
// Preserve builder-created profile jobs (edit form has no profile editor)
|
|
||||||
if (existing?.extract_mode === "profile" && existing.heuristic_profile) {
|
|
||||||
return {
|
|
||||||
channel: data.channel.trim(),
|
|
||||||
limit: data.limit,
|
|
||||||
extract_mode: "profile",
|
|
||||||
heuristic_profile: existing.heuristic_profile,
|
|
||||||
sample_post: existing.sample_post ?? null,
|
|
||||||
};
|
|
||||||
}
|
|
||||||
return {
|
|
||||||
channel: data.channel.trim(),
|
|
||||||
limit: data.limit,
|
|
||||||
extract_mode: data.extract_mode,
|
|
||||||
};
|
|
||||||
}
|
|
||||||
if (sourceType === "crawl4ai") {
|
|
||||||
return {
|
|
||||||
urls: parseLines(data.urlsText),
|
|
||||||
domain_profile: data.domain_profile.trim() || "generic_news",
|
|
||||||
extract_mode: data.extract_mode,
|
|
||||||
};
|
|
||||||
}
|
|
||||||
return {
|
|
||||||
urls: parseLines(data.urlsText),
|
|
||||||
texts: parseLines(data.textsText),
|
|
||||||
input_mode: data.input_mode,
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
const formHint = computed(() => {
|
|
||||||
if (form.value.source_type === "telegram") {
|
|
||||||
return form.value.extract_mode === "llm"
|
|
||||||
? "LLM (DeepSeek): неструктурированные посты канала разбираются в locality/date/coords/description."
|
|
||||||
: "Активные Telegram-парсеры подхватываются real-time listener; batch забирает последние N постов (эвристики).";
|
|
||||||
}
|
|
||||||
if (form.value.source_type === "crawl4ai") {
|
|
||||||
return form.value.extract_mode === "llm"
|
|
||||||
? "Crawl4AI + LLM (DeepSeek): страница → structured event. Нужен DEEPSEEK_API_KEY и cp-workers-web."
|
|
||||||
: "Crawl4AI обходит URL и нормализует страницы эвристиками. Обрабатывает cp-workers-web.";
|
|
||||||
}
|
|
||||||
return "VIINA-адаптер извлекает инциденты из новостных URL/текстов. Обрабатывает cp-workers-nlp.";
|
|
||||||
});
|
|
||||||
|
|
||||||
async function loadJobs() {
|
async function loadJobs() {
|
||||||
try {
|
try {
|
||||||
jobs.value = await fetchJobs();
|
jobs.value = await fetchJobs();
|
||||||
@@ -190,68 +93,51 @@ async function loadJobs() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
async function handleSubmit() {
|
async function loadEntities() {
|
||||||
if (form.value.source_type === "telegram" && !form.value.channel.trim()) {
|
try {
|
||||||
error.value = "Укажите канал Telegram";
|
const [chs, prs] = await Promise.all([fetchChannels(), fetchProfiles()]);
|
||||||
return;
|
channels.value = chs;
|
||||||
}
|
profiles.value = prs;
|
||||||
if (form.value.source_type === "crawl4ai" && !parseLines(form.value.urlsText).length) {
|
} catch {
|
||||||
error.value = "Укажите хотя бы один URL";
|
/* selects stay empty */
|
||||||
return;
|
|
||||||
}
|
|
||||||
if (form.value.source_type === "viina") {
|
|
||||||
const urls = parseLines(form.value.urlsText);
|
|
||||||
const texts = parseLines(form.value.textsText);
|
|
||||||
if (form.value.input_mode === "urls" && !urls.length) {
|
|
||||||
error.value = "Укажите URL для VIINA";
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
if (form.value.input_mode === "texts" && !texts.length) {
|
|
||||||
error.value = "Укажите тексты для VIINA";
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
if (form.value.input_mode === "mixed" && !urls.length && !texts.length) {
|
|
||||||
error.value = "Укажите URL или тексты";
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function handleCreatePair() {
|
||||||
|
if (!pairForm.value.channel_id || !pairForm.value.profile_id) {
|
||||||
|
error.value = "Выберите канал и профиль";
|
||||||
|
return;
|
||||||
|
}
|
||||||
submitting.value = true;
|
submitting.value = true;
|
||||||
error.value = "";
|
error.value = "";
|
||||||
try {
|
try {
|
||||||
await createJob({
|
await createJob({
|
||||||
source_type: form.value.source_type,
|
channel_id: pairForm.value.channel_id,
|
||||||
source_config: buildSourceConfig(form.value.source_type, form.value),
|
profile_id: pairForm.value.profile_id,
|
||||||
interval_seconds: form.value.interval_seconds,
|
limit: pairForm.value.limit,
|
||||||
|
interval_seconds: pairForm.value.interval_seconds,
|
||||||
is_active: true,
|
is_active: true,
|
||||||
});
|
});
|
||||||
form.value.channel = "";
|
pairForm.value.channel_id = null;
|
||||||
form.value.urlsText = "";
|
pairForm.value.profile_id = null;
|
||||||
form.value.textsText = "";
|
|
||||||
await loadJobs();
|
await loadJobs();
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
error.value = err instanceof Error ? err.message : "Не удалось создать задание";
|
error.value = err instanceof Error ? err.message : "Не удалось создать связку";
|
||||||
} finally {
|
} finally {
|
||||||
submitting.value = false;
|
submitting.value = false;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
function openEdit(job: ParseJob) {
|
function openEdit(job: ParseJob) {
|
||||||
|
if (!isPairJob(job)) {
|
||||||
|
error.value = "Редактируются только связки канал+профиль. Остальные задания можно удалить.";
|
||||||
|
return;
|
||||||
|
}
|
||||||
editingJob.value = job;
|
editingJob.value = job;
|
||||||
editForm.value = {
|
editForm.value = {
|
||||||
channel: channelFromConfig(job.source_config),
|
channel_id: job.channel_id ?? null,
|
||||||
|
profile_id: job.profile_id ?? null,
|
||||||
limit: limitFromConfig(job.source_config),
|
limit: limitFromConfig(job.source_config),
|
||||||
urlsText: urlsFromConfig(job.source_config).join("\n"),
|
|
||||||
textsText: textsFromConfig(job.source_config).join("\n"),
|
|
||||||
domain_profile:
|
|
||||||
typeof job.source_config.domain_profile === "string"
|
|
||||||
? job.source_config.domain_profile
|
|
||||||
: "generic_news",
|
|
||||||
input_mode:
|
|
||||||
job.source_config.input_mode === "texts" || job.source_config.input_mode === "mixed"
|
|
||||||
? job.source_config.input_mode
|
|
||||||
: "urls",
|
|
||||||
extract_mode: job.source_config.extract_mode === "llm" ? "llm" : "heuristic",
|
|
||||||
interval_seconds: job.interval_seconds,
|
interval_seconds: job.interval_seconds,
|
||||||
is_active: job.is_active,
|
is_active: job.is_active,
|
||||||
};
|
};
|
||||||
@@ -262,19 +148,18 @@ function closeEdit() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
async function handleSaveEdit() {
|
async function handleSaveEdit() {
|
||||||
if (!editingJob.value) return;
|
if (!editingJob.value || !isPairJob(editingJob.value)) return;
|
||||||
const sourceType = editingJob.value.source_type;
|
if (!editForm.value.channel_id || !editForm.value.profile_id) {
|
||||||
|
error.value = "Выберите канал и профиль";
|
||||||
if (sourceType === "telegram" && !editForm.value.channel.trim()) {
|
|
||||||
error.value = "Укажите канал Telegram";
|
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
submitting.value = true;
|
submitting.value = true;
|
||||||
error.value = "";
|
error.value = "";
|
||||||
try {
|
try {
|
||||||
await updateJob(editingJob.value.id, {
|
await updateJob(editingJob.value.id, {
|
||||||
source_config: buildSourceConfig(sourceType, editForm.value, editingJob.value.source_config),
|
channel_id: editForm.value.channel_id,
|
||||||
|
profile_id: editForm.value.profile_id,
|
||||||
|
limit: editForm.value.limit,
|
||||||
interval_seconds: editForm.value.interval_seconds,
|
interval_seconds: editForm.value.interval_seconds,
|
||||||
is_active: editForm.value.is_active,
|
is_active: editForm.value.is_active,
|
||||||
});
|
});
|
||||||
@@ -333,6 +218,7 @@ function formatDate(value: string | null): string {
|
|||||||
|
|
||||||
onMounted(() => {
|
onMounted(() => {
|
||||||
loadJobs();
|
loadJobs();
|
||||||
|
loadEntities();
|
||||||
pollTimer = setInterval(loadJobs, 5000);
|
pollTimer = setInterval(loadJobs, 5000);
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -346,82 +232,42 @@ onUnmounted(() => {
|
|||||||
<h2 class="page-heading">Парсеры</h2>
|
<h2 class="page-heading">Парсеры</h2>
|
||||||
|
|
||||||
<section class="card">
|
<section class="card">
|
||||||
<h3>Новый парсер</h3>
|
<h3>Новая связка (канал + профиль)</h3>
|
||||||
<p class="hint">
|
<p class="hint">
|
||||||
{{ formHint }}
|
Создайте
|
||||||
Дубликаты по <code>source_url</code> не записываются.
|
<router-link to="/channels">канал</router-link>
|
||||||
Статичный парсер по образцу поста —
|
и
|
||||||
<router-link to="/parser-builder">Конструктор</router-link>.
|
<router-link to="/parser-profiles">профиль</router-link>,
|
||||||
|
затем свяжите их здесь. В Redis уходит плоский
|
||||||
|
<code>extract_mode=profile</code> + <code>heuristic_profile</code>.
|
||||||
</p>
|
</p>
|
||||||
<form class="admin-form" @submit.prevent="handleSubmit">
|
<form class="admin-form" @submit.prevent="handleCreatePair">
|
||||||
<div class="form-row">
|
<div class="form-row">
|
||||||
<label>
|
<label>
|
||||||
Тип источника
|
Канал
|
||||||
<select v-model="form.source_type">
|
<select v-model.number="pairForm.channel_id" required>
|
||||||
<option v-for="opt in SOURCE_TYPES" :key="opt.value" :value="opt.value">
|
<option :value="null" disabled>Выберите…</option>
|
||||||
{{ opt.label }}
|
<option v-for="ch in activeChannels" :key="ch.id" :value="ch.id">
|
||||||
|
{{ ch.name }} (@{{ ch.channel }})
|
||||||
</option>
|
</option>
|
||||||
</select>
|
</select>
|
||||||
</label>
|
</label>
|
||||||
|
|
||||||
<template v-if="form.source_type === 'telegram'">
|
|
||||||
<label>
|
<label>
|
||||||
Канал
|
Профиль
|
||||||
<input v-model="form.channel" type="text" placeholder="creamy_caprice" required />
|
<select v-model.number="pairForm.profile_id" required>
|
||||||
</label>
|
<option :value="null" disabled>Выберите…</option>
|
||||||
<label>
|
<option v-for="p in readyProfiles" :key="p.id" :value="p.id">
|
||||||
Лимит
|
{{ p.name }} (#{{ p.id }})
|
||||||
<input v-model.number="form.limit" type="number" min="1" max="1000" />
|
</option>
|
||||||
</label>
|
|
||||||
<label>
|
|
||||||
Извлечение
|
|
||||||
<select v-model="form.extract_mode">
|
|
||||||
<option value="heuristic">Эвристики (структурированные посты)</option>
|
|
||||||
<option value="llm">LLM DeepSeek (неструктурированные)</option>
|
|
||||||
</select>
|
</select>
|
||||||
</label>
|
</label>
|
||||||
</template>
|
|
||||||
|
|
||||||
<template v-else-if="form.source_type === 'crawl4ai'">
|
|
||||||
<label class="span-2">
|
|
||||||
URL (по одному в строке)
|
|
||||||
<textarea v-model="form.urlsText" rows="3" placeholder="https://example.com/news/…" required />
|
|
||||||
</label>
|
|
||||||
<label>
|
<label>
|
||||||
Domain profile
|
Лимит batch
|
||||||
<input v-model="form.domain_profile" type="text" placeholder="generic_news" />
|
<input v-model.number="pairForm.limit" type="number" min="1" max="1000" />
|
||||||
</label>
|
</label>
|
||||||
<label>
|
|
||||||
Извлечение
|
|
||||||
<select v-model="form.extract_mode">
|
|
||||||
<option value="heuristic">Эвристики (regex)</option>
|
|
||||||
<option value="llm">LLM DeepSeek (Crawl4AI)</option>
|
|
||||||
</select>
|
|
||||||
</label>
|
|
||||||
</template>
|
|
||||||
|
|
||||||
<template v-else>
|
|
||||||
<label>
|
|
||||||
Режим ввода
|
|
||||||
<select v-model="form.input_mode">
|
|
||||||
<option value="urls">URLs</option>
|
|
||||||
<option value="texts">Texts</option>
|
|
||||||
<option value="mixed">Mixed</option>
|
|
||||||
</select>
|
|
||||||
</label>
|
|
||||||
<label v-if="form.input_mode !== 'texts'" class="span-2">
|
|
||||||
URL (по одному в строке)
|
|
||||||
<textarea v-model="form.urlsText" rows="3" placeholder="https://example.com/article…" />
|
|
||||||
</label>
|
|
||||||
<label v-if="form.input_mode !== 'urls'" class="span-2">
|
|
||||||
Тексты (по одному блоку в строке)
|
|
||||||
<textarea v-model="form.textsText" rows="3" placeholder="Текст статьи…" />
|
|
||||||
</label>
|
|
||||||
</template>
|
|
||||||
|
|
||||||
<label>
|
<label>
|
||||||
Интервал
|
Интервал
|
||||||
<select v-model.number="form.interval_seconds">
|
<select v-model.number="pairForm.interval_seconds">
|
||||||
<option v-for="opt in INTERVAL_OPTIONS" :key="opt.value" :value="opt.value">
|
<option v-for="opt in INTERVAL_OPTIONS" :key="opt.value" :value="opt.value">
|
||||||
{{ opt.label }}
|
{{ opt.label }}
|
||||||
</option>
|
</option>
|
||||||
@@ -429,16 +275,24 @@ onUnmounted(() => {
|
|||||||
</label>
|
</label>
|
||||||
</div>
|
</div>
|
||||||
<div class="form-actions">
|
<div class="form-actions">
|
||||||
<button type="submit" class="btn btn-primary" :disabled="submitting">
|
<button
|
||||||
{{ submitting ? "Создание..." : "Создать парсер" }}
|
type="submit"
|
||||||
|
class="btn btn-primary"
|
||||||
|
:disabled="submitting || !pairForm.channel_id || !pairForm.profile_id"
|
||||||
|
>
|
||||||
|
{{ submitting ? "Создание..." : "Создать связку" }}
|
||||||
</button>
|
</button>
|
||||||
</div>
|
</div>
|
||||||
</form>
|
</form>
|
||||||
|
<p v-if="!activeChannels.length || !readyProfiles.length" class="hint">
|
||||||
|
Нет активных каналов или готовых профилей —
|
||||||
|
сначала добавьте их на страницах Каналы / Профили.
|
||||||
|
</p>
|
||||||
<p v-if="error" class="form-error">{{ error }}</p>
|
<p v-if="error" class="form-error">{{ error }}</p>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
<section class="card">
|
<section class="card">
|
||||||
<h3>Парсеры <span v-if="loading" class="muted">(загрузка...)</span></h3>
|
<h3>Связки <span v-if="loading" class="muted">(загрузка...)</span></h3>
|
||||||
<div class="table-wrap">
|
<div class="table-wrap">
|
||||||
<table class="admin-table">
|
<table class="admin-table">
|
||||||
<thead>
|
<thead>
|
||||||
@@ -470,16 +324,18 @@ onUnmounted(() => {
|
|||||||
<td>{{ formatDate(job.last_run_at) }}</td>
|
<td>{{ formatDate(job.last_run_at) }}</td>
|
||||||
<td class="error-cell">{{ job.last_error ?? "—" }}</td>
|
<td class="error-cell">{{ job.last_error ?? "—" }}</td>
|
||||||
<td class="actions-cell">
|
<td class="actions-cell">
|
||||||
<button class="btn btn-sm" type="button" @click="openEdit(job)">Изменить</button>
|
<button
|
||||||
|
v-if="isPairJob(job)"
|
||||||
|
class="btn btn-sm"
|
||||||
|
type="button"
|
||||||
|
@click="openEdit(job)"
|
||||||
|
>
|
||||||
|
Изменить
|
||||||
|
</button>
|
||||||
<button
|
<button
|
||||||
class="btn btn-sm"
|
class="btn btn-sm"
|
||||||
type="button"
|
type="button"
|
||||||
:disabled="!canForceRun(job) || forcingJobId === job.id"
|
:disabled="!canForceRun(job) || forcingJobId === job.id"
|
||||||
:title="
|
|
||||||
canForceRun(job)
|
|
||||||
? 'Принудительно выполнить запрос постов'
|
|
||||||
: 'Парсер уже в очереди или выполняется'
|
|
||||||
"
|
|
||||||
@click="handleForceRun(job.id)"
|
@click="handleForceRun(job.id)"
|
||||||
>
|
>
|
||||||
{{ forcingJobId === job.id ? "Запуск..." : "Запустить" }}
|
{{ forcingJobId === job.id ? "Запуск..." : "Запустить" }}
|
||||||
@@ -501,69 +357,29 @@ onUnmounted(() => {
|
|||||||
|
|
||||||
<div v-if="editingJob" class="modal-backdrop" @click.self="closeEdit">
|
<div v-if="editingJob" class="modal-backdrop" @click.self="closeEdit">
|
||||||
<div class="modal card">
|
<div class="modal card">
|
||||||
<h3>Редактировать парсер #{{ editingJob.id }} ({{ editingJob.source_type }})</h3>
|
<h3>Редактировать связку #{{ editingJob.id }}</h3>
|
||||||
<form class="admin-form" @submit.prevent="handleSaveEdit">
|
<form class="admin-form" @submit.prevent="handleSaveEdit">
|
||||||
<div class="form-row">
|
<div class="form-row">
|
||||||
<template v-if="editingJob.source_type === 'telegram'">
|
|
||||||
<label>
|
<label>
|
||||||
Канал
|
Канал
|
||||||
<input v-model="editForm.channel" type="text" required />
|
<select v-model.number="editForm.channel_id" required>
|
||||||
|
<option v-for="ch in editChannelOptions" :key="ch.id" :value="ch.id">
|
||||||
|
{{ ch.name }} (@{{ ch.channel }}){{ ch.is_active ? "" : " — выкл." }}
|
||||||
|
</option>
|
||||||
|
</select>
|
||||||
</label>
|
</label>
|
||||||
<label>
|
<label>
|
||||||
Лимит
|
Профиль
|
||||||
|
<select v-model.number="editForm.profile_id" required>
|
||||||
|
<option v-for="p in editProfileOptions" :key="p.id" :value="p.id">
|
||||||
|
{{ p.name }} (#{{ p.id }})
|
||||||
|
</option>
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
Лимит batch
|
||||||
<input v-model.number="editForm.limit" type="number" min="1" max="1000" />
|
<input v-model.number="editForm.limit" type="number" min="1" max="1000" />
|
||||||
</label>
|
</label>
|
||||||
<label v-if="!isProfileJob(editingJob)">
|
|
||||||
Извлечение
|
|
||||||
<select v-model="editForm.extract_mode">
|
|
||||||
<option value="heuristic">Эвристики</option>
|
|
||||||
<option value="llm">LLM DeepSeek</option>
|
|
||||||
</select>
|
|
||||||
</label>
|
|
||||||
<p v-else class="hint profile-hint">
|
|
||||||
Режим: <code>profile</code> (статичный профиль из конструктора).
|
|
||||||
Сменить правила — через повторную генерацию в «Конструктор».
|
|
||||||
</p>
|
|
||||||
<label v-if="isProfileJob(editingJob)" class="span-2">
|
|
||||||
Профиль (только просмотр)
|
|
||||||
<textarea class="profile-ro" :value="profileJson(editingJob)" rows="8" readonly />
|
|
||||||
</label>
|
|
||||||
</template>
|
|
||||||
<template v-else-if="editingJob.source_type === 'crawl4ai'">
|
|
||||||
<label class="span-2">
|
|
||||||
URL (по одному в строке)
|
|
||||||
<textarea v-model="editForm.urlsText" rows="3" required />
|
|
||||||
</label>
|
|
||||||
<label>
|
|
||||||
Domain profile
|
|
||||||
<input v-model="editForm.domain_profile" type="text" />
|
|
||||||
</label>
|
|
||||||
<label>
|
|
||||||
Извлечение
|
|
||||||
<select v-model="editForm.extract_mode">
|
|
||||||
<option value="heuristic">Эвристики</option>
|
|
||||||
<option value="llm">LLM DeepSeek</option>
|
|
||||||
</select>
|
|
||||||
</label>
|
|
||||||
</template>
|
|
||||||
<template v-else>
|
|
||||||
<label>
|
|
||||||
Режим ввода
|
|
||||||
<select v-model="editForm.input_mode">
|
|
||||||
<option value="urls">URLs</option>
|
|
||||||
<option value="texts">Texts</option>
|
|
||||||
<option value="mixed">Mixed</option>
|
|
||||||
</select>
|
|
||||||
</label>
|
|
||||||
<label v-if="editForm.input_mode !== 'texts'" class="span-2">
|
|
||||||
URL
|
|
||||||
<textarea v-model="editForm.urlsText" rows="3" />
|
|
||||||
</label>
|
|
||||||
<label v-if="editForm.input_mode !== 'urls'" class="span-2">
|
|
||||||
Тексты
|
|
||||||
<textarea v-model="editForm.textsText" rows="3" />
|
|
||||||
</label>
|
|
||||||
</template>
|
|
||||||
<label>
|
<label>
|
||||||
Интервал
|
Интервал
|
||||||
<select v-model.number="editForm.interval_seconds">
|
<select v-model.number="editForm.interval_seconds">
|
||||||
@@ -597,19 +413,6 @@ onUnmounted(() => {
|
|||||||
line-height: 1.5;
|
line-height: 1.5;
|
||||||
}
|
}
|
||||||
|
|
||||||
.form-row .span-2 {
|
|
||||||
grid-column: span 2;
|
|
||||||
}
|
|
||||||
|
|
||||||
.form-row textarea {
|
|
||||||
width: 100%;
|
|
||||||
font: inherit;
|
|
||||||
padding: 0.4rem 0.5rem;
|
|
||||||
border: 1px solid #d1d5db;
|
|
||||||
border-radius: 4px;
|
|
||||||
resize: vertical;
|
|
||||||
}
|
|
||||||
|
|
||||||
.source-cell {
|
.source-cell {
|
||||||
max-width: 280px;
|
max-width: 280px;
|
||||||
overflow: hidden;
|
overflow: hidden;
|
||||||
@@ -625,11 +428,6 @@ onUnmounted(() => {
|
|||||||
margin-right: 0.25rem;
|
margin-right: 0.25rem;
|
||||||
}
|
}
|
||||||
|
|
||||||
.btn-danger {
|
|
||||||
color: #b91c1c;
|
|
||||||
border-color: #fecaca;
|
|
||||||
}
|
|
||||||
|
|
||||||
.modal-backdrop {
|
.modal-backdrop {
|
||||||
position: fixed;
|
position: fixed;
|
||||||
inset: 0;
|
inset: 0;
|
||||||
@@ -643,8 +441,6 @@ onUnmounted(() => {
|
|||||||
.modal {
|
.modal {
|
||||||
width: min(560px, 92vw);
|
width: min(560px, 92vw);
|
||||||
margin: 0;
|
margin: 0;
|
||||||
max-height: 90vh;
|
|
||||||
overflow: auto;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
.checkbox-row {
|
.checkbox-row {
|
||||||
@@ -654,16 +450,12 @@ onUnmounted(() => {
|
|||||||
margin-top: 1.5rem;
|
margin-top: 1.5rem;
|
||||||
}
|
}
|
||||||
|
|
||||||
.profile-hint {
|
.error-cell {
|
||||||
flex: 1 1 100%;
|
max-width: 200px;
|
||||||
margin: 0;
|
overflow: hidden;
|
||||||
}
|
text-overflow: ellipsis;
|
||||||
|
white-space: nowrap;
|
||||||
.profile-ro {
|
font-size: 0.8rem;
|
||||||
width: 100%;
|
color: #b91c1c;
|
||||||
font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
|
|
||||||
font-size: 0.75rem;
|
|
||||||
background: #f8f8f8;
|
|
||||||
color: #444;
|
|
||||||
}
|
}
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -70,7 +70,7 @@ async def run(job_id, source_config, *, ctx) -> tuple[list[dict], str | None]
|
|||||||
- `heuristic_profile` в `source_config` (схема `contracts/heuristic_profile.py`);
|
- `heuristic_profile` в `source_config` (схема `contracts/heuristic_profile.py`);
|
||||||
- интерпретатор: `workers/heuristic_profile.py` (те же правила, что preview в ЦА);
|
- интерпретатор: `workers/heuristic_profile.py` (те же правила, что preview в ЦА);
|
||||||
- Telegram batch + listener применяют профиль без DeepSeek;
|
- Telegram batch + listener применяют профиль без DeepSeek;
|
||||||
- генерация профиля — только в админке (`/admin/parser-builder/generate`).
|
- генерация профиля — только в админке (`/admin/parser-profiles/generate`).
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
flowchart LR
|
flowchart LR
|
||||||
|
|||||||
@@ -56,7 +56,7 @@ class HeuristicProfile(BaseModel):
|
|||||||
|
|
||||||
|
|
||||||
def target_field_specs() -> list[dict[str, str]]:
|
def target_field_specs() -> list[dict[str, str]]:
|
||||||
"""Fixed target table for parser-builder UI (roadmap: custom tables later)."""
|
"""Fixed target table for profile UI (roadmap: custom tables later)."""
|
||||||
return [
|
return [
|
||||||
{"name": "title", "type": "string", "description": "Short event title"},
|
{"name": "title", "type": "string", "description": "Short event title"},
|
||||||
{"name": "description", "type": "string", "description": "Event summary / body"},
|
{"name": "description", "type": "string", "description": "Event summary / body"},
|
||||||
|
|||||||
@@ -184,7 +184,7 @@ async def run(job_id, source_config, *, ctx) -> tuple[list[dict], str | None]
|
|||||||
|
|
||||||
Опционально: `extract_mode: llm` (DeepSeek) для telegram/crawl4ai в batch.
|
Опционально: `extract_mode: llm` (DeepSeek) для telegram/crawl4ai в batch.
|
||||||
|
|
||||||
**Конструктор парсера** (`/parser-builder`): менеджер вставляет образец поста → DeepSeek один раз строит `HeuristicProfile` (статичные regex/правила) → job с `extract_mode=profile`. Runtime (batch + listener) применяет только профиль, без LLM. Кастомные пользовательские таблицы — roadmap (подмена `target-fields`).
|
**Профиль + канал:** UI `/parser-profiles` (Generate/Preview/CRUD) и `/channels`; связка на `/parsers` → `ParseJob` с FK. При enqueue ЦА flatten в `extract_mode=profile` + `heuristic_profile`. Runtime (batch + listener) применяет только профиль, без LLM. Кастомные пользовательские таблицы — roadmap.
|
||||||
|
|
||||||
Подробности: [centers/parsing/ARCHITECTURE.md](../centers/parsing/ARCHITECTURE.md), [centers/analytics/ARCHITECTURE.md](../centers/analytics/ARCHITECTURE.md).
|
Подробности: [centers/parsing/ARCHITECTURE.md](../centers/parsing/ARCHITECTURE.md), [centers/analytics/ARCHITECTURE.md](../centers/analytics/ARCHITECTURE.md).
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -63,7 +63,7 @@ Listener добавляет флаг `listener: true` на стороне CA API
|
|||||||
|
|
||||||
Реестр: `CONFIG_MODELS` + `parse_source_config(source_type, raw)`.
|
Реестр: `CONFIG_MODELS` + `parse_source_config(source_type, raw)`.
|
||||||
|
|
||||||
Профиль конструктора: `contracts/heuristic_profile.py` (`HeuristicProfile`, `apply_profile`).
|
Профиль парсера: `contracts/heuristic_profile.py` (`HeuristicProfile`, `apply_profile`). Сущности Profile/Channel живут в БД ЦА; в Redis уходит плоский `heuristic_profile`.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
+2
-2
@@ -5,7 +5,7 @@
|
|||||||
- Docker + Docker Compose
|
- Docker + Docker Compose
|
||||||
- Файл `.env` (из `.env.example`)
|
- Файл `.env` (из `.env.example`)
|
||||||
- Для Telegram: `data/telegram.session` + `TELEGRAM_API_ID` / `TELEGRAM_API_HASH`
|
- Для Telegram: `data/telegram.session` + `TELEGRAM_API_ID` / `TELEGRAM_API_HASH`
|
||||||
- Опционально: `DEEPSEEK_API_KEY` для `extract_mode: llm` (воркеры) и конструктора парсера на `ca-api`
|
- Опционально: `DEEPSEEK_API_KEY` для `extract_mode: llm` (воркеры) и Generate профиля на `ca-api`
|
||||||
|
|
||||||
## Быстрый старт
|
## Быстрый старт
|
||||||
|
|
||||||
@@ -109,7 +109,7 @@ docker compose logs -f cp-workers-web
|
|||||||
| Frontend 000 / нет контейнеров | `docker compose up -d` (без `--build`, если registry недоступен, но образы уже есть) |
|
| Frontend 000 / нет контейнеров | `docker compose up -d` (без `--build`, если registry недоступен, но образы уже есть) |
|
||||||
| Job не берётся | Смотреть `ENABLED_ADAPTERS` / `WORKER_FAMILIES` нужного сервиса |
|
| Job не берётся | Смотреть `ENABLED_ADAPTERS` / `WORKER_FAMILIES` нужного сервиса |
|
||||||
| LLM не работает | `DEEPSEEK_API_KEY` в `.env`, перезапуск `cp-workers` / `cp-workers-web` |
|
| LLM не работает | `DEEPSEEK_API_KEY` в `.env`, перезапуск `cp-workers` / `cp-workers-web` |
|
||||||
| Generate в конструкторе 503 | `DEEPSEEK_API_KEY` в `.env` + `env_file` у `ca-api`, перезапуск `ca-api` |
|
| Generate профиля 503 | `DEEPSEEK_API_KEY` в `.env` + `env_file` у `ca-api`, перезапуск `ca-api` |
|
||||||
| Изменения UI не видны | Пересобрать `ca-frontend` |
|
| Изменения UI не видны | Пересобрать `ca-frontend` |
|
||||||
|
|
||||||
## Границы при разработке
|
## Границы при разработке
|
||||||
|
|||||||
Reference in New Issue
Block a user