feat: per-sink delivery, journal forwarding, and operational API
CI / lint (push) Canceled after 0s
CI / test (push) Canceled after 0s
CI / release (push) Canceled after 0s

Persistence:
- one delivery state per sink, with claim, crash recovery, and index replacement
- retention for delivered and dead records
- LMDB writes moved off the event loop

Pipeline:
- one attempt per dispatch, with the next try scheduled by next_retry_at
- backpressure through max_queue_size
- reload re-reads YAML; SIGTERM drains and SIGHUP reloads

Plugins:
- journald source, enricher and filter processors, Loki sink
- built-ins register when entry points are missing

API and CLI:
- health, metrics, per-sink queue, and dead-letter replay
- startup prints the pipeline on one line

Tests:
- dispatch, plugins, metrics, CLI, and persistence coverage
This commit is contained in:
ahp
2026-10-14 01:58:07 +03:30
parent 5e081144d5
commit 06d233054b
38 changed files with 3282 additions and 557 deletions
+29 -9
View File
@@ -17,8 +17,8 @@ Daena is a modular, extensible Linux service that:
## Architecture
```
Sources → [Processors] → Pipeline → Persistence (LMDB) → Sink Dispatcher → Remote
↻ retry queue
Sources → [Processors] → LMDB (one delivery per sink) → dispatchers
↻ per-sink retry, no head-of-line sleep
```
Key design decisions:
@@ -67,8 +67,9 @@ asyncio.run(main())
# Run tests
uv run pytest tests/ -v
# Start gRPC server
# Start gRPC server. Startup prints the pipeline. --verbose logs each record.
uv run daena-serve
uv run daena-serve --verbose
# Use the CLI (point at a running daena gRPC API)
uv run daena status
@@ -160,14 +161,15 @@ The gRPC API runs on `127.0.0.1:8642` with the following RPCs:
| RPC | Description |
|-----|-------------|
| `GetStatus` | Service status and version |
| `GetHealth` | Health check (backend, plugins, runtime) |
| `GetHealth` | Health, including oldest inflight age (`ok`, `degraded`, `down`) |
| `GetConfig` | Loaded configuration as JSON |
| `ListPlugins` | Registered sources, processors, and sinks |
| `GetQueueState` | Queue state counts (pending, delivered, failed, dead) |
| `GetQueueState` | Delivery counts, including a per-sink breakdown |
| `ListPendingRecords` | Pending records with pagination |
| `ListDeadRecords` | Dead-letter records with pagination |
| `ListSinks` | Configured sinks |
| `ReloadConfig` | Reload service configuration at runtime |
| `ReloadConfig` | Re-read the config file and restart the pipeline |
| `ReplayDead` | Requeue dead-letter deliveries |
### Reflect the gRPC service
@@ -182,9 +184,12 @@ grpcurl -plaintext 127.0.0.1:8642 describe daena.v1.Daena
Daena provides a CLI that connects to the gRPC server:
```bash
daena status # Show service status
daena health # Check health
daena queue # Inspect queue state
daena --target 127.0.0.1:8642 status
daena health # ok / degraded / down
daena queue # Inspect queue state, per sink
daena records pending
daena records dead
daena replay # Requeue dead-letter deliveries
daena sinks # List configured sinks
daena plugins # List registered plugins
daena config # Show loaded configuration
@@ -250,6 +255,21 @@ my_source = "my_sources:MySource"
Same pattern applies for `Processor` and `Sink` base classes.
Built-in plugins:
| Kind | Name | Role |
|------|------|------|
| source | `dummy` | Interval test events (`count` stops after N) |
| source | `journald` | `journalctl --follow` with an optional cursor file |
| processor | `passthrough` | No-op |
| processor | `enricher` | Hostname and static fields |
| processor | `filter` | Drop or keep by field equality |
| sink | `dummy` | Accept locally |
| sink | `http` | JSON POST |
| sink | `loki` | Loki push API |
`SIGHUP` re-reads the config file (systemd `ExecReload`). `SIGTERM` drains and closes LMDB. Prometheus text is served at `http://127.0.0.1:8643/metrics` when metrics are enabled. Each sink has its own delivery state, and a failed attempt is scheduled with `next_retry_at` instead of blocking the dispatcher.
## Branch workflow
- `main` — stable integration branch