feat: per-sink delivery, journal forwarding, and operational API
Persistence: - one delivery state per sink, with claim, crash recovery, and index replacement - retention for delivered and dead records - LMDB writes moved off the event loop Pipeline: - one attempt per dispatch, with the next try scheduled by next_retry_at - backpressure through max_queue_size - reload re-reads YAML; SIGTERM drains and SIGHUP reloads Plugins: - journald source, enricher and filter processors, Loki sink - built-ins register when entry points are missing API and CLI: - health, metrics, per-sink queue, and dead-letter replay - startup prints the pipeline on one line Tests: - dispatch, plugins, metrics, CLI, and persistence coverage
This commit is contained in:
@@ -17,8 +17,8 @@ Daena is a modular, extensible Linux service that:
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Sources → [Processors] → Pipeline → Persistence (LMDB) → Sink Dispatcher → Remote
|
||||
↻ retry queue
|
||||
Sources → [Processors] → LMDB (one delivery per sink) → dispatchers
|
||||
↻ per-sink retry, no head-of-line sleep
|
||||
```
|
||||
|
||||
Key design decisions:
|
||||
@@ -67,8 +67,9 @@ asyncio.run(main())
|
||||
# Run tests
|
||||
uv run pytest tests/ -v
|
||||
|
||||
# Start gRPC server
|
||||
# Start gRPC server. Startup prints the pipeline. --verbose logs each record.
|
||||
uv run daena-serve
|
||||
uv run daena-serve --verbose
|
||||
|
||||
# Use the CLI (point at a running daena gRPC API)
|
||||
uv run daena status
|
||||
@@ -160,14 +161,15 @@ The gRPC API runs on `127.0.0.1:8642` with the following RPCs:
|
||||
| RPC | Description |
|
||||
|-----|-------------|
|
||||
| `GetStatus` | Service status and version |
|
||||
| `GetHealth` | Health check (backend, plugins, runtime) |
|
||||
| `GetHealth` | Health, including oldest inflight age (`ok`, `degraded`, `down`) |
|
||||
| `GetConfig` | Loaded configuration as JSON |
|
||||
| `ListPlugins` | Registered sources, processors, and sinks |
|
||||
| `GetQueueState` | Queue state counts (pending, delivered, failed, dead) |
|
||||
| `GetQueueState` | Delivery counts, including a per-sink breakdown |
|
||||
| `ListPendingRecords` | Pending records with pagination |
|
||||
| `ListDeadRecords` | Dead-letter records with pagination |
|
||||
| `ListSinks` | Configured sinks |
|
||||
| `ReloadConfig` | Reload service configuration at runtime |
|
||||
| `ReloadConfig` | Re-read the config file and restart the pipeline |
|
||||
| `ReplayDead` | Requeue dead-letter deliveries |
|
||||
|
||||
### Reflect the gRPC service
|
||||
|
||||
@@ -182,9 +184,12 @@ grpcurl -plaintext 127.0.0.1:8642 describe daena.v1.Daena
|
||||
Daena provides a CLI that connects to the gRPC server:
|
||||
|
||||
```bash
|
||||
daena status # Show service status
|
||||
daena health # Check health
|
||||
daena queue # Inspect queue state
|
||||
daena --target 127.0.0.1:8642 status
|
||||
daena health # ok / degraded / down
|
||||
daena queue # Inspect queue state, per sink
|
||||
daena records pending
|
||||
daena records dead
|
||||
daena replay # Requeue dead-letter deliveries
|
||||
daena sinks # List configured sinks
|
||||
daena plugins # List registered plugins
|
||||
daena config # Show loaded configuration
|
||||
@@ -250,6 +255,21 @@ my_source = "my_sources:MySource"
|
||||
|
||||
Same pattern applies for `Processor` and `Sink` base classes.
|
||||
|
||||
Built-in plugins:
|
||||
|
||||
| Kind | Name | Role |
|
||||
|------|------|------|
|
||||
| source | `dummy` | Interval test events (`count` stops after N) |
|
||||
| source | `journald` | `journalctl --follow` with an optional cursor file |
|
||||
| processor | `passthrough` | No-op |
|
||||
| processor | `enricher` | Hostname and static fields |
|
||||
| processor | `filter` | Drop or keep by field equality |
|
||||
| sink | `dummy` | Accept locally |
|
||||
| sink | `http` | JSON POST |
|
||||
| sink | `loki` | Loki push API |
|
||||
|
||||
`SIGHUP` re-reads the config file (systemd `ExecReload`). `SIGTERM` drains and closes LMDB. Prometheus text is served at `http://127.0.0.1:8643/metrics` when metrics are enabled. Each sink has its own delivery state, and a failed attempt is scheduled with `next_retry_at` instead of blocking the dispatcher.
|
||||
|
||||
## Branch workflow
|
||||
|
||||
- `main` — stable integration branch
|
||||
|
||||
Reference in New Issue
Block a user