feat: implement daena MVP core service
Complete implementation of the daena log/event collection service:
Core runtime:
- DaenaRuntime service lifecycle (start/stop/reload)
- Pipeline orchestrator with source/persist/dispatch flow
- LMDB persistence backend with key-prefix partitioning
- Plugin discovery via Python entry points
Pipeline:
- Source/Processor/Sink ABC interfaces with DeliveryResult
- DummySource for testing and development
- DummySink and HTTPSink implementations
- PassthroughProcessor
- Record dataclass for in-memory records
- StoredRecord model with RecordState tracking
REST API (FastAPI, v1):
- GET /system/{status,health,config}
- GET /plugins
- GET /queue, /queue/{pending,dead}
- GET /sinks
- Versioned under /api/v1 prefix
CLI (Typer):
- status, health, queue, sinks, plugins, config, version
Persistence (LMDB):
- Single-database key-prefix design (no named DBs)
- Index keys: idx:{state}:{ts}\x00{id}
- Record keys: rec:{id}
- Full CRUD: put/get/ack/nack/dead/update_state
- Pending/retry/dead-letter queries with state filtering
- delete_old for TTL-based retention
Deployment:
- systemd unit file (deploy/daena.service)
- Example YAML config (config/daena.yaml)
- Entry points: daena (CLI), daena-serve (uvicorn server)
Quality:
- 23 unit tests covering config, persistence, pipeline
- ruff linting and formatting pass clean
- Py3.13 compatible, uv-managed, pyenv-pinned
- GPG-signed commit on dev branch
This commit is contained in:
@@ -1,2 +1,212 @@
|
||||
# daena
|
||||
Collect it. Keep it. Ship it. That’s Daena! 🤠
|
||||
|
||||
Collect, buffer, and forward logs and events from Linux to central observability backends.
|
||||
|
||||
## Overview
|
||||
|
||||
Daena is a modular, extensible Linux service that:
|
||||
|
||||
- Collects logs, events, and telemetry from pluggable sources
|
||||
- Normalizes and processes records through a configurable pipeline
|
||||
- Persists data locally in LMDB for durability, replay, and buffering
|
||||
- Forwards records reliably to one or more remote sinks
|
||||
- Exposes a REST API for management and inspection
|
||||
- Provides a CLI for operational tasks
|
||||
- Degrades gracefully under failures with retry, backoff, and dead-letter queues
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Sources → [Processors] → Pipeline → Persistence (LMDB) → Sink Dispatcher → Remote
|
||||
↻ retry queue
|
||||
```
|
||||
|
||||
Key design decisions:
|
||||
|
||||
- **Plugin architecture**: Sources, processors, and sinks are loaded via
|
||||
Python entry points. Add new ones by installing a package or registering
|
||||
in `pyproject.toml`.
|
||||
- **Pluggable persistence**: The `Backend` protocol abstracts storage.
|
||||
MVP uses LMDB. Future: NATS JetStream.
|
||||
- **Reliability-first**: Records survive restarts, sink failures trigger
|
||||
exponential backoff, and permanently failed records land in a dead-letter
|
||||
queue.
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
# Prerequisites: pyenv with Python 3.13, uv installed
|
||||
|
||||
# Clone and set up
|
||||
git clone <repo> && cd daena
|
||||
pyenv local 3.13.5
|
||||
uv sync --extra dev
|
||||
|
||||
# Run with dummy source/sink (no external dependencies)
|
||||
uv run python -c "
|
||||
import asyncio
|
||||
from daena.config import DaenaConfig, SourceConfig, SinkConfig
|
||||
from daena.core.runtime import DaenaRuntime
|
||||
|
||||
async def main():
|
||||
config = DaenaConfig()
|
||||
config.sources = [SourceConfig(name='test', type='dummy',
|
||||
options={'interval': 1.0, 'message': 'hello'})]
|
||||
config.sinks = [SinkConfig(name='log', type='dummy')]
|
||||
config.persistence.lmdb.path = '/tmp/daena_data'
|
||||
|
||||
runtime = DaenaRuntime(config)
|
||||
await runtime.start()
|
||||
await asyncio.sleep(5)
|
||||
print(f'Records: {runtime.backend.count()}')
|
||||
await runtime.stop()
|
||||
|
||||
asyncio.run(main())
|
||||
"
|
||||
|
||||
# Run tests
|
||||
uv run pytest tests/ -v
|
||||
|
||||
# Use the CLI (point at a running daena API)
|
||||
uv run daena status
|
||||
uv run daena health
|
||||
uv run daena queue
|
||||
```
|
||||
|
||||
## Project structure
|
||||
|
||||
```
|
||||
daena/
|
||||
├── pyproject.toml # Project metadata, dependencies, entry points
|
||||
├── config/daena.yaml # Example configuration
|
||||
├── deploy/daena.service # systemd unit file
|
||||
├── src/daena/
|
||||
│ ├── app.py # `daena-serve` entry point (uvicorn)
|
||||
│ ├── config.py # YAML → pydantic config
|
||||
│ ├── logutil.py # structlog setup
|
||||
│ ├── exceptions.py # Domain exceptions
|
||||
│ ├── api/ # FastAPI REST API (v1 versioned)
|
||||
│ ├── cli/ # Typer CLI
|
||||
│ ├── core/
|
||||
│ │ ├── runtime.py # Service lifecycle
|
||||
│ │ └── pipeline.py # Collection → persist → dispatch
|
||||
│ ├── persistence/
|
||||
│ │ ├── backend.py # Abstract Backend protocol
|
||||
│ │ ├── models.py # RecordState, StoredRecord
|
||||
│ │ └── lmdb.py # LMDB implementation
|
||||
│ ├── pipeline/
|
||||
│ │ ├── base.py # Source/Processor/Sink ABCs
|
||||
│ │ ├── record.py # In-memory Record dataclass
|
||||
│ │ ├── sources/ # Built-in sources (dummy)
|
||||
│ │ ├── sinks/ # Built-in sinks (dummy, http)
|
||||
│ │ └── processors/ # Built-in processors (passthrough)
|
||||
│ └── plugins/
|
||||
│ ├── loader.py # Entry-point discovery
|
||||
│ └── registry.py # Plugin registry
|
||||
└── tests/
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Daena loads configuration from `/etc/daena/daena.yaml`, `./daena.yaml`, or
|
||||
`./config/daena.yaml`. See `config/daena.yaml` for a full example.
|
||||
|
||||
```yaml
|
||||
logging:
|
||||
level: info
|
||||
format: json
|
||||
|
||||
api:
|
||||
http:
|
||||
host: 127.0.0.1
|
||||
port: 8642
|
||||
|
||||
persistence:
|
||||
backend: lmdb
|
||||
lmdb:
|
||||
path: /var/lib/daena/data
|
||||
map_size: 104857600
|
||||
|
||||
sources:
|
||||
- name: system_logs
|
||||
type: journald # future
|
||||
options: {}
|
||||
|
||||
processors:
|
||||
- name: add_hostname
|
||||
type: enricher # future
|
||||
options: {}
|
||||
|
||||
sinks:
|
||||
- name: central
|
||||
type: http
|
||||
options:
|
||||
url: https://logs.example.com/ingest
|
||||
```
|
||||
|
||||
## REST API
|
||||
|
||||
The API runs on `http://127.0.0.1:8642/api/v1`:
|
||||
|
||||
| Endpoint | Description |
|
||||
|----------|-------------|
|
||||
| `GET /system/status` | Service status |
|
||||
| `GET /system/health` | Health check |
|
||||
| `GET /system/config` | Loaded configuration |
|
||||
| `GET /plugins` | Registered plugins |
|
||||
| `GET /queue` | Queue state counts |
|
||||
| `GET /queue/pending` | Pending records |
|
||||
| `GET /queue/dead` | Dead-letter records |
|
||||
| `GET /sinks` | Configured sinks |
|
||||
|
||||
## Deployment
|
||||
|
||||
### systemd
|
||||
|
||||
```bash
|
||||
# Create daena user
|
||||
sudo useradd -r -s /sbin/nologin -d /var/lib/daena daena
|
||||
|
||||
# Install package
|
||||
uv pip install .
|
||||
|
||||
# Set up directories
|
||||
sudo mkdir -p /etc/daena /var/lib/daena /var/log/daena /run/daena
|
||||
sudo cp config/daena.yaml /etc/daena/
|
||||
sudo cp deploy/daena.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now daena
|
||||
```
|
||||
|
||||
## Extending
|
||||
|
||||
Add a new source by creating a class that inherits `daena.pipeline.base.Source`
|
||||
and registering it as an entry point:
|
||||
|
||||
```python
|
||||
# my_sources.py
|
||||
from daena.pipeline.base import Source
|
||||
from daena.pipeline.record import Record
|
||||
|
||||
class MySource(Source):
|
||||
async def start(self):
|
||||
pass
|
||||
async def stop(self):
|
||||
pass
|
||||
```
|
||||
|
||||
```toml
|
||||
# pyproject.toml
|
||||
[project.entry-points."daena.sources"]
|
||||
my_source = "my_sources:MySource"
|
||||
```
|
||||
|
||||
Same pattern applies for `Processor` and `Sink` base classes.
|
||||
|
||||
## Branch workflow
|
||||
|
||||
- `main` — stable integration branch
|
||||
- `dev` — active development branch
|
||||
|
||||
Feature work happens on `dev` or topic branches. Merges to `main` represent
|
||||
stable release checkpoints.
|
||||
|
||||
Reference in New Issue
Block a user