daena/README.md

daena

Collect, buffer, and forward logs and events from Linux to central observability backends.

Overview

Daena is a modular, extensible Linux service that:

• Collects logs, events, and telemetry from pluggable sources
• Normalizes and processes records through a configurable pipeline
• Persists data locally in LMDB for durability, replay, and buffering
• Forwards records reliably to one or more remote sinks
• Exposes a REST API for management and inspection
• Provides a CLI for operational tasks
• Degrades gracefully under failures with retry, backoff, and dead-letter queues

Architecture

Sources → [Processors] → Pipeline → Persistence (LMDB) → Sink Dispatcher → Remote
                                                              ↻ retry queue

Key design decisions:

• Plugin architecture: Sources, processors, and sinks are loaded via Python entry points. Add new ones by installing a package or registering in pyproject.toml.
• Pluggable persistence: The Backend protocol abstracts storage. MVP uses LMDB. Future: NATS JetStream.
• Reliability-first: Records survive restarts, sink failures trigger exponential backoff, and permanently failed records land in a dead-letter queue.

Quick start

# Prerequisites: pyenv with Python 3.13, uv installed

# Clone and set up
git clone <repo> && cd daena
pyenv local 3.13.5
uv sync --extra dev

# Run with dummy source/sink (no external dependencies)
uv run python -c "
import asyncio
from daena.config import DaenaConfig, SourceConfig, SinkConfig
from daena.core.runtime import DaenaRuntime

async def main():
    config = DaenaConfig()
    config.sources = [SourceConfig(name='test', type='dummy',
        options={'interval': 1.0, 'message': 'hello'})]
    config.sinks = [SinkConfig(name='log', type='dummy')]
    config.persistence.lmdb.path = '/tmp/daena_data'

    runtime = DaenaRuntime(config)
    await runtime.start()
    await asyncio.sleep(5)
    print(f'Records: {runtime.backend.count()}')
    await runtime.stop()

asyncio.run(main())
"

# Run tests
uv run pytest tests/ -v

# Use the CLI (point at a running daena API)
uv run daena status
uv run daena health
uv run daena queue

Project structure

daena/
├── pyproject.toml              # Project metadata, dependencies, entry points
├── config/daena.yaml           # Example configuration
├── deploy/daena.service        # systemd unit file
├── src/daena/
│   ├── app.py                  # `daena-serve` entry point (uvicorn)
│   ├── config.py               # YAML → pydantic config
│   ├── logutil.py              # structlog setup
│   ├── exceptions.py           # Domain exceptions
│   ├── api/                    # FastAPI REST API (v1 versioned)
│   ├── cli/                    # Typer CLI
│   ├── core/
│   │   ├── runtime.py          # Service lifecycle
│   │   └── pipeline.py         # Collection → persist → dispatch
│   ├── persistence/
│   │   ├── backend.py          # Abstract Backend protocol
│   │   ├── models.py           # RecordState, StoredRecord
│   │   └── lmdb.py             # LMDB implementation
│   ├── pipeline/
│   │   ├── base.py             # Source/Processor/Sink ABCs
│   │   ├── record.py           # In-memory Record dataclass
│   │   ├── sources/            # Built-in sources (dummy)
│   │   ├── sinks/              # Built-in sinks (dummy, http)
│   │   └── processors/         # Built-in processors (passthrough)
│   └── plugins/
│       ├── loader.py           # Entry-point discovery
│       └── registry.py         # Plugin registry
└── tests/

Configuration

Daena loads configuration from /etc/daena/daena.yaml, ./daena.yaml, or ./config/daena.yaml. See config/daena.yaml for a full example.

logging:
  level: info
  format: json

api:
  http:
    host: 127.0.0.1
    port: 8642

persistence:
  backend: lmdb
  lmdb:
    path: /var/lib/daena/data
    map_size: 104857600

sources:
  - name: system_logs
    type: journald       # future
    options: {}

processors:
  - name: add_hostname
    type: enricher       # future
    options: {}

sinks:
  - name: central
    type: http
    options:
      url: https://logs.example.com/ingest

REST API

The API runs on http://127.0.0.1:8642/api/v1:

Endpoint	Description
GET /system/status	Service status
GET /system/health	Health check
GET /system/config	Loaded configuration
GET /plugins	Registered plugins
GET /queue	Queue state counts
GET /queue/pending	Pending records
GET /queue/dead	Dead-letter records
GET /sinks	Configured sinks

Deployment

systemd

# Create daena user
sudo useradd -r -s /sbin/nologin -d /var/lib/daena daena

# Install package
uv pip install .

# Set up directories
sudo mkdir -p /etc/daena /var/lib/daena /var/log/daena /run/daena
sudo cp config/daena.yaml /etc/daena/
sudo cp deploy/daena.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now daena

Extending

Add a new source by creating a class that inherits daena.pipeline.base.Source and registering it as an entry point:

# my_sources.py
from daena.pipeline.base import Source
from daena.pipeline.record import Record

class MySource(Source):
    async def start(self):
        pass
    async def stop(self):
        pass

# pyproject.toml
[project.entry-points."daena.sources"]
my_source = "my_sources:MySource"

Same pattern applies for Processor and Sink base classes.

Branch workflow

• main — stable integration branch
• dev — active development branch

Feature work happens on dev or topic branches. Merges to main represent stable release checkpoints.