83 lines
2.7 KiB
Markdown
83 lines
2.7 KiB
Markdown
# Operations
|
|
|
|
## Health
|
|
|
|
```bash
|
|
curl -k "$PUBLIC_URL/api/health" # {"status":"ok"}
|
|
kubectl -n pargar get pods,ingressroute,svc
|
|
kubectl -n pargar logs deploy/pargar-backend --tail=100
|
|
```
|
|
|
|
## Backup
|
|
|
|
### Compose
|
|
|
|
```bash
|
|
./scripts/backup.sh [outdir]
|
|
```
|
|
|
|
Output: `postgres.sql.gz` + RustFS data archive (when possible).
|
|
|
|
### Helm / k3s
|
|
|
|
With `backup.enabled=true`:
|
|
|
|
- CronJob `pargar-pg-backup` on `backup.schedule` (default daily 03:00 UTC)
|
|
- Files on PVC `pargar-backups`
|
|
- Auto-delete older than `backup.retentionDays`
|
|
|
|
Manual restore (summary):
|
|
|
|
```bash
|
|
# Example: copy dump from pod/PVC then
|
|
gunzip -c pargar-YYYYMMDD.sql.gz | psql "$DATABASE_URL"
|
|
```
|
|
|
|
Back up object storage separately (bucket versioning / RustFS PVC snapshot).
|
|
|
|
## Metrics and tracing
|
|
|
|
- `GET /metrics` — without `METRICS_TOKEN` and outside DevMode → 404
|
|
- Header `X-Metrics-Token` or `Authorization: Bearer …`
|
|
- OTLP: `OTEL_EXPORTER_OTLP_ENDPOINT` + `OTEL_SERVICE_NAME`
|
|
|
|
## Updating images on k3s
|
|
|
|
```bash
|
|
./scripts/k3s-load-images.sh
|
|
./scripts/helm-k3s.sh
|
|
kubectl -n pargar rollout restart deploy/pargar-backend deploy/pargar-frontend
|
|
kubectl -n pargar rollout status deploy/pargar-backend
|
|
```
|
|
|
|
The `local` tag uses `IfNotPresent`; always restart after import.
|
|
|
|
## Common troubleshooting
|
|
|
|
| Symptom | Likely cause | Action |
|
|
|---------|--------------|--------|
|
|
| Seed video will not play | `sample.mp4` missing in storage | Check `sample media` logs; ensure file is in the image; restart backend |
|
|
| Sudden 401 after upgrade | JWT secret changed | Do not rotate the secret unless intentional; users must log in again |
|
|
| Backend CrashLoop after changing DB password in Secret | Old PVC password ≠ new Secret | Restore previous Secret password or re-seed DB |
|
|
| Repeated 429 | Rate limit | Wait; honor `Retry-After`; DevMode behaves differently |
|
|
| Empty CAPTCHA in tests | `CAPTCHA_EXPOSE_ANSWER` only with DevMode | Tests set both true |
|
|
| Helm lint/template differs from cluster | `lookup` empty without API | Install/upgrade against a real cluster |
|
|
| Frontend does not show changes | Stale image | Rebuild + import + rollout restart |
|
|
|
|
## Logging
|
|
|
|
- Requests with `request_id`, `user_id`, status, duration
|
|
- Security events: login, recovery, password reset, …
|
|
- Critical-path internals: `writeInternalErr` (log cause, generic user message)
|
|
|
|
## Useful scripts
|
|
|
|
| Script | Purpose |
|
|
|--------|---------|
|
|
| `scripts/compose-dev.sh` | Development Compose |
|
|
| `scripts/compose-prod.sh` | Production Compose |
|
|
| `scripts/k3s-load-images.sh` | Build + import into k3s containerd |
|
|
| `scripts/helm-k3s.sh` | helm upgrade --install |
|
|
| `scripts/backup.sh` | Compose backup |
|
|
| `scripts/regenerate-sample-mp4.sh` | Rebuild color bars with ffmpeg |
|