Sentinel

An API for gathering Linux server / Docker Engine metrics.

This will be used in coolify.io.

Features

Real-time system metrics collection (CPU, Memory)
Docker container metrics tracking
Historical metrics storage with SQLite
REST API for querying metrics
Configurable data retention
Push metrics to external endpoints
Debug mode with verbose logging and a database statistics endpoint
Optional on-box web/traffic analytics computed from the reverse-proxy access log (see Traffic Analytics)

Quick Start

Prerequisites

Rust 1.97 or higher (for development)
Docker (for container metrics)
Linux environment (production deployment)

Installation

Using Docker (Recommended)

docker run -d \
  -p 8888:8888 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -e TOKEN=your-secret-token \
  -e PUSH_ENDPOINT=https://coolify.example.com \
  ghcr.io/coollabsio/sentinel:latest

Using Cargo

# Clone the repository
git clone https://github.com/coollabsio/sentinel.git
cd sentinel

# Run the application
TOKEN=your-secret-token cargo run

Using cargo-watch (Development with hot reload)

# Install cargo-watch if not already installed
cargo install cargo-watch

# Run with hot reload
TOKEN=your-secret-token cargo watch -x run

Configuration

Sentinel is configured using environment variables:

Required Variables

Variable	Description	Example
`TOKEN`	Authentication token for API access	`your-secret-token`
`PUSH_ENDPOINT`	Coolify base URL that receives Sentinel pushes	`https://coolify.example.com`

Optional Variables

Variable	Default	Description
`PUSH_INTERVAL_SECONDS`	60	Interval for pushing metrics
`COLLECTOR_ENABLED`	`false`	Enable/disable metrics collection
`COLLECTOR_REFRESH_RATE_SECONDS`	5	Metrics collection interval
`COLLECTOR_RETENTION_PERIOD_DAYS`	7	How long to keep metrics in database
`STORAGE_ENABLED`	`true`	Enable/disable storage (disk + container storage) collection
`STORAGE_REFRESH_RATE_SECONDS`	300	Interval for the cheap per-mount filesystem stats
`STORAGE_VOLUMES_ENABLED`	`true`	Enable/disable per-container volume and bind mount sizes
`STORAGE_VOLUMES_REFRESH_RATE_SECONDS`	900	Interval for the expensive volume walk (kept separate so it can't hammer host I/O)
`HOST_MOUNT_PREFIX`	(empty)	Path prefix under which host paths are mounted into Sentinel's container. When set, bind mount sources are `du`-walked under it; when empty, bind mounts are not sized
`DOCKER_HOST`	`/var/run/docker.sock`	Docker Engine API socket; `unix://` URLs and bare paths only — e.g. `unix:///run/user/1000/podman/podman.sock` for rootless Podman
`DEBUG`	`false`	Enable verbose logging, `human_friendly_time` fields, and the `/api/stats` route
`PORT`	`8888`	HTTP server port
`SENTINEL_COMMAND_RETENTION_DAYS`	7	Days to keep completed control-command results for replay
`SENTINEL_COMMAND_MAX_RECORDS`	100000	Maximum completed control-command results; active records are not removed

Traffic Analytics Variables

Inert unless Sentinel is built with the traffic Cargo feature. See Traffic Analytics below for the full feature description.

Variable	Default	Description
`TRAFFIC_ENABLED`	`false`	Opt-in toggle for the whole subsystem (mirrors `COLLECTOR_ENABLED`)
`TRAFFIC_ACCESS_LOG_PATH`	`/data/coolify/proxy/access.log`	Reverse-proxy JSON access log to tail
`TRAFFIC_PROXY_TYPE`	`auto`	`traefik`, `caddy`, or `auto` (sniffs the format; Nginx is deferred)
`TRAFFIC_TOPN`	`50`	Top-N cap per dimension (paths, countries, browsers, ...); overflow folds into a `__other__` row
`TRAFFIC_IP_MODE`	`full`	How the `ip` dimension stores client IPs: `full`, `anonymized` (only the IPv4 /24 or IPv6 /48 network, as `1.2.3.0/24`), or `off` (no `ip` rows, and no city either). At start, Sentinel deletes stored `ip` rows that the mode does not allow, and with `off` also the stored cities. Unique-visitor counts always use a HyperLogLog sketch, which stores no IPs
`TRAFFIC_SAMPLE_THRESHOLD`	`0` (off)	Events/sec above which to start sampling under extreme load
`TRAFFIC_RETENTION_1M_HOURS`	`48`	Safety net, not a queryable window — see note below
`TRAFFIC_RETENTION_1H_DAYS`	`30`	How long hourly rollups are kept; this is your real fine-grained history
`TRAFFIC_RETENTION_1D_DAYS`	`395`	How long daily rollups are kept before deletion (~13 months)
`GEOIP_ENABLED`	`true`	Enable/disable country and city enrichment. The database downloads in the background and is reused from disk on the next start
`GEOIP_DB_URL`	(unset)	Explicit GeoIP database URL; when set, used with no fallback
`GEOIP_MAXMIND_LICENSE_KEY`	(unset)	MaxMind license key; when set, GeoLite2 is downloaded directly from MaxMind, taking priority over `GEOIP_DB_URL`
`GEOIP_MAXMIND_EDITION`	`GeoLite2-City`	MaxMind edition ID used when `GEOIP_MAXMIND_LICENSE_KEY` is set. A country edition gives country data only
`GEOIP_REFRESH_DAYS`	`30`	How often to re-check the GeoIP database for updates (a conditional request; cheap even when unchanged)

Note: TRAFFIC_RETENTION_1M_HOURS is a backlog safety net, not a queryable window. Hourly compaction deletes per-minute rows as it folds them into the hourly tier, so the minute table normally holds only the last hour or two — this cap just bounds how far it can grow if compaction falls behind. Longer ranges are served from the hourly and daily tiers.

In development mode (cargo run/cargo build debug profile, or SENTINEL_DEVELOPMENT=1), PUSH_ENDPOINT defaults to http://localhost:8000.

Storage collection & host paths. Disk stats only cover filesystems Sentinel's container can see. Named volumes are sized by the Docker daemon (/system/df), so they need no host mounts (volumes of drivers other than local count as 0). Bind mounts count as 0 unless you mount their host paths (read-only is fine) and point HOST_MOUNT_PREFIX at them (e.g. /host); a missing path then logs one warning. Large trees (/, /var, home dirs) are walked every cycle and can be slow; the walk is capped, or set STORAGE_VOLUMES_ENABLED=false to skip all volume sizing.

Example Configuration

export TOKEN=your-secret-token
export PUSH_ENDPOINT=https://coolify.example.com
export COLLECTOR_REFRESH_RATE_SECONDS=30
export COLLECTOR_RETENTION_PERIOD_DAYS=14
export DEBUG=false

./sentinel

API Reference

Sentinel provides a comprehensive REST API for retrieving system and Docker container metrics.

Quick API Overview

GET /api/health - Health check
GET /api/version - Get service version
GET /api/cpu/current - Current CPU usage
GET /api/cpu/history - Historical CPU data
GET /api/memory/current - Current memory usage
GET /api/memory/history - Historical memory data
GET /api/container/:id/cpu/history - Container CPU history
GET /api/container/:id/memory/history - Container memory history

Authentication

Metrics and debug API requests require a Bearer token. Health and version endpoints remain public for probes:

curl -H "Authorization: Bearer YOUR_TOKEN" \
  http://localhost:8888/api/cpu/current

Complete API Documentation

For detailed API documentation including request/response examples, query parameters, and error responses, see API.md and the OpenAPI spec openapi.json.

OpenAPI Specification

openapi.json is generated from the route annotations in code, and a test fails in CI when it drifts. Open it in any OpenAPI tool (for example https://editor.swagger.io). See API.md for how to regenerate it.

Traffic Analytics

Sentinel can optionally compute Cloudflare-style web analytics (requests, bandwidth, status classes, latency percentiles, unique visitors, geo, browser/OS/device, bot classification, referers, top paths) on the server from the reverse-proxy's JSON access log (Traefik or Caddy).

Only per-minute aggregate rollups are stored, never raw request rows, so storage doesn't grow with traffic. Rollups compact minute → hourly → daily with per-tier retention. Coolify pulls the aggregates the same way it does CPU/memory metrics; see the Traffic Analytics tag in openapi.json for the query endpoints.

The feature is opt-in at runtime (TRAFFIC_ENABLED=true) and gated at compile time by the traffic Cargo feature. The published Docker images already build with --features traffic; if you build Sentinel yourself, pass cargo build --features traffic or the traffic endpoints will 404.

Coolify integration

This repo implements the agent side only. Wiring it into Coolify — JSON access logs on the proxy, retaining Cloudflare headers, mounting the log into the container, passing the TRAFFIC_*/GEOIP_* env vars, and the pull/render UI — is tracked separately.

App attribution. Traffic is keyed per app. For Traefik the app UUID is parsed from the router name. For Caddy, if the access-log entry carries a coolify_app_id field (a custom log field Coolify can inject), that UUID is used — identical semantics to Traefik; otherwise attribution falls back to the request host, so hand-configured Caddy sites still work.

Architecture

Sentinel follows a service-oriented architecture with these components:

Core Services

API Server (crates/api/) - Axum-based HTTP server exposing metrics endpoints
Collector Service (crates/collector/) - Periodically collects system and Docker metrics
Push Service (crates/push/) - Sends metrics to external endpoints
Database Layer (crates/store/) - SQLite storage (rusqlite, bundled) with automatic cleanup

Data Flow

Docker Engine ──┐
                ├──> Collector Service ──> SQLite Database ──> API Server ──> Clients
System Stats ───┘                              │
                                               └──> Push Service ──> External Endpoint

Development

Project Structure

sentinel/
├── src/               # Application entry point (main.rs)
├── crates/
│   ├── api/          # HTTP API and routes (axum)
│   ├── collector/    # Metrics collection service
│   ├── push/         # Push service
│   ├── store/        # Database layer (rusqlite)
│   ├── config/       # Configuration management
│   └── docker/       # Docker Engine client (bollard)
├── Cargo.toml         # Workspace manifest
├── Dockerfile          # Docker build configuration
├── API.md              # API documentation
└── openapi.json        # OpenAPI spec, generated from route annotations

Building

# Build binary
cargo build --release

# Build Docker image
docker build -t sentinel .

# Run tests
cargo test --workspace

# Format code
cargo fmt --all

# Run linter
cargo clippy --workspace --all-targets -- -D warnings

Test with Coolify development

Coolify's development testing host uses the same Docker daemon, and its Sentinel page already accepts a custom development image.

# Start Coolify development first from the Coolify repository.
spin up

# Build Sentinel in this repository.
./scripts/coolify-dev.sh build

The image receives a build-time version such as 1.0.0-dev+9b1cd1a.dirty, so startup logs and /api/version clearly distinguish it from a release build. Set SENTINEL_DEV_VERSION to override this value.

In Coolify, open Servers → localhost → Sentinel, set Custom Sentinel Docker Image (Dev Only) to sentinel:dev, then enable or restart Sentinel. Verify the running container afterwards:

./scripts/coolify-dev.sh smoke
docker logs coolify-sentinel

The smoke command checks Docker health plus authenticated history access from inside the Sentinel container. The Coolify UI heartbeat confirms that push delivery to /api/v1/sentinel/push succeeds.

For an isolated end-to-end check, keep the Coolify-managed coolify-sentinel container running and execute:

./scripts/coolify-dev.sh integration

This starts a temporary candidate container using the development server's existing token and endpoint, then checks a custom listener port, health, authentication, collection, and a real push to Coolify before removing the container.

Dependencies

Key dependencies used in the project:

axum: HTTP web framework
bollard: Docker API client
sysinfo: System metrics collection
rusqlite (bundled): SQLite database driver
tokio: Async runtime and concurrent service management

Deployment

Docker Compose Example

version: '3.8'

services:
  sentinel:
    image: ghcr.io/coollabsio/sentinel:latest
    ports:
      - "8888:8888"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - sentinel-data:/app/db
    environment:
      TOKEN: ${SENTINEL_TOKEN}
      PUSH_ENDPOINT: ${PUSH_ENDPOINT}
      COLLECTOR_ENABLED: "true"
      COLLECTOR_REFRESH_RATE_SECONDS: 30
      COLLECTOR_RETENTION_PERIOD_DAYS: 14
    restart: unless-stopped

volumes:
  sentinel-data:

Systemd Service Example

[Unit]
Description=Sentinel Metrics Service
After=network.target docker.service
Requires=docker.service

[Service]
Type=simple
User=sentinel
Environment="TOKEN=your-secret-token"
Environment="PUSH_ENDPOINT=https://coolify.example.com"
ExecStart=/usr/local/bin/sentinel
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Monitoring and Debugging

Health Check

curl http://localhost:8888/api/health

Database Statistics (Debug Mode)

When DEBUG=true, access database statistics:

curl -H "Authorization: Bearer YOUR_TOKEN" \
  http://localhost:8888/api/stats

Contributing

This project is built for Coolify.io. Contributions are welcome!

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

License

See LICENSE file for details.

Support

For issues and questions:

GitHub Issues: https://github.com/coollabsio/sentinel/issues
Coolify Discord: https://discord.gg/coolify

Name		Name	Last commit message	Last commit date
.github		.github
crates		crates
docs		docs
scripts		scripts
src		src
tests		tests
.dockerignore		.dockerignore
.env.development		.env.development
.envrc		.envrc
.gitignore		.gitignore
AGENTS.md		AGENTS.md
API.md		API.md
BENCHMARK.md		BENCHMARK.md
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
Dockerfile		Dockerfile
Dockerfile.flux		Dockerfile.flux
Dockerfile.host		Dockerfile.host
LICENSE		LICENSE
README.md		README.md
build.sh		build.sh
flake.lock		flake.lock
flake.nix		flake.nix
justfile		justfile
openapi.json		openapi.json
rust-toolchain.toml		rust-toolchain.toml

Latest commit

History

Folders and files

Repository files navigation