sluice

A central syslog browser: rsyslog collects, PostgreSQL stores, and a single Go binary serves a filterable view of it.

It is a deliberate replacement for LogAnalyzer, keeping what that tool got right — one dense list of log lines with filters above it, and a collector that does not depend on the web UI being alive — on a stack that is still maintained and that scales past a few million rows.

Targets Enterprise Linux 10 (Rocky, Alma, RHEL).

How it fits together

  devices ──udp/tcp 514──▶  rsyslog  ──http batches──▶  sluice ingest  ──COPY──▶  PostgreSQL
                              │                          (127.0.0.1:9514)             │
                       disk-assisted queue                                            │
                       (survives outages)                                             │
                                                          sluice web  ◀──queries──────┘
                                                        (127.0.0.1:8080)
                                                               │
                                                          nginx (TLS)

rsyslog is the only component on the wire. It batches up to 500 events into one HTTP POST, which sluice writes as a single COPY and only then acknowledges. If sluice or PostgreSQL is unavailable the endpoint answers 503, rsyslog holds the batch in its disk-assisted queue and replays it on recovery — so the web service can be restarted or upgraded at any time without losing an event.

sluice ingest and sluice web are separate units of the same binary for the same reason: restarting the UI must never pause collection.

Why the schema looks like this

Three decisions do most of the work, and all three are things LogAnalyzer's single flat SystemEvents table could not do.

Daily range partitions. Retention is a DROP TABLE on a whole partition: instant, and it leaves no dead tuples for autovacuum to chase. A DELETE FROM events WHERE ... on a large table is what eventually makes the naive design unusable.

Partitioned on receipt time, not device time. Device clocks lie — a switch with no working NTP will stamp messages years out. Partitioning on that value scatters rows into partitions retention will never reclaim. Receipt time is monotonic and always sane, so it drives partitioning, retention and paging; the device's own timestamp is stored beside it, displayed by default, and flagged in the UI when the two disagree by more than a minute.

The trigram index is added only once a partition is sealed. A GIN index is the most expensive thing you can maintain on a high-rate append table. The partition being written to today has none, so ingest is fast; sluice maintain adds one to each partition after it stops receiving writes, so substring search over history is indexed. Partition pruning keeps an unindexed scan of the current day bounded to one day of data.

Paging is keyset-based on the (received_at, id) primary key rather than LIMIT/OFFSET, so page 500 costs the same as page 1.

Requirements

• PostgreSQL 16 or newer, with pg_trgm available
• rsyslog with omhttp and mmjsonparse. On EL10 omhttp.so is part of the rsyslog package itself — there is no rsyslog-omhttp — and only rsyslog-mmjsonparse needs installing separately.
• Go 1.25+ to build. EL10's AppStream Go is 1.26, so the stock golang package is enough; no golang-toolset needed.

Install

From the RPM

On an EL10 machine with rpm-build and golang installed:

make rpm
sudo dnf install ~/rpmbuild/RPMS/x86_64/sluice-0.1.0-1.el10.x86_64.rpm

Building the RPM in a container

You do not need an EL10 machine, or to install a build toolchain on the one you have. testlab/ carries an AlmaLinux 10 image that builds the package the way a packager would, on the distribution it targets:

cd testlab
docker compose --profile tools build rpmbuild
docker compose --profile tools run --rm rpmbuild

The binary and source RPMs land in testlab/out/, and the run prints the toolchain versions and rpm -qlvp for each package so the contents can be checked before installing anything.

The image copies the source in at build time rather than mounting it, so that what gets packaged is a clean tree — rebuild it after changing anything. If the working tree has uncommitted changes it packages those rather than HEAD, and says so; make rpm on its own always packages HEAD.

Nothing else in the lab needs to be running for this.

Signing it

The build produces an unsigned package. Installing one on a host that checks signatures fails:

Package sluice-0.1.0-1.el10.x86_64.rpm is not signed
Error: GPG check FAILED

That check is localpkg_gpgcheck, and a central log collector is not somewhere to turn it off. Sign the package instead.

Once, on the build host — RSA rather than the current gpg default of Ed25519, because rpm builds vary in whether they can verify EdDSA:

gpg --quick-gen-key 'sluice packaging <you@example.org>' rsa4096 sign never
gpg --armor --export 'sluice packaging' > RPM-GPG-KEY-sluice

Then for each build:

make rpm
make rpm-sign RPM_GPG_NAME='sluice packaging'
rpm -K ~/rpmbuild/RPMS/x86_64/sluice-*.rpm     # "digests signatures OK"

Once, on each server that will install it:

sudo rpm --import RPM-GPG-KEY-sluice

After which dnf install accepts the package with the check left on. Until the key is imported the server still refuses it, which is the point.

If you only need it installed once on a machine you already trust, sudo dnf install --nogpgcheck ./sluice-*.rpm skips the check — a decision about that one package on that one host, not a setting to leave changed.

testlab/scripts/check-rpm-signing.sh runs this end to end against a throwaway key: it confirms the unsigned build really is unsigned, signs it, checks a host without the key still refuses it, and checks that importing the key makes it install.

From source

make build
sudo make install
sudo groupadd -r sluice
sudo useradd -r -g sluice -d / -s /sbin/nologin sluice

sluice writes nothing to disk — its state is in PostgreSQL and its logs go to the journal — so the account needs no home directory.

Set up

1. Database

sudo dnf install postgresql-server postgresql-contrib
sudo postgresql-setup --initdb
sudo systemctl enable --now postgresql

sudo -u postgres createuser sluice
sudo -u postgres createdb -O sluice sluice
sudo -u postgres psql -d sluice -c 'CREATE EXTENSION IF NOT EXISTS pg_trgm'

Creating the extension as postgres up front means the sluice role does not need CREATE on the database; the migration's own CREATE EXTENSION IF NOT EXISTS then becomes a no-op.

The default DSN uses a Unix socket with peer authentication, so no password ends up in a config file.

2. Configuration

sudo install -d -m 0750 -o root -g sluice /etc/sluice
sudo install -m 0640 -o root -g sluice \
    /usr/share/doc/sluice/sluice.toml.example /etc/sluice/sluice.toml
sluice gentoken
sudo vi /etc/sluice/sluice.toml   # paste the token into ingest.token

This file holds the ingest token, and a database password too if you point database.dsn at a remote server, so it is installed 0640 root:sluice. Confirm the mode after editing — an editor that writes a replacement file rather than editing in place will use your umask, which is usually 0644.

3. Schema and first user

sudo -u sluice sluice migrate
sudo -u sluice sluice maintain          # creates the first partitions
sudo -u sluice sluice user add -admin $USER

4. Services

sudo systemctl enable --now sluice-ingest.service sluice-web.service
sudo systemctl enable --now sluice-maintain.timer

5. rsyslog

There is no rsyslog-omhttp package on EL10 — omhttp.so ships inside the rsyslog package itself. Only mmjsonparse is a separate install.

sudo dnf install rsyslog rsyslog-mmjsonparse
sudo install -m 0600 -o root -g root \
    /usr/share/doc/sluice/60-sluice.conf /etc/rsyslog.d/
sudo vi /etc/rsyslog.d/60-sluice.conf    # set httpheadervalue to the token
sudo systemctl restart rsyslog

This file ends up holding the ingest token in plaintext, so it is installed 0600 rather than the 0644 usual for /etc/rsyslog.d. rsyslogd reads it as root and does not need more. Left world-readable, the token would let any local account post fabricated events to the collector — forged entries in the audit trail everything else here exists to keep trustworthy. Check the mode again after editing: some editors write a fresh file with the default umask.

Open the syslog ports, and let rsyslog reach the ingest port:

sudo firewall-cmd --permanent --add-port=514/udp --add-port=514/tcp
sudo firewall-cmd --reload

# Required. SELinux confines rsyslog to syslogd_t, which may not connect to an
# unlabelled high port; without this rsyslog cannot reach the ingest endpoint
# at all and no events arrive.
sudo semanage port -a -t syslogd_port_t -p tcp 9514

If you are diagnosing this rather than applying it up front, note that the denial is not attributed to rsyslogd. omhttp runs the action in its own queue thread, named after the action, and ausearch -c matches that thread name — so the obvious command finds nothing:

sudo ausearch -m avc -c rsyslogd -ts recent      # says "<no matches>"

Look for it either without the filter, or under the thread name, or in the journal where setroubleshoot spells it out:

sudo ausearch -m avc -ts recent
sudo ausearch -m avc -c 'rs:sluice queue' -ts recent
sudo journalctl -t setroubleshoot -S -15min

The denial reads SELinux is preventing /usr/sbin/rsyslogd from name_connect access on the tcp_socket port 9514.

6. TLS

The shipped nginx configuration expects a certificate at /etc/pki/tls/certs/sluice.crt and its key at /etc/pki/tls/private/sluice.key. Make one first, then install nginx.

A self-signed certificate

For an internal name — the usual case, since a .lan or .fritz.box host has no publicly resolvable name for a CA to validate:

sudo openssl req -x509 -newkey rsa:4096 -sha256 -days 825 -nodes \
    -keyout /etc/pki/tls/private/sluice.key \
    -out    /etc/pki/tls/certs/sluice.crt \
    -subj   "/CN=logs.example.lan" \
    -addext "subjectAltName=DNS:logs.example.lan,IP:192.0.2.10" \
    -addext "basicConstraints=critical,CA:FALSE" \
    -addext "keyUsage=critical,digitalSignature,keyEncipherment" \
    -addext "extendedKeyUsage=serverAuth"

sudo chmod 0600 /etc/pki/tls/private/sluice.key
sudo restorecon -v /etc/pki/tls/certs/sluice.crt /etc/pki/tls/private/sluice.key

subjectAltName is the part that matters: browsers have ignored the common name since 2017 and will reject a certificate without a matching SAN entry, so list every name and address the UI will be reached by. 825 days is the longest validity clients still accept for a manually trusted certificate.

openssl already writes the key 0600, but /etc/pki/tls/private is 0755 on EL10 — the key's own mode is the only thing protecting it, so the chmod is worth keeping if you ever generate the key elsewhere and copy it in.

Browsers will warn once, because nothing signed it. Either accept it, or trust it — on a Linux client:

sudo cp sluice.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust

Or a real one

If the host does have a publicly resolvable name and port 80 reachable from the internet, use a CA instead and skip the warnings and the renewals:

sudo dnf install certbot python3-certbot-nginx
sudo certbot --nginx -d logs.example.com

certbot rewrites the certificate paths in the nginx configuration itself and installs a renewal timer.

Then nginx

sudo dnf install nginx
sudo install -m 0644 /usr/share/doc/sluice/nginx-sluice.conf /etc/nginx/conf.d/sluice.conf
sudo vi /etc/nginx/conf.d/sluice.conf    # server_name, and the paths if they differ
sudo setsebool -P httpd_can_network_connect on
sudo nginx -t
sudo systemctl enable --now nginx

# Without this the UI is reachable only from the machine itself: nginx is
# listening, but firewalld drops the connection before it gets there.
sudo firewall-cmd --permanent --add-service=http --add-service=https
sudo firewall-cmd --reload

httpd_can_network_connect is what lets nginx reach sluice on loopback; without it SELinux blocks the proxy pass, the same way it blocks rsyslog without the port label above.

After replacing a certificate later, sudo nginx -t && sudo systemctl reload nginx — reload rather than restart, so open connections are not dropped.

Operating it

Every subcommand that touches the configuration or the database has to run as the sluice user. Two separate reasons, both of which produce a confusing error otherwise:

• the config is 0640 root:sluice inside a 0750 directory, so no other unprivileged account can even see it;
• the default DSN uses peer authentication over the Unix socket with no username, so PostgreSQL takes the role from the OS user. As root that is the role root, which does not exist.

So sudo -u sluice, not sudo:

sudo -u sluice sluice user list             # accounts
sudo -u sluice sluice user passwd max       # change a password
sudo -u sluice sluice maintain -v           # run retention now, verbosely

curl -s http://127.0.0.1:9514/healthz       # ingest is up, database reachable
journalctl -u sluice-ingest -f

Retention, lookahead and the trigram index are all set under [retention] in /etc/sluice/sluice.toml. Changes take effect on the next maintenance run; lowering days drops the newly expired partitions the next time the timer fires.

Delivery guarantees

Delivery is at least once. If a batch commits but the acknowledgement is lost on the way back to rsyslog, rsyslog will resend it and those events appear twice. This is the right trade for a log server — the alternative loses events instead — but it is worth knowing before treating a count as exact.

The event count above the table stops at 200,000 and shows + when it has been capped. Counting every matching row on a large table is precisely the "row counting" option that made LogAnalyzer's UI hang, so sluice will not do it.

Development

make check    # gofmt, go vet, staticcheck, go test
make build

Run against a throwaway database without installing anything:

podman run -d --rm --name sluice-dev \
    -e POSTGRES_PASSWORD=dev -e POSTGRES_DB=sluice -p 5432:5432 \
    docker.io/library/postgres:16-alpine

umask 077          # the file below holds a database password
cat > dev.toml <<'CONF'
[database]
dsn = "postgres://postgres:dev@127.0.0.1:5432/sluice"
[ingest]
listen = "127.0.0.1:9514"
# Short and guessable tokens are refused, in development too: the check exists
# so that a hand-typed placeholder cannot reach production. `sluice gentoken`
# emits a real one.
token = "dev-only-Kq3wP7mZ2xR9tV5nB8cF4hJ6"
[web]
listen = "127.0.0.1:8080"
secure_cookies = false
CONF

./sluice migrate  -config dev.toml
./sluice maintain -config dev.toml
./sluice user add -config dev.toml -admin dev
./sluice ingest -config dev.toml -v &
./sluice web    -config dev.toml -v &

curl -X POST -H 'X-Sluice-Token: dev-only-Kq3wP7mZ2xR9tV5nB8cF4hJ6' -d '[{"ts":"","host":"test","fac":1,"sev":3,"app":"demo","msg":"hello"}]' \
    http://127.0.0.1:9514/ingest

The testlab

testlab/ runs the whole pipeline — rsyslog, PostgreSQL, ingest and web — in containers on AlmaLinux 10, using the distribution's own rsyslog and Go. It exists because the interesting questions about this project are not answerable by unit tests: whether the shipped rsyslog template produces what the decoder expects, and whether omhttp really replays a refused batch in full.

cd testlab
./setup.sh                  # build and start; UI on :8080, syslog on :1514
./scripts/run-checks.sh     # answer the open questions
docker compose down -v      # tear down

See testlab/README.md for what each check proves, what it found, and what it deliberately cannot tell you (SELinux, and anything about scale).

Layout

Path	Contents
cmd/sluice	Subcommand dispatch and operator tooling
internal/model	The event type and the RFC 5424 code points
internal/ingest	HTTP endpoint and the rsyslog wire format
internal/store	Every SQL statement, in one package
internal/db	Connection pool and embedded migrations
internal/maintain	Partition creation, sealing and retention
internal/web	UI handlers, templates and assets (all embedded)
internal/auth	Argon2id hashing and token generation
deploy/	rsyslog, systemd and nginx configuration
packaging/rpm	RPM spec