No description
  • Go 55.5%
  • TypeScript 37.1%
  • Just 2.8%
  • Shell 2.2%
  • CSS 1.2%
  • Other 1.2%
Find a file
Max Fiedler 96515d5bc6 Sign release checksums with minisign
just release signs SHA256SUMS with the trusted comment "loom <version>"
and verifies it against the committed minisign.pub; it refuses to start
without minisign, the secret key or minisign.pub. README "Releases"
covers verifying a download, checking the key ID out of band, and
creating and replacing the key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 09:09:40 +01:00
cmd/loom Add basic auth for the API and UI 2026-10-30 07:53:17 +01:00
deploy Add Dockerfile and a compose file for the whole stack 2026-10-30 08:01:37 +01:00
internal Add basic auth for the API and UI 2026-10-30 07:53:17 +01:00
schema Keep connections 400 days 2026-10-30 09:02:07 +01:00
web Add edge drill-down and sankey; mark milestone 5 done 2026-10-29 22:21:23 +01:00
.dockerignore Add Dockerfile and a compose file for the whole stack 2026-10-30 08:01:37 +01:00
.gitignore Add Dockerfile and a compose file for the whole stack 2026-10-30 08:01:37 +01:00
CHANGELOG.md Sign release checksums with minisign 2026-10-30 09:09:40 +01:00
CLAUDE.md Scan what ships before a release 2026-10-30 09:07:27 +01:00
Dockerfile Scan what ships before a release 2026-10-30 09:07:27 +01:00
go.mod Scan what ships before a release 2026-10-30 09:07:27 +01:00
go.sum Scan what ships before a release 2026-10-30 09:07:27 +01:00
Justfile Sign release checksums with minisign 2026-10-30 09:09:40 +01:00
README.md Sign release checksums with minisign 2026-10-30 09:09:40 +01:00
ROADMAP.md Sign release checksums with minisign 2026-10-30 09:09:40 +01:00

Loom

NetFlow-fed network communication map. Loom collects flow records from the devices that already see your traffic (VMware distributed switches, physical switches, firewalls), stores the basic facts of every connection, and shows who talks to whom as an interactive map with drill-down per endpoint.

Out of scope: packet inspection, per-flow forensics, long-term byte accounting.

What it does

  • Overview map. Endpoints are nodes, traffic is edges. Buckets collapse many addresses into one node. An "Internet" bucket exists by default; double-click expands it into its networks by ASN.
  • Drill-down. Click a node to see its peers, the ports used, and when each connection was first and last seen. Click an edge for the same per connection.
  • Mapping workflow. Addresses that appear in the data are assigned to logical nodes (a VM, a physical host, a bucket) by the user. Unassigned addresses are listed so nothing hides.
  • Tagging. Rules (CIDR, port, protocol) attach tags to connections. Views and search filter by tag. Rules are evaluated at query time, so a new rule applies to all history without reprocessing.

Architecture

exporters --UDP 2055--> loom binary (goflow2 decoder) --batch insert--> ClickHouse
                              |                                            ^
                        HTTP API + static UI  <------- queries ------------+
Layer Choice Why
Collector goflow2 as a Go library Decodes NetFlow v5, v9, IPFIX and sFlow including template caches and sampling options. That is the only hard part of a collector and it is already solved.
Storage ClickHouse GROUP BY over hundreds of millions of rows for the map. Materialized views aggregate on insert. ip_trie dictionaries map addresses to nodes at query time. TTL is the retention policy.
Backend Go, one binary Embeds the collector, the API and the built frontend. One deployable.
Frontend TypeScript, React, Vite, Cytoscape.js, Apache ECharts Cytoscape compound nodes are the buckets, with expand and collapse. ECharts for timelines and the endpoint sankey. React because both libraries have maintained React wrappers and it is the framework most contributors know.

Rejected: Akvorado (needs Kafka, UI is ISP-style AS graphs, not a node map), ntopng (NetFlow needs the commercial nProbe), ElastiFlow (closed source), a custom collector (see goflow2 row).

Data model

A flow record carries: exporter address, ingress and egress interface index, source and destination IP, source and destination port, protocol, bytes, packets, first and last seen. It carries no VM name, no hostname, no topology.

The identity of an endpoint in the data is the IP address. The interface index only says where on the exporter the traffic was seen (a switch port, a port group). It is kept as an attribute for "seen on switch X port Y", it is not the node. The user maps addresses to nodes, not interfaces.

Tables (ClickHouse):

  • flows_raw: records as decoded by goflow2, kept 7 days (about 3-6 MiB a day in the milestone 1 setup). Debugging and validation only.
  • connections: one row per (day, exporter, client_if, server_if, client, server, port, proto) with first_seen, last_seen, and bytes and packets per direction. Filled by a materialized view over flows_raw. The day column keeps "was this used in period X" answerable and gives retention a key: rows are kept at least 400 days and dropped a whole part at a time, usually a month. Times are the collector clock, not the exporter's: a switch without NTP or a forged record would otherwise move rows to 1970.
  • connections_dedup (view): connections per day, deduplicated across exporters, with the node of each side and the tags of every matching rule. Queries read this.
  • nodes, node_members: a logical node (name, kind: vm, physical, bucket) and the CIDRs that belong to it, one node per CIDR. Feeds the node_trie dictionary; the most specific prefix wins.
  • tag_rules: (client_cidr, server_cidr, port, proto) to tag. Matched by address range, not ip_trie, because ip_trie returns only the longest match and several rules may tag one connection.
  • internal_ranges: RFC 1918, CGNAT, link-local, multicast, broadcast, unspecified, loopback, IPv6 ULA, and your own public prefixes. Any address outside these is "Internet".
  • service_ports: (port, proto) that is always the server side, see Direction below.
  • asn: prefix to ASN and name, loaded daily by the binary from iptoasn.com (-asn-url, empty turns it off). iptoasn lists ranges, so each is split into the fewest CIDRs that cover it, a million in all. asn_trie (ip_trie, about 80 MiB in ClickHouse) names a network AS15169 GOOGLE, so the Internet bucket splits into networks without range lists. A load is a new version, listed in asn_loads once complete; the dictionary reads only listed versions, so a failed load keeps the previous one, also across a restart. One node per ASN: Google's cloud (AS396982) is a network of its own. shortcut: an expanded Internet shows its 30 biggest networks, the rest stay on Internet; page or filter them if one that matters is cut.

Direction: flow records are unidirectional. The reply arrives as a separate record whose destination port is ephemeral. Loom treats a port listed in service_ports as the service side (server), otherwise the side with the lower port number, and keeps bytes per direction. Equal ports make the lower address the server. Protocols without ports (ICMP, IGMP) get port 0, since exporters put the ICMP type there, and so fall back to the lower address.

service_ports exists for services above the clients' ephemeral range (32768 and up on Linux, 49152 and up on Windows), where the client often has the lower port. In the milestone 1 data that was 16 % of vDS and 21 % of Sophos TCP/UDP bytes, almost all Fritz!Box TR-064 on TCP 49000 (seeded). A listed port wins only against a peer port >= 1024, because a client can pick a listed port as its own source port to reach a service like :443. The list is read at ingest: an added port applies to flows received from then on, so on the day of the change a conversation shows up in both orientations. shortcut: wrong for peer-to-peer, for unlisted services above the ephemeral range, and when a client's source port is a listed port and its server's port is >= 1024; add the port when a service shows up.

A high port that many different peer ports talk to is a candidate. Check the service on the host before listing it: a listed port applies to every address, and forged flows can make any port look like a candidate.

SELECT addr, port, proto, uniq(peer_port) AS peer_ports
FROM (SELECT src_addr AS addr, src_port AS port, dst_port AS peer_port, proto FROM flows_raw
      UNION ALL
      SELECT dst_addr, dst_port, src_port, proto FROM flows_raw)
WHERE proto IN (6, 17) AND port >= 32768
  AND (port, proto) NOT IN (SELECT port, proto FROM service_ports)
GROUP BY ALL HAVING peer_ports >= 5 ORDER BY peer_ports DESC;

Double counting: the same packet is exported by every device that sees it. Primary mitigation is the setup rule below. Safety net in connections_dedup: per (day, client, server, port, proto) take max() over observation points (exporters), not sum(); the interfaces of one exporter are summed. shortcut: underestimates when two exporters see disjoint traffic for the same tuple, revisit if asymmetric paths appear. Within one exporter, input and output monitors on two access ports of the same switch (IOS-XE below) report a flow between those ports twice with the same interfaces, and the two records add up. shortcut: overcounts traffic between two monitored ports of one switch, revisit by keying on the flow direction if that traffic matters.

Exporter setup

One observation point per traffic path. Decide per segment which device exports and leave the others off for that segment.

UDP has no authentication: firewall the collector's UDP 2055 to the exporter source addresses, or anyone reaching the port can inject fake flows, and list the same addresses in -exporters.

  • VM traffic: export from the vDS. Do not also export from the physical switch ports facing the ESXi hosts.
  • Standalone physical devices: export from their access port on the switch. Never on trunks or uplinks between switches, that duplicates every flow that already has an access-port observation.
  • Firewall: it sees everything crossing zones and duplicates vDS and switch exports for internal-to-Internet traffic. Keep it for segments no other exporter covers; the max() dedup absorbs the overlap.

VMware vSphere 8.x (vDS)

  • License: Enterprise Plus (covered). Format: IPFIX only (NetFlow v10).
  • Switch level: distributed switch, Configure, Settings, NetFlow:
    • Collector: loom host, UDP 2055.
    • Switch IP address: set it. Without it every ESXi host appears as its own exporter.
    • Observation Domain ID: leave empty. If NSX IPFIX is also in use, the two IDs must differ.
    • Active flow export timeout 60 s, idle 15 s. Defaults are fine.
    • Sampling rate: 0 means every packet, N means N packets dropped after each sampled one. Set 0 explicitly, do not trust the default. Raise it only if host CPU shows the cost.
    • Process internal flows only: disabled. Enable it only in the alternative setup where the physical switch exports the ESXi uplinks; the vDS then adds just the host-local VM-to-VM traffic the switch cannot see.
  • Port group level: Configure, Policies, Monitoring, NetFlow: Enabled. Enable on VM port groups only. Leave uplink and vmkernel port groups (vMotion, vSAN) off unless you want that traffic.

Cisco (IOS-XE, Flexible NetFlow)

Catalyst 9k syntax. Verify per platform: many smaller Catalyst models have no NetFlow at all, and collect interface output is rejected on some ASICs (drop the line if so). Export v9 or IPFIX to UDP 2055.

flow record LOOM
 match ipv4 protocol
 match ipv4 source address
 match ipv4 destination address
 match transport source-port
 match transport destination-port
 match interface input
 collect interface output
 collect counter bytes long
 collect counter packets long
 collect timestamp absolute first
 collect timestamp absolute last
flow exporter LOOM
 destination <loom-ip>
 source <mgmt-interface>
 transport udp 2055
 export-protocol netflow-v9
 template data timeout 60
flow monitor LOOM
 record LOOM
 exporter LOOM
 cache timeout active 60
 cache timeout inactive 15
interface GigabitEthernet1/0/12
 ip flow monitor LOOM input
 ip flow monitor LOOM output

Apply input and output on the access ports of standalone devices so both directions of their conversations are seen. Never apply on uplinks.

Cisco (classic IOS 15.2E, Flexible NetFlow)

Catalyst 2960-X/XR syntax from the 15.2(7)E configuration guide, tested on a 2960-X with 15.2(7)E12. Needs the Cisco ONE for Access license (show license); without it the 2960-X only has sampled NetFlow Lite. The 2960-L has no NetFlow. Other 15.2E platforms (2960-CX, 3560-CX, IE) have their own guide; check the points below there. CEF must be enabled (show ip cef summary).

Differences from IOS-XE, imposed by the platform:

  • Export is NetFlow v9 only.
  • Monitors attach in the input direction only. An access port shows what the device sends, nothing it receives. The connection is on the map either way (the lower port marks the service side), but its bytes are one-sided unless the peer's port or another exporter (vDS, Sophos) sees the other direction.
  • No match interface. The interface is a collected field, which changes nothing because each monitor sits on one port.
  • Timestamps are sys-uptime (relative to boot). goflow2 turns them into wall-clock time from the switch clock in the v9 header, so the switch needs NTP: a wrong clock shifts every flow by the same offset.
  • cache timeout inactive is at least 30 s.
  • No monitors on port channels: apply on the member ports.
  • The exporter source must have an IP address and be up: use the management SVI, not a switchport. That SVI address is the one to allow to the collector's UDP 2055.
  • A monitor cannot be changed while attached. To edit the record, remove ip flow monitor from every port first.
  • collect interface output returns nothing for broadcast and multicast flows; drop the line if the switch rejects it.
flow record LOOM
 match ipv4 protocol
 match ipv4 source address
 match ipv4 destination address
 match transport source-port
 match transport destination-port
 collect interface input
 collect interface output
 collect counter bytes long
 collect counter packets long
 collect timestamp sys-uptime first
 collect timestamp sys-uptime last
flow exporter LOOM
 destination <loom-ip>
 source Vlan<mgmt-vlan>
 transport udp 2055
 export-protocol netflow-v9
 template data timeout 60
flow monitor LOOM
 record LOOM
 exporter LOOM
 cache timeout active 60
 cache timeout inactive 30
interface GigabitEthernet1/0/12
 ip flow monitor LOOM input

Placement as above (access ports of standalone devices, never uplinks), but input only.

Sophos Firewall (SFOS)

  • Format: NetFlow v5. IPv4 only, no application fields.
  • Configure under System, Administration, NetFlow (older XG via console system netflow): server = loom host, port 2055. Up to five collectors.
  • Only traffic of firewall rules with "Log firewall traffic" enabled is exported. Enable it on the rules you want to see.
  • Exported addresses are pre-NAT (verified in milestone 1): internal hosts appear with their own address, not the WAN address.

Deployment

deploy/stack.yml runs ClickHouse and the loom binary, built from the Dockerfile, on a host with Docker and the Compose plugin (2.20 or newer):

git clone <this repository> loom && cd loom
mkdir -p deploy/secrets && chmod 700 deploy/secrets
(umask 077; openssl rand -hex 32 > deploy/secrets/clickhouse_loom_password)
(umask 077; openssl rand -hex 32 > deploy/secrets/http_password)
setfacl -m u:65532:r deploy/secrets/*   # the loom image's user
echo 'LOOM_EXPORTERS=192.168.40.0/24,10.1.1.1' > deploy/.env
docker compose -f deploy/stack.yml up -d --build

just setup writes the two secrets as above. Compose passes file secrets as bind mounts with their host permissions, so the files stay 600 and an ACL (setfacl, package acl) lets the image's user read them.

The binary runs on the host's network, so exporter source addresses arrive unchanged. It shares the host's network namespace: UDP 2055 is the host's, guarded by the host firewall alone ("Exporter setup"), and the container reaches the host's loopback services. The API and UI are on 127.0.0.1:8080 and ask for the password in deploy/secrets/http_password (any user name). LOOM_HTTP=:8080 in deploy/.env opens them to the network, where the password travels in clear text with every request unless a TLS proxy sits in front (see "Development"). The container runs as an unprivileged user on a read-only filesystem without capabilities. The stack shares the loom project and volume with deploy/compose.yml, so a development volume carries over; it needs schema/07_loom_user.sh re-run once if it predates milestone 6 (see below). After an update: git pull, then the same up command.

Releases

A release is loom-<version>-linux-amd64, loom-<version>-linux-arm64, SHA256SUMS, and SHA256SUMS.minisig: a minisign signature of the checksums with the key whose public half is minisign.pub in this repository.

To check a download, take minisign.pub from the tag you install (git show <version>:minisign.pub > minisign.pub), not from the newest commit. Its first line ends in the key ID: compare it with the key ID the maintainer publishes outside this repository, since whoever could swap the release here could swap the key too. Then, in the directory holding the downloaded files:

minisign -Vm SHA256SUMS -p minisign.pub   # prints "Trusted comment: loom <version>"
sha256sum -c --ignore-missing SHA256SUMS

The trusted comment is signed with the checksums: it must name the version you meant to download, or an older, valid release was passed off as this one. no file was verified from sha256sum means no downloaded file has a name SHA256SUMS lists: keep the names as released.

Maintainers: just release signs with ~/.minisign/minisign.key, created once at the repository root with minisign -G -p minisign.pub (it asks for a password and will not overwrite an existing key). Commit minisign.pub and publish its key ID somewhere besides this repository. To replace the key, announce the new key ID in that same place, signed with the old key unless the old key is the reason; tags keep the minisign.pub they were signed with. just release refuses to start without minisign, the secret key or minisign.pub, and checks the new signature against minisign.pub.

Development

Planned layout and commands are in CLAUDE.md, milestones in ROADMAP.md.

Development platform

Linux is the reference platform. What to install, and what for:

Tool Version Used for
Go 1.25 or newer backend build and tests
staticcheck latest (go install honnef.co/go/tools/cmd/staticcheck@latest) Go lint gate
govulncheck latest (go install golang.org/x/vuln/cmd/govulncheck@latest) known vulnerabilities in called Go code, lint gate
Node.js with npm 22 LTS frontend build, lint, tests
Docker with Compose plugin 2.20 or newer ClickHouse container, deploy/stack.yml
shellcheck any shell scripts under deploy/
hadolint any Dockerfile lint
setfacl (package acl) any lets the loom image read the secrets (just setup)
trivy, dockle current image scan (just scan, part of just release)
minisign any release signature (just release)
just 1.17 or newer Justfile recipes; optional, they wrap the commands below

git is assumed present. Nothing else needs a system package: ClickHouse runs in a container (clickhouse-client via docker compose exec), goflow2 is a library inside the binary.

mkdir -p -m 700 deploy/secrets
(umask 077; openssl rand -hex 32 > deploy/secrets/clickhouse_loom_password)
docker compose -f deploy/compose.yml up -d --wait
(cd web && npm ci && npm run build)   # embedded at go build time
go run ./cmd/loom -exporters <exporter addresses>   # Ctrl-C stops; -h lists the flags

The Justfile wraps these and the lint, test, build and release commands (just lists them): just setup once, then just up and just run -exporters ..., with just web for frontend work. just release writes static Linux binaries for amd64 and arm64 with a signed SHA256SUMS to bin/release/ (see "Releases").

The binary fetches the iptoasn dump (9 MB) at start when the stored one is a day old or missing, then daily, over HTTPS: it needs outbound access to iptoasn.com, or -asn-url '' to do without networks.

The binary serves the UI at / and the API under /api/, on http://127.0.0.1:8080. It embeds the frontend as built when it was compiled: run npm run build before go build, or the UI answers 404 and the log warns. For frontend work, npm run dev in web/ serves it with hot reload on localhost:5173 and passes /api to the binary.

The binary receives flows on UDP 2055 on every host address, so exporter source addresses arrive unchanged. Restrict the port in the host firewall (INPUT chain) to the exporter source addresses; anyone else reaching it can inject flows or fill the disk. List the same addresses or CIDRs in -exporters (-exporters 192.168.40.0/24,10.1.1.1, required; 0.0.0.0/0,::/0 takes any sender): the binary drops packets from any other sender before decoding, so a stray sender costs no template memory, and logs once a minute how many it dropped and the last sender, which names an exporter you forgot (check it before listing it, the address may be forged). It is not authentication, since a host that forges an exporter's address passes, nor flood protection, since dropped packets have already taken their place in the receive queue: the firewall rule stays.

The API and UI listen on 127.0.0.1:8080. On any other address they ask for a password (HTTP basic auth, any user name), and the binary refuses to start without one:

(umask 077; openssl rand -hex 32 > deploy/secrets/http_password)
go run ./cmd/loom -exporters <exporter addresses> \
  -http :8080 -http-password-file deploy/secrets/http_password

Basic auth sends the password with every request: put TLS in front (a reverse proxy) or use an SSH tunnel, never plain HTTP across a network you do not trust; the binary warns at start when it is plain HTTP beyond loopback. Failed attempts are not limited, so passwords shorter than 16 characters are refused; wrong ones are logged once a minute with a count and the last client address (wrong API password). Without a password, the API answers only requests addressed to an IP address or localhost, which shuts out DNS rebinding from a web page; with one, any host name works, because a rebinding page has no credentials for its own origin.

ClickHouse is published on 127.0.0.1:8123 only. The binary connects as loom (API, ASN load) and loom_ingest (collector), created by schema/07_loom_user.sh from the secret file with only the grants each needs. default stays passwordless and reachable only from inside the container; check that from the host, it must answer Authentication failed:

curl -s 'http://127.0.0.1:8123/?query=SELECT+1'

The container gets 4 GiB (mem_limit, no swap). loom gets 1 GB per query and 2 GB for all its queries, 20 concurrent queries, 100 MB per result, 3 GB of spill files and 60 s per query (300 s for the ASN load), so API reads cannot starve the collector; raise memory in deploy/compose.yml and schema/07_loom_user.sh together. System logs expire after 30 days. On a volume from before milestone 6, ClickHouse keeps the old log tables renamed, without the TTL: list them with SELECT name FROM system.tables WHERE database = 'system' AND match(name, '_log_[0-9]+$') and DROP TABLE them.

ClickHouse stops inserts and merges while the filesystem holding its volume has less than 5 GiB free (deploy/clickhouse.xml). The binary then logs flows dropped, the line to alert on. This floor, not the retention, is the control against a flood of forged flows: their connections rows would only expire after 400 days. Free space with the oldest days of flows_raw first; a month of connections is history nothing can rebuild, so drop one only when the raw days are not enough:

docker compose -f deploy/compose.yml exec clickhouse clickhouse-client --query \
  "ALTER TABLE flows_raw DROP PARTITION '2026-10-01'"   # connections: 202610

schema/ is applied only when the ClickHouse volume is empty. After changing it, docker compose -f deploy/compose.yml down -v (deletes stored flows). A volume from before the loom user keeps its data with the command below; run the same after changing the secret file to rotate the password, and once on a volume from before milestone 6 for the limits.

docker compose -f deploy/compose.yml up -d --wait
docker compose -f deploy/compose.yml exec clickhouse /docker-entrypoint-initdb.d/07_loom_user.sh

A volume from before connections had a TTL keeps it forever; give it the retention once:

docker compose -f deploy/compose.yml exec clickhouse clickhouse-client --query "
  ALTER TABLE connections MODIFY SETTING ttl_only_drop_parts = 1;
  ALTER TABLE connections MODIFY TTL day + INTERVAL 400 DAY"

connections_dedup is a view and holds no data: after a change to schema/06_connections_dedup.sql (milestone 4 added seen_on, milestone 5 the networks), update it in place. A volume from before milestone 5 needs schema/05_asn.sql first, and 07_loom_user.sh after it for the grants:

docker compose -f deploy/compose.yml exec clickhouse \
  clickhouse-client --queries-file /docker-entrypoint-initdb.d/05_asn.sql
docker compose -f deploy/compose.yml exec clickhouse \
  clickhouse-client --queries-file /docker-entrypoint-initdb.d/06_connections_dedup.sql
docker compose -f deploy/compose.yml exec clickhouse /docker-entrypoint-initdb.d/07_loom_user.sh

Records are inserted every 5 s or every 10 000 records. While ClickHouse is unavailable a batch is retried for about a minute; meanwhile the UDP queue fills and further packets are dropped, which the log counts.

The log shows flow packet rejected with template not found for a v9 or IPFIX exporter until that exporter's first template arrives, typically within a minute of enabling it; records before that are dropped. Errors that keep repeating for minutes mean the templates do not reach the collector. NetFlow v5 has no templates.

go test ./... includes an end-to-end test that starts a throwaway ClickHouse container (about 20 s); go test -short ./... skips it.

API

Reads take from and to as YYYY-MM-DD (collector-clock days, inclusive, at most 366 of them) where listed and answer with a JSON array of objects as ClickHouse renders them: addresses as IPv6, IPv4 as ::ffff:a.b.c.d. POST and PUT take form fields (application/x-www-form-urlencoded) or a query string, DELETE a query string; writes answer 204. With -http-password-file, every request needs the password (curl -u loom:<password>). Browser requests from another site are refused, reads too. Queries stop after 60 s. Node names Internet, addresses and AS<number> names are reserved for what has no node. Rows name an Internet side's network in client_asn and server_asn, empty if it has none.

Method and path Fields Notes
GET /api/graph from, to, tag traffic between nodes per port and protocol; with tag, only connections that carry it
GET /api/graph/addresses from, to, tag the same per address, with each side's node
GET /api/connections node, from, to, tag conversations of one node as the graph names it (node, unassigned address, Internet, network), with tags and seen_on: [exporter, client_if, server_if] per exporter and interface pair
GET /api/unassigned from, to internal addresses with traffic and no node
GET /api/node address the node an address has on the graph: its node, the address itself if internal without one, or Internet with its network in asn
GET, POST /api/nodes name, kind (vm, physical, bucket) POST answers 201 {"id": ...}
PUT, DELETE /api/nodes/{id} name, kind
GET, POST, DELETE /api/members node_id, cidr POST of a mapped CIDR moves it; DELETE takes cidr
GET, POST, DELETE /api/tag-rules tag, client_cidr, server_cidr, port, proto defaults as in tag_rules; DELETE matches every field
GET, POST, DELETE /api/internal-ranges cidr
curl -s 'localhost:8080/api/graph?from=2026-10-01&to=2026-10-31'
curl -s -X POST localhost:8080/api/tag-rules -d tag=dns -d server_cidr=192.168.40.2/32 -d port=53

SQL

Check per exporter:

docker compose -f deploy/compose.yml exec clickhouse clickhouse-client --query "
  SELECT type, sampler_address, count(), sum(bytes), max(sampling_rate),
         fromUnixTimestamp64Nano(toInt64(min(time_flow_start_ns))) AS first,
         fromUnixTimestamp64Nano(toInt64(max(time_flow_end_ns))) AS last
  FROM flows_raw GROUP BY ALL"

Queries against connections_dedup (run them in clickhouse-client as above). Always filter on day, and on client/server where it applies, inside the same query: those filters reach the primary key.

-- Who talks to whom, per node, in a date range
SELECT client_node, server_node, groupUniqArray(port) AS ports,
       sum(bytes_to_server) AS up, sum(bytes_to_client) AS down
FROM connections_dedup
WHERE day BETWEEN '2026-10-01' AND '2026-10-31'
GROUP BY client_node, server_node ORDER BY up + down DESC;

-- Peers of one address, with ports and first and last seen
WITH toIPv6('192.168.40.10') AS y
SELECT if(client = y, server, client) AS peer, client = y AS y_is_client,
       port, proto, min(first_seen), max(last_seen),
       sum(bytes_to_server + bytes_to_client) AS bytes
FROM connections_dedup
WHERE (client = y OR server = y) AND day >= today() - 7
GROUP BY ALL ORDER BY bytes DESC;

-- Map addresses to a node; the dictionary reloads within a minute.
-- Run once: a second run adds a second 'fritzbox' node.
INSERT INTO nodes (name, kind) VALUES ('fritzbox', 'physical');
INSERT INTO node_members SELECT id, '192.168.178.1/32' FROM nodes WHERE name = 'fritzbox';
SYSTEM RELOAD DICTIONARY node_trie;

-- Tag rule: applies to all stored history at once. Defaults match anything.
INSERT INTO tag_rules (tag, server_cidr, port) VALUES ('dns', '192.168.40.2/32', 53);
SELECT * FROM connections_dedup WHERE day >= today() - 1 AND has(tags, 'dns');

-- Service on a high port: applies to flows received from now on
INSERT INTO service_ports VALUES (32400, 6);