- Go 55.5%
- TypeScript 37.1%
- Just 2.8%
- Shell 2.2%
- CSS 1.2%
- Other 1.2%
just release signs SHA256SUMS with the trusted comment "loom <version>" and verifies it against the committed minisign.pub; it refuses to start without minisign, the secret key or minisign.pub. README "Releases" covers verifying a download, checking the key ID out of band, and creating and replacing the key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|---|---|---|
| cmd/loom | ||
| deploy | ||
| internal | ||
| schema | ||
| web | ||
| .dockerignore | ||
| .gitignore | ||
| CHANGELOG.md | ||
| CLAUDE.md | ||
| Dockerfile | ||
| go.mod | ||
| go.sum | ||
| Justfile | ||
| README.md | ||
| ROADMAP.md | ||
Loom
NetFlow-fed network communication map. Loom collects flow records from the devices that already see your traffic (VMware distributed switches, physical switches, firewalls), stores the basic facts of every connection, and shows who talks to whom as an interactive map with drill-down per endpoint.
Out of scope: packet inspection, per-flow forensics, long-term byte accounting.
What it does
- Overview map. Endpoints are nodes, traffic is edges. Buckets collapse many addresses into one node. An "Internet" bucket exists by default; double-click expands it into its networks by ASN.
- Drill-down. Click a node to see its peers, the ports used, and when each connection was first and last seen. Click an edge for the same per connection.
- Mapping workflow. Addresses that appear in the data are assigned to logical nodes (a VM, a physical host, a bucket) by the user. Unassigned addresses are listed so nothing hides.
- Tagging. Rules (CIDR, port, protocol) attach tags to connections. Views and search filter by tag. Rules are evaluated at query time, so a new rule applies to all history without reprocessing.
Architecture
exporters --UDP 2055--> loom binary (goflow2 decoder) --batch insert--> ClickHouse
| ^
HTTP API + static UI <------- queries ------------+
| Layer | Choice | Why |
|---|---|---|
| Collector | goflow2 as a Go library | Decodes NetFlow v5, v9, IPFIX and sFlow including template caches and sampling options. That is the only hard part of a collector and it is already solved. |
| Storage | ClickHouse | GROUP BY over hundreds of millions of rows for the map. Materialized views aggregate on insert. ip_trie dictionaries map addresses to nodes at query time. TTL is the retention policy. |
| Backend | Go, one binary | Embeds the collector, the API and the built frontend. One deployable. |
| Frontend | TypeScript, React, Vite, Cytoscape.js, Apache ECharts | Cytoscape compound nodes are the buckets, with expand and collapse. ECharts for timelines and the endpoint sankey. React because both libraries have maintained React wrappers and it is the framework most contributors know. |
Rejected: Akvorado (needs Kafka, UI is ISP-style AS graphs, not a node map), ntopng (NetFlow needs the commercial nProbe), ElastiFlow (closed source), a custom collector (see goflow2 row).
Data model
A flow record carries: exporter address, ingress and egress interface index, source and destination IP, source and destination port, protocol, bytes, packets, first and last seen. It carries no VM name, no hostname, no topology.
The identity of an endpoint in the data is the IP address. The interface index only says where on the exporter the traffic was seen (a switch port, a port group). It is kept as an attribute for "seen on switch X port Y", it is not the node. The user maps addresses to nodes, not interfaces.
Tables (ClickHouse):
flows_raw: records as decoded by goflow2, kept 7 days (about 3-6 MiB a day in the milestone 1 setup). Debugging and validation only.connections: one row per(day, exporter, client_if, server_if, client, server, port, proto)withfirst_seen,last_seen, and bytes and packets per direction. Filled by a materialized view overflows_raw. The day column keeps "was this used in period X" answerable and gives retention a key: rows are kept at least 400 days and dropped a whole part at a time, usually a month. Times are the collector clock, not the exporter's: a switch without NTP or a forged record would otherwise move rows to 1970.connections_dedup(view):connectionsper day, deduplicated across exporters, with the node of each side and the tags of every matching rule. Queries read this.nodes,node_members: a logical node (name, kind: vm, physical, bucket) and the CIDRs that belong to it, one node per CIDR. Feeds thenode_triedictionary; the most specific prefix wins.tag_rules:(client_cidr, server_cidr, port, proto)to tag. Matched by address range, notip_trie, becauseip_triereturns only the longest match and several rules may tag one connection.internal_ranges: RFC 1918, CGNAT, link-local, multicast, broadcast, unspecified, loopback, IPv6 ULA, and your own public prefixes. Any address outside these is "Internet".service_ports:(port, proto)that is always the server side, see Direction below.asn: prefix to ASN and name, loaded daily by the binary from iptoasn.com (-asn-url, empty turns it off). iptoasn lists ranges, so each is split into the fewest CIDRs that cover it, a million in all.asn_trie(ip_trie, about 80 MiB in ClickHouse) names a networkAS15169 GOOGLE, so the Internet bucket splits into networks without range lists. A load is a new version, listed inasn_loadsonce complete; the dictionary reads only listed versions, so a failed load keeps the previous one, also across a restart. One node per ASN: Google's cloud (AS396982) is a network of its own.shortcut: an expanded Internet shows its 30 biggest networks, the rest stay on Internet; page or filter them if one that matters is cut.
Direction: flow records are unidirectional. The reply arrives as a separate
record whose destination port is ephemeral. Loom treats a port listed in
service_ports as the service side (server), otherwise the side with the
lower port number, and keeps bytes per direction. Equal ports make the
lower address the server. Protocols without ports (ICMP, IGMP) get port 0,
since exporters put the ICMP type there, and so fall back to the lower
address.
service_ports exists for services above the clients' ephemeral range
(32768 and up on Linux, 49152 and up on Windows), where the client often
has the lower port. In the milestone 1 data that was 16 % of vDS and 21 %
of Sophos TCP/UDP bytes, almost all Fritz!Box TR-064 on TCP 49000 (seeded).
A listed port wins only against a peer port >= 1024, because a client can
pick a listed port as its own source port to reach a service like :443.
The list is read at ingest: an added port applies to flows received from
then on, so on the day of the change a conversation shows up in both
orientations. shortcut: wrong for peer-to-peer, for unlisted services above the ephemeral range, and when a client's source port is a listed port and its server's port is >= 1024; add the port when a service shows up.
A high port that many different peer ports talk to is a candidate. Check the service on the host before listing it: a listed port applies to every address, and forged flows can make any port look like a candidate.
SELECT addr, port, proto, uniq(peer_port) AS peer_ports
FROM (SELECT src_addr AS addr, src_port AS port, dst_port AS peer_port, proto FROM flows_raw
UNION ALL
SELECT dst_addr, dst_port, src_port, proto FROM flows_raw)
WHERE proto IN (6, 17) AND port >= 32768
AND (port, proto) NOT IN (SELECT port, proto FROM service_ports)
GROUP BY ALL HAVING peer_ports >= 5 ORDER BY peer_ports DESC;
Double counting: the same packet is exported by every device that sees it.
Primary mitigation is the setup rule below. Safety net in
connections_dedup: per (day, client, server, port, proto) take max()
over observation points (exporters), not sum(); the interfaces of one
exporter are summed. shortcut: underestimates when two exporters see disjoint traffic for the same tuple, revisit if asymmetric paths appear.
Within one exporter, input and output monitors on two access ports of
the same switch (IOS-XE below) report a flow between those ports twice with
the same interfaces, and the two records add up. shortcut: overcounts traffic between two monitored ports of one switch, revisit by keying on the flow direction if that traffic matters.
Exporter setup
One observation point per traffic path. Decide per segment which device exports and leave the others off for that segment.
UDP has no authentication: firewall the collector's UDP 2055 to the exporter
source addresses, or anyone reaching the port can inject fake flows, and
list the same addresses in -exporters.
- VM traffic: export from the vDS. Do not also export from the physical switch ports facing the ESXi hosts.
- Standalone physical devices: export from their access port on the switch. Never on trunks or uplinks between switches, that duplicates every flow that already has an access-port observation.
- Firewall: it sees everything crossing zones and duplicates vDS and switch
exports for internal-to-Internet traffic. Keep it for segments no other
exporter covers; the
max()dedup absorbs the overlap.
VMware vSphere 8.x (vDS)
- License: Enterprise Plus (covered). Format: IPFIX only (NetFlow v10).
- Switch level: distributed switch, Configure, Settings, NetFlow:
- Collector: loom host, UDP 2055.
- Switch IP address: set it. Without it every ESXi host appears as its own exporter.
- Observation Domain ID: leave empty. If NSX IPFIX is also in use, the two IDs must differ.
- Active flow export timeout 60 s, idle 15 s. Defaults are fine.
- Sampling rate: 0 means every packet, N means N packets dropped after each sampled one. Set 0 explicitly, do not trust the default. Raise it only if host CPU shows the cost.
- Process internal flows only: disabled. Enable it only in the alternative setup where the physical switch exports the ESXi uplinks; the vDS then adds just the host-local VM-to-VM traffic the switch cannot see.
- Port group level: Configure, Policies, Monitoring, NetFlow: Enabled. Enable on VM port groups only. Leave uplink and vmkernel port groups (vMotion, vSAN) off unless you want that traffic.
Cisco (IOS-XE, Flexible NetFlow)
Catalyst 9k syntax. Verify per platform: many smaller Catalyst models have no
NetFlow at all, and collect interface output is rejected on some ASICs
(drop the line if so). Export v9 or IPFIX to UDP 2055.
flow record LOOM
match ipv4 protocol
match ipv4 source address
match ipv4 destination address
match transport source-port
match transport destination-port
match interface input
collect interface output
collect counter bytes long
collect counter packets long
collect timestamp absolute first
collect timestamp absolute last
flow exporter LOOM
destination <loom-ip>
source <mgmt-interface>
transport udp 2055
export-protocol netflow-v9
template data timeout 60
flow monitor LOOM
record LOOM
exporter LOOM
cache timeout active 60
cache timeout inactive 15
interface GigabitEthernet1/0/12
ip flow monitor LOOM input
ip flow monitor LOOM output
Apply input and output on the access ports of standalone devices so both
directions of their conversations are seen. Never apply on uplinks.
Cisco (classic IOS 15.2E, Flexible NetFlow)
Catalyst 2960-X/XR syntax from the
15.2(7)E configuration guide,
tested on a 2960-X with 15.2(7)E12. Needs the Cisco ONE for Access license (show license); without it the
2960-X only has sampled NetFlow Lite. The 2960-L has no NetFlow. Other 15.2E
platforms (2960-CX, 3560-CX, IE) have their own guide; check the points
below there. CEF must be enabled (show ip cef summary).
Differences from IOS-XE, imposed by the platform:
- Export is NetFlow v9 only.
- Monitors attach in the input direction only. An access port shows what the device sends, nothing it receives. The connection is on the map either way (the lower port marks the service side), but its bytes are one-sided unless the peer's port or another exporter (vDS, Sophos) sees the other direction.
- No
match interface. The interface is a collected field, which changes nothing because each monitor sits on one port. - Timestamps are
sys-uptime(relative to boot). goflow2 turns them into wall-clock time from the switch clock in the v9 header, so the switch needs NTP: a wrong clock shifts every flow by the same offset. cache timeout inactiveis at least 30 s.- No monitors on port channels: apply on the member ports.
- The exporter
sourcemust have an IP address and be up: use the management SVI, not a switchport. That SVI address is the one to allow to the collector's UDP 2055. - A monitor cannot be changed while attached. To edit the record, remove
ip flow monitorfrom every port first. collect interface outputreturns nothing for broadcast and multicast flows; drop the line if the switch rejects it.
flow record LOOM
match ipv4 protocol
match ipv4 source address
match ipv4 destination address
match transport source-port
match transport destination-port
collect interface input
collect interface output
collect counter bytes long
collect counter packets long
collect timestamp sys-uptime first
collect timestamp sys-uptime last
flow exporter LOOM
destination <loom-ip>
source Vlan<mgmt-vlan>
transport udp 2055
export-protocol netflow-v9
template data timeout 60
flow monitor LOOM
record LOOM
exporter LOOM
cache timeout active 60
cache timeout inactive 30
interface GigabitEthernet1/0/12
ip flow monitor LOOM input
Placement as above (access ports of standalone devices, never uplinks), but input only.
Sophos Firewall (SFOS)
- Format: NetFlow v5. IPv4 only, no application fields.
- Configure under System, Administration, NetFlow (older XG via console
system netflow): server = loom host, port 2055. Up to five collectors. - Only traffic of firewall rules with "Log firewall traffic" enabled is exported. Enable it on the rules you want to see.
- Exported addresses are pre-NAT (verified in milestone 1): internal hosts appear with their own address, not the WAN address.
Deployment
deploy/stack.yml runs ClickHouse and the loom binary, built from the
Dockerfile, on a host with Docker and the Compose plugin (2.20 or newer):
git clone <this repository> loom && cd loom
mkdir -p deploy/secrets && chmod 700 deploy/secrets
(umask 077; openssl rand -hex 32 > deploy/secrets/clickhouse_loom_password)
(umask 077; openssl rand -hex 32 > deploy/secrets/http_password)
setfacl -m u:65532:r deploy/secrets/* # the loom image's user
echo 'LOOM_EXPORTERS=192.168.40.0/24,10.1.1.1' > deploy/.env
docker compose -f deploy/stack.yml up -d --build
just setup writes the two secrets as above. Compose passes file secrets
as bind mounts with their host permissions, so the files stay 600 and an
ACL (setfacl, package acl) lets the image's user read them.
The binary runs on the host's network, so exporter source addresses
arrive unchanged. It shares the host's network namespace: UDP 2055 is the
host's, guarded by the host firewall alone ("Exporter setup"), and the
container reaches the host's loopback services. The API and UI are on
127.0.0.1:8080 and ask for the password in
deploy/secrets/http_password (any user name). LOOM_HTTP=:8080 in
deploy/.env opens them to the network, where the password travels in
clear text with every request unless a TLS proxy sits in front (see
"Development").
The container runs as an unprivileged user on a read-only filesystem
without capabilities. The stack shares the loom project and volume with
deploy/compose.yml, so a development volume carries over; it needs
schema/07_loom_user.sh re-run once if it predates milestone 6 (see
below). After an update: git pull, then the same up command.
Releases
A release is loom-<version>-linux-amd64, loom-<version>-linux-arm64,
SHA256SUMS, and SHA256SUMS.minisig: a
minisign signature of the
checksums with the key whose public half is minisign.pub in this
repository.
To check a download, take minisign.pub from the tag you install
(git show <version>:minisign.pub > minisign.pub), not from the newest
commit. Its first line ends in the key ID: compare it with the key ID the
maintainer publishes outside this repository, since whoever could swap
the release here could swap the key too. Then, in the directory holding
the downloaded files:
minisign -Vm SHA256SUMS -p minisign.pub # prints "Trusted comment: loom <version>"
sha256sum -c --ignore-missing SHA256SUMS
The trusted comment is signed with the checksums: it must name the
version you meant to download, or an older, valid release was passed off
as this one. no file was verified from sha256sum means no downloaded
file has a name SHA256SUMS lists: keep the names as released.
Maintainers: just release signs with ~/.minisign/minisign.key,
created once at the repository root with minisign -G -p minisign.pub
(it asks for a password and will not overwrite an existing key). Commit
minisign.pub and publish its key ID somewhere besides this repository.
To replace the key, announce the new key ID in that same place, signed
with the old key unless the old key is the reason; tags keep the
minisign.pub they were signed with. just release refuses to start
without minisign, the secret key or minisign.pub, and checks the new
signature against minisign.pub.
Development
Planned layout and commands are in CLAUDE.md, milestones in ROADMAP.md.
Development platform
Linux is the reference platform. What to install, and what for:
| Tool | Version | Used for |
|---|---|---|
| Go | 1.25 or newer | backend build and tests |
staticcheck |
latest (go install honnef.co/go/tools/cmd/staticcheck@latest) |
Go lint gate |
govulncheck |
latest (go install golang.org/x/vuln/cmd/govulncheck@latest) |
known vulnerabilities in called Go code, lint gate |
| Node.js with npm | 22 LTS | frontend build, lint, tests |
| Docker with Compose plugin | 2.20 or newer | ClickHouse container, deploy/stack.yml |
shellcheck |
any | shell scripts under deploy/ |
hadolint |
any | Dockerfile lint |
setfacl (package acl) |
any | lets the loom image read the secrets (just setup) |
trivy, dockle |
current | image scan (just scan, part of just release) |
minisign |
any | release signature (just release) |
just |
1.17 or newer | Justfile recipes; optional, they wrap the commands below |
git is assumed present. Nothing else needs a system package: ClickHouse
runs in a container (clickhouse-client via docker compose exec),
goflow2 is a library inside the binary.
mkdir -p -m 700 deploy/secrets
(umask 077; openssl rand -hex 32 > deploy/secrets/clickhouse_loom_password)
docker compose -f deploy/compose.yml up -d --wait
(cd web && npm ci && npm run build) # embedded at go build time
go run ./cmd/loom -exporters <exporter addresses> # Ctrl-C stops; -h lists the flags
The Justfile wraps these and the lint, test, build and release commands
(just lists them): just setup once, then just up and
just run -exporters ..., with just web for frontend work.
just release writes static Linux binaries for amd64 and arm64 with a
signed SHA256SUMS to bin/release/ (see "Releases").
The binary fetches the iptoasn dump (9 MB) at start when the stored one
is a day old or missing, then daily, over HTTPS: it needs outbound access
to iptoasn.com, or -asn-url '' to do without networks.
The binary serves the UI at / and the API under /api/, on
http://127.0.0.1:8080. It embeds the frontend as built when it was
compiled: run npm run build before go build, or the UI answers 404 and
the log warns. For frontend work, npm run dev in web/ serves it with
hot reload on localhost:5173 and passes /api to the binary.
The binary receives flows on UDP 2055 on every host address, so exporter
source addresses arrive unchanged. Restrict the port in the host firewall
(INPUT chain) to the exporter source addresses; anyone else reaching it can
inject flows or fill the disk. List the same addresses or CIDRs in
-exporters (-exporters 192.168.40.0/24,10.1.1.1, required;
0.0.0.0/0,::/0 takes any sender): the binary drops packets from any other
sender before decoding, so a stray sender costs no template memory, and
logs once a minute how many it dropped and the last sender, which names an
exporter you forgot (check it before listing it, the address may be
forged). It is not authentication, since a host that forges an exporter's
address passes, nor flood protection, since dropped packets have already
taken their place in the receive queue: the firewall rule stays.
The API and UI listen on 127.0.0.1:8080. On any other address they ask
for a password (HTTP basic auth, any user name), and the binary refuses to
start without one:
(umask 077; openssl rand -hex 32 > deploy/secrets/http_password)
go run ./cmd/loom -exporters <exporter addresses> \
-http :8080 -http-password-file deploy/secrets/http_password
Basic auth sends the password with every request: put TLS in front (a
reverse proxy) or use an SSH tunnel, never plain HTTP across a network you
do not trust; the binary warns at start when it is plain HTTP beyond
loopback. Failed attempts are not limited, so passwords shorter than 16
characters are refused; wrong ones are logged once a minute with a count
and the last client address (wrong API password). Without a password, the API answers only
requests addressed to an IP address or localhost, which shuts out DNS
rebinding from a web page; with one, any host name works, because a
rebinding page has no credentials for its own origin.
ClickHouse is published on 127.0.0.1:8123 only. The binary connects as
loom (API, ASN load) and loom_ingest (collector), created by
schema/07_loom_user.sh from the secret file with only the grants each
needs. default stays passwordless and reachable only from
inside the container; check that from the host, it must answer
Authentication failed:
curl -s 'http://127.0.0.1:8123/?query=SELECT+1'
The container gets 4 GiB (mem_limit, no swap). loom gets 1 GB per
query and 2 GB for all its queries, 20 concurrent queries, 100 MB per
result, 3 GB of spill files and 60 s per query (300 s for the ASN load),
so API reads cannot starve the collector; raise memory in
deploy/compose.yml and schema/07_loom_user.sh together. System logs
expire after 30 days. On a volume from before milestone 6, ClickHouse
keeps the old log tables renamed, without the TTL: list them with
SELECT name FROM system.tables WHERE database = 'system' AND match(name, '_log_[0-9]+$')
and DROP TABLE them.
ClickHouse stops inserts and merges while the filesystem holding its
volume has less than 5 GiB free (deploy/clickhouse.xml). The binary then
logs flows dropped, the line to alert on. This floor, not the retention,
is the control against a flood of forged flows: their connections rows
would only expire after 400 days. Free space with the oldest days of
flows_raw first; a month of connections is history nothing can
rebuild, so drop one only when the raw days are not enough:
docker compose -f deploy/compose.yml exec clickhouse clickhouse-client --query \
"ALTER TABLE flows_raw DROP PARTITION '2026-10-01'" # connections: 202610
schema/ is applied only when the ClickHouse volume is empty. After
changing it, docker compose -f deploy/compose.yml down -v (deletes stored
flows). A volume from before the loom user keeps its data with the
command below; run the same after changing the secret file to rotate the
password, and once on a volume from before milestone 6 for the limits.
docker compose -f deploy/compose.yml up -d --wait
docker compose -f deploy/compose.yml exec clickhouse /docker-entrypoint-initdb.d/07_loom_user.sh
A volume from before connections had a TTL keeps it forever; give it
the retention once:
docker compose -f deploy/compose.yml exec clickhouse clickhouse-client --query "
ALTER TABLE connections MODIFY SETTING ttl_only_drop_parts = 1;
ALTER TABLE connections MODIFY TTL day + INTERVAL 400 DAY"
connections_dedup is a view and holds no data: after a change to
schema/06_connections_dedup.sql (milestone 4 added seen_on, milestone
5 the networks), update it in place. A volume from before milestone 5
needs schema/05_asn.sql first, and 07_loom_user.sh after it for the
grants:
docker compose -f deploy/compose.yml exec clickhouse \
clickhouse-client --queries-file /docker-entrypoint-initdb.d/05_asn.sql
docker compose -f deploy/compose.yml exec clickhouse \
clickhouse-client --queries-file /docker-entrypoint-initdb.d/06_connections_dedup.sql
docker compose -f deploy/compose.yml exec clickhouse /docker-entrypoint-initdb.d/07_loom_user.sh
Records are inserted every 5 s or every 10 000 records. While ClickHouse is unavailable a batch is retried for about a minute; meanwhile the UDP queue fills and further packets are dropped, which the log counts.
The log shows flow packet rejected with template not found for a v9 or
IPFIX exporter until that exporter's first template arrives, typically
within a minute of enabling it; records before that are dropped. Errors
that keep repeating for minutes mean the templates do not reach the
collector. NetFlow v5 has no templates.
go test ./... includes an end-to-end test that starts a throwaway
ClickHouse container (about 20 s); go test -short ./... skips it.
API
Reads take from and to as YYYY-MM-DD (collector-clock days,
inclusive, at most 366 of them) where listed and answer with a JSON array
of objects as ClickHouse renders them: addresses as IPv6, IPv4 as
::ffff:a.b.c.d.
POST and PUT take form fields (application/x-www-form-urlencoded) or a
query string, DELETE a query string; writes answer 204. With
-http-password-file, every request needs the password (curl -u loom:<password>). Browser requests from another site are refused, reads
too. Queries stop after 60 s. Node names
Internet, addresses and AS<number> names are reserved for what has no
node. Rows name an Internet side's network in client_asn and
server_asn, empty if it has none.
| Method and path | Fields | Notes |
|---|---|---|
GET /api/graph |
from, to, tag | traffic between nodes per port and protocol; with tag, only connections that carry it |
GET /api/graph/addresses |
from, to, tag | the same per address, with each side's node |
GET /api/connections |
node, from, to, tag | conversations of one node as the graph names it (node, unassigned address, Internet, network), with tags and seen_on: [exporter, client_if, server_if] per exporter and interface pair |
GET /api/unassigned |
from, to | internal addresses with traffic and no node |
GET /api/node |
address | the node an address has on the graph: its node, the address itself if internal without one, or Internet with its network in asn |
GET, POST /api/nodes |
name, kind (vm, physical, bucket) |
POST answers 201 {"id": ...} |
PUT, DELETE /api/nodes/{id} |
name, kind | |
GET, POST, DELETE /api/members |
node_id, cidr | POST of a mapped CIDR moves it; DELETE takes cidr |
GET, POST, DELETE /api/tag-rules |
tag, client_cidr, server_cidr, port, proto | defaults as in tag_rules; DELETE matches every field |
GET, POST, DELETE /api/internal-ranges |
cidr |
curl -s 'localhost:8080/api/graph?from=2026-10-01&to=2026-10-31'
curl -s -X POST localhost:8080/api/tag-rules -d tag=dns -d server_cidr=192.168.40.2/32 -d port=53
SQL
Check per exporter:
docker compose -f deploy/compose.yml exec clickhouse clickhouse-client --query "
SELECT type, sampler_address, count(), sum(bytes), max(sampling_rate),
fromUnixTimestamp64Nano(toInt64(min(time_flow_start_ns))) AS first,
fromUnixTimestamp64Nano(toInt64(max(time_flow_end_ns))) AS last
FROM flows_raw GROUP BY ALL"
Queries against connections_dedup (run them in clickhouse-client as
above). Always filter on day, and on client/server where it applies,
inside the same query: those filters reach the primary key.
-- Who talks to whom, per node, in a date range
SELECT client_node, server_node, groupUniqArray(port) AS ports,
sum(bytes_to_server) AS up, sum(bytes_to_client) AS down
FROM connections_dedup
WHERE day BETWEEN '2026-10-01' AND '2026-10-31'
GROUP BY client_node, server_node ORDER BY up + down DESC;
-- Peers of one address, with ports and first and last seen
WITH toIPv6('192.168.40.10') AS y
SELECT if(client = y, server, client) AS peer, client = y AS y_is_client,
port, proto, min(first_seen), max(last_seen),
sum(bytes_to_server + bytes_to_client) AS bytes
FROM connections_dedup
WHERE (client = y OR server = y) AND day >= today() - 7
GROUP BY ALL ORDER BY bytes DESC;
-- Map addresses to a node; the dictionary reloads within a minute.
-- Run once: a second run adds a second 'fritzbox' node.
INSERT INTO nodes (name, kind) VALUES ('fritzbox', 'physical');
INSERT INTO node_members SELECT id, '192.168.178.1/32' FROM nodes WHERE name = 'fritzbox';
SYSTEM RELOAD DICTIONARY node_trie;
-- Tag rule: applies to all stored history at once. Defaults match anything.
INSERT INTO tag_rules (tag, server_cidr, port) VALUES ('dns', '192.168.40.2/32', 53);
SELECT * FROM connections_dedup WHERE day >= today() - 1 AND has(tags, 'dns');
-- Service on a high port: applies to flows received from now on
INSERT INTO service_ports VALUES (32400, 6);