Watchtower Log in

Every Decipher project’s logs, in one place

Decipher, ipmine, Sanctum, Scheme and the rest ship their logs here; their servers send what they use. Watchtower keeps it by project, turns it into metrics and archives old days.

Nothing sent is dropped Stored once, even when resent Archived before it leaves

Everything in one place

Logs

Search every project’s logs by text, level, service, status, route or request id, or watch them live.

Errors

The same exception through the same code is one group — open, resolved or ignored; a resolved one that comes back reopens.

Metrics

Events by level, requests by status, p50 to p99 latency, the busiest, slowest and failing routes, from an hour to 30 days.

Traces

Every event of one request, across every project it touched, on a single timeline.

Servers

An agent on each server sends CPU, memory, disks, network, every service and every Postgres database, every 30 seconds.

What happens to a log line

1 A project sends a batchGzipped NDJSON with its token; the project keeps the batch until it is answered.
2 Stored, then answered200 comes only after the batch is committed; anything else and the sender tries again.
3 Kept once, by dayEach event’s id and time are its key, so a retry stores nothing twice.
4 Folded into metricsEvery 10 seconds the worker adds it to minute stats, routes and error groups.
5 Archived and checkedDays three days old go to zstd files; each is read back and checked before its day leaves Postgres.

What is never lost

A batch not yet answered — the sender keeps it and sends it again.
A line that is not JSON — stored anyway, as a warning with its raw text.
A value too long — cut to fit, and the cut is noted on the event.
A day with no checked archive — it stays in Postgres until it has one.

The one approximation: percentiles over more than a minute are the minutes’ percentiles averaged by request count, marked ≈ — close, not exact.