Observability

Once code is released, observability is how you know what it is actually doing: which requests are slow, which queries are heavy, where an error really started. A session replay shows you one user’s experience; observability shows you the system’s. The backend is instrumented with OpenTelemetry, and we send traces and metrics over OTLP to SigNoz, which stores them in ClickHouse.

OpenTelemetry is vendor-neutral: the code emits standard traces and metrics, and the backend they land in is a configuration detail the application never knows about.

What is instrumented for free

The OTel SDK starts once at boot (apps/api/src/instrumentation.ts) with auto-instrumentation, so a lot is captured without any per-feature work:

HTTP · every inbound request and outbound call, with timing.
Postgres · every query, with enhanced reporting, so slow or chatty queries surface.
tRPC procedures · each procedure is a span, so one trace follows a call from the edge down to the database.

The service reports as systeric-api, so its traces group under one name.

Adding your own signal

Auto-instrumentation tells you a request was slow. A custom span or metric tells you why, in your domain’s language. The pattern is already in the codebase: small metrics helpers per area (roadmap-metrics, editor-metrics, assistant-metrics, concurrency-metrics, and more).

Add a span around a meaningful business operation, not every function. It earns its place when you will ask “how long did that take” or “did that even run” in production.
Add a metric for a number you will want to chart or alert on: a queue depth, a retry count, a cache hit rate.
Name it for the dashboard. The name is how you find it in SigNoz months later. Be specific and consistent.

Keep correlation intact: a trace id runs through the request, so a structured log can be tied back to the exact trace. Log that id, do not invent a parallel one.

What to watch

Instrument so you can answer the questions you will actually have at 2am: is it up, is it slow, is it erroring, and where. Traces answer “where in the path”, metrics answer “how much and how often”. Point alerts at the few signals that mean a user is hurting, error rate, latency, saturation, not at every metric you emit. Alerting on noise trains people to ignore alerts, which is worse than no alert at all.