Keeping small services observable without a vendor

Observability platforms are priced for organizations with an on-call rotation and a budget line. If you run a handful of services for yourself, the useful subset is much smaller than the product category suggests: you need to know when something is down, and you need enough history to tell whether it is getting worse.

Three signals, in order of value

1. Is it up? A boolean per service, checked from outside.
2. Is it slow? A latency number per service, sampled on a schedule.
3. Is it full? Disk, memory, connection count.

Everything else — traces, per-request metrics, log aggregation — is worth adding only after these three are solid, and for a personal deployment they often never are.

Check from outside, not inside

A health check running on the same host answers "is the process alive", which is not the question. The question is "can a client reach it". Those diverge constantly: the process is fine but the reverse proxy is misconfigured, the certificate expired, the tunnel dropped, the firewall rule changed.

So the check has to originate somewhere else. The cheapest version is a cron job on a second machine:

*/5 * * * * curl -fsS -m 10 -o /dev/null https://service.example/health \
  || echo "$(date -Is) FAIL" >> /var/log/uptime.log

-f is what makes this work — without it curl exits zero on a 500 and the check passes while the service is broken.

Check the thing that actually breaks

An endpoint that returns 200 as long as the web server is running is nearly useless. The health endpoint should touch the dependencies the service needs: the database, the disk, the upstream API. If any of them is down, the health check should fail, because from a user's perspective the service is down.

The counter-argument is that this makes health checks flaky. That is the point. Flaky dependencies produce flaky service, and you want to know before your users do.

Latency is the leading indicator

Availability is a lagging signal: by the time a check fails, the failure has already happened. Latency degrades first. Recording response time on the same schedule gives you warning:

curl -fsS -m 10 -o /dev/null -w "%{time_total}\n" https://service.example/

A service whose p50 climbs from 80 ms to 400 ms over a week is about to fail, and the number tells you that days before the check goes red.

Alerting without a platform

For one person, the alerting channel is usually a chat message or an email. Both are one HTTP POST:

curl -fsS -X POST -d "text=service down" "$WEBHOOK_URL"

Two rules keep this from becoming noise. Alert only on state transitions, not on every failed check — a service that is down for an hour should page once, not twelve times. And alert on recovery too, otherwise you never learn the incident ended.

# state file makes transitions cheap
[ -f /tmp/svc.down ] || { notify "down"; touch /tmp/svc.down; }

Retention

Log files that grow forever eventually fill the disk, which is a self-inflicted outage caused by the monitoring. Cap them at write time rather than relying on a rotation job that might not run:

if [ "$(wc -l < "$LOG")" -gt 5000 ]; then
  tail -2000 "$LOG" > "$LOG.tmp" && mv "$LOG.tmp" "$LOG"
fi

Keeping 2000 lines of a five-minute-interval check is about a week of history, which is enough to answer "was this happening yesterday".