Collecting Metrics
SharpMUSH reports how it is doing as numbers: players online, how long commands and functions take, how full the world’s disk is, and whether messages between its parts are piling up. This page shows how to collect those numbers with Prometheus, look at them in Grafana, and get an alert when one goes wrong.
It is written for the Docker Compose stacks: the one-click install from
Launch a Server, and the files in deploy/ described in the
Deployment Reference.
Where the metrics are
Section titled “Where the metrics are”Two of SharpMUSH’s programs publish metrics. Each answers GET /metrics with plain HTTP, in the
Prometheus text format, on a port of its own:
| Program | Container | Port | Setting |
|---|---|---|---|
| Game server | sharpmush-server | 9092 | Metrics:Port (environment variable Metrics__Port) |
| Connection server | sharpmush-socketserver | 9091 | ConnectionServer:MetricsPort (ConnectionServer__MetricsPort) |
The rendering worker has no metrics.
Nothing is sent anywhere: a scraper has to come and read the numbers. Prometheus does that here, but anything that reads the Prometheus format can.
A metrics port answers /metrics and nothing else, and no other port answers /metrics, so a
reverse proxy cannot publish them by accident. Setting a metrics port to 0 turns metrics off for
that program: no port answers /metrics.
The ports are not on the internet:
| Stack | 9092 | 9091 |
|---|---|---|
| One-click install | The server’s loopback address (127.0.0.1:9092) | The server’s loopback address (127.0.0.1:9091) |
deploy/docker-compose.prod.yml or docker-compose.cloudflare.yml | The server’s loopback address | The game’s Docker network only |
To check the game server is publishing, run this on the machine the game runs on:
curl -s http://127.0.0.1:9092/metrics | grep '^sharpmush_players_logged_in'Output:
sharpmush_players_logged_in{otel_scope_name="SharpMUSH",otel_scope_version="1.0.0"} 0Run Prometheus and Grafana beside the game
Section titled “Run Prometheus and Grafana beside the game”The simplest setup runs Prometheus and Grafana as a second Compose project on the same machine, joined to the game’s Docker network. Prometheus then reads both programs by container name, and neither metrics port has to be published anywhere new.
-
Find the name of the game’s network:
Terminal window docker network ls --filter name=sharpmush-net --format '{{.Name}}'A one-click install prints
sharpmush_sharpmush-net. A stack started from thedeploy/folder printsdeploy_sharpmush-net. -
Make a folder for the monitoring files, next to the game’s files but not inside them:
Terminal window mkdir ~/sharpmush-monitoring && cd ~/sharpmush-monitoring -
Save this as
prometheus.yml. It reads both programs every 15 seconds:global:scrape_interval: 15sscrape_configs:- job_name: sharpmush-serverstatic_configs:- targets: ["sharpmush-server:9092"]- job_name: sharpmush-socketserverstatic_configs:- targets: ["sharpmush-socketserver:9091"]rule_files:- /etc/prometheus/alerts.yml -
Save the rules from Alerts below as
alerts.yml. -
Save this as
grafana-datasource.yml, so Grafana starts already connected to Prometheus:apiVersion: 1datasources:- name: Prometheustype: prometheusaccess: proxyurl: http://prometheus:9090isDefault: true -
Save this as
compose.yml. If step 1 printed another network name, put it in place ofsharpmush_sharpmush-neton the last lines:name: sharpmush-monitoringservices:prometheus:image: prom/prometheus:v3.5.0command:- --config.file=/etc/prometheus/prometheus.yml- --storage.tsdb.retention.time=30dvolumes:- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro- ./alerts.yml:/etc/prometheus/alerts.yml:ro- prometheus-data:/prometheusports:- "127.0.0.1:9090:9090"networks:- gamerestart: unless-stoppedgrafana:image: grafana/grafana:12.1.0volumes:- ./grafana-datasource.yml:/etc/grafana/provisioning/datasources/prometheus.yml:ro- grafana-data:/var/lib/grafanaports:- "127.0.0.1:3000:3000"networks:- gamerestart: unless-stoppedvolumes:prometheus-data:grafana-data:networks:game:name: sharpmush_sharpmush-netexternal: truePrometheus keeps 30 days of readings in its own volume. Both ports are on the loopback address only.
-
Start them:
Terminal window docker compose up -d -
Check that Prometheus can read both programs:
Terminal window curl -s http://127.0.0.1:9090/api/v1/targets | grep -o '"health":"[a-z]*"'Output:
"health":"up""health":"up"A
"health":"down"means Prometheus cannot reach that program. Check the network name incompose.ymland that the game is running.
Open Grafana
Section titled “Open Grafana”Grafana listens on the game machine’s loopback address, so reach it through an ssh tunnel. On your own computer:
ssh -N -L 3000:127.0.0.1:3000 root@<your-server-ip>Leave that running and open http://localhost:3000. Use the same kind of tunnel, with port 9090,
to open Prometheus itself.
The timings dashboard
Section titled “The timings dashboard”SharpMUSH ships a Grafana dashboard for function and command timings: the slowest functions and commands, where the time goes, and the call and failure rates. To add it:
-
Download it on your own computer:
Terminal window curl -fsSL https://raw.githubusercontent.com/SharpMUSH/SharpMUSH/main/deploy/grafana/sharpmush-invocations.json -o sharpmush-invocations.json -
In Grafana, open Dashboards, then New, then Import.
-
Select Upload dashboard JSON file, pick the file, then select Import.
The dashboard is called SharpMUSH function and command timings. Its Prometheus menu picks the data source, and starts on the one set up above. Its Function and Command menus narrow every panel to the names you pick.
Alerts
Section titled “Alerts”These rules cover the failures an operator most needs to hear about. Save them as alerts.yml:
groups: - name: sharpmush rules: - alert: SharpMUSHDown expr: up{job=~"sharpmush-.*"} == 0 for: 2m annotations: summary: "{{ $labels.job }} has not answered a scrape for 2 minutes" - alert: SharpMUSHBackupWillNotStart expr: sharpmush_storage_backup_fits == 0 annotations: summary: "The next backup would not find enough free disk" - alert: SharpMUSHMapNearlyFull expr: sharpmush_storage_bytes{kind="map_headroom"} < ignoring(kind) 2 * sharpmush_storage_bytes{kind="live"} annotations: summary: "The world is close to its map limit; raise SHARPMUSH_LIGHTNING_MAPSIZE" - alert: SharpMUSHBusStreamFilling expr: max by (stream) (sharpmush_messaging_stream_bytes{stream=~"SHARPMUSH-.*"} / sharpmush_messaging_stream_max_bytes) > 0.8 for: 10m annotations: summary: "Bus stream {{ $labels.stream }} is over 80% of its byte budget" - alert: SharpMUSHBusRefusing expr: sum by (stream, reason) (increase(sharpmush_messaging_publish_rejected_total[10m])) > 0 annotations: summary: "The bus refused messages on {{ $labels.stream }} ({{ $labels.reason }})" - alert: SharpMUSHMessagesDropped expr: sum by (job, consumer, reason) (increase(sharpmush_messaging_handler_terminal_failures_total[10m])) > 0 annotations: summary: "{{ $labels.consumer }} gave up on messages ({{ $labels.reason }})" - alert: SharpMUSHBacklog expr: sum by (job, consumer) (sharpmush_messaging_consumer_pending) > 1000 for: 5m annotations: summary: "{{ $labels.consumer }} has over 1000 messages waiting"A firing alert shows on Prometheus’s Alerts page. To have one sent to you by email, Discord or another service, set up a contact point in Grafana’s Alerting section, or run Prometheus’s Alertmanager.
Storage and History Retention explains the storage figures behind the backup and map alerts, and has two more disk alerts.
What the metrics mean
Section titled “What the metrics mean”Prometheus adds job and instance labels to every series, and SharpMUSH adds otel_scope_name and
otel_scope_version. Counters end in _total, timings in _milliseconds, and sizes are in bytes.
Both programs publish the player, connection and health gauges below, but only one of them keeps
each up to date. Read players, connections and sharpmush_server_health from
job="sharpmush-server", and sharpmush_connectionserver_health from
job="sharpmush-socketserver". The other program reports a fixed 0 for players and connections, and a
fixed 1 for the other’s health.
Players and connections
Section titled “Players and connections”| Metric | What it is | Labels |
|---|---|---|
sharpmush_players_logged_in | Connections logged in to a character now; a character connected twice counts twice | |
sharpmush_connections_active | Connections now, logged in or not | |
sharpmush_connection_events_total | Connections opened and closed, logins and logouts, from the game server only | event_type: connected, disconnected, logged_in, logged_out, account_mode |
sharpmush_server_health | 1 while the game server runs | |
sharpmush_connectionserver_health | 1 while the connection server runs |
A health gauge cannot report 0 for a program that has stopped, so alert on Prometheus’s own up
instead, as SharpMUSHDown does.
Logins in the last hour:
sum(increase(sharpmush_connection_events_total{event_type="logged_in"}[1h]))Function and command timings
Section titled “Function and command timings”| Metric | Labels |
|---|---|
sharpmush_function_invocation_duration_milliseconds | function_name, success |
sharpmush_command_invocation_duration_milliseconds | command_name, success |
Each is a histogram of the time one call took, from the game server. A call’s time is its own work:
the time spent on its arguments and on calls inside it is counted where it was spent. A label holds
the built-in’s name in capitals, whatever the caller typed, so @pe is counted as @PEMIT. The
Deployment Reference has the details.
The slowest functions, at the 95th percentile:
topk(10, histogram_quantile(0.95, sum by (le, function_name) (rate(sharpmush_function_invocation_duration_milliseconds_bucket[5m]))))The world’s storage
Section titled “The world’s storage”From the game server.
| Metric | What it is | Labels |
|---|---|---|
sharpmush_storage_bytes | The world’s sizes: map limit, file, disk used, live data, free space and what a backup needs | kind |
sharpmush_storage_backup_fits | 1 when the next backup would find enough free disk, 0 when it would not start | |
sharpmush_storage_attribute_index_bytes | Space each attribute index takes | index |
sharpmush_storage_stale_readers_cleared_total | Read slots freed because the program holding them died mid-read | |
sharpmush_storage_writer_wait_milliseconds | How long each write waited for its turn | |
sharpmush_storage_writer_hold_milliseconds | How long each write took, saving to disk included | |
sharpmush_storage_writer_queued | Writes waiting now | |
sharpmush_storage_read_duration_milliseconds | How long each read stayed open | |
sharpmush_storage_reads_active | Reads open now |
Writes run one at a time. A rising wait means writes are queueing behind each other:
histogram_quantile(0.99, sum by (le) (rate(sharpmush_storage_writer_wait_milliseconds_bucket[5m])))The kind values of sharpmush_storage_bytes are listed in
Storage and History Retention.
From the game server, once feeds hold lines.
| Metric | What it is | Labels |
|---|---|---|
sharpmush_feed_lines | Lines held by each kind of feed | kind |
sharpmush_feed_stored_bytes | Space those lines take in the world | kind |
The message bus
Section titled “The message bus”The game server and the connection server pass commands and output to each other through NATS. Both publish these, and each reports the consumers it reads from.
| Metric | What it is | Labels |
|---|---|---|
sharpmush_messaging_stream_bytes | Bytes a stream holds | stream |
sharpmush_messaging_stream_max_bytes | A stream’s byte budget, or -1 for none | stream |
sharpmush_messaging_stream_messages | Messages a stream holds | stream |
sharpmush_messaging_consumer_pending | Messages not yet delivered to a consumer | stream, consumer |
sharpmush_messaging_consumer_ack_pending | Messages delivered and not yet finished | stream, consumer |
sharpmush_messaging_consumer_redelivered | Messages a consumer was given more than once | stream, consumer |
sharpmush_messaging_publish_rejected_total | Messages the bus refused or did not confirm in time | stream, reason |
sharpmush_messaging_handler_retries_total | Messages tried again after a failure | consumer |
sharpmush_messaging_handler_terminal_failures_total | Messages given up on | consumer, reason |
sharpmush_messaging_handler_duplicates_total | Messages delivered again after they were handled, and skipped | consumer |
sharpmush_messaging_jetstream_storage_used_bytes | Disk the bus uses | |
sharpmush_messaging_jetstream_storage_available_bytes | Disk the bus may still use, when it has a limit |
The streams are read every 30 seconds. SHARPMUSH-CS carries players’ input to the game server and
SHARPMUSH-MS carries output back. When one of them is full, the bus refuses new messages on it
(sharpmush_messaging_publish_rejected_total with reason="full") until the other side catches up;
nothing already queued is lost. Each gets 512 MiB unless SHARPMUSH_NATS_MAX_BYTES (in bytes) is set
on both programs. TERMINAL_REPLAY_V2 keeps recent browser terminal output for reconnects and drops
its oldest output when full, so it is normal for it to sit near its budget.
.NET and HTTP
Section titled “.NET and HTTP”Both programs also publish the standard .NET runtime metrics (dotnet_*: memory, garbage
collection, threads, CPU time) and ASP.NET Core’s web server metrics (http_server_*, kestrel_*).
CPU use of each program, in cores:
sum by (job) (rate(dotnet_process_cpu_time_seconds_total[5m]))