Skip to content

Collecting Metrics

SharpMUSH reports how it is doing as numbers: players online, how long commands and functions take, how full the world’s disk is, and whether messages between its parts are piling up. This page shows how to collect those numbers with Prometheus, look at them in Grafana, and get an alert when one goes wrong.

It is written for the Docker Compose stacks: the one-click install from Launch a Server, and the files in deploy/ described in the Deployment Reference.

How metrics get from SharpMUSH to Grafana Inside the game's Docker network, Prometheus reads /metrics from the game server on port 9092 and from the connection server on port 9091 every 15 seconds, and keeps the readings. Grafana reads Prometheus. The rendering worker has no metrics. Your browser reaches Grafana through an ssh tunnel to port 3000 on the server's loopback address. The game's Docker network (sharpmush-net) Game server sharpmush-server:9092 Connection server sharpmush-socketserver:9091 Rendering worker no metrics Prometheus reads every 15 s, keeps 30 days GET /metrics GET /metrics Grafana 127.0.0.1:3000 queries Your browser on your computer ssh tunnel

Two of SharpMUSH’s programs publish metrics. Each answers GET /metrics with plain HTTP, in the Prometheus text format, on a port of its own:

ProgramContainerPortSetting
Game serversharpmush-server9092Metrics:Port (environment variable Metrics__Port)
Connection serversharpmush-socketserver9091ConnectionServer:MetricsPort (ConnectionServer__MetricsPort)

The rendering worker has no metrics.

Nothing is sent anywhere: a scraper has to come and read the numbers. Prometheus does that here, but anything that reads the Prometheus format can.

A metrics port answers /metrics and nothing else, and no other port answers /metrics, so a reverse proxy cannot publish them by accident. Setting a metrics port to 0 turns metrics off for that program: no port answers /metrics.

The ports are not on the internet:

Stack90929091
One-click installThe server’s loopback address (127.0.0.1:9092)The server’s loopback address (127.0.0.1:9091)
deploy/docker-compose.prod.yml or docker-compose.cloudflare.ymlThe server’s loopback addressThe game’s Docker network only

To check the game server is publishing, run this on the machine the game runs on:

Terminal window
curl -s http://127.0.0.1:9092/metrics | grep '^sharpmush_players_logged_in'

Output:

sharpmush_players_logged_in{otel_scope_name="SharpMUSH",otel_scope_version="1.0.0"} 0

Run Prometheus and Grafana beside the game

Section titled “Run Prometheus and Grafana beside the game”

The simplest setup runs Prometheus and Grafana as a second Compose project on the same machine, joined to the game’s Docker network. Prometheus then reads both programs by container name, and neither metrics port has to be published anywhere new.

  1. Find the name of the game’s network:

    Terminal window
    docker network ls --filter name=sharpmush-net --format '{{.Name}}'

    A one-click install prints sharpmush_sharpmush-net. A stack started from the deploy/ folder prints deploy_sharpmush-net.

  2. Make a folder for the monitoring files, next to the game’s files but not inside them:

    Terminal window
    mkdir ~/sharpmush-monitoring && cd ~/sharpmush-monitoring
  3. Save this as prometheus.yml. It reads both programs every 15 seconds:

    global:
    scrape_interval: 15s
    scrape_configs:
    - job_name: sharpmush-server
    static_configs:
    - targets: ["sharpmush-server:9092"]
    - job_name: sharpmush-socketserver
    static_configs:
    - targets: ["sharpmush-socketserver:9091"]
    rule_files:
    - /etc/prometheus/alerts.yml
  4. Save the rules from Alerts below as alerts.yml.

  5. Save this as grafana-datasource.yml, so Grafana starts already connected to Prometheus:

    apiVersion: 1
    datasources:
    - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
  6. Save this as compose.yml. If step 1 printed another network name, put it in place of sharpmush_sharpmush-net on the last lines:

    name: sharpmush-monitoring
    services:
    prometheus:
    image: prom/prometheus:v3.5.0
    command:
    - --config.file=/etc/prometheus/prometheus.yml
    - --storage.tsdb.retention.time=30d
    volumes:
    - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    - ./alerts.yml:/etc/prometheus/alerts.yml:ro
    - prometheus-data:/prometheus
    ports:
    - "127.0.0.1:9090:9090"
    networks:
    - game
    restart: unless-stopped
    grafana:
    image: grafana/grafana:12.1.0
    volumes:
    - ./grafana-datasource.yml:/etc/grafana/provisioning/datasources/prometheus.yml:ro
    - grafana-data:/var/lib/grafana
    ports:
    - "127.0.0.1:3000:3000"
    networks:
    - game
    restart: unless-stopped
    volumes:
    prometheus-data:
    grafana-data:
    networks:
    game:
    name: sharpmush_sharpmush-net
    external: true

    Prometheus keeps 30 days of readings in its own volume. Both ports are on the loopback address only.

  7. Start them:

    Terminal window
    docker compose up -d
  8. Check that Prometheus can read both programs:

    Terminal window
    curl -s http://127.0.0.1:9090/api/v1/targets | grep -o '"health":"[a-z]*"'

    Output:

    "health":"up"
    "health":"up"

    A "health":"down" means Prometheus cannot reach that program. Check the network name in compose.yml and that the game is running.

Grafana listens on the game machine’s loopback address, so reach it through an ssh tunnel. On your own computer:

Terminal window
ssh -N -L 3000:127.0.0.1:3000 root@<your-server-ip>

Leave that running and open http://localhost:3000. Use the same kind of tunnel, with port 9090, to open Prometheus itself.

SharpMUSH ships a Grafana dashboard for function and command timings: the slowest functions and commands, where the time goes, and the call and failure rates. To add it:

  1. Download it on your own computer:

    Terminal window
    curl -fsSL https://raw.githubusercontent.com/SharpMUSH/SharpMUSH/main/deploy/grafana/sharpmush-invocations.json -o sharpmush-invocations.json
  2. In Grafana, open Dashboards, then New, then Import.

  3. Select Upload dashboard JSON file, pick the file, then select Import.

The dashboard is called SharpMUSH function and command timings. Its Prometheus menu picks the data source, and starts on the one set up above. Its Function and Command menus narrow every panel to the names you pick.

These rules cover the failures an operator most needs to hear about. Save them as alerts.yml:

groups:
- name: sharpmush
rules:
- alert: SharpMUSHDown
expr: up{job=~"sharpmush-.*"} == 0
for: 2m
annotations:
summary: "{{ $labels.job }} has not answered a scrape for 2 minutes"
- alert: SharpMUSHBackupWillNotStart
expr: sharpmush_storage_backup_fits == 0
annotations:
summary: "The next backup would not find enough free disk"
- alert: SharpMUSHMapNearlyFull
expr: sharpmush_storage_bytes{kind="map_headroom"} < ignoring(kind) 2 * sharpmush_storage_bytes{kind="live"}
annotations:
summary: "The world is close to its map limit; raise SHARPMUSH_LIGHTNING_MAPSIZE"
- alert: SharpMUSHBusStreamFilling
expr: max by (stream) (sharpmush_messaging_stream_bytes{stream=~"SHARPMUSH-.*"} / sharpmush_messaging_stream_max_bytes) > 0.8
for: 10m
annotations:
summary: "Bus stream {{ $labels.stream }} is over 80% of its byte budget"
- alert: SharpMUSHBusRefusing
expr: sum by (stream, reason) (increase(sharpmush_messaging_publish_rejected_total[10m])) > 0
annotations:
summary: "The bus refused messages on {{ $labels.stream }} ({{ $labels.reason }})"
- alert: SharpMUSHMessagesDropped
expr: sum by (job, consumer, reason) (increase(sharpmush_messaging_handler_terminal_failures_total[10m])) > 0
annotations:
summary: "{{ $labels.consumer }} gave up on messages ({{ $labels.reason }})"
- alert: SharpMUSHBacklog
expr: sum by (job, consumer) (sharpmush_messaging_consumer_pending) > 1000
for: 5m
annotations:
summary: "{{ $labels.consumer }} has over 1000 messages waiting"

A firing alert shows on Prometheus’s Alerts page. To have one sent to you by email, Discord or another service, set up a contact point in Grafana’s Alerting section, or run Prometheus’s Alertmanager.

Storage and History Retention explains the storage figures behind the backup and map alerts, and has two more disk alerts.

Prometheus adds job and instance labels to every series, and SharpMUSH adds otel_scope_name and otel_scope_version. Counters end in _total, timings in _milliseconds, and sizes are in bytes.

Both programs publish the player, connection and health gauges below, but only one of them keeps each up to date. Read players, connections and sharpmush_server_health from job="sharpmush-server", and sharpmush_connectionserver_health from job="sharpmush-socketserver". The other program reports a fixed 0 for players and connections, and a fixed 1 for the other’s health.

MetricWhat it isLabels
sharpmush_players_logged_inConnections logged in to a character now; a character connected twice counts twice
sharpmush_connections_activeConnections now, logged in or not
sharpmush_connection_events_totalConnections opened and closed, logins and logouts, from the game server onlyevent_type: connected, disconnected, logged_in, logged_out, account_mode
sharpmush_server_health1 while the game server runs
sharpmush_connectionserver_health1 while the connection server runs

A health gauge cannot report 0 for a program that has stopped, so alert on Prometheus’s own up instead, as SharpMUSHDown does.

Logins in the last hour:

sum(increase(sharpmush_connection_events_total{event_type="logged_in"}[1h]))
MetricLabels
sharpmush_function_invocation_duration_millisecondsfunction_name, success
sharpmush_command_invocation_duration_millisecondscommand_name, success

Each is a histogram of the time one call took, from the game server. A call’s time is its own work: the time spent on its arguments and on calls inside it is counted where it was spent. A label holds the built-in’s name in capitals, whatever the caller typed, so @pe is counted as @PEMIT. The Deployment Reference has the details.

The slowest functions, at the 95th percentile:

topk(10, histogram_quantile(0.95, sum by (le, function_name) (rate(sharpmush_function_invocation_duration_milliseconds_bucket[5m]))))

From the game server.

MetricWhat it isLabels
sharpmush_storage_bytesThe world’s sizes: map limit, file, disk used, live data, free space and what a backup needskind
sharpmush_storage_backup_fits1 when the next backup would find enough free disk, 0 when it would not start
sharpmush_storage_attribute_index_bytesSpace each attribute index takesindex
sharpmush_storage_stale_readers_cleared_totalRead slots freed because the program holding them died mid-read
sharpmush_storage_writer_wait_millisecondsHow long each write waited for its turn
sharpmush_storage_writer_hold_millisecondsHow long each write took, saving to disk included
sharpmush_storage_writer_queuedWrites waiting now
sharpmush_storage_read_duration_millisecondsHow long each read stayed open
sharpmush_storage_reads_activeReads open now

Writes run one at a time. A rising wait means writes are queueing behind each other:

histogram_quantile(0.99, sum by (le) (rate(sharpmush_storage_writer_wait_milliseconds_bucket[5m])))

The kind values of sharpmush_storage_bytes are listed in Storage and History Retention.

From the game server, once feeds hold lines.

MetricWhat it isLabels
sharpmush_feed_linesLines held by each kind of feedkind
sharpmush_feed_stored_bytesSpace those lines take in the worldkind

The game server and the connection server pass commands and output to each other through NATS. Both publish these, and each reports the consumers it reads from.

MetricWhat it isLabels
sharpmush_messaging_stream_bytesBytes a stream holdsstream
sharpmush_messaging_stream_max_bytesA stream’s byte budget, or -1 for nonestream
sharpmush_messaging_stream_messagesMessages a stream holdsstream
sharpmush_messaging_consumer_pendingMessages not yet delivered to a consumerstream, consumer
sharpmush_messaging_consumer_ack_pendingMessages delivered and not yet finishedstream, consumer
sharpmush_messaging_consumer_redeliveredMessages a consumer was given more than oncestream, consumer
sharpmush_messaging_publish_rejected_totalMessages the bus refused or did not confirm in timestream, reason
sharpmush_messaging_handler_retries_totalMessages tried again after a failureconsumer
sharpmush_messaging_handler_terminal_failures_totalMessages given up onconsumer, reason
sharpmush_messaging_handler_duplicates_totalMessages delivered again after they were handled, and skippedconsumer
sharpmush_messaging_jetstream_storage_used_bytesDisk the bus uses
sharpmush_messaging_jetstream_storage_available_bytesDisk the bus may still use, when it has a limit

The streams are read every 30 seconds. SHARPMUSH-CS carries players’ input to the game server and SHARPMUSH-MS carries output back. When one of them is full, the bus refuses new messages on it (sharpmush_messaging_publish_rejected_total with reason="full") until the other side catches up; nothing already queued is lost. Each gets 512 MiB unless SHARPMUSH_NATS_MAX_BYTES (in bytes) is set on both programs. TERMINAL_REPLAY_V2 keeps recent browser terminal output for reconnects and drops its oldest output when full, so it is normal for it to sit near its budget.

Both programs also publish the standard .NET runtime metrics (dotnet_*: memory, garbage collection, threads, CPU time) and ASP.NET Core’s web server metrics (http_server_*, kestrel_*). CPU use of each program, in cores:

sum by (job) (rate(dotnet_process_cpu_time_seconds_total[5m]))