Sourcegulldocs
/

Alerts

Every alert Sourcegull's Prometheus rules can fire, with what to do about it. The rules are generated from crates/monitoring into deploy/docker/prometheus/sourcegull-alerts.yml (in the Compose bundle, to load into your Prometheus) and the Helm chart's PrometheusRule (metrics.prometheusRule.enabled=true); README.md shows how to set them up. See dashboards.md for the panels they watch.

Each alert is named sourcegull_<service>_<observable> and carries the labels severity (warning: needs a look soon; critical: users are affected or will be shortly), service, owner and sourcegull_alert (the anchor below). A warning and a critical rule share a name; inhibit the warning while the critical one fires. Alertmanager routing is up to your Prometheus setup.

web_http_5xx_ratio

web: share of requests answered with 5xx

Descriptions

  • warning: web: share of requests answered with 5xx above 2% for 5m
  • critical: web: share of requests answered with 5xx above 10% for 10m

Next steps

  • Find the failing routes: in Grafana, run sum by (route, status) (rate(sourcegull_http_requests_total{role=~"web|all",status=~"5.."}[5m])).
  • Read the web logs for error entries around that time (docker compose logs web, kubectl logs -l app.kubernetes.io/component=web). Failed responses carry an x-request-id that appears in the log lines.
  • Many 502/503 from GraphQL or search often mean Postgres or the repo nodes are unreachable: check the Postgres and repo dashboards.
  • Panel: share of requests answered with 5xx on the Sourcegull: web dashboard.

Query

promql
(sum(rate(sourcegull_http_requests_total{role=~"web|all",route!~"/metrics|/healthz|/readyz",status=~"5.."}[5m])) or 0 * sum(rate(sourcegull_http_requests_total{role=~"web|all",route!~"/metrics|/healthz|/readyz"}[5m])))
/ sum(rate(sourcegull_http_requests_total{role=~"web|all",route!~"/metrics|/healthz|/readyz"}[5m]))

Owner: platform

web_http_p90_latency

web: 90th percentile request duration by route

Descriptions

  • warning: web: 90th percentile request duration by route above 5s for 15m

Next steps

  • Check which routes are slow on this panel; GraphQL (/.api/graphql) has its own panel below.
  • Slow GraphQL requests and searches are logged above observability.logSlowGraphQLRequests / observability.logSlowSearches and listed under Site admin > Slow requests.
  • Check Postgres (connection saturation, long transactions) and CPU usage of the web instances; add web replicas if they are CPU-bound.
  • Panel: 90th percentile request duration by route on the Sourcegull: web dashboard.

Query

promql
histogram_quantile(0.9, sum by (le, route) (rate(sourcegull_http_request_duration_seconds_bucket{role=~"web|all",route!~"/metrics|/healthz|/readyz|/.api/search/stream|/.api/compute/stream|/.api/blame/.*|/.api/scip/upload|/.api/files/.*|/.executors/.*"}[5m])))

Owner: platform

web_search_stream_p90_duration

web: 90th percentile streaming search duration

Descriptions

  • warning: web: 90th percentile streaming search duration above 20s for 15m

Time from the request until the search stream ends, as users see it. Histogram buckets end at 60 s.

Next steps

  • Check the repo dashboard: index queue length and index errors (unindexed repositories are searched without the index, which is slower), and the repo node instances.
  • Look for expensive queries under Site admin > Slow requests (observability.logSlowSearches).
  • Check CPU and memory of the repo nodes; the index's hot set should fit in RAM (see the Helm sizing guide).
  • Panel: 90th percentile streaming search duration on the Sourcegull: web dashboard.

Query

promql
histogram_quantile(0.9, sum by (le) (rate(sourcegull_http_request_duration_seconds_bucket{role=~"web|all",route="/.api/search/stream"}[5m])))

Owner: search

web_graphql_p90_duration

web: 90th percentile GraphQL request duration

Descriptions

  • warning: web: 90th percentile GraphQL request duration above 5s for 15m

Next steps

  • Find the slow operations under Site admin > Slow requests (enable observability.logSlowGraphQLRequests).
  • Check the Postgres dashboard for connection saturation and long transactions.
  • Panel: 90th percentile GraphQL request duration on the Sourcegull: web dashboard.

Query

promql
histogram_quantile(0.9, sum by (le) (rate(sourcegull_http_request_duration_seconds_bucket{role=~"web|all",route="/.api/graphql"}[5m])))

Owner: platform

web_webhook_rejection_ratio

web: share of code host webhooks rejected

Descriptions

  • warning: web: share of code host webhooks rejected above 50% for 30m

Next steps

  • 401/403: the secret token in GitLab does not match the webhook's secret. Compare them on the code host connection page (Site admin > Code host connections > Webhooks) and in GitLab (project or group > Settings > Webhooks).
  • 404: the webhook was deleted in Sourcegull but GitLab still sends to it; remove it in GitLab.
  • Recent deliveries and their errors are listed under Site admin > Webhook logs.
  • Panel: share of code host webhooks rejected on the Sourcegull: web dashboard.

Query

promql
(sum(rate(sourcegull_http_requests_total{role=~"web|all",route=~"/.api/gitlab-webhooks|/.api/webhooks/.*",status!~"2.."}[30m])) or 0 * sum(rate(sourcegull_http_requests_total{role=~"web|all",route=~"/.api/gitlab-webhooks|/.api/webhooks/.*"}[30m])))
/ sum(rate(sourcegull_http_requests_total{role=~"web|all",route=~"/.api/gitlab-webhooks|/.api/webhooks/.*"}[30m]))

Owner: source

web_instance_up

web: instances answering Prometheus scrapes

Descriptions

  • critical: web: instances answering Prometheus scrapes below 1 for 5m

1 while Prometheus can scrape the instance's internal listener (:6060), 0 when it cannot.

Next steps

  • Check the web containers or pods (docker compose ps web, kubectl get pods -l app.kubernetes.io/component=web) and their logs for a crash or failed start.
  • A web instance refuses to start while database migrations are pending: run sourcegull migrate up (the Helm chart's migrate job does this on upgrade).
  • If the instance runs but cannot be scraped, check that Prometheus reaches port 6060 (NetworkPolicy, Compose network).
  • Panel: instances answering Prometheus scrapes on the Sourcegull: web dashboard.

Query

promql
up{role=~"web|all"}

Owner: platform

repo_disk_free_ratio

repo: free disk space on the data volume

Descriptions

  • warning: repo: free disk space on the data volume below 15% for 10m
  • critical: repo: free disk space on the data volume below 5% for 5m

Measured by the repo node janitor (2 minutes after start, then hourly). The node pauses clones when free space is below both SOURCEGULL_REPO_PAUSE_CLONES_FREE_PERCENT (default 10%) of the volume and SOURCEGULL_REPO_PAUSE_CLONES_FREE_GIB (default 20 GiB); Site admin > Repo nodes shows the threshold.

Next steps

  • Grow the node's volume (Helm: repo.persistence.size on a storage class that allows expansion; Compose: the repo-0-data volume's disk).
  • Or add a repo node so repositories spread over more disks (Helm: repo.replicaCount).
  • Check Site admin > Repo nodes and Repositories for unexpectedly large repositories; excluding them in the code host connection frees their space after the next sync.
  • The repo node janitor removes repositories deleted from Sourcegull and maintains the object stores; its errors appear in the repo node's logs.
  • Panel: free disk space on the data volume on the Sourcegull: repo dashboard.

Query

promql
min by (instance) (sourcegull_repo_disk_free_bytes{role=~"repo|all"} / sourcegull_repo_disk_total_bytes{role=~"repo|all"})

Owner: source

repo_repo_update_error_ratio

repo: share of clones and fetches failing

Descriptions

  • warning: repo: share of clones and fetches failing above 20% for 30m
  • critical: repo: share of clones and fetches failing above 80% for 30m

Next steps

  • Site admin > Repositories lists the failing repositories with their last error (filter: Failed fetch).
  • If most updates fail, the code host is probably unreachable or the connection's token expired or lost access: test the connection under Site admin > Code host connections.
  • Check the repo node logs for repo update failed entries; corrupt clones are re-cloned from Site admin > Repositories.
  • Panel: share of clones and fetches failing on the Sourcegull: repo dashboard.

Query

promql
(sum by (kind) (rate(sourcegull_repo_updates_total{role=~"repo|all",outcome="error"}[30m])) or 0 * sum by (kind) (rate(sourcegull_repo_updates_total{role=~"repo|all"}[30m])))
/ sum by (kind) (rate(sourcegull_repo_updates_total{role=~"repo|all"}[30m]))

Owner: source

repo_update_queue_oldest

repo: age of the oldest queued clone or fetch

Descriptions

  • warning: repo: age of the oldest queued clone or fetch above 2h for 30m

Next steps

  • Check that every repo node is up (instances panel) and processing: sourcegull_repo_updates_total should grow.
  • A large backlog after adding a code host connection drains by itself; to speed it up raise gitMaxConcurrentClones in the site config (read when a repo node starts) and SOURCEGULL_REPO_MAX_CONCURRENT_FETCHES on the repo nodes, or add repo nodes.
  • Queued and failed jobs are listed under Site admin > Background jobs (repo_update_jobs).
  • Panel: age of the oldest queued clone or fetch on the Sourcegull: repo dashboard.

Query

promql
max(sourcegull_queue_oldest_queued_seconds{queue="repo_update_jobs"})

Owner: source

repo_repos_corrupted

repo: repositories marked corrupt

Descriptions

  • warning: repo: repositories marked corrupt above 0 for 5m

Next steps

  • Re-clone the repositories from Site admin > Repositories (filter: Corrupted).
  • Repeated corruption on one node points to its disk or file system: check the node's kernel and volume logs.
  • Panel: repositories marked corrupt on the Sourcegull: repo dashboard.

Query

promql
max(sourcegull_repos_corrupted)

Owner: source

repo_index_queue_length

repo: repositories waiting to be indexed

Descriptions

  • warning: repo: repositories waiting to be indexed above 500 for 1h

Next steps

  • After a large import the queue drains by itself; check the build rate on the index updates panel.
  • If it does not shrink, check the repo node logs for indexing failed and its CPU usage; raise the node's CPU or add repo nodes.
  • Searches over unindexed repositories still work but are slower.
  • Panel: repositories waiting to be indexed on the Sourcegull: repo dashboard.

Query

promql
max by (instance) (sourcegull_index_queue_length{role=~"repo|all"})

Owner: search

repo_index_error_ratio

repo: share of index builds failing

Descriptions

  • warning: repo: share of index builds failing above 20% for 30m

Next steps

  • Site admin > Repositories shows the index status and error of each repository (filter: Not indexed).
  • Check the repo node logs for indexing failed; disk full (see the disk panels) is a common cause.
  • Panel: share of index builds failing on the Sourcegull: repo dashboard.

Query

promql
(sum(rate(sourcegull_index_updates_total{role=~"repo|all",outcome="error"}[30m])) or 0 * sum(rate(sourcegull_index_updates_total{role=~"repo|all"}[30m])))
/ sum(rate(sourcegull_index_updates_total{role=~"repo|all"}[30m]))

Owner: search

repo_repo_moves_failed

repo: repository moves that gave up

Descriptions

  • warning: repo: repository moves that gave up above 0 for 30m

Next steps

  • Site admin > Repo nodes lists the failed moves with their errors; the repositories stay on their old node and keep working.
  • Common causes: the old node is unreachable from the new one (gRPC port 3178), or the new node's disk is full.
  • Fix the cause, then use Retry failed moves on that page.
  • Panel: repository moves that gave up on the Sourcegull: repo dashboard.

Query

promql
max(sourcegull_repo_moves{state="failed"})

Owner: source

repo_instance_up

repo: instances answering Prometheus scrapes

Descriptions

  • critical: repo: instances answering Prometheus scrapes below 1 for 5m

1 while Prometheus can scrape the instance's internal listener (:6060), 0 when it cannot.

Next steps

  • Check the repo pods or containers (kubectl get pods -l app.kubernetes.io/component=repo, docker compose ps repo-0) and their logs.
  • A repo node refuses to start while migrations are pending, and when its data volume is not writable.
  • While a node is down, searches skip its repositories (results say so) and its clones and fetches wait.
  • Panel: instances answering Prometheus scrapes on the Sourcegull: repo dashboard.

Query

promql
up{role=~"repo|all"}

Owner: platform

worker_queue_oldest_queued

worker: age of the oldest queued job by queue

Descriptions

  • warning: worker: age of the oldest queued job by queue above 6h for 30m

Next steps

  • Check that the workers are up (instances panel) and their logs for errors from the queue's handler.
  • Site admin > Background jobs shows the queue's oldest jobs and recent failures; a single job type failing and retrying can hold up the rest.
  • Queues that do much work per job (trend backfills, search jobs) can lag after large changes; add worker replicas or raise the feature's concurrency setting (e.g. insights.query.worker.concurrency, batchChanges.reconcilerConcurrency).
  • Panel: age of the oldest queued job by queue on the Sourcegull: worker dashboard.

Query

promql
max by (queue) (sourcegull_queue_oldest_queued_seconds{queue!~"codeintel_index_jobs|batch_spec_workspace_execution_jobs|repo_update_jobs"})

Owner: platform

worker_upload_processing_failure_ratio

worker: share of SCIP upload processing runs failing

Descriptions

  • warning: worker: share of SCIP upload processing runs failing above 25% for 30m

Next steps

  • The repository's Code graph > Uploads page shows each failed upload's error.
  • unknown commit errors mean the repo node has not fetched the uploaded commit yet; they are retried. Check the repo dashboard if they persist.
  • Check the worker logs for processing an upload failed and the blob store (blob dashboard) for errors reading upload files.
  • Panel: share of SCIP upload processing runs failing on the Sourcegull: worker dashboard.

Query

promql
(sum(rate(sourcegull_codeintel_upload_processing_seconds_count{outcome!="completed"}[1h])) or 0 * sum(rate(sourcegull_codeintel_upload_processing_seconds_count[1h])))
/ sum(rate(sourcegull_codeintel_upload_processing_seconds_count[1h]))

Owner: code-intel

worker_codeintel_failing_repos_growth

worker: growth in repositories whose precise indexing fails, over 6 hours

Descriptions

  • warning: worker: growth in repositories whose precise indexing fails, over 6 hours above 0 for 1h

Next steps

  • Site admin > Index coverage, filtered to Failing, lists the repositories with the latest failure message; each links to the repository's Code graph page with the job's logs.
  • Many repositories failing at once usually means the executors or an indexer image broke: check the executor dashboard and an index job's logs.
  • Code navigation in a failing repository answers from its last completed index or from search, so results can be incomplete until indexing succeeds again.
  • Panel: growth in repositories whose precise indexing fails, over 6 hours on the Sourcegull: worker dashboard.

Query

promql
max(sourcegull_codeintel_coverage_repos{state="failing"}) - max(sourcegull_codeintel_coverage_repos{state="failing"} offset 6h)

Owner: code-intel

worker_reconciler_failure_ratio

worker: share of changeset reconciliations failing

Descriptions

  • warning: worker: share of changeset reconciliations failing above 20% for 30m

Next steps

  • The bulk change's page lists the failed changesets with the code host's error.
  • Most failures are code host permissions: the bulk change's credential (user or site credential) needs api scope and Developer access to the project.
  • Check the code host's rate limits and status; the outbound requests log (Site admin > Outbound requests) shows recent calls when outboundRequestLogLimit is set.
  • Panel: share of changeset reconciliations failing on the Sourcegull: worker dashboard.

Query

promql
(sum(rate(sourcegull_batches_changesets_reconciled_total{outcome="failed"}[30m])) or 0 * sum(rate(sourcegull_batches_changesets_reconciled_total[30m])))
/ sum(rate(sourcegull_batches_changesets_reconciled_total[30m]))

Owner: batches

worker_changeset_sync_error_ratio

worker: share of changeset syncs failing

Descriptions

  • warning: worker: share of changeset syncs failing above 50% for 1h

Next steps

  • The syncer reads merge request state from GitLab; check the code host connection (Site admin > Code host connections > Test) and its token.
  • Worker logs show the failing changesets and errors.
  • Panel: share of changeset syncs failing on the Sourcegull: worker dashboard.

Query

promql
(sum(rate(sourcegull_batches_changeset_syncs_total{outcome="error"}[1h])) or 0 * sum(rate(sourcegull_batches_changeset_syncs_total[1h])))
/ sum(rate(sourcegull_batches_changeset_syncs_total[1h]))

Owner: batches

worker_code_monitor_delivery_error_ratio

worker: share of watch notifications failing by kind

Descriptions

  • warning: worker: share of watch notifications failing by kind above 50% for 30m

Next steps

  • Email: check the SMTP settings (email.smtp and email.address in the site config) and the worker logs for SMTP errors.
  • Slack, Mattermost and webhooks: the receiving URL rejects or does not answer; the watch's page shows the last errors. Fix or remove the action.
  • Telegram: the Bot API refuses the message (wrong bot token, or the bot is not in the chat) or is unreachable from the worker; the watch's page shows the Bot API's answer.
  • Panel: share of watch notifications failing by kind on the Sourcegull: worker dashboard.

Query

promql
(sum by (kind) (rate(sourcegull_code_monitor_deliveries_total{outcome="error"}[1h])) or 0 * sum by (kind) (rate(sourcegull_code_monitor_deliveries_total[1h])))
/ sum by (kind) (rate(sourcegull_code_monitor_deliveries_total[1h]))

Owner: code-insights

worker_outbound_webhook_error_ratio

worker: share of outbound webhook deliveries failing

Descriptions

  • warning: worker: share of outbound webhook deliveries failing above 50% for 30m

Next steps

  • Site admin > Outbound webhooks shows each webhook's recent deliveries with status and error.
  • Fix or remove the receiving endpoint; failed deliveries are retried with backoff.
  • Panel: share of outbound webhook deliveries failing on the Sourcegull: worker dashboard.

Query

promql
(sum(rate(sourcegull_outbound_webhook_deliveries_total{outcome="error"}[1h])) or 0 * sum(rate(sourcegull_outbound_webhook_deliveries_total[1h])))
/ sum(rate(sourcegull_outbound_webhook_deliveries_total[1h]))

Owner: platform

worker_oob_migration_errors

worker: out-of-band migration errors in the last hour

Descriptions

  • warning: worker: out-of-band migration errors in the last hour above 0

Next steps

  • Site admin > Migrations lists the migration's last errors.
  • Re-encryption errors usually mean a key is missing from the key file (SOURCEGULL_KEYS_FILE) or SOURCEGULL_KEYS: keep old keys there until the migration completes.
  • Upgrades refuse to pass a version whose out-of-band migrations are incomplete, so fix the cause before upgrading.
  • Panel: out-of-band migration errors in the last hour on the Sourcegull: worker dashboard.

Query

promql
sum by (id) (increase(sourcegull_oob_migration_errors_total[1h]))

Owner: platform

worker_instance_up

worker: instances answering Prometheus scrapes

Descriptions

  • critical: worker: instances answering Prometheus scrapes below 1 for 5m

1 while Prometheus can scrape the instance's internal listener (:6060), 0 when it cannot.

Next steps

  • Check the worker containers or pods (docker compose ps worker, kubectl get pods -l app.kubernetes.io/component=worker) and their logs.
  • A worker refuses to start while migrations are pending.
  • While no worker runs, queues grow and singleton jobs (syncs, janitors, schedulers) stop; they resume when a worker is back.
  • Panel: instances answering Prometheus scrapes on the Sourcegull: worker dashboard.

Query

promql
up{role=~"worker|all"}

Owner: platform

executor_jobs_without_executors

executor: queued executor jobs while no executor is active

Descriptions

  • warning: executor: queued executor jobs while no executor is active above 0 for 15m

The number of queued executor jobs when no executor has sent a heartbeat for a minute, else 0.

Next steps

  • Start executors: Compose docker compose --profile executors up -d, Helm executor.enabled=true.
  • If executors run, check their logs: a wrong SOURCEGULL_EXECUTOR_TOKEN (must equal executors.accessToken in the site config) or an unreachable SOURCEGULL_EXECUTOR_FRONTEND_URL stops heartbeats.
  • Without executors, disable auto-indexing (Site admin > Code graph policies) and server-side bulk changes so jobs stop piling up.
  • Panel: queued executor jobs while no executor is active on the Sourcegull: executor dashboard.

Query

promql
sum(max by (queue) (sourcegull_queue_jobs{queue=~"codeintel_index_jobs|batch_spec_workspace_execution_jobs",state="queued"}))
* on() (max(sourcegull_executors_active) == bool 0)

Owner: platform

executor_queue_oldest_queued

executor: age of the oldest queued executor job by queue

Descriptions

  • warning: executor: age of the oldest queued executor job by queue above 1h for 30m

Next steps

  • Executors are busy or missing: compare running jobs with the executors' capacity (SOURCEGULL_EXECUTOR_MAX_JOBS / Helm executor.maxConcurrentJobs per executor).
  • Add executors (Helm executor.replicaCount) or raise their concurrency if the nodes have room.
  • Executors only take queues listed in SOURCEGULL_EXECUTOR_QUEUES (Helm executor.queues); check the queue is in the list.
  • Panel: age of the oldest queued executor job by queue on the Sourcegull: executor dashboard.

Query

promql
max by (queue) (sourcegull_queue_oldest_queued_seconds{queue=~"codeintel_index_jobs|batch_spec_workspace_execution_jobs"})

Owner: platform

executor_api_5xx_ratio

executor: share of executor API requests answered with 5xx

Descriptions

  • warning: executor: share of executor API requests answered with 5xx above 5% for 15m

Next steps

  • Read the web logs for errors on /.executors/ routes; git requests are served from the repo nodes, so check the repo dashboard too.
  • Upload failures of batch step results can come from the blob store: check the blob dashboard.
  • Panel: share of executor API requests answered with 5xx on the Sourcegull: executor dashboard.

Query

promql
(sum(rate(sourcegull_http_requests_total{role=~"web|all",route=~"/.executors/.*",status=~"5.."}[15m])) or 0 * sum(rate(sourcegull_http_requests_total{role=~"web|all",route=~"/.executors/.*"}[15m])))
/ sum(rate(sourcegull_http_requests_total{role=~"web|all",route=~"/.executors/.*"}[15m]))

Owner: platform

blob_error_ratio

blob: share of blob requests answered with 5xx

Descriptions

  • warning: blob: share of blob requests answered with 5xx above 1% for 10m
  • critical: blob: share of blob requests answered with 5xx above 10% for 10m

Next steps

  • Read the blob role's logs for blob store request failed.
  • A full or read-only volume is the usual cause: check the free space of the blob volume (Helm blobstore.persistence.size).
  • Panel: share of blob requests answered with 5xx on the Sourcegull: blob dashboard.

Query

promql
(sum(rate(sourcegull_blob_requests_total{role=~"blob|all",status=~"5.."}[10m])) or 0 * sum(rate(sourcegull_blob_requests_total{role=~"blob|all"}[10m])))
/ sum(rate(sourcegull_blob_requests_total{role=~"blob|all"}[10m]))

Owner: platform

blob_instance_up

blob: instances answering Prometheus scrapes

Descriptions

  • critical: blob: instances answering Prometheus scrapes below 1 for 5m

1 while Prometheus can scrape the instance's internal listener (:6060), 0 when it cannot.

Next steps

  • Check the blob pod or container (kubectl get pods -l app.kubernetes.io/component=blob, docker compose ps blob) and its logs.
  • While it is down, SCIP uploads, bulk change file uploads and search job downloads fail.
  • Panel: instances answering Prometheus scrapes on the Sourcegull: blob dashboard.

Query

promql
up{role=~"blob|all"}

Owner: platform

postgres_postgres_up

postgres: workers reaching Postgres

Descriptions

  • critical: postgres: workers reaching Postgres below 1 for 5m

1 when the last metrics collection by a worker (every 30 s) reached Postgres, else 0.

Next steps

  • Check that Postgres runs and accepts connections (pg_isready), and its logs (disk full, too many connections, crash recovery).
  • Check SOURCEGULL_DATABASE_URL and the network path (Kubernetes NetworkPolicy, external database firewall).
  • All roles need Postgres: expect web, worker and repo alerts until it is back.
  • Panel: workers reaching Postgres on the Sourcegull: postgres dashboard.

Query

promql
max(sourcegull_postgres_up)

Owner: platform

postgres_connection_saturation

postgres: share of max_connections in use by Sourcegull's database

Descriptions

  • warning: postgres: share of max_connections in use by Sourcegull's database above 80% for 5m
  • critical: postgres: share of max_connections in use by Sourcegull's database above 95% for 5m

Next steps

  • Each Sourcegull process opens a pool per role; scaling out web or worker replicas multiplies connections. Compare the pools panel below with max_connections.
  • Raise max_connections in Postgres (and its memory accordingly), or put PgBouncer in transaction mode in front of it.
  • Lower the pool sizes (SOURCEGULL_DB_MAX_CONNECTIONS, SOURCEGULL_DB_WORKER_MAX_CONNECTIONS) of roles that hold many idle connections.
  • Panel: share of max_connections in use by Sourcegull's database on the Sourcegull: postgres dashboard.

Query

promql
sum(max by (state) (sourcegull_postgres_connections))
/ max(sourcegull_postgres_max_connections)

Owner: platform

postgres_oldest_transaction

postgres: age of the oldest open transaction

Descriptions

  • warning: postgres: age of the oldest open transaction above 30m for 5m

Next steps

  • Find it: SELECT pid, usename, application_name, state, xact_start, query FROM pg_stat_activity WHERE xact_start IS NOT NULL ORDER BY xact_start LIMIT 5;. Sourcegull's connections name their role in application_name.
  • Long transactions block vacuum and can hold locks other queries wait on. End it with SELECT pg_terminate_backend(<pid>); if it is stuck.
  • Schema migrations run in a transaction and can legitimately take long during an upgrade.
  • Panel: age of the oldest open transaction on the Sourcegull: postgres dashboard.

Query

promql
max(sourcegull_postgres_oldest_transaction_seconds)

Owner: platform

postgres_pool_saturation

postgres: share of the connection pool in use by process

Descriptions

  • warning: postgres: share of the connection pool in use by process above 90% for 15m

Next steps

  • Requests wait for a connection when the pool is exhausted: raise the pool size (SOURCEGULL_DB_MAX_CONNECTIONS, or SOURCEGULL_DB_WORKER_MAX_CONNECTIONS for the worker pool) if Postgres has spare connections (see connection saturation).
  • A pool that stays full while traffic is normal points to slow queries or long transactions: check the oldest transaction panel and the slow request logs.
  • Panel: share of the connection pool in use by process on the Sourcegull: postgres dashboard.

Query

promql
max by (role, instance, pool) (sourcegull_db_pool_connections{state="in_use"} / ignoring(state) sourcegull_db_pool_connections{state="max"})

Owner: platform

Edit this page on GitHub