Skip to main content

Maintenance And Troubleshooting

Use this page after ProPR is already running.

Updating​

The canonical update flow is described in Deployment; the short version:

For launcher-based installs, pull the newer launcher image and restart the stack:

docker pull propr/launcher:latest

Stop the running launcher (Ctrl-C, or stop its container — it tears down the stack containers on shutdown), then start it again with your usual docker run command from Production Deployment. The launcher's manifest pins exact image versions, so the newer launcher pulls the newer service images and unified agent image.

For source-based Compose deployments:

git pull
npm ci
npm run compose:up

npm run compose:up runs scripts/compose.sh up, which rebuilds and restarts the stack from docker-compose.yml plus docker-compose.dev.yml. Use npm run compose:build to force a full image rebuild and npm run compose:down to stop the stack.

Upgrading Safely​

Service images and the unified agent image are pinned to the release version of the control plane that starts them: the CLI and launcher manifests pin exact image versions, so choosing the control-plane version chooses the whole stack version. propr/launcher:latest and a plain npm update -g propr-cli track the newest release; to pin, install an exact CLI version (npm install -g propr-cli@<version>) and update deliberately.

A safe upgrade, in order:

  1. Take a backup — at minimum the SQLite database and .env.
  2. Update the control plane and restart (the flows above); persistent state in data/, repos/, and the Redis volume is unaffected.
  3. From a source checkout, apply pending database migrations with npm run db:migrate.

After the upgrade, the API's public /api/compatibility endpoint reports the stack version and the API/UI compatibility contract — the hosted UI checks it before login and stops at a clear version-mismatch screen rather than running against an incompatible stack. /api/status includes the same metadata for authenticated diagnostics.

The unified agent image replaces the older per-agent image families. After confirming the upgraded worker is healthy, reclaim disk used by stale local agent images with docker image ls 'propr/*agent*' and remove only obsolete per-agent repositories that are no longer referenced by your running containers or configuration.

If installation-wide runtime packages are enabled, verify their derived images after the upgrade or any base-image replacement:

propr runtime packages verify

This check is read-only and uses the same current base-image configuration as agent execution. It exits nonzero for stale lineage, missing derived images or packages, pinned-version mismatches, incorrect ProPR labels/final users, and desired-versus-active drift. Repair an unhealthy profile with propr runtime packages apply --wait, then verify again. Use --json for structured per-image diagnostics. A disabled/empty profile succeeds explicitly without starting a check container.

Refreshing agent image package pins​

Dockerfile.agent pins its Debian packages with *_VERSION_PREFIX build args and verifies the GitHub CLI apt keyring against GITHUBCLI_KEYRING_SHA256. Two external events can break the base-stage build (which surfaces as a failed local build fallback when the registry image is unavailable):

  • Debian moves a package beyond a pinned prefix. The build fails with No apt version found for <package> with prefix <prefix>. Refresh the affected *_VERSION_PREFIX args from the current node:22-bookworm-slim archive (docker run --rm node:22-bookworm-slim sh -c 'apt-get update -qq && apt-cache madison <package>') and rebuild.
  • GitHub rotates the CLI keyring. The sha256sum -c step fails. Re-download https://cli.github.com/packages/githubcli-archive-keyring.gpg, verify it out of band, and update GITHUBCLI_KEYRING_SHA256.

Only gh has a latest-version fallback (its repo drops old releases); every other pin fails the build hard, so treat a failing agent-image build in CI as a signal to refresh pins rather than a transient error.

Rollback: re-pin the previous version (npm install -g propr-cli@<previous>, or the previous launcher image) and restart from the runtime directory. Treat database migrations as forward-only in practice: the Knex migrations define down steps, but no packaged command runs them — npm run db:migrate only applies migrate:latest — so the reliable way to roll back the database is restoring the SQLite backup taken in step 1.

Database Migrations​

The SQLite schema is managed with Knex migrations (knexfile.ts, migrations in packages/core/src/db/migrations/). After updating a source checkout, apply pending migrations:

npm run db:migrate

This runs npx knex migrate:latest --knexfile ./knexfile.ts against the database at DB_FILENAME (default ./data/propr.sqlite).

Resetting Queue State​

If jobs are stuck in failed or processing states, reset the queue:

# Development (tsx, debug logging)
npm run daemon:reset:dev

# Against a production build
npm run daemon:reset

The reset clears all Redis queue data (waiting, active, completed, and failed jobs) and removes processing/done state labels from GitHub issues, so issues can be reprocessed from a clean state. npm run worker:reset and npm run worker:reset:dev perform the equivalent reset before starting a worker.

Logs​

Service processes log structured JSON to stdout. For launcher installs, the containers are named with the propr- prefix:

docker ps
docker logs -f propr-daemon
docker logs -f propr-worker
docker logs -f propr-api

Other locations:

  • The host logs/ directory (PROPR_LOGS_DIR) is mounted into the service containers at /usr/src/app/logs.
  • Agent session logs are written under /tmp/claude-logs on the host (shared with the worker containers and the agent containers they start), and are surfaced per task in the Web UI task detail view and through the /api/execution/... endpoints.
  • Per-LLM-call records are stored in the SQLite llm_logs table and shown on the LLM Log page; see Metrics.
  • Set LOG_LEVEL=debug in .env for more verbose service logs.

Backups​

Back up:

  • The SQLite database (data/propr.sqlite, including -wal/-shm files) — the primary application state. Copy it while the stack is stopped, or use sqlite3 propr.sqlite ".backup backup.sqlite" for a consistent snapshot of a live database (WAL mode is enabled)
  • Production .env and the GitHub App private key (or the secret source that produces them)
  • ProPR's managed agent credential root if you use direct login (~/.propr/agent-credentials for CLI-started, native and Compose installs, or PROPR_DATA_DIR/agent-credentials for the launcher container)
  • The propr-redis-data Docker volume if you want queue state and sessions to survive a restore
  • Logs, if you need history

Git clones and worktrees (under /tmp/git-processor by default) can contain unpublished commits, stashes, ignored/private files and recovery state. Include these directories, their Git registrations, and salvage records/bundles in a recoverable backup; a remote clone alone is insufficient. Do not back up only repos/ and logs/ — they do not contain the application state. The runtime directory these paths live in is described in Deployment → Runtime Directory Layout.

Repository And Worktree Cleanup​

ProPR keeps Git state under /tmp/git-processor on the host. The launcher mounts that path at the same location in every service container, so agent containers can bind-mount worktrees by their host path:

  • /tmp/git-processor/clones/ — cached clones, one per monitored repository (GIT_CLONES_BASE_PATH)
  • /tmp/git-processor/worktrees/ — per-task worktrees (GIT_WORKTREES_BASE_PATH)

Worktrees are retained after completion. Cleanup may detach HEAD to free the task branch, but does not remove the checkout, branch history or worktree registration. WORKTREE_RETENTION_STRATEGY=always_delete is a legacy setting and does not authorize deletion; expiry and mtimes are never recovery proofs. Do not delete managed directories by hand to resolve disk pressure.

The instance storage API provides supported dry-run and execute controls. Both require an authenticated instance administrator and, for POST, JSON plus X-ProPR-Storage: 1. Use your established instance API session:

  1. Read GET /api/instance/storage/v1/capabilities to obtain instanceId.
  2. Read GET /api/instance/storage/v1/policy and GET /api/instance/storage/v1/usage. Usage includes the latest bounded generated-payload inspection, reclaimable bytes, protected candidates, coverage gaps and last cleanup result. These observations may be stale or partial; unknown usage is not zero.
  3. Dry-run with POST /api/instance/storage/v1/cleanup/preview, body {"instanceId":"…","categories":["worktrees"]}. This does not delete data or enable policy.
  4. After separately reviewed policy enablement (PUT /api/instance/storage/v1/policy, body {"instanceId":"…","enabled":true,"idleHours":24}), repeat the preview and review candidates. Execute its short-lived confirmation with POST /api/instance/storage/v1/cleanup/execute, body {"instanceId":"…","confirmation":"…"}. Read GET /api/instance/storage/v1/cleanup/operations/<confirmation>?instanceId=… for its durable result. Confirmation is consumed before effects and cannot be replayed.

Generated cleanup is opt-in, with a minimum 24 hours since actual last use even for manual execution. The instance-wide durable use clock is deliberately conservative: any protected task/repository/read/queue activity resets it. Active readers, queued/failed/recovering tasks, unknown queue/database authority, and abandoned lifecycle claims prevent cleanup. Pressure never weakens these rules. Policy remains disabled by default; implementation does not enable it or clean live storage.

Eligible payloads must be created empty by a trusted producer through GeneratedPayloadStorage.generate, with explicit regeneration semantics and a complete content fingerprint stored outside the checkout. Only node_modules, .propr/cache, .propr/build, and .propr/test-cache are allowlisted. Existing directories cannot be adopted. Producers must generate only refetchable/public or reproducible output, never copy secrets/customer inputs into these directories. Ordinary agent installs and historical worktrees remain protected because their origin is unknown.

For newly provisioned managed worktrees, a supported optional producer is:

npx tsx scripts/generate-worktree-dependencies.ts /tmp/git-processor/worktrees/<owner>/<repo>/<task>

This refuses existing node_modules, requires an npm v2/v3 lockfile with integrity-pinned public registry dependencies, and runs npm ci with scripts and bin links disabled. It does not enable policy or mark an active task completed. Task lifecycle completion later marks generated records eligible; failed and salvage-retained runs remain protected. This producer is intentionally narrower than ordinary package installation.

Each preview rotates through at most ten records, with bounded enumeration, a 100,000-entry/512 MiB payload proof budget and five-second filesystem proof window; API operations also have a 60-second deadline. Large or unreadable payloads remain protected. Fresh content, tracked-file, Git integrity, inode, link and mount proofs are required at execution. Only exact proven generated files are unlinked; empty payload roots and all Git/customer files remain. Partial interruption retains the record and fails closed on a subsequent changed fingerprint; obtain a fresh preview rather than replaying a confirmation. No whole-worktree deletion is implemented.

Worker admission reserves 2 GiB free space plus 1 GiB per admitted task by default (including durable outstanding reservations). Configure WORKTREE_ADMISSION_RESERVE_BYTES and WORKTREE_TASK_BUDGET_BYTES for your workload. Insufficient/unknown capacity rejects new worker admission without deleting files. These are admission estimates, not filesystem quotas: provision filesystem/container write quotas for a hard ceiling on arbitrary task writes. Unregistered/oversized payloads, abandoned claims, non-Linux hosts and unrestricted task writers remain explicit coverage gaps; increase disk capacity or investigate protected data through supported recovery/backup procedures.

Rejected Pushes And The Salvage Ladder​

When the final push from a worktree fails — implementation runs, PR follow-ups, /fix, ultrafix cycles and merge-conflict jobs — ProPR does not discard the agent's commits with the worktree. It tries, in order:

  1. Retry the push once with a freshly refreshed installation token (tokens are short-lived and can expire during long runs).
  2. Rescue ref: push the same commits to refs/propr/rescue/<taskId>--<timestamp> on the same remote (the UTC creation time, for example 20261006T120000Z, is part of the name). This works for non-fast-forward rejections and transient branch-level failures because the ref is new; it does not help when push protection rejects the commits themselves.
  3. Bundle: write a git bundle to PUSH_RESCUE_BUNDLE_DIR (default <DATA_DIR>/rescue/<owner>/<repo>/<taskId>.bundle).
  4. Keep the worktree: override WORKTREE_RETENTION_STRATEGY for that task, detach its HEAD (so the branch is free for the next job) and record the retention in PUSH_RESCUE_WORKTREE_RECORD_DIR, outside the checkout (an informational .retention-info.json with "reason": "push_salvage" is also written into the worktree).

The rung that succeeded is recorded on the task timeline, and the task fails with the push diagnosis — the rejection class, GitHub's unblock URL for push protection, and the exact recovery command — in the task detail view, propr task get <task-id> (pushFailure in --json) and the GitHub failure comment. See Troubleshooting → A Push Was Rejected for recovering each kind.

Retention. The daemon deletes rescue refs created more than PUSH_RESCUE_RETENTION_DAYS (default 14) days ago — measured from the timestamp in the ref name, not from the commit dates — from every monitored repository, together with bundles of the same age, every PUSH_RESCUE_SWEEP_INTERVAL_MS (default 6 hours) and once at startup. Set the retention to 0 to keep them until you delete them yourself. Retained worktrees are not age-deleted. Their protection is recorded through a record in PUSH_RESCUE_WORKTREE_RECORD_DIR (default <DATA_DIR>/rescue-worktrees); the .retention-info.json written into the worktree is informational and does not by itself keep a worktree. To delete one rescue ref by hand:

git push origin --delete refs/propr/rescue/<taskId>--<timestamp>

Rescue refs live outside refs/heads, so GitHub never lists them as branches, normal clones do not fetch them, and ProPR ignores pushes to them for branch discovery and merge-conflict detection.

Common Issues​

Symptom-by-symptom diagnosis has moved to the dedicated Troubleshooting page. It covers the scenarios that used to live here — Dashboard Shows Unauthorized, Worker Not Processing Issues, and Metrics Not Updating — plus setup failures, agent run failures, stuck queues, and hosted UI tunnel problems.

Debugging​

Start with:

docker ps
docker logs <container>

Use Redis inspection carefully on production systems. Avoid KEYS '*'; use scan-based queries instead:

# Launcher install
docker exec propr-redis redis-cli --scan --pattern 'bull:*'

# Source Compose install
docker compose exec redis redis-cli --scan --pattern 'bull:*'

Security Checks​

  • Use strong secrets (SESSION_SECRET, GH_WEBHOOK_SECRET, SYSTEM_TASK_SECRET).
  • Enable HTTPS in production.
  • Restrict access to the Docker socket.
  • Restrict mounted credential directories.
  • Keep images updated.
  • Test restore procedures.

Tuning​

Increase concurrency only when the bottleneck is clear. The relevant knobs:

  • WORKER_CONCURRENCY — concurrent jobs per worker (code default 5; the shipped .env.example sets 2)
  • POLLING_INTERVAL_MS (default 60000) — issue polling frequency
  • Agent timeouts: CLAUDE_TIMEOUT_MS, CODEX_TIMEOUT_MS, ANTIGRAVITY_TIMEOUT_MS, OPENCODE_TIMEOUT_MS, VIBE_TIMEOUT_MS

Watch queue depth, task duration, provider rate limits, disk usage, CPU, and memory. For provider capacity pressure, see the Agent Tank usage sidebar.

Teardown​

To remove ProPR from a host completely:

  1. Stop the stack. propr stop stops and removes the stack containers (use --keep only if you want them retained). For launcher installs, stop the launcher container — it stops and removes the stack containers on shutdown. For source Compose installs, run npm run compose:down.

  2. Remove the Redis volume. Stopping the stack leaves queue state and sessions behind in a Docker volume:

    docker volume rm propr-redis-data     # CLI/launcher installs; the prefix follows PROPR_STACK (default: propr)

    For source Compose installs, docker compose down --volumes removes the compose-managed redis_data volume.

  3. Revoke hosted credentials (relay mode and hosted UI tunnel only), before deleting .env. From the stack directory:

    propr relay list
    propr relay revoke <id>

    Relay tokens can also be managed from the ProPR Connect dashboard. If you used the hosted UI tunnel, deprovision the tunnel in ProPR Connect — PROPR_UI_TUNNEL_TOKEN is a live Cloudflare credential.

  4. Delete the runtime directory — data/, logs/, repos/, plus .env and the GitHub App private key (own-App mode):

    rm -rf /srv/propr     # or your propr-deploy directory

    Launcher-container agent credentials are already below data/. For a CLI-started, native, or Compose install that used direct agent login, separately remove ~/.propr/agent-credentials after backing up or revoking those provider accounts; do not remove the rest of ~/.propr unless you also intend to discard other CLI state.

  5. Uninstall the GitHub App. Remove the ProPR GitHub App installation (the shared hosted App, or your own) from the organization or repositories in GitHub settings. If you registered a GitHub OAuth App only for this install's dashboard login, delete that too.

  6. Uninstall the CLI:

    npm uninstall -g propr-cli    # use the same sudo convention as the install

Optionally reclaim disk from the pulled images: list them with docker images 'propr/*' and remove with docker rmi.