feat: add System Status page with Celery queue monitoring and purge

[valentijnscholten]

Summary

Add a Celery Status page showing the status of the broker and the worker(s) and the number of tasks in the queue. The queue can also be purged.
A separate API call and button to get more details such as the number of tasks per "task name" and a purge button per "task name".

• New System Status page (/system_status) accessible to superusers from the navigation menu, alongside System Settings. Displays Celery worker liveness, pending queue length, and active task timeout/expiry settings.
• Separate Redis Broker and Celery Worker status badges so it is immediately clear which component is failing. Both show a spinner while loading.
• Loading spinners on page load and after clicking Refresh — all status indicators reset to a spinner state before re-fetching.
• Purge queue button to remove all pending tasks (with a warning about re-running deduplication via python manage.py dedupe). Reloads the page after a successful purge.
• View Details button (next to Refresh) — triggers an O(N) full queue scan and renders a per-task breakdown table. Disabled and shown with a warning when the broker is unreachable. Shows a spinner while loading.
  • Table columns: #, task name, count, oldest queue position, newest queue position, expiry timestamps (human-readable local time + time-left indicator), per-row purge button.
  • Rows sorted by oldest queue position ascending.
• Per-task purge — each row in the breakdown table has a Purge button that removes all queued tasks with that task name via a targeted Redis LREM pipeline, without affecting other tasks.
• Celery Settings moved into its own panel section below the status panel.
• Four superuser-only REST API endpoints (same permission guards as SystemSettingsViewSet):
  • GET /api/v2/celery/status/ — worker status, broker status, queue length, and task config
  • POST /api/v2/celery/queue/purge/ — purge all pending tasks from the broker queue
  • GET /api/v2/celery/queue/details/ — per-task breakdown (O(N) queue scan)
  • POST /api/v2/celery/queue/task/purge/ — purge all queued tasks matching a given task name
• Fix get_celery_worker_status() to use app.control.ping() via Celery's pidbox control channel instead of dispatching a task. The old approach would hang indefinitely when the task queue was clogged.
• New environment variables in settings.dist.py:
  • DD_CELERY_TASK_TIME_LIMIT (default: 43200 = 12 h) — hard kill via SIGKILL after this many seconds
  • DD_CELERY_TASK_SOFT_TIME_LIMIT (default: disabled) — raises SoftTimeLimitExceeded for graceful cleanup
  • DD_CELERY_TASK_DEFAULT_EXPIRES (default: 43200 = 12 h) — silently discard tasks that have been waiting in the queue longer than this before any worker picks them up
• Client-side AJAX rendering of the status panel so that dojo-pro can consume the same API endpoints without duplicating the view logic.

[Maffooch]

This is a great start! Would it be possible to iterate over the task names, and get a count of those currently queued? That would allow for easier debugging into where bottlenecks are

[valentijnscholten]

This is a great start! Would it be possible to iterate over the task names, and get a count of those currently queued? That would allow for easier debugging into where bottlenecks are

Redis doesn't provide a lightweight way to do this. The implementation would have to iterate over all entries in the queue and get the task name & count them. I didn't add that as it would be slow probably for clogged instances. But I could do a test to see if it's acceptable and put this behind a "Statistics" button to trigger it only when the user needs the extra level of detail.

[valentijnscholten]

This is a great start! Would it be possible to iterate over the task names, and get a count of those currently queued? That would allow for easier debugging into where bottlenecks are

Redis doesn't provide a lightweight way to do this. The implementation would have to iterate over all entries in the queue and get the task name & count them. I didn't add that as it would be slow probably for clogged instances. But I could do a test to see if it's acceptable and put this behind a "Statistics" button to trigger it only when the user needs the extra level of detail.

@Maffooch Added a View Details button which will render a table with number of tasks per tasks, some info and a per task name purge button. Getting the table with details is not very slow, couple of seconds even with 100k tasks queued.