Configura filas de parsing e atualiza seed de logs

README.md

@@ -106,6 +106,7 @@ task_search_log_files.delay(
     from_date="2021-05-21",
     until_date="2021-05-21",
     trigger_validation=True,
+    parse_queue_name="parse_xlarge",
 )
 ```
 
@@ -150,6 +151,20 @@ Use `compose=production.yml` or another Compose file when needed:
 make ps compose=production.yml
 ```
 
+Parsing behavior can be adjusted through comma-separated environment variables:
+
+| Variable | Default |
+|---|---|
+| `DEFAULT_PARSE_QUEUE` | `parse_small` |
+| `PARSING_METADATA_CACHE_COLLECTIONS` | All active log collections |
+| `PARSING_METADATA_CACHE_RELEASE_COLLECTIONS` | `scl` |
+| `YEAR_PARTITIONED_COLLECTIONS` | `chl,col,mex,scl` |
+
+The worker image starts one Celery process per container. Its entrypoint accepts
+`CELERY_WORKER_QUEUES`, `CELERY_WORKER_CONCURRENCY`,
+`CELERY_WORKER_PREFETCH_MULTIPLIER`, `CELERY_WORKER_NAME`, and
+`CELERY_WORKER_LOG_LEVEL`. An empty queue setting consumes Celery's default queue.
+
 Run one test path:
 
 ```bash
@@ -172,14 +187,17 @@ Metadata synchronization keeps sources and documents updated from ArticleMeta, O
 
 Configure the default schedule manually in Wagtail/Admin through `django-celery-beat`
 `PeriodicTask` records. Exact cron times may vary by installation, but the default
-operational setup should include:
+operational setup should include the entries below. Every task that triggers
+parsing can receive an explicit `parse_queue_name`. If omitted, the downstream
+worker uses `DEFAULT_PARSE_QUEUE` (`parse_small` by default). The periodic task
+itself runs on `load`, while `parse_queue_name` selects the parsing worker.
 
 | Task | Suggested schedule | Notes |
 |---|---|---|
-| `[Metadata] Daily Sync Routine (Auto)` | Daily, early morning | Refreshes sources and documents before log processing. Use the `load` queue. |
-| `[Log Pipeline] Daily Routine (Auto)` | Daily, after metadata sync | Runs Search -> Validate -> Parse -> Export for new logs. Use the `load` queue. |
-| `[Metrics] Resume Log Exports` | Every 15-30 minutes | Retries errored or stale daily metric export jobs. |
-| `[Metrics] Resume Stale Parsing Logs` | Every 30-60 minutes | Marks stale `PAR` logs for retry. |
+| Individual `[Metadata] Sync ...` tasks | Daily, early morning | Stagger the required ArticleMeta, OPAC, Books, Preprints, and Dataverse collectors on the `load` queue. |
+| `[Log Pipeline] 1. Search Logs (Manual)` | Daily, after metadata sync | Create one entry per collection group. Use `load` as the task queue and set `parse_queue_name` for parsing. |
+| `[Metrics] Resume Log Exports` | Every 15-30 minutes | Retries errored or stale daily metric export jobs. Accepts `queue_name`; otherwise uses `DEFAULT_PARSE_QUEUE`. |
+| `[Metrics] Resume Stale Parsing Logs` | Every 30-60 minutes | Marks stale `PAR` logs for retry. Accepts `queue_name`; otherwise uses `DEFAULT_PARSE_QUEUE`. |
 | `[Metrics] Cleanup Daily Payloads` | Daily or weekly | Removes old exported daily payload files. |
 | `[Reports] Populate All Reports` | Daily, after log processing | Refreshes weekly, monthly, and yearly log report tables. |
 

VERSION

@@ -1 +1 @@
-2.3.6
+2.3.7

collection/tasks.py

@@ -1,11 +1,7 @@
-from django.contrib.auth import get_user_model
-
 from collection.models import Collection
 from config import celery_app
 from core.utils.request_utils import _get_user
 
-User = get_user_model()
-
 
 @celery_app.task(bind=True, name="[Collection] Load Collection Data")
 def task_load_collections(self, user_id=None, username=None):

compose/local/django/Dockerfile

@@ -1,10 +1,10 @@
-ARG PYTHON_VERSION=3.11-bullseye
+ARG PYTHON_VERSION=3.11-bookworm
 
 # define an alias for the specfic python version used in this file.
-FROM python:${PYTHON_VERSION} as python
+FROM python:${PYTHON_VERSION} AS python
 
 # Python build stage
-FROM python as python-build-stage
+FROM python AS python-build-stage
 
 ARG BUILD_ENVIRONMENT=local
 
@@ -27,7 +27,7 @@ RUN pip wheel --wheel-dir /usr/src/app/wheels -r ${BUILD_ENVIRONMENT}.txt
 
 
 # Python 'run' stage
-FROM python as python-run-stage
+FROM python AS python-run-stage
 
 ARG BUILD_ENVIRONMENT=local
 ARG APP_HOME=/app

compose/local/django/celery/worker/start

@@ -1,34 +1,20 @@
 #!/bin/bash
 
 set -o errexit
+set -o pipefail
 set -o nounset
 
-# Worker padrão
-watchgod celery.__main__.main --args -A config.celery_app worker -l INFO --concurrency=1 -n worker.default@%h &
+worker_args=(
+  -A config.celery_app
+  worker
+  -l "${CELERY_WORKER_LOG_LEVEL:-INFO}"
+  --concurrency="${CELERY_WORKER_CONCURRENCY:-1}"
+  --prefetch-multiplier="${CELERY_WORKER_PREFETCH_MULTIPLIER:-1}"
+  -n "${CELERY_WORKER_NAME:-worker.default@%h}"
+)
 
-# Worker para load e validação de dados
-watchgod celery.__main__.main --args -A config.celery_app worker -l INFO --concurrency=4 -Q load -n worker.load@%h &
+if [ -n "${CELERY_WORKER_QUEUES:-}" ]; then
+  worker_args+=(-Q "${CELERY_WORKER_QUEUES}")
+fi
 
-# Worker para scl
-watchgod celery.__main__.main --args -A config.celery_app worker -l INFO --concurrency=1 -Q parse_xlarge -n worker.parse_xlarge@%h &
-
-# Worker para chl col mex
-watchgod celery.__main__.main --args -A config.celery_app worker -l INFO --concurrency=1 -Q parse_large -n worker.parse_large@%h &
-
-# Worker para cri esp psi prt ven
-watchgod celery.__main__.main --args -A config.celery_app worker -l INFO --concurrency=1 -Q parse_medium -n worker.parse_medium@%h &
-
-# Worker para arg bol cub data ecu per preprints pry rve spa sss sza ury wid
-watchgod celery.__main__.main --args -A config.celery_app worker -l INFO --concurrency=1 -Q parse_small -n worker.parse_small@%h &
-
-# Workers seriais adicionais para backfill paralelo de colecoes pequenas
-watchgod celery.__main__.main --args -A config.celery_app worker -l INFO --concurrency=1 -Q parse_small_1 -n worker.parse_small_1@%h &
-watchgod celery.__main__.main --args -A config.celery_app worker -l INFO --concurrency=1 -Q parse_small_2 -n worker.parse_small_2@%h &
-watchgod celery.__main__.main --args -A config.celery_app worker -l INFO --concurrency=1 -Q parse_small_3 -n worker.parse_small_3@%h &
-watchgod celery.__main__.main --args -A config.celery_app worker -l INFO --concurrency=1 -Q parse_small_4 -n worker.parse_small_4@%h &
-watchgod celery.__main__.main --args -A config.celery_app worker -l INFO --concurrency=1 -Q parse_small_5 -n worker.parse_small_5@%h &
-watchgod celery.__main__.main --args -A config.celery_app worker -l INFO --concurrency=1 -Q parse_small_6 -n worker.parse_small_6@%h &
-watchgod celery.__main__.main --args -A config.celery_app worker -l INFO --concurrency=1 -Q parse_small_7 -n worker.parse_small_7@%h &
-watchgod celery.__main__.main --args -A config.celery_app worker -l INFO --concurrency=1 -Q parse_small_8 -n worker.parse_small_8@%h &
-
-wait
+exec watchgod celery.__main__.main --args "${worker_args[@]}"

compose/local/django/start

@@ -6,4 +6,4 @@ set -o nounset
 
 
 python manage.py migrate
-python manage.py runserver_plus 0.0.0.0:8000
+exec python manage.py runserver_plus 0.0.0.0:8000

compose/production/django/Dockerfile

@@ -1,10 +1,10 @@
-ARG PYTHON_VERSION=3.11-bullseye
+ARG PYTHON_VERSION=3.11-bookworm
 
 # define an alias for the specfic python version used in this file.
-FROM python:${PYTHON_VERSION} as python
+FROM python:${PYTHON_VERSION} AS python
 
 # Python build stage
-FROM python as python-build-stage
+FROM python AS python-build-stage
 
 ARG BUILD_ENVIRONMENT=production
 
@@ -28,7 +28,7 @@ RUN pip wheel --wheel-dir /usr/src/app/wheels  \
 
 
 # Python 'run' stage
-FROM python as python-run-stage
+FROM python AS python-run-stage
 
 ARG BUILD_ENVIRONMENT=production
 ARG APP_HOME=/app

compose/production/django/celery/worker/start

@@ -4,32 +4,17 @@ set -o errexit
 set -o pipefail
 set -o nounset
 
-# Worker padrão
-celery -A config.celery_app worker -l INFO --concurrency=1 -n worker.default@%h &
-
-# Worker para load e validação de dados
-celery -A config.celery_app worker -l INFO --concurrency=4 -Q load -n worker.load@%h &
-
-# Worker para parse_scl (coleções extra-grandes)
-celery -A config.celery_app worker -l INFO --concurrency=1 -Q parse_xlarge -n worker.parse_xlarge@%h &
-
-# Worker para chl col mex (coleções grandes)
-celery -A config.celery_app worker -l INFO --concurrency=1 -Q parse_large -n worker.parse_large@%h &
-
-# Worker para cri esp psi prt ven (coleções médias)
-celery -A config.celery_app worker -l INFO --concurrency=1 -Q parse_medium -n worker.parse_medium@%h &
-
-# Worker para arg bol cub data ecu per preprints pry rve spa sss sza ury wid (coleções pequenas)
-celery -A config.celery_app worker -l INFO --concurrency=1 -Q parse_small -n worker.parse_small@%h &
-
-# Workers seriais adicionais para backfill paralelo de colecoes pequenas
-celery -A config.celery_app worker -l INFO --concurrency=1 -Q parse_small_1 -n worker.parse_small_1@%h &
-celery -A config.celery_app worker -l INFO --concurrency=1 -Q parse_small_2 -n worker.parse_small_2@%h &
-celery -A config.celery_app worker -l INFO --concurrency=1 -Q parse_small_3 -n worker.parse_small_3@%h &
-celery -A config.celery_app worker -l INFO --concurrency=1 -Q parse_small_4 -n worker.parse_small_4@%h &
-celery -A config.celery_app worker -l INFO --concurrency=1 -Q parse_small_5 -n worker.parse_small_5@%h &
-celery -A config.celery_app worker -l INFO --concurrency=1 -Q parse_small_6 -n worker.parse_small_6@%h &
-celery -A config.celery_app worker -l INFO --concurrency=1 -Q parse_small_7 -n worker.parse_small_7@%h &
-celery -A config.celery_app worker -l INFO --concurrency=1 -Q parse_small_8 -n worker.parse_small_8@%h &
-
-wait
+worker_args=(
+  -A config.celery_app
+  worker
+  -l "${CELERY_WORKER_LOG_LEVEL:-INFO}"
+  --concurrency="${CELERY_WORKER_CONCURRENCY:-1}"
+  --prefetch-multiplier="${CELERY_WORKER_PREFETCH_MULTIPLIER:-1}"
+  -n "${CELERY_WORKER_NAME:-worker.default@%h}"
+)
+
+if [ -n "${CELERY_WORKER_QUEUES:-}" ]; then
+  worker_args+=(-Q "${CELERY_WORKER_QUEUES}")
+fi
+
+exec celery "${worker_args[@]}"

compose/production/django/start

@@ -26,4 +26,5 @@ if compress_enabled; then
   # NOTE this command will fail if django-compressor is disabled
   python /app/manage.py compress
 fi
+
 /usr/local/bin/gunicorn config.wsgi --bind 0.0.0.0:5000 --chdir=/app --timeout 1000 --workers 3 --worker-connections=1000 --worker-class=gevent