⚠️ refactor(datashare-python): normalize error reporting

[ClemDoum]

Description

⚠️ breaking datashare-python
Addresses #89,

Aggregated results and errors reporting normalization

This PRs aims at factorizing and normalizing error reporting and aggregated statistics reporting accross workers.
Prior to it the passport worker and extract worker were reporting aggregated stats and errors in their own format.
Since the ASR worker was also expected to report those too, it was the occasion to factorize.

Errors are now reported with a minimal Problem Detail RFC implementation:

class Error(BaseModel):
    title: str
    detail: str | None

When errors are serialiazed in the output payload, details are omitted for security + payload size.

At a workflow levels the number of processed documents (total or successful) can be reported using ProcessingReport or ProcessingReportWithPages if want page level information:

class ProcessingReportWithPages(BaseModel):
	n_pages: int
	n_docs: int

Errors at the document or page level can be reported using ErrorReport.

The error report contains workflows level stats about the number of failed docs / page but all errors per failing document.
Since a document can be processed in several piece (typically when processed by page or page batch), there can be many different errors for a single document. In the ErrorReport errors are grouped per document inside the DocProcessingErrors:

class DocProcessingErrors(BaseModel):
    doc_id: str
    root_document: str
    project: str
    errors: Annotated[list[Error]]

Hence the error report looks like:

class ErrorReport(BaseModel):
	n_docs: int
	errors: list[DocProcessingErrors]

Intermediate results tracing factorization

A document can be transformed many times before an error occur. Additionnally, many different errors can affect the same document as documents can be split into pages or page chunks to be processed. This makes error tracking and aggregration a bit tidious.

To help error aggregation and to improve debugging, a intermediate result mechanism was implemented (ProcessingResult) or more precisely refactored from the existing one in the passport worker in order to serialize intermediate results with trace to their parent (FromParent/WithParent helper classes).

This makes it very easy backtrack to the original Document from the index which was the root of the intermediate results and thus makes it easy to debug + aggregate errors at the document level.

To help with this logic a few objects were introduced/refactored in addition to the existing ones.

Document: represents a Datashare document from the index. Files from the index are stored somewhere, either on the filesystem either or as an artifact.

A document which has been located on the filesystem is called DatashareFile, this file can be transforme into subsequent intermediate results like this:

Document --(file localization)--> DatashareFile --(transformation A)--> ResultA(WithParent[DatashareFile]) --(transformation B)--> ResultB(WithParent[ResultA]) ....

All intermediate results are created using the from_parentclass method which lets us backtrack to the original document.

A generic WorkerFile was create for the common case the the intermediate result is a file persisted on the workdir (passport pages converted to pngs, document converted as PDF, embedded document symlinked to the workdir with a propery file name and extension...)

Changes

datashare-python

Changed

• renamed DocumentLocation to FileLocation

Added

• added the WorkerFilePath object to reprensent a file located on the worker filesystem
• introcuded ErrorSource, ErrorSourceWithPages
• added ProcessingError to hold an Error and its source
• added ProcessingReport, ProcessingReportWithPages, DocProcessingErrors

asr-worker

Added

• report preprocessing, inference and postprocessing errors into an error report

Changed

• split activities into seperate modeles, refactored the workflow into smaller pieces