Uploader improvements and misc fixes

[adkinsrs]

This pull request introduces several improvements and fixes across the codebase, focusing on standardizing and validating observation metadata, improving resource management for consumers, and enhancing spatial data handling. The most important changes are outlined below.

Observation Metadata Standardization and Validation

• Replaced the old observation metadata sanitization and categorization logic in anndata_processor.py with a new _sanitize_and_flag_obs_columns method. This method standardizes and sanitizes the obs table, flags ambiguous columns (e.g., numeric columns that may be categorical), and records them in metadata.json for uploader review. The new logic is consistently applied across all supported data formats, improving downstream data quality and user experience. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12]
• Updated the spatial data processing pipeline to standardize and flag ambiguous observation columns before writing the Zarr store, ensuring consistency with the main AnnData processing workflow. [1] [2]

Resource Management and Reliability

• Added documentation and example configuration for setting per-worker memory caps and RabbitMQ consumer timeouts. This prevents runaway memory usage and duplicate job processing by the projectr-consumer, improving reliability and stability of the system.

Spatial Data Handling Improvements

• Added a zarr_path property to the SpatialHandler class, with getter and setter, to consistently manage the Zarr file path. Updated the get_adata method in analysis.py to load AnnData directly from the Zarr file, improving reliability and error handling. [1] [2] [3]
• Ensured that Ensembl IDs in adata.var are made unique before downstream analysis in both the primary analysis pipeline and spatial data QC/embedding computation, preventing errors caused by duplicate IDs. [1] [2]

Dependency and Documentation Updates

• Removed the unused kaleido dependency from requirements and installation scripts. [1] [2]
• Updated Apache and systemd setup documentation for more robust and streamlined service management, including enabling new consumers and reloading the systemd daemon. [1] [2]

These changes collectively improve data integrity, reliability, and maintainability of the system.

[adkinsrs]

Will address later

[adkinsrs]

Will address later

[adkinsrs]

Will address later