Dynamic Middleware Management API Implementation

[keshxvdayal]

A dynamic middleware management system for proTES that allows runtime modification of the middleware stack without service restarts. The system provides RESTful API endpoints to list, add, remove, and reorder middlewares, with changes persisted to the database and reflected immediately in request processing.

[keshxvdayal]

Implementation Plan
1. Backend API Endpoints (COMPLETED)
GET /api/middlewares - List all configured middlewares with their order and metadata
POST /api/middlewares - Add new middleware (local or from external sources like GitHub)
DELETE /api/middlewares/ - Remove a middleware by ID
PUT /api/middlewares/order - Reorder the middleware stack by providing ordered list of IDs

2. Dynamic Middleware Loading (COMPLETED)
Middleware classes are dynamically imported at runtime using importlib
Support for loading middleware from local class paths (e.g., 'pro_tes.middleware.TaskLog')
Support for fetching middleware code from GitHub URLs
Validation ensures all middleware classes inherit from AbstractMiddleware
Each middleware addition triggers immediate reload of the in-memory stack

3. Order Field Logic (COMPLETED)
When order field is not specified in POST request, middleware is appended to the end of the stack
When order field is specified, middleware is inserted at that position and existing middlewares at that position or higher have their order incremented by 1
Reordering endpoint accepts array of middleware IDs in desired order and updates all order values accordingly

4. Fallback Groups Support (COMPLETED)
API accepts class_path as either a string (single middleware) or list of strings (fallback group)
Fallback groups allow multiple middlewares at the same position where first is tried, and if it fails, next is attempted
Current implementation stores fallback groups in database and middleware_handler processes them correctly

5. Persistence (COMPLETED)
All middleware configurations stored in MongoDB 'middlewares' collection
Each middleware document contains: name, class_path, order, source (local/github)
On service startup, middleware stack is automatically loaded from database
reload_middleware_stack() function reconstructs in-memory middleware list from database after any modification

6. Immediate Effect (COMPLETED)
Changes to middleware stack via API are immediately reflected in request processing
reload_middleware_stack() is called after every add, delete, or reorder operation
No service restart required for middleware changes to take effect

7. Auto-Dependency Installation (IN PROGRESS)
When fetching middleware from GitHub, detect presence of requirements.txt, pyproject.toml, or setup.py in the repository
Parse dependency specifications from these files
Automatically install dependencies using subprocess calls to pip
Log installation progress and errors for debugging
Handle edge cases: missing dependency files, installation failures, version conflicts

8. Documentation (PENDING)
Document all API endpoints with request/response examples in service README
Explain order field behavior and fallback group mechanism
Provide examples of local and external middleware addition
Document persistence and immediate effect behavior
Include troubleshooting section for common issues

9. Testing and Validation (PENDING)

Test with existing middlewares from pro_tes.middleware package
Test with custom external middleware from GitHub repositories
Verify persistence by restarting service and checking middleware stack
Verify immediate effect by monitoring request logs before and after middleware changes
Test edge cases: duplicate names, duplicate class paths, invalid middleware code, GitHub fetch failures
Capture screenshots demonstrating each operation for PR review

Screenshots Testing all core functionality:

• List middlewares (GET)
• Add middleware without order (appends)
• Add middleware with order (inserts)
• Fallback groups
• Get specific middleware
• Duplicate prevention
• Reordering
• Deletion

[uniqueg]

The general plan is fine, thanks @keshxvdayal . But a few important comments:

1. This is too big a feature for a single issue. Please break it down into subtasks, where each of the points you are listing is at most one subtask. Raise a PR for each subtask individually. E.g., let's first merge the API endpoints, then the dynamic middleware loading etc. Provide a more detailed plan for each subtask, for example, I will create an OpenAPI document for defining the endpoint operations, then will include that in the FOCA configuration, then will implement controllers for each operation etc. That way, we can catch problems early.
2. Things are complete if, and only if, they are documented, tested, reviewed and merged. You mark subtasks as completed even though there aren't even code changes proposed for these. As mentioned in the call, please push all code all the time. There should not be a situation where the state in your local copy differs more than a day's worth from the feature branch on the remote. If you follow that, then there will also be no difference between your perception of completeness and ours.
3. We are following the principles of continuous documentation and continuous testing. This means that docs and testing can never be an afterthought. Updating docs and tests has to be an ongoing effort and part of every individual subtask. You can of course work on multiple subtasks/branches/features etc. at the same time, if you can manage, especially when waiting for reviews, but your work on each subtask/PR is not done until it's merged - and that won't happen if the necessary unit and regression tests are not written and the docs are provided/updated, where applicable.

So, let's not make this another huge PR, but please break it down and let's do one step at a time, following the practices described above and others listed in our contributor guidelines. Btw, please re-review them. For example, we use PR titles following the Conventional Commit specification.

[keshxvdayal]

Thanks for the detailed feedback. Understood 👍
I’ll break this into smaller, well-defined subtasks, provide a detailed plan for each, and raise separate PRs accordingly (with docs and tests included per subtask). I’ll also ensure code is pushed continuously and follow the contributor guidelines, including Conventional Commit PR titles.