Enhance catalog management, caching, and error handling in pChronicle - #138
Merged
Merged
Conversation
…cking - Added new `status` module to define `CatalogConsistency`, `CatalogState`, and `CatalogStatus` for improved catalog state management. - Updated various modules to utilize the new catalog state definitions, enhancing clarity and consistency in catalog operations. - Refactored error handling in the CLI server to include execution stages, improving error reporting and debugging capabilities. This commit aims to enhance the robustness of catalog management and provide clearer insights into catalog states during operations.
- Introduced `lance-io` and `object_store` dependencies to improve object storage capabilities. - Added `BlockCache` and `CacheConfig` for efficient caching of object store reads, enhancing performance. - Implemented a new `lance_cache` benchmark to evaluate caching performance. - Refactored dataset access methods to utilize the new caching mechanisms, improving data retrieval efficiency. - Updated various modules to integrate the new caching features, ensuring consistency across data operations. This commit aims to optimize data handling and improve the overall performance of the pChronicle application.
…ties - Introduced `ManifestRefreshReport` to track the status of mount refresh operations, including partial refreshes. - Updated `refresh_mount` method to return a report on the number of directories refreshed and whether the refresh was complete. - Enhanced `BrowseStatus` to include a `partial` field, indicating if the dataset view is incomplete. - Modified `QueryDatasetSummary` to include browsing status, improving dataset visibility in the UI. - Updated various modules to reflect changes in manifest caching and browsing, ensuring consistency across the application. This commit aims to improve the efficiency and user experience of dataset management and browsing in the pChronicle application.
- Refactored the dataset loading logic to handle errors from the catalog query more gracefully. - Added error reporting to notify users when catalog discovery fails, ensuring a better user experience. - Preserved the functionality for all-dataset requests, allowing for continued usability even when some scans fail. This commit aims to enhance the robustness of dataset loading and improve user feedback during errors.
- Introduced a `RUNS_SCAN_TIMEOUT` constant to enforce a timeout on run scan operations, improving responsiveness and preventing long-running queries from blocking. - Updated the `load_run_summaries` function to utilize the timeout, ensuring that both catalog building and run summary retrieval are subject to this limit. - Enhanced error handling to provide clearer feedback when timeouts occur, improving user experience during dataset loading. This commit aims to improve the robustness and reliability of the server's handling of run scans.
- Introduced a new `with_runs_deadline` function to manage timeouts for run-related operations, enhancing responsiveness and error handling. - Updated the `RUNS_SCAN_TIMEOUT` constant to 50 seconds, ensuring timely cancellation of long-running tasks. - Enhanced the `ApiError` struct with a `runs_timeout` method to provide clearer error messages when timeouts occur. - Added tests to verify the timeout behavior and error reporting for run scans, improving reliability and user feedback. This commit aims to improve the server's handling of run operations by enforcing timeouts and providing structured error responses.
- Introduced a new signal for tracking the generation of run requests, allowing for better management of concurrent requests. - Updated the load_runs function to utilize the generation signal, ensuring that only the latest request's results are processed. - Enhanced error handling and loading state management to prevent outdated responses from interfering with the current request. This commit aims to improve the responsiveness and accuracy of run data loading in the workspace component.
…ties - Introduced `list_for_browse` and `refresh_for_browse` methods to improve the efficiency of browsing datasets without probing every child immediately. - Updated `list_impl` and `list_nav_children` methods to support conditional probing of child datasets, optimizing performance during directory navigation. - Enhanced the `BrowseCoordinator` to leverage the new browsing capabilities, allowing for quicker access to dataset summaries without unnecessary requests. This commit aims to improve the user experience and performance of dataset browsing in the pChronicle application.
- Updated the `AimdState` structure to manage concurrency per endpoint and bucket, improving remote operation handling. - Refactored the `state_for` function to incorporate concurrency limits, ensuring better resource management during I/O operations. - Introduced a new `is_transient_error` function to classify errors more accurately, enhancing the robustness of error handling in remote operations. - Improved logging for manifest cache persistence and dataset browsing, providing clearer insights into operation statuses. This commit aims to optimize the performance and reliability of object store interactions in the pChronicle application.
- Updated `CatalogTree` and `BrowseStatus` structs to include default values, improving initialization consistency. - Refactored the `label` method in `QueryDatasetSummary` for better readability and maintainability. - Introduced a new signal for tracking catalog generation, enhancing concurrency control during dataset loading. - Improved error handling and loading state management in the `load_catalog_tree` function to ensure accurate responses based on request generation. This commit aims to optimize dataset browsing and improve the user experience in the pChronicle application.
- Updated the `load_runs` function to use `generation.peek()` instead of directly reading the signal, avoiding potential subscription to its own writes and preventing infinite request loops. - This change enhances the stability of the reactive effect by ensuring proper management of request generations. This commit aims to improve the reliability of run data loading in the workspace component.
…nagement - Introduced `StoreConfig` struct to encapsulate S3 configuration parameters, improving clarity and usability. - Added `with_background_object_store_io` function to manage background I/O operations independently from foreground tasks, enhancing concurrency control. - Updated `DatasetLocation` and `DatasetMount` to support backend configuration, allowing for more flexible dataset management. - Refactored object store interaction methods to utilize the new configuration, ensuring consistent behavior across different storage backends. This commit aims to improve the configurability and performance of object store interactions in the pChronicle application.
…acking - Introduced a new `request_progress` module to provide bounded, opt-in diagnostics for API requests, allowing users to track execution phases and outcomes. - Added functionality to capture and report the state of requests, including phases such as authentication, execution, and response. - Enhanced middleware to integrate request progress tracking, ensuring that diagnostic information is available for ongoing requests. - Updated the UI to display request diagnostics, improving user visibility into API interactions and their statuses. This commit aims to enhance the observability and user experience of API requests in the pChronicle application.
- Updated the logic in the `App` function to prevent unnecessary refetching of runs when navigating to the detail page, ensuring that the existing run data is retained. - Introduced a condition to check if the detail page is accessed without any loaded runs, enhancing performance and user experience by avoiding redundant data requests. This commit aims to improve the efficiency of run data management in the workspace component.
- Refactored the worker pool to implement a more efficient slot management system, allowing for up to 4 workers per scope and 8 workers server-wide. - Introduced a cleanup mechanism that reclaims idle workers after 120 seconds, with a reaper task running every 30 seconds to maintain optimal resource usage. - Updated the lease mechanism to allow concurrent requests in the same scope to utilize different execution workers, improving performance and resource allocation. - Enhanced documentation to reflect the new worker management capabilities and timeout settings. This commit aims to optimize worker utilization and improve the responsiveness of the worker pool in the pChronicle application.
Unqualified text search now targets only `message_value`; field selectors carry their own columns, and `#content` is an alias for the message body. Preview SQL partitions matches per visible run so every page row gets evidence, and previews render around the hit rather than truncating the whole field.
Increase the remote concurrency defaults and overlap adjacent block fetches to reduce latency when reading uncached objects. Batch FTS predicate searches across datasets and open all storyline datasets concurrently.
- Introduced `RetryPatience` enum to define retry behavior for interactive and batch operations, allowing for better control over request timeouts. - Updated `with_object_store_retries` to utilize the new patience settings, optimizing retry strategies based on workload type. - Enhanced `PersistentCache` to include compaction logic, reducing fragmentation and improving write performance. - Added methods for managing foreground demand in object store operations, ensuring that interactive requests are prioritized over background tasks. This commit aims to improve the efficiency and reliability of object store interactions in the pChronicle application.
- Cleaned up formatting in `object_store_io_gate.rs` and `persistent_cache.rs` to enhance code readability by aligning method calls and reducing line breaks. - Simplified the `lane` assignment in the `Gate` struct for better clarity. - Improved the structure of `upsert` method calls in `PersistentCache` to maintain consistent formatting. This commit aims to enhance code maintainability and readability across the object store and cache management components in the pChronicle application.
- Simplified error handling in various modules by removing unnecessary conversions, enhancing clarity and reducing boilerplate code. - Improved cache management logic in `opendal_store.rs` and `persistent_cache.rs` to streamline operator insertion and batch processing. - Enhanced readability by restructuring conditional statements and aligning method calls. This commit aims to enhance code maintainability and performance across the pChronicle application.
- Added `RetryPatience` and `set_retry_patience` to `opendal_store` for improved retry behavior in object store operations. - Introduced `PersistentCache` export in `mod.rs` to streamline cache management. - Refactored hash generation in `fingerprint` and `backend_identity_from_env` methods to use `unwrap_or_default()` for safer error handling. This commit aims to improve the configurability and reliability of object store interactions and cache management in the pChronicle application.
- Updated the `public_mounts` method to utilize `libraries_for_public` and enhance the creation of `DatasetMount` instances with backend configuration. - Refactored dataset listing and access control logic to directly use the `CatalogSnapshot`, improving clarity and reducing reliance on optional states. - Simplified credential retrieval for public datasets, streamlining the request handling process. This commit aims to improve the structure and efficiency of catalog access control and dataset management in the pChronicle application.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.