You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Track the gap between generic stream primitives and a complete local-file upload/download workflow in language bindings.
Problem and Solution
AWS CLI 2.33.15 provides a concrete application: its upload/download submitters pass a filename to the native transfer manager. The file destination allows the manager to own scheduling and writes; the application does not reconstruct an ordered array of Range responses. OpenDAL applications should have an idiomatic, bounded-memory way to complete the same file task without assembling another service-specific transfer manager.
Prioritize a native file-input path where it removes language-level copying or source preparation, and a file-output path where positioned writes can exploit independent ranges.
Reuse core Reader/Writer, file-service and existing transfer capabilities as appropriate. Keep generic stream transformations supported.
Specify source mutation/replay, destination overwrite/temporary-file publication, close/durability, cancellation, retry ownership, metadata and memory limits before finalizing method names.
Let an actual caller and a measured capability gap justify each binding addition. Avoid introducing a binding-specific S3 scheduler solely to satisfy a benchmark.
Validation must compare the same final file/object result using the SDK native file API and OpenDAL best public path. Preserve integrity and include the agreed local I/O and completion semantics. Separate time to completion, CPU, memory and temporary-disk cost. No speedup is asserted by this tracking issue.
Feature Description
Track the gap between generic stream primitives and a complete local-file upload/download workflow in language bindings.
Problem and Solution
AWS CLI 2.33.15 provides a concrete application: its upload/download submitters pass a filename to the native transfer manager. The file destination allows the manager to own scheduling and writes; the application does not reconstruct an ordered array of Range responses. OpenDAL applications should have an idiomatic, bounded-memory way to complete the same file task without assembling another service-specific transfer manager.
Scope:
Validation must compare the same final file/object result using the SDK native file API and OpenDAL best public path. Preserve integrity and include the agreed local I/O and completion semantics. Separate time to completion, CPU, memory and temporary-disk cost. No speedup is asserted by this tracking issue.
Related: #8251, #8246, #8248 and #8250.
Additional Context
Source reference. OpenDAL API inventory was checked at
b6cf44f7b8a1523409e0e998e478c996ac970f03.