Skip to content

tracking: add idiomatic bounded-memory local-file transfer APIs to bindings #8254

Description

@Xuanwo

Feature Description

Track the gap between generic stream primitives and a complete local-file upload/download workflow in language bindings.

Problem and Solution

AWS CLI 2.33.15 provides a concrete application: its upload/download submitters pass a filename to the native transfer manager. The file destination allows the manager to own scheduling and writes; the application does not reconstruct an ordered array of Range responses. OpenDAL applications should have an idiomatic, bounded-memory way to complete the same file task without assembling another service-specific transfer manager.

Scope:

  • Inventory the best existing public path in Python, Java, Node.js and Go before adding APIs. Existing stream wrappers and io.Copy remain valid capabilities (new feature: add efficient io.Copy paths to Go Reader and Writer #8248); do not claim they are absent.
  • Prioritize a native file-input path where it removes language-level copying or source preparation, and a file-output path where positioned writes can exploit independent ranges.
  • Reuse core Reader/Writer, file-service and existing transfer capabilities as appropriate. Keep generic stream transformations supported.
  • Specify source mutation/replay, destination overwrite/temporary-file publication, close/durability, cancellation, retry ownership, metadata and memory limits before finalizing method names.
  • Let an actual caller and a measured capability gap justify each binding addition. Avoid introducing a binding-specific S3 scheduler solely to satisfy a benchmark.

Validation must compare the same final file/object result using the SDK native file API and OpenDAL best public path. Preserve integrity and include the agreed local I/O and completion semantics. Separate time to completion, CPU, memory and temporary-disk cost. No speedup is asserted by this tracking issue.

Related: #8251, #8246, #8248 and #8250.

Additional Context

Source reference. OpenDAL API inventory was checked at b6cf44f7b8a1523409e0e998e478c996ac970f03.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions