Feature Description
Reduce the synchronous Python File writer's adapter buffering and repeated runtime entry while preserving the existing file-like API.
Problem and proposed direction
At 4b132b0d33c8a5b84ff83f6fcfbab264860c5b3d, the path is File.write -> StdWriter.write_all -> StdWriter.write -> Handle.block_on(FuturesAsyncWriter.write). The FuturesAsyncWriter stages borrowed input through a 256 KiB FlexBuf, even when Python supplies an immutable 8 MiB bytes object. A large File.write can therefore require repeated adapter writes and runtime entries before the storage writer handles multipart chunking.
The binding could retain the core blocking Writer and pass an owned Buffer retaining the immutable Python bytes. py_bytes_like_into_buffer already provides an owner-preserving conversion for whole-object writes. This is a proposal to remove intermediate work, not a claim of measured benefit from an unimplemented change.
Python File, StdWriter, FuturesAsyncWriter.
Observed workload
Measured OpenDAL commit: 4b132b0d33c8a5b84ff83f6fcfbab264860c5b3d; harness commit: c4c203859ed72d9a0ffac7438486a2d205649102. Real S3 in us-east-2, c7i.8xlarge with a 4-vCPU/1-GiB client limit, six formal repeats. Boto3 1.43.88 uses native APIs, without a transfer manager. Values below are medians, not significance claims. The harness and raw archive are currently private, so these numbers are descriptive evidence, not a publicly reproducible benchmark package.
Uploading a 59,158,238-byte local file uses 8 MiB producer blocks, 8 MiB multipart parts and CRC32C on both sides, followed by independently verified readback. At C4, median elapsed time was 0.419 s for OpenDAL File and 0.370 s for Boto3, while process CPU was lower for OpenDAL (0.075 s versus 0.124 s). These measurements motivate investigation but do not isolate the adapter as the cause of the elapsed-time gap.
Acceptance criteria
- Preserve accepted inputs, write-all byte counts, small-write coalescing, flush behavior, close errors and context-manager completion across services.
- Retain immutable Python owners through buffered, in-flight and retried writes. Do not weaken isolation for mutable inputs.
- Validate tiny, exact-part, non-multiple and mixed-size writes with full readback.
- Measure copy/allocation work, runtime entries and retained memory, then compare real-S3 file uploads under unchanged conditions; request timing should separate dispatch and completion delays from copying.
Related: #8246 intentionally covers AsyncFile and leaves the synchronous flush contract separate. #8159 covers releasing the GIL during blocking File I/O; that work is complementary and should remain separately attributable.
Feature Description
Reduce the synchronous Python File writer's adapter buffering and repeated runtime entry while preserving the existing file-like API.
Problem and proposed direction
At
4b132b0d33c8a5b84ff83f6fcfbab264860c5b3d, the path isFile.write -> StdWriter.write_all -> StdWriter.write -> Handle.block_on(FuturesAsyncWriter.write). The FuturesAsyncWriter stages borrowed input through a 256 KiB FlexBuf, even when Python supplies an immutable 8 MiB bytes object. A large File.write can therefore require repeated adapter writes and runtime entries before the storage writer handles multipart chunking.The binding could retain the core blocking Writer and pass an owned Buffer retaining the immutable Python bytes.
py_bytes_like_into_bufferalready provides an owner-preserving conversion for whole-object writes. This is a proposal to remove intermediate work, not a claim of measured benefit from an unimplemented change.Python File, StdWriter, FuturesAsyncWriter.
Observed workload
Measured OpenDAL commit:
4b132b0d33c8a5b84ff83f6fcfbab264860c5b3d; harness commit:c4c203859ed72d9a0ffac7438486a2d205649102. Real S3 in us-east-2, c7i.8xlarge with a 4-vCPU/1-GiB client limit, six formal repeats. Boto3 1.43.88 uses native APIs, without a transfer manager. Values below are medians, not significance claims. The harness and raw archive are currently private, so these numbers are descriptive evidence, not a publicly reproducible benchmark package.Uploading a 59,158,238-byte local file uses 8 MiB producer blocks, 8 MiB multipart parts and CRC32C on both sides, followed by independently verified readback. At C4, median elapsed time was 0.419 s for OpenDAL File and 0.370 s for Boto3, while process CPU was lower for OpenDAL (0.075 s versus 0.124 s). These measurements motivate investigation but do not isolate the adapter as the cause of the elapsed-time gap.
Acceptance criteria
Related: #8246 intentionally covers AsyncFile and leaves the synchronous flush contract separate. #8159 covers releasing the GIL during blocking File I/O; that work is complementary and should remain separately attributable.