Skip to content
This repository was archived by the owner on Oct 19, 2026. It is now read-only.

Fix Windows Unicode file paths in native I/O - #333

Merged
heitzmann merged 1 commit into
heitzmann:mainfrom
CAOShurong:fix-windows-unicode-paths
Oct 15, 2026
Merged

heitzmann merged 1 commit into
heitzmann:mainfrom
CAOShurong:fix-windows-unicode-paths

Conversation

@CAOShurong

Copy link
Copy Markdown
Contributor

Fixes #292.

On Windows, Python's filesystem converter passes UTF-8 filenames to the native
code, but narrow fopen interprets them using the ANSI codepage. This can write
to a different filename, fail for an existing Unicode parent directory, or
overwrite an unrelated ANSI-decoded filename.

Route the native file-open calls through one helper that converts valid UTF-8
paths to UTF-16 and uses _wfopen on Windows. Other platforms still call
fopen unchanged. Stream modes, serialization and Python converters are
unchanged; invalid UTF-8 is rejected rather than replaced.

The regression tests cover GDS/OAS/SVG output, streaming and raw-cell reads,
metadata, existing files moved independently with Python, str/Path/bytes
arguments, supplementary/decomposed characters and failure paths.

Validation on Windows 11, CPython 3.13.1, ANSI codepage 936, MSVC:

  • Built and non-editably installed unmodified main and the patched wheels using
    the existing Windows CI configuration. The new tests fail 149 cases on main
    (52 pass); the patch passes all 201 cases repeatedly. All 91 original test
    outcomes are unchanged; the full suite passes 292 tests.
  • Ran python -m pytest <checkout>/tests/file_io_test.py -q and
    python -m pytest <checkout>/tests -q from outside the source package,
    with --basetemp=<fresh-directory> after the default local temporary directory
    produced fixture permission errors.
  • Built the wheel/sdist, checked source/package/installed bytes, and ran native
    C++ plus installed Python GDS/OAS/SVG success and error workflows.
  • Black and changed-line C++ formatting checks pass. Whole-file clang-format
    still reports the same five pre-existing diagnostics on unmodified main.

Linux/macOS and other Python versions have not been run locally. An existing
Windows gds_timestamp(path, new_timestamp) failure also reproduces on
unmodified main and is unchanged; this patch covers timestamp reads and file
opening, not that separate timestamp-update issue.

@heitzmann
heitzmann merged commit c8d6bf8 into heitzmann:main Oct 15, 2026
6 checks passed
@heitzmann

Copy link
Copy Markdown
Owner

Thanks @CAOShurong

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Special characters/Umlaut in write_gds filename

2 participants