Add native response file support by bkhouri · Pull Request #909 · apple/swift-argument-parser

bkhouri · 2026-05-27T05:11:47Z

Add native response file support in Swift Argument Parser for all ParsableCommand and AsyncParsableCommand commands. When at least one @file reference is present in the input, parser error messages now include a multi-line at <file>:<line> / included from <file>:<line> block that pinpoints the offending argument and its full include chain.

Fixes: #846
Relates to #41

Checklist

I've added at least one test that validates that my change is working, if appropriate
I've followed the code style of the rest of the project
I've read the Contribution Guidelines
I've updated the documentation if necessary

rauhul · 2026-05-28T20:31:00Z

IMO this sort of feature will need to come with some sort of source location feature too

bkhouri · 2026-07-10T19:15:36Z

IMO this sort of feature will need to come with some sort of source location feature too

Thanks for this feedback. A source location feature has been added.

bkhouri · 2026-07-10T19:24:59Z

marking as Draft to prevent accidentally merging before it's ready.

bkhouri · 2026-07-21T22:19:50Z

#931 augments the source location information.

allevato · 2026-07-22T00:25:48Z

Apologies if I missed it, but can you add a test for quoted empty string behavior in response files?

For example, these files:

-foo
''
-bar

and

-foo
""
-bar

should both expand to three arguments: ["-foo", "", "-bar"].

This was on my mind because a few months ago I fixed a bug in LLVM's response file parser that was incorrectly dropping empty arguments (instead of preserving them as gcc would): llvm/llvm-project#187566

In fact, some of the other test cases I added might also serve as good tests of quotation handling for this implementation, too.

bkhouri · 2026-07-22T18:32:07Z

Apologies if I missed it, but can you add a test for quoted empty string behavior in response files?

Thanks for the feedback @allevato . I added more test coverage around empty quoted string s, and some with a new line character (\n) in a quoted string.

natecook1000

Thanks, @bkhouri! Initial feedback here – I haven't reviewed the tests yet, will do that next.

natecook1000 · 2026-08-21T13:43:42Z

+    // Ordering (least → greatest):
+    //   .argumentIndex(a) < .argumentIndex(b) when a < b
+    //   .argumentIndex(_) < .responseFile(_,_) < .defaultValue
+    // Two `.responseFile` values are ordered by lexicographically
+    // comparing their flattened chain step arrays.


IIUC, the arguments in a response file are expanded "in place", such that they take the position of the @file command-line argument that includes them. Are there any issues with that in-place expansion leading to an actual ordering of arguments that's different from this ordering, where all the arguments from expanded response files are positioned after all command-line arguments?

I don't believe there are ordering issues. Every expanded argument (CLI or response-file origin) is assigned a sequential post-expansion InputIndex in SplitArguments.swift, and all origins use .argumentIndex(SplitArguments.Index). Natural < ordering on .argumentIndex therefore reflects the true in-place expansion order — a response-file arg at post-expansion index 5 sorts between CLI arg at index 4 and CLI arg at index 6. Chain provenance for user-facing error messages lives separately in FormattingContext.responseFileChains keyed by the same post-expansion InputIndex.

There are also tests that help verify the ordering:

in ResponseFileExpanderTests.swift:112-238 that validate the ordering within the response files processing is as expected.

in ResponseFileEndToEndTests.swift, responseFileWithMixedArgsLastWinsCLI:122 parses a sample CLI [@file, --count 100], where the response file sets --count 2. The test asserts count == 100, which comes after the response file on the command line.

in ResponseFileEndToEndTests.swift, responseFileWithMixedArgsLastWinsResponseFile:141 is the same as the previous, but reversed the order on the command line.

I added additional automated test that will validate the CLI argument expansion is as expected.

natecook1000 · 2026-08-21T14:14:47Z

+  fileprivate func resolveFileURL(
+    _ fileName: String,
+    relativeTo parentURL: URL? = nil
+  ) -> URL {


I think this method is duplicating an existing API: https://developer.apple.com/documentation/foundation/url/init(filepath:directoryhint:relativeto:)

I can't recall why I didn't use the existing API.

natecook1000 · 2026-08-21T14:15:38Z

+  /// On POSIX, that means it starts with `/`. On Windows, that includes
+  /// drive-rooted paths (`C:\...` / `C:/...`), UNC paths (`\\server\...`),
+  /// and drive-relative-absolute paths (starting with `\` or `/`).
+  fileprivate static func isAbsolutePath(_ path: String) -> Bool {


Let's try hard not to rewrite our own version of this logic – is this necessary if resolveFileURL is simplified?

Add native response file support in Swift Argument Parser for all `ParsableCommand` and `AsyncParsableCommand` commands. When at least one `@file` reference is present in the input, parser error messages now include a multi-line `at <file>:<line>` / `included from <file>:<line>` block that pinpoints the offending argument and its full include chain. Fixes #846 Relates to #41

Response files make it easy to produce argument lists far larger than the OS allows on the command line, which exposed two quadratic-time bugs in the parser: - `SplitArguments.removeAll(in:)` iterated `InputOrigin`'s underlying `Set` in arbitrary hash order. `remove(at:)` is only O(1) when positions arrive in ascending order (it scans forward from `firstUnused`); out-of-order removal degraded every call to an O(n) scan. Iterate the existing sorted `elements` accessor instead. - `hasTerminatorBetween`, called once per option to rule out an intervening `--`, scanned every remaining argument even when no terminator existed. Since `elements` is sorted by index, stop scanning once the target index is reached. Also harden `ParsedValues.update` for array values: boxing the array in `Any` inside a dictionary kept a second live reference to the buffer during `append`, defeating copy-on-write and forcing a full copy per element. Added scaling tests (2,000 vs. 20,000 arguments) for repeated options and positional arguments, via both a response file and direct argv, that fail if parsing time grows super-linearly. Before this change, 20,000 repeated options took over five minutes; after, the full suite of 391 tests runs in ~1 second.

natecook1000

Thanks, @bkhouri! Some notes below – let me know if you have any questions…

natecook1000 · 2026-10-16T16:12:09Z

+- **Nested quotes** — a double-quoted string may contain unescaped single
+  quotes, and a single-quoted string may contain unescaped double quotes.
+  Only the outer quote style needs to be escaped or switched.


This bullet isn't like the other two – could it just be a short paragraph underneath the two bullets?

natecook1000 · 2026-10-16T16:12:48Z

+Whitespace inside a quoted value (of either style) is preserved and treated
+as part of the value — it does not split the argument. Outside of quotes,
+runs of spaces and tabs separate arguments as usual.


What about line breaks?

natecook1000 · 2026-10-16T16:13:51Z

+##### `@file` inside a quoted argument
+
+The response-file prefix is only interpreted when it starts an *unquoted*
+token. A quoted string that happens to begin with `@` is passed through as a
+literal value (this differs from the `@@`-escape mechanism described in
+[Literal At Signs](#literal-at-signs), which only applies to unquoted
+tokens):
+
+```
+--username "@admin"    # value is "@admin"; no file is opened
+--pattern  '@daily'    # value is "@daily"; single quotes protect it too
+```


This should reference the user's chosen response file prefix, rather than (or in addition to) "@" specifically.

Actually, this doesn't seem worthy of its own section. Maybe a sentence in the Nested Response Files section?

natecook1000 · 2026-10-16T16:15:57Z

+#### Equals Format
+
+Arguments can use the equals format:
+
+```
+--input=input.txt
+--output=output.txt
+--count=42
+```


Let's just include this "equals" format in the examples above.

natecook1000 · 2026-10-16T16:17:30Z

+Error: Maximum nesting depth (32) exceeded for response files
+```
+
+### Examples


I think we can scrap all of these examples.

natecook1000 · 2026-10-22T20:34:34Z

+//
+// The "gate inactive" tests act as a regression guard — they should pass
+// today and continue to pass after the feature lands. The "gate active"
+// tests are expected to FAIL until the feature is implemented.


Requesting these kinds of comments — about process rather than documenting what the file describes — be removed.

Suggested change

//

// The "gate inactive" tests act as a regression guard — they should pass

// today and continue to pass after the feature lands. The "gate active"

// tests are expected to FAIL until the feature is implemented.

natecook1000 · 2026-10-22T20:37:58Z

+// MARK: - Configuration Options
+
+extension ResponseFileEndToEndTests {
+  // These tests will verify that response file support can be configured
+  // The actual configuration mechanism will be defined during implementation
+
+  @Test func disableResponseFileSupport() throws {
+    // Test that response file support can be disabled per command
+    // This would be implemented as a configuration option
+    // For now, this is a placeholder for the feature
+
+    // When disabled, @file should be treated as a literal argument
+    // Implementation details TBD
+  }
+
+  @Test func responseFileSearchPaths() throws {
+    // Test that response files can be searched in multiple directories
+    // Implementation details TBD
+  }
+}
+


natecook1000 · 2026-10-22T20:43:29Z

+    @Test() func repeatedOptionParsingScalesLinearlyWithResponseFileSize()
+      async throws
+    {
+      func content(count: Int) -> String {
+        (0..<count).map { "--tag\nvalue\($0)\n" }.joined()
+      }
+
+      let small = try await responseFileParseDuration(
+        RepeatedOptionCommand.self, content: content(count: 2_000),
+        fileName: "repeated-option-small.txt")
+      let large = try await responseFileParseDuration(
+        RepeatedOptionCommand.self, content: content(count: 20_000),
+        fileName: "repeated-option-large.txt")
+
+      #expect(
+        large < small * 30,
+        """
+        Parsing time scaled super-linearly with input size: \(small) for \
+        2,000 repeated options vs \(large) for 20,000. A 10x increase in \
+        input should cost roughly 10x the time, not ~100x, which would \
+        indicate an O(n^2) hot path.
+        """
+      )
+    }
+
+    @Test() func positionalArgumentParsingScalesLinearlyWithResponseFileSize()
+      async throws
+    {
+      func content(count: Int) -> String {
+        (0..<count).map { "file\($0).txt\n" }.joined()
+      }
+
+      let small = try await responseFileParseDuration(
+        PositionalCommand.self, content: content(count: 2_000),
+        fileName: "positional-small.txt")
+      let large = try await responseFileParseDuration(
+        PositionalCommand.self, content: content(count: 20_000),
+        fileName: "positional-large.txt")
+
+      #expect(
+        large < small * 30,
+        """
+        Parsing time scaled super-linearly with input size: \(small) for \
+        2,000 positional arguments vs \(large) for 20,000. A 10x increase \
+        in input should cost roughly 10x the time, not ~100x, which would \
+        indicate an O(n^2) hot path.
+        """
+      )
+    }
+  }


I think you're covering this in the ...+Performance.swift file above – let's just run these perf tests once.

natecook1000 · 2026-10-22T20:50:03Z

+        }
+        result.append(char)
+      case "'":
+        if !inDoubleQuotes {


Does \ escaping only work inside double quotes? I would have thought that

abc\"def

would include a bare non-quoting double quote: abc"def.

natecook1000 · 2026-10-22T21:04:07Z

    helpNames: NameSpecification? = nil,
-    aliases: [String] = []
+    aliases: [String] = [],
+    responseFilePrefix: Character? = nil


I think a more specific API would be more clear about the configuration, and also do a better job of pushing people toward the convention. How about:

enum ResponseFile { case disabled case enabled(prefix: Character) static var enabled: { .enabled(prefix: "@") } var prefixCharacter: Character? { ... } var isEnabled: Bool { .... } }

Then we make .disabled the default, and just passing responseFile: .enabled gives the correct "@" prefix, without people needing to look it up.

bkhouri force-pushed the t/main/gh846_native_response_file_support branch from 72be887 to 2aaaca8 Compare May 27, 2026 06:02

bkhouri self-assigned this May 27, 2026

bkhouri mentioned this pull request May 28, 2026

appendingPathComponent() API behaves incorrectly on Windows platform swiftlang/swift-foundation#2005

Closed

cmcgee1024 reviewed May 28, 2026

View reviewed changes

Comment thread Sources/ArgumentParser/Documentation.docc/Articles/ResponseFiles.md Outdated

cmcgee1024 reviewed May 28, 2026

View reviewed changes

Comment thread Sources/ArgumentParser/Documentation.docc/Articles/ResponseFiles.md

cmcgee1024 reviewed May 28, 2026

View reviewed changes

Comment thread Sources/ArgumentParser/Documentation.docc/Articles/ResponseFiles.md

cmcgee1024 reviewed May 28, 2026

View reviewed changes

Comment thread Sources/ArgumentParser/Documentation.docc/Articles/ResponseFiles.md

bkhouri force-pushed the t/main/gh846_native_response_file_support branch 12 times, most recently from c4a089d to 5cf5af6 Compare July 10, 2026 19:09

bkhouri marked this pull request as ready for review July 10, 2026 19:23

bkhouri requested review from natecook1000 and rauhul as code owners July 10, 2026 19:23

bkhouri marked this pull request as draft July 10, 2026 19:24

bkhouri commented Jul 14, 2026

View reviewed changes

Comment thread Sources/ArgumentParser/Documentation.docc/Articles/ExperimentalFeatures.md Outdated

Comment thread Sources/ArgumentParser/Documentation.docc/Articles/ResponseFiles.md

Comment thread Sources/ArgumentParser/Documentation.docc/Articles/ExperimentalFeatures.md Outdated

bkhouri force-pushed the t/main/gh846_native_response_file_support branch 3 times, most recently from 368e955 to cd020b9 Compare July 14, 2026 13:53

bkhouri marked this pull request as ready for review July 21, 2026 22:18

bkhouri marked this pull request as draft July 21, 2026 22:36

bkhouri force-pushed the t/main/gh846_native_response_file_support branch 3 times, most recently from f297509 to 8879fe4 Compare July 21, 2026 23:56

bkhouri force-pushed the t/main/gh846_native_response_file_support branch 3 times, most recently from 5dff61a to 2065f86 Compare July 22, 2026 18:29

bkhouri marked this pull request as ready for review July 24, 2026 21:18

bkhouri force-pushed the t/main/gh846_native_response_file_support branch 2 times, most recently from 0a1ae06 to 2e3f6c3 Compare August 19, 2026 19:07

natecook1000 reviewed Aug 21, 2026

View reviewed changes

bkhouri force-pushed the t/main/gh846_native_response_file_support branch 5 times, most recently from cd1ab10 to 07f2035 Compare August 21, 2026 21:07

bkhouri force-pushed the t/main/gh846_native_response_file_support branch 2 times, most recently from d69a318 to 2b09395 Compare September 1, 2026 02:29

bkhouri added 2 commits October 16, 2026 10:46

Support new line in a quoted string within a response file

b03e5b7

bkhouri force-pushed the t/main/gh846_native_response_file_support branch from 2b09395 to 83c9a14 Compare October 16, 2026 14:46

Addressing PR comments

c11dedb

bkhouri force-pushed the t/main/gh846_native_response_file_support branch from 83c9a14 to c11dedb Compare October 16, 2026 14:50

bkhouri force-pushed the t/main/gh846_native_response_file_support branch from 5e79423 to 1bcde41 Compare October 20, 2026 12:38

natecook1000 reviewed Oct 22, 2026

View reviewed changes

Conversation

bkhouri commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rauhul commented May 28, 2026

Uh oh!

bkhouri commented Jul 10, 2026

Uh oh!

bkhouri commented Jul 10, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bkhouri commented Jul 21, 2026

Uh oh!

allevato commented Jul 22, 2026

Uh oh!

bkhouri commented Jul 22, 2026

Uh oh!

natecook1000 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

natecook1000 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

bkhouri commented May 27, 2026 •

edited

Loading