[Bug]: Linux (COSMIC Wayland): capture fails with "error alloc buffers: Invalid argument", then records black frames #1445

Environment

• OpenScreen 2.1.2 (.deb, amd64); 2.2.0-rc.2 contains no related change
• Pop!_OS 24.04, COSMIC on Wayland
• cosmic-comp 0.11792508925, xdg-desktop-portal-cosmic 1.11.01792525517
• PipeWire 1.6.8, Mesa 26.1.6, kernel 7.2.6
• GPU: AMD iGPU 1002:1114 (single GPU, /dev/dri/renderD128)

What happens

Starting a recording fails immediately with:

OpenScreen could not start screen capture: error alloc buffers: Invalid argument

The shared-memory retry from #1069 runs, but it fails the same way.

Other apps can capture the screen on the same session. Kooha (GStreamer pipewiresrc) records fine.

Log (OPENSCREEN_PIPEWIRE_DEBUG=1)

[osc-dmabuf] negotiated 1920x1080 uses_dmabuf=1 modifier=0x2000000084abb04
{"code":"stream-state","error":"error alloc buffers: Invalid argument","state":"error"}
capture-retry-shared-memory: PipeWire buffer allocation failed with DMA-BUF; retrying screen capture with shared-memory buffers.
[osc-dmabuf] negotiated 1920x1080 uses_dmabuf=1 modifier=0x0
{"code":"stream-state","error":"error alloc buffers: Invalid argument","state":"error"}
pipewire-capture-failed: OpenScreen could not start screen capture: error alloc buffers: Invalid argument

PipeWire's own log shows the Buffers params that do not intersect:

params Spa:Enum:ParamId:Buffers: 0:0 Invalid argument (input param)
  buffers: Range 4 [2..16], blocks: 1
  dataType: Flags 8                 <- consumer (OpenScreen): DmaBuf only
params Spa:Enum:ParamId:Buffers: 1:0 Invalid argument (output param)
  dataType: Flags 4                 <- producer (xdg-desktop-portal-cosmic): MemFd only
  size: 8294400, stride: 7680, align: 16, blocks: 1, buffers: Range 4 [1..]

Root cause

There are two problems.

1. The buffer type is inferred from the modifier

osc_on_param_changed (csrc/pw_shim.c) sets
uses_dmabuf = (format.flags & SPA_VIDEO_FLAG_MODIFIER) != 0. In video mode it then requests dataType = 1 << SPA_DATA_DmaBuf only.

xdg-desktop-portal-cosmic does accept the EnumFormat object that carries a modifier, including tiled modifiers and LINEAR. But it only offers MemFd buffers for it. The intersection is therefore empty, and PipeWire returns -EINVAL.

The shared-memory retry negotiates the same modifier-carrying format again (now LINEAR, 0x0). So it requests DmaBuf-only again and fails for the same reason.

2. MemFd buffers on a modifier format are black

I patched the helper locally so that it advertises MemPtr|MemFd|DmaBuf. With that, buffers negotiate as MemFd and frames arrive (1118 frames in 18 s, no errors). However, every frame is solid black. This happens with both the tiled modifier and LINEAR, and with both the Vulkan and the software encoder.

COSMIC apparently fills memfd buffers only when the negotiated format has no modifier.

Workaround that works

I patched these two things in the binary helper:

1. Always request MemPtr|MemFd|DmaBuf in the Buffers param.
2. Pass only the shm EnumFormat object (no modifier) to pw_stream_connect, with n_params = 1 and no dmabuf-first swap.

With both, COSMIC negotiates uses_dmabuf=0 modifier=0x0. The recording is correct with the Vulkan encoder.

Suggested fix

• Derive the buffer dataType from what the producer offers rather than from SPA_VIDEO_FLAG_MODIFIER. At minimum, include MemFd/MemPtr alongside DmaBuf when the modifier is LINEAR.
• Make capture-retry-shared-memory actually drop the dmabuf EnumFormat object. Then the retry negotiates a format without a modifier, instead of the same modifier-carrying format again.
• Optionally, add an env switch to disable the dmabuf EnumFormat entirely, for example OPENSCREEN_PIPEWIRE_DISABLE_DMABUF. Today there is only OPENSCREEN_PIPEWIRE_FORCE_DMABUF.

Related: #646 / #1069 (same error message; the shm retry added there does not cover this case).