Compile whole VU1 programs and run sound IRX modules natively (DQ8 fork)

ps2xAnalyzer/CMakeLists.txt

@@ -27,8 +27,8 @@ add_library(ps2_analyzer_lib STATIC ${PS2ANALYZER_LIB_SOURCES})
 
 target_include_directories(ps2_analyzer_lib PUBLIC
     ${CMAKE_CURRENT_SOURCE_DIR}/include
-    ${CMAKE_SOURCE_DIR}/ps2xRecomp/include
-    ${CMAKE_SOURCE_DIR}/ps2xRuntime/include
+    ${CMAKE_CURRENT_SOURCE_DIR}/../ps2xRecomp/include
+    ${CMAKE_CURRENT_SOURCE_DIR}/../ps2xRuntime/include
 )
 
 target_link_libraries(ps2_analyzer_lib PUBLIC
@@ -50,7 +50,7 @@ install(TARGETS ps2_analyzer ps2_analyzer_lib
     ARCHIVE DESTINATION lib
 )
 
-include("${CMAKE_SOURCE_DIR}/ps2xRuntime/cmake/ReleaseMode.cmake")
+include("${CMAKE_CURRENT_SOURCE_DIR}/../ps2xRuntime/cmake/ReleaseMode.cmake")
 
 if(CMAKE_BUILD_TYPE STREQUAL "Release" OR CMAKE_BUILD_TYPE STREQUAL "RelWithDebInfo")
     EnableFastReleaseMode(ps2_analyzer_lib)

ps2xIOP/CMakeLists.txt

@@ -16,6 +16,12 @@ add_library(ps2_iop STATIC
     src/modules/clfile.cpp
     src/modules/sound_update_stub.cpp
     src/modules/sdrdrv.cpp
+    src/native_iop.cpp
+    src/lle/cpu.cpp
+    src/lle/irx.cpp
+    src/lle/iop.cpp
+    src/lle/kernel.cpp
+    src/lle/spu2.cpp
 )
 
 target_compile_features(ps2_iop PUBLIC cxx_std_20)

ps2xIOP/README.md

@@ -4,9 +4,11 @@
 `ps2xRuntime`. It implements the behavior that games expect from IOP services
 exposed through SIF RPC and DMA.
 
-This subsystem does not emulate the IOP's R3000A CPU and does not load or
-execute IRX binaries. Its scope is the RPC/DMA behavior needed by recompiled
-games.
+The services below do not emulate the IOP's R3000A CPU and do not load or
+execute IRX binaries. Their scope is the RPC/DMA behavior needed by recompiled
+games. A runtime can additionally run a game's own modules on a small emulated
+IOP, which is how sound drivers are handled; see
+[Native IRX modules](#native-irx-modules).
 
 > [!IMPORTANT]
 > `ps2_iop`/`ps2x::iop` is a C++20 static library linked into the runtime.
@@ -121,6 +123,69 @@ services, and every active service receives each SIF transfer notification.
 Implementations must filter the relevant SID/function or transfer
 kind/phase/address range themselves.
 
+## Native IRX modules
+
+Some drivers are easier to run than to rewrite. A sound driver is the usual
+case: its RPC protocol is the only part the EE sees, but what the game hears
+comes from years of tuning in how it drives the SPU2. `NativeIop`
+([`native_iop.h`](include/ps2x/iop/native_iop.h)) loads a game's own IRX files
+onto an emulated IOP and lets their RPC servers answer the game directly.
+
+The emulator under [`src/lle/`](src/lle) is only as large as sound drivers
+need:
+
+| File | Contents |
+| --- | --- |
+| `cpu.*` | R3000A interpreter, with load delay slots and no caches or MMU |
+| `irx.*` | IRX parsing, relocation and import stub discovery |
+| `kernel.cpp` | High-level kernel: threads, semaphores, event flags, mailboxes, alarms, hardware timers, interrupts, the IOP heap, SIF DMA and RPC, and the sysclib calls the drivers use |
+| `spu2.*` | Both SPU2 cores: 48 ADPCM voices with ADSR, noise, pitch modulation, reverb, AutoDMA input and 2 MiB of sound memory |
+| `iop.*` | The pieces together: memory map, DMA channels 4 and 7, interrupts and the clock |
+
+An import the kernel does not provide returns zero, and an instruction the CPU
+does not implement stops its thread with a log message. Both mean a driver
+needs more than has been written so far.
+
+### Timing
+
+IOP time is counted at 36.864 MHz and advances only as the SPU2 produces
+samples, 768 cycles for each 48 kHz stereo pair. The host audio callback calls
+`render()`, so the IOP runs in step with the audio device rather than with the
+EE. Work that answers an RPC runs without moving the clock, until every thread
+is waiting again, and so do interrupt, alarm and timer handlers. Charging that
+work to the clock let heavy RPC traffic push the drivers' timers ahead of the
+audio, and music played fast.
+
+With no audio device, or while it is paused, an RPC that waits on IOP time
+steps the IOP itself and discards the samples. A call that still cannot finish
+gives up after two seconds and logs a warning.
+
+### Hooking it up
+
+The runtime names the modules to run natively before the game starts. The
+native path then takes over in three places:
+
+- `sceSifLoadModule` of a named module also loads it on the emulated IOP. The
+  high-level module tracker still hands out the id the game sees.
+- `IopSubsystem::handleRpc` gives an RPC to a server a native module
+  registered before any high-level service with the same SID.
+- IOP heap allocation and EE-to-IOP `sceSifSetDma` use the emulated IOP's
+  memory, since native modules read what the game sends them from there.
+
+IOP modules that bind back to an EE server, like SDRDRV's callback thread,
+wait until the EE has registered it (`IopHost::hasGuestRpcServer`). Answering
+the bind early left the thread spinning and starved the rest of the driver.
+
+On the runtime side, [`ps2_native_iop.h`](../ps2xRuntime/include/runtime/ps2_native_iop.h)
+takes the module list, opens a 48 kHz stereo stream once the first module
+loads, and sets the volume. Volume zero keeps the stream running: games wait
+on their sound drivers, so the IOP has to keep going even when nothing is
+heard. `PS2X_AUDIO_DUMP=path` keeps a raw copy of everything played
+(48 kHz, stereo, signed 16-bit) for comparing against a reference offline.
+
+The SPU2 input path and the DMA registers have tests in
+[`ps2_native_iop_tests.cpp`](../ps2xTest/src/ps2_native_iop_tests.cpp).
+
 ## Linking the static library 
 
 ```cmake

ps2xIOP/include/ps2x/iop/iop_host.h

@@ -82,6 +82,13 @@ namespace ps2x::iop
         virtual int32_t memoryCard(const MemoryCardRequest &request) = 0;
 
         virtual bool hasGuestFunction(uint32_t address) const = 0;
+        // Whether the EE has registered an RPC server under this id, which is
+        // what an IOP module binding to it waits for.
+        virtual bool hasGuestRpcServer(uint32_t sid) const
+        {
+            (void)sid;
+            return false;
+        }
         virtual bool invokeGuestFunction(uint64_t callToken,
                                          uint32_t address,
                                          uint32_t a0,

ps2xIOP/include/ps2x/iop/iop_subsystem.h

@@ -1,6 +1,7 @@
 #pragma once
 
 #include "ps2x/iop/iop_host.h"
+#include "ps2x/iop/native_iop.h"
 #include "ps2x/iop/iop_types.h"
 
 #include <filesystem>
@@ -33,6 +34,9 @@ namespace ps2x::iop
 
         [[nodiscard]] DebugSnapshot debugSnapshot() const;
 
+        // Original IRX modules running on an emulated IOP; see NativeIop.
+        [[nodiscard]] NativeIop &native();
+
     private:
         class Impl;
         std::unique_ptr<Impl> m_impl;

ps2xIOP/include/ps2x/iop/native_iop.h

@@ -0,0 +1,53 @@
+#pragma once
+
+#include "ps2x/iop/iop_host.h"
+
+#include <cstdint>
+#include <memory>
+#include <string>
+#include <string_view>
+#include <vector>
+
+namespace ps2x::iop
+{
+    // Runs a game's own IRX modules, sound drivers mostly, on an emulated IOP
+    // with an SPU2. The audio callback calls render() while the EE thread loads
+    // modules and makes calls, so every entry point takes the same lock.
+    class NativeIop
+    {
+    public:
+        explicit NativeIop(IopHost &host);
+        ~NativeIop();
+
+        NativeIop(const NativeIop &) = delete;
+        NativeIop &operator=(const NativeIop &) = delete;
+
+        // Module file names (for example "LIBSD.IRX") that run natively.
+        // Once any are set, IOP memory handed to the EE is this IOP's.
+        void setModules(std::vector<std::string> names);
+        bool enabled() const;
+        bool claims(std::string_view guestPath) const;
+        // Load a claimed module; arguments are NUL-separated as sceSifLoadModule passes them.
+        int32_t load(std::string_view guestPath, std::string_view arguments);
+
+        // A server a native module registered, with its record and receive buffer.
+        bool server(uint32_t sid, uint32_t &record, uint32_t &buffer) const;
+        // Blocking SIF RPC. The reply is written to receiveAddress in EE memory.
+        bool call(uint32_t sid, uint32_t function, uint32_t sendAddress, uint32_t sendSize,
+                  uint32_t receiveAddress, uint32_t receiveSize);
+
+        // IOP memory as the EE sees it through SIF DMA and the IOP heap calls.
+        bool write(uint32_t address, const void *data, uint32_t size);
+        bool read(uint32_t address, void *data, uint32_t size);
+        uint32_t allocate(uint32_t size);
+        void release(uint32_t address);
+
+        // Fill interleaved 48 kHz stereo frames, running the IOP in step.
+        void render(int16_t *frames, uint32_t count);
+        bool active() const;
+
+    private:
+        class Impl;
+        std::unique_ptr<Impl> m_impl;
+    };
+}

ps2xIOP/src/iop_subsystem.cpp

@@ -68,7 +68,8 @@ namespace ps2x::iop
     {
     public:
         explicit Impl(IopHost &hostRef)
-            : host(hostRef), pluginCatalog(hostRef), coreServices(detail::createCoreServices(hostRef)), profiles(detail::createBuiltinProfiles())
+            : host(hostRef), pluginCatalog(hostRef), coreServices(detail::createCoreServices(hostRef)), profiles(detail::createBuiltinProfiles()),
+              native(hostRef)
         {
             rebuildRoutes();
         }
@@ -118,6 +119,7 @@ namespace ps2x::iop
         std::string activeProvider;
         std::string lastError;
         bool routesValid = true;
+        NativeIop native;
     };
 
     IopSubsystem::IopSubsystem(IopHost &host)
@@ -279,6 +281,18 @@ namespace ps2x::iop
 
     RpcResult IopSubsystem::handleRpc(const RpcRequest &request)
     {
+        // A server registered by a native module replaces any high-level one.
+        uint32_t record = 0, buffer = 0;
+        if (m_impl->native.server(request.sid, record, buffer))
+        {
+            RpcResult result;
+            result.handled = m_impl->native.call(request.sid, request.function, request.send.address,
+                                                 request.send.size, request.receive.address,
+                                                 request.receive.size);
+            result.resultAddress = request.receive.address;
+            result.serverDispatchPolicy = ServerDispatchPolicy::Suppress;
+            return result;
+        }
         const auto it = m_impl->routes.find(request.sid);
         if (it == m_impl->routes.end() || !it->second)
         {
@@ -305,6 +319,11 @@ namespace ps2x::iop
         }
     }
 
+    NativeIop &IopSubsystem::native()
+    {
+        return m_impl->native;
+    }
+
     DebugSnapshot IopSubsystem::debugSnapshot() const
     {
         DebugSnapshot snapshot;

ps2xIOP/src/lle/bus.h

@@ -0,0 +1,70 @@
+#pragma once
+
+#include <cstdint>
+#include <cstring>
+#include <vector>
+
+namespace ps2x::iop::lle
+{
+    // Hardware registers behind 0x1F801000 and the SPU2 window.
+    class IoDevice
+    {
+    public:
+        virtual ~IoDevice() = default;
+        virtual uint32_t ioRead(uint32_t physical, unsigned bytes) = 0;
+        virtual void ioWrite(uint32_t physical, uint32_t value, unsigned bytes) = 0;
+    };
+
+    // The IOP's physical address space: 2 MiB of RAM (mirrored through the
+    // first 8 MiB), the 1 KiB scratchpad and memory-mapped hardware.
+    class Bus
+    {
+    public:
+        static constexpr uint32_t kRamBytes = 2u << 20;
+
+        explicit Bus(IoDevice &io) : m_io(io), m_ram(kRamBytes, 0u), m_scratchpad(1024u, 0u) {}
+
+        uint8_t *ram() { return m_ram.data(); }
+        const uint8_t *ram() const { return m_ram.data(); }
+
+        uint32_t fetch(uint32_t address) { return read32(address); }
+
+        uint8_t read8(uint32_t address) { return static_cast<uint8_t>(read<1>(address)); }
+        uint16_t read16(uint32_t address) { return static_cast<uint16_t>(read<2>(address)); }
+        uint32_t read32(uint32_t address) { return read<4>(address); }
+        void write8(uint32_t address, uint8_t value) { write<1>(address, value); }
+        void write16(uint32_t address, uint16_t value) { write<2>(address, value); }
+        void write32(uint32_t address, uint32_t value) { write<4>(address, value); }
+
+    private:
+        template <unsigned Bytes>
+        uint32_t read(uint32_t address)
+        {
+            const uint32_t physical = address & 0x1FFFFFFFu;
+            uint32_t value = 0;
+            if (physical < 0x00800000u)
+                std::memcpy(&value, &m_ram[physical & (kRamBytes - Bytes)], Bytes);
+            else if ((physical & ~0x3FFu) == 0x1F800000u)
+                std::memcpy(&value, &m_scratchpad[physical & (0x400u - Bytes)], Bytes);
+            else
+                value = m_io.ioRead(physical, Bytes);
+            return value;
+        }
+
+        template <unsigned Bytes>
+        void write(uint32_t address, uint32_t value)
+        {
+            const uint32_t physical = address & 0x1FFFFFFFu;
+            if (physical < 0x00800000u)
+                std::memcpy(&m_ram[physical & (kRamBytes - Bytes)], &value, Bytes);
+            else if ((physical & ~0x3FFu) == 0x1F800000u)
+                std::memcpy(&m_scratchpad[physical & (0x400u - Bytes)], &value, Bytes);
+            else
+                m_io.ioWrite(physical, value, Bytes);
+        }
+
+        IoDevice &m_io;
+        std::vector<uint8_t> m_ram;
+        std::vector<uint8_t> m_scratchpad;
+    };
+}