IssueTracker

LLVM-7.0 (and LLVM-10.0) try to map readonly data pages with READ+EXECUTE permissions unconditionally.
This makes certain tests, but not all of them, fail on Fuchsia.

I'm testing SwiftShader on Fuchsia in its default configuration (i.e. using LLVM 7.0 as the Reactor JIT).
This platform is much stricter than others when it comes to mapping memory pages as executables: by default this is simply refused, unless the page comes from a crypto-signed filesystem, totally preventing JITs from working.

There is an escape hatch, though, which is to call zx_vmo_replace_as_executable() when _allocating_ the VMO kernel object (that is used to back the pages). This must happen before mapping any page.

On Fuchsia, this is done in the implementation of allocateMemoryPages() which looks like:

void *allocateMemoryPages(size_t bytes, int permissions, bool need_exec) {
    ...
    // Create VMO object to hold the pages.
    zx_handle_t vmo;
    zx_vmo_create(length, 0, &vmo);

    // Allow EXECUTE mapping if |need_exec| is set.
    if (need_exec)
        zx_vmo_replace_as_executable(vmo, ZX_HANDLE_INVALID, &vmo);

    // Map pages into process' address space (a.k.a. VMAR).
    zx_vaddr_t reservation;
    zx_vmar_map(zx_vmar_root_self(), permissionsToZxVmOptions(permissions), 0, vmo, 0, length, &reservation);
}

Later, the protection of the pages can be changed through protectMemoryPages() which looks like:

void protectMemoryPages(void *memory, size_t bytes, int permissions) {
    ....
    zx_status_t status = zx_vmar_protect(
        zx_vmar_root_self(), permissionsToZxVmOptions(permissions),
        reinterpret_cast<zx_vaddr_t>(memory), bytes);

    ASSERT(status == ZX_OK);
}

Unfortunately, the ASSERT() above actually triggers for *some* of the Vulkan tests I run on Fuchsia (e.g. vk-triangle-test, which is a basic pair of vertex + fragment shaders that draw a gradient-filled triangle), but not all of them (e.g. the VulkanUnitTests run from a Chromium check out just run fine, and they exercise Compute operations).

Digging further, these functions are called from LLVMJIT.cpp::MemoryMapper::{allocateMappedMemory, protectMappedMemory} which are callbacks invoked at runtime by LLVM. The first function takes a |purpose| argument describing the intended purpose for the block and a |flags| mask or READ|WRITE|EXEC permissions.

Adding logs shows that when running vk-triangle-test, the following happens (in this specific order):

- allocateMappedMemory(purpose=Code, size=4096, flags=RW) is called to allocate a first page.
- allocateMappedMemory(purpose=ROData, size=4096, flags=RW) is called to allocate a second page.
- protectMemoryPages(flags=RE) is called to remap the first page as READ+EXECUTABLE. This succeeds.
- protectMemoryPages(flags=RE) is called to remap the second page as READ+EXECUTABLE.

The latter operation fails and triggers the assert, because the second page was created with purpose ROData and hence should not be executable.

Digging even deeper, this looks like a bug in LLVM 7.0, i.e. from third_party/llvm-7.0/llvm/lib/ExecutionEngine/SectionMemoryManager.cpp:

bool SectionMemoryManager::finalizeMemory(std::string *ErrMsg) {
  ...
  // Make code memory executable.
  ec = applyMemoryGroupPermissions(CodeMem,
                                   sys::Memory::MF_READ | sys::Memory::MF_EXEC);
  ...

  // Make read-only data memory read-only.
  ec = applyMemoryGroupPermissions(RODataMem,
                                   sys::Memory::MF_READ | sys::Memory::MF_EXEC);   <===== HERE
  ...
}

Changing the last applyMemoryGroupPermissions() call to only use sys::Memory::MF_READ seems to fix the issue.

This code path is also implemented this way in LLVM-10.0

------ WARN TRACES -----
../src/Reactor/LLVMJIT.cpp:255 WARNING: MemoryMapper::allocateMappedMemory(purpose=Code, length=4096, flags=RW)
../src/Reactor/ExecutableMemory.cpp:301 WARNING: allocateMemoryPages(size=4096, permissions=RW) -> [0x1463db46000, 0x1463db47000)
../src/Reactor/LLVMJIT.cpp:255 WARNING: MemoryMapper::allocateMappedMemory(purpose=ROData, length=4096, flags=RW)
../src/Reactor/ExecutableMemory.cpp:301 WARNING: allocateMemoryPages(size=4096, permissions=RW) -> [0x310212a8000, 0x310212a9000)
../src/Reactor/LLVMJIT.cpp:289 WARNING: MemoryMapper::protectMappedMemory(range=[0x1463db46000,0x1463db467f3), flags=RE)
../src/Reactor/ExecutableMemory.cpp:356 WARNING: protectMemoryPages(range=[0x1463db46000, 0x1463db47000), permissions=RE)
../src/Reactor/LLVMJIT.cpp:289 WARNING: MemoryMapper::protectMappedMemory(range=[0x310212a8000,0x310212a8020), flags=RE)
../src/Reactor/ExecutableMemory.cpp:356 WARNING: protectMemoryPages(range=[0x310212a8000, 0x310212a9000), permissions=RE)
../src/Reactor/ExecutableMemory.cpp:361 WARNING: ASSERT(status == ZX_OK)

../src/Reactor/LLVMJIT.cpp:289 WARNING: MemoryMapper::protectMappedMemory(range=[0x310212a8000,0x310212a8080), flags=RE)
../src/Reactor/ExecutableMemory.cpp:356 WARNING: protectMemoryPages(range=[0x310212a8000, 0x310212a9000), permissions=RE)
../src/Reactor/ExecutableMemory.cpp:361 WARNING: ASSERT(status == ZX_OK)

../src/Reactor/LLVMJIT.cpp:255 WARNING: MemoryMapper::allocateMappedMemory(purpose=Code, length=4096, flags=RW)
../src/Reactor/ExecutableMemory.cpp:301 WARNING: allocateMemoryPages(size=4096, permissions=RW) -> [0x369d28cd000, 0x369d28ce000)
../src/Reactor/LLVMJIT.cpp:255 WARNING: MemoryMapper::allocateMappedMemory(purpose=ROData, length=4096, flags=RW)
../src/Reactor/ExecutableMemory.cpp:301 WARNING: allocateMemoryPages(size=4096, permissions=RW) -> [0xb883710000, 0xb883711000)
../src/Reactor/LLVMJIT.cpp:289 WARNING: MemoryMapper::protectMappedMemory(range=[0x369d28cd000,0x369d28cd893), flags=RE)
../src/Reactor/ExecutableMemory.cpp:356 WARNING: protectMemoryPages(range=[0x369d28cd000, 0x369d28ce000), permissions=RE)
../src/Reactor/LLVMJIT.cpp:289 WARNING: MemoryMapper::protectMappedMemory(range=[0xb883710000,0xb88371000c), flags=RE)
../src/Reactor/ExecutableMemory.cpp:356 WARNING: protectMemoryPages(range=[0xb883710000, 0xb883711000), permissions=RE)
../src/Reactor/ExecutableMemory.cpp:361 WARNING: ASSERT(status == ZX_OK)

../src/Reactor/LLVMJIT.cpp:289 WARNING: MemoryMapper::protectMappedMemory(range=[0xb883710000,0xb883710030), flags=RE)
../src/Reactor/ExecutableMemory.cpp:356 WARNING: protectMemoryPages(range=[0xb883710000, 0xb883711000), permissions=RE)
../src/Reactor/ExecutableMemory.cpp:361 WARNING: ASSERT(status == ZX_OK)

../src/Reactor/LLVMJIT.cpp:255 WARNING: MemoryMapper::allocateMappedMemory(purpose=Code, length=4096, flags=RW)
../src/Reactor/ExecutableMemory.cpp:301 WARNING: allocateMemoryPages(size=4096, permissions=RW) -> [0x251f216a000, 0x251f216b000)
../src/Reactor/LLVMJIT.cpp:255 WARNING: MemoryMapper::allocateMappedMemory(purpose=ROData, length=4096, flags=RW)
../src/Reactor/ExecutableMemory.cpp:301 WARNING: allocateMemoryPages(size=4096, permissions=RW) -> [0x142efbd8000, 0x142efbd9000)
../src/Reactor/LLVMJIT.cpp:289 WARNING: MemoryMapper::protectMappedMemory(range=[0x251f216a000,0x251f216a91d), flags=RE)
../src/Reactor/ExecutableMemory.cpp:356 WARNING: protectMemoryPages(range=[0x251f216a000, 0x251f216b000), permissions=RE)
../src/Reactor/LLVMJIT.cpp:289 WARNING: MemoryMapper::protectMappedMemory(range=[0x142efbd8000,0x142efbd8050), flags=RE)
../src/Reactor/ExecutableMemory.cpp:356 WARNING: protectMemoryPages(range=[0x142efbd8000, 0x142efbd9000), permissions=RE)
../src/Reactor/ExecutableMemory.cpp:361 WARNING: ASSERT(status == ZX_OK)

../src/Reactor/LLVMJIT.cpp:289 WARNING: MemoryMapper::protectMappedMemory(range=[0x142efbd8000,0x142efbd805c), flags=RE)
../src/Reactor/ExecutableMemory.cpp:356 WARNING: protectMemoryPages(range=[0x142efbd8000, 0x142efbd9000), permissions=RE)
../src/Reactor/ExecutableMemory.cpp:361 WARNING: ASSERT(status == ZX_OK)