IssueTracker

• RISC-V branch for Swiftshader: https://github.com/developandplay/swiftshader/tree/rv64

With the latest commit on my RISC-V branch of Swiftshader I'm finally getting to the point of producing a JIT related error within Swiftshader (still using llvm10):

[ RUN      ] DrawTest.VertexShaderNoPositionOutput
vk::DynamicLoader exception: Failed to load vulkan library!
Falling back to loading SwiftShader directly (i.e. no validation layers)
JIT session error: Symbols not found: [ __muldi3 ]
JIT session error: Symbols not found: [ __addsf3, __divdi3, __divsf3, __eqsf2, __fixsfdi, __floatsisf, __gesf2, __gtsf2, __ltsf2, __moddi3, __muldi3, __mulsf3, __subsf3, __unordsf2 ]
JIT session error: Symbols not found: [ __fixsfdi, __floatsisf, __gtsf2, __ltsf2, __muldi3, __mulsf3 ]
[16737.053636] Thread<00>[7888]: unhandled signal 11 code 0x2 at 0x0000003f7c0f93b0

I am now checking if this is due to missing relocations or another issue specific to LLVM JIT on RISC-V.

Thanks for opening a tracker issue for this!

LLVM obtains the address of external symbols through the Resolver class, so try adding them there. Note we trim the leading underscores.

Apparently jitTargetMachineBuilder was just lacking the right features for floating point operations. After adding:

+                jitTargetMachineBuilder.getFeatures().AddFeature("+m");
+                jitTargetMachineBuilder.getFeatures().AddFeature("+a");
+                jitTargetMachineBuilder.getFeatures().AddFeature("+f");
+                jitTargetMachineBuilder.getFeatures().AddFeature("+d");
+                jitTargetMachineBuilder.getFeatures().AddFeature("+c");

the test now passes:

[----------] 1 test from DrawTest
[ RUN      ] DrawTest.VertexShaderNoPositionOutput
vk::DynamicLoader exception: Failed to load vulkan library!
Falling back to loading SwiftShader directly (i.e. no validation layers)
[       OK ] DrawTest.VertexShaderNoPositionOutput (5155 ms)
[----------] 1 test from DrawTest (5156 ms total)

However I'm not sure what that vk::DynamicLoader exception: Failed to load vulkan library! warning is about. It does not occur on my x86 system.

On top of that the remaining tests from the vk-unittests test suite are also passing:

[----------] Global test environment tear-down
[==========] 139 tests from 3 test suites ran. (60747 ms total)
[  PASSED  ] 139 tests.

This is a bit surprising to me considering that llvm10 JIT does not support any relocations for RISC-V. For reference even the most basic "hello world" using LLVM JIT requires relocations.

So I guess there would be three possible explanations:

• The way Swiftshader uses LLVM JIT just does not lead to any relocations being generated ever.
• The test cases in vk-unittests don't lead to relocations being generated (but other uses of Swiftshader might?).
• The test cases have some fundamental issue on RISC-V which makes them pass even though they should not.

I would be happy if you could add your thoughts.

So to see if Swiftshader now actually works I tried headless the compute and rendering examples from Sascha Willems.

Unfortunately both of them fail (non deterministic either Illegal instruction or Segmentation fault) right after submitting the actual compute/render command buffer:

• https://github.com/SaschaWillems/Vulkan/blob/master/examples/computeheadless/computeheadless.cpp#L475
• https://github.com/SaschaWillems/Vulkan/blob/master/examples/renderheadless/renderheadless.cpp#L681

Example output from the headless rendering:

Running headless rendering example
GPU: SwiftShader Device (LLVM 10.0.0)
[93802.782734] Thread<02>[20046]: unhandled signal 4 code 0x1 at 0x0000003f9241d0c8
[93802.783081] Thread<00>[20044]: unhandled signal 4 code 0x1 at 0x0000003f9241d0c8
....
[93802.802700] status: 0000000000004020 badaddr: 0000000000000000 cause: 0000000000000002
Illegal instruction

Is there maybe a compile setting specific to Swiftshader that I could be missing? If not do you have advice on how I could go about debugging this issue?

That's a great start! I wonder if LLVM 10 supports the relocations for Position Independent Code (PIC), which are just relative offsets and don't require any 'linking' relocations for external symbols? I believe that theory could be tested by running ReactorUnitTests, where only some of the tests use external symbols.

Note that upgrading all of SwiftShader to a newer version of LLVM could be a lot of work and affects numerous projects, so to avoid such complication I'd like to reiterate my suggestion from the e-mail thread: Add a CMake build option to download a newer version of LLVM as a submodule, an link against it wholesale.

Thanks for pointing out the ReactorUnitTests. I just ran it and for the Swizzle4 testcase I'm also running into an Illegal instruction. I'll see what this test case is about and what is the cause for the failure.

While I was at it I also ran the system-unittest which fails for the following test cases:

[  FAILED  ] 4 tests, listed below:
[  FAILED  ] MemFd.AllocatingConstructor
[  FAILED  ] MemFd.ExplicitAllocation
[  FAILED  ] MemFd.Close
[  FAILED  ] MemFd.ExportImportFd

I will investigate those too.

Regarding the Update SwiftShader to use llvm-14 in the original in the first post:

For now my plan is to investigate this further only when I have confirmed that my issues are definitely related to missing relocations or some other missing llvm-14 feature. That way I can then implement them prioritised by their usage in Swiftshader. If so I'll follow your advice to make it a CMake build option. On first glance it seems like this comes down to replacing the link to libllvm.a with all the individual libLLVM*.a library files that the regular build of llvm creates.

So I did walk through the remaining test cases in ReactorUnitTests.

The following test cases crash with an Illegal instruction error:

• Swizzle4
• Shuffle
• MinMax
• NotNeg
• Call*
• CToReactorTCastTestTypes for std::pair<float, Float>
• Fibonacci
• Coroutines*
• IntrinsicTest_Float*
• FRem
• Multithreaded_Coroutine
• Arithmetic_ConstantArgs

Out of the remaining 94 test cases another 5 of them failed:

[==========] 94 tests from 31 test suites ran. (111757 ms total)
[  PASSED  ] 89 tests.
[  FAILED  ] 5 tests, listed below:
[  FAILED  ] ReactorUnitTests.RoundInt
[  FAILED  ] ReactorUnitTests.Args_2Mixed
[  FAILED  ] ReactorUnitTests.Args_4Mixed
[  FAILED  ] ReactorUnitTests.Args_5Mixed
[  FAILED  ] ReactorUnitTests.Args_GreaterThan5Mixed

 5 FAILED TESTS

So I finally managed to link with llvm-13. Besides a few minor changes it was pretty straight forward.

Now when upgrading to llvm-14 I'm running into an issue as gtest;gtest_main are used by llvm and swiftshader so the target definitions overlap. I'm not sure if llvm-13 didn't use Google Test or why exactly this is occuring to me.

I'll see if I can disable the gtest;gtest_main targets for llvm and resolve this conflict.

Seems like I was building llvm with LLVM_INCLUDE_TESTS off for llvm-13 and LLVM_INCLUDE_TESTS on for llvm-14.

Now I managed to link against llvm-14 as well. The behavior for the ReactorUnitTests tests remains the same. I will check back with the llvm developers to see if they have some tips on debugging the jit crashes.

For the lli tool they were able to adjust the output such that instead of simply crashing it would spit out the missing relocation. I'll check if I can do something similar for the current version of swiftshader.

After talking to the llvm developers I now know that for RISC-V we need to use llvm::orc::ObjectLinkingLayer instead of llvm::orc::RTDyldObjectLinkingLayer.

With this change and llvm-14 now quite a few more unit tests pass or at least fail gracefully. There still some left however that cause a segmentation fault. I will make a list and look into them over the next few days.

Most excitingly now the examples from Sascha's repository work. So here are the first triangles rendered with Swiftshader on RISC-V! Finally seeing some visual progress is super cool.

That's amazing progress! Feel free to upload a patch to build with LLVM 14.

I've found that ReactorUnitTests.RoundInt actually also fails with LLVM 10 when we disable our x86-specific code path, due to RoundIntClamped() returning 0 for an out-of-range input . That even happens when using both an upper and lower clamp operation. I'll try to find a more robust baseline implementation.

Patch: https://swiftshader-review.googlesource.com/c/SwiftShader/+/62728. Could you give that a try? Please sign in to that site to be able to participate in code reviews and upload changes.

Sorry for the late response...I was busy with exams this week.

I tried your patch and it resolves the ReactorUnitTests.RoundInt issue. Also commented that in the review tool. Is there something else I should do on there?

As for adding the LLVM 14 changes: Would you add the full llvm repo as a submodule? Currently I'm just providing the path of either my LLVM 14 build or the system LLVM (13) to find_package in my CMakeLists.txt for Reactor. Probably the submodule way is better in terms of reproducability.

Addditionally: (How) would I have to adjust the BUILD.gn for Reactor?

As for adding the LLVM 14 changes: Would you add the full llvm repo as a submodule? Currently I'm just providing the path of either my LLVM 14 build or the system LLVM (13) to find_package in my CMakeLists.txt for Reactor. Probably the submodule way is better in terms of reproducability.

Yes, as a submodule would be great. FYI, we're set up to have all submodules be optional (mandatory third-party dependencies are an integral part of our repo using git subtree).

Addditionally: (How) would I have to adjust the BUILD.gn for Reactor?

No, that's the build file for Chromium, which can keep using Subzero and LLVM 10 as JIT compilers. But thanks for checking!

Ok so I separated the LLVM update changes from the RISC-V changes and put the LLVM changes up for review. I'm looking forward to your feedback on https://swiftshader-review.googlesource.com/c/SwiftShader/+/63308

Now for the RISC-V specific changes I would also like to commit them as soon as possible. Since they touch quite a few files having them upstreamed would help me with merge conflicts when rebasing.

Therefore I was wondering what the smallest possible commit would be. Right now quite a few test cases are still failing but the triangle demo works as shown above. This is probably due to issues/relocations on the LLVM side (but could also be on the Swiftshader side).

Additionally I was wondering whether there a way to imstrument gtest to only run a certain subset of test cases? Currently I'm commenting out the test cases that lead to crashes but there has to be a better way.

Also is there a setting for gtest to pass on the console output that Swiftshader generates? This would help with debugging those tests that fail gracefully.

Therefore I was wondering what the smallest possible commit would be.

Feel free to upload a work-in-progress patch and I'll help see how we can best approach readying it for commit.

Additionally I was wondering whether there a way to imstrument gtest to only run a certain subset of test cases? Currently I'm commenting out the test cases that lead to crashes but there has to be a better way.

Yes, the --gtest_filter command line argument can be used to exclude or include specific tests.

Also is there a setting for gtest to pass on the console output that Swiftshader generates? This would help with debugging those tests that fail gracefully.

It redirects stdout/stderr to not disturb the actual test result output, but you can programmatically make it capture it and retrieve it: https://stackoverflow.com/questions/3803465/how-to-capture-stdout-stderr-with-googletest.

Turns out after reversing all the update script adjustments the changes are quite minimal. I put out https://swiftshader-review.googlesource.com/c/SwiftShader/+/64168 for a start. I'll add more information soon on which tests exactly are passing/failing on that commit.

So if I look at the log I would assume that those tests that fail are related to an issue with the calling convention.

I presented this work at the RISC-V Europe Summit in Barcelona last week. It got some interest from RISC-V startups. The paper and poster can be found here. .

Most recent commit used for the conference can be found here.

As part of my Master thesis I have now added support for the RISC-V vector extension. More details on the progress can be found at: https://issuetracker.google.com/issues/307592533