IssueTracker

I believe https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/5346490 should make this work, though I also noticed that we're not passing any form of --jobs to x.py, so RBE's "more parallelism" benefit doesn't really exist as it stands.

On the other hand, passing --jobs=$(get_makeopts_jobs) applies that limit to both Rust's and LLVM's builds. cargo -j1000 isn't being a good citizen on machines that are also running make -j$(ncpu).

Maybe there's room for a local patch, and/or advocating upstream for --llvm-jobs, depending on whether reclient can be made to work with the rustc invocations here.

Hrmm,

FAILED: lib/Demangle/CMakeFiles/LLVMDemangle.dir/MicrosoftDemangle.cpp.o 
/var/tmp/portage/dev-lang/rust-host-1.75.0-r1/files/reclient_compiler_launcher.sh /usr/bin/x86_64-pc-linux-gnu-clang++ -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/var/tmp/portage/dev-lang/rust-host-1.75.0-r1/work/build/x86_64-pc-linux-gnu/llvm/build/lib/Demangle -I/var/tmp/portage/dev-lang/rust-host-1.75.0-r1/work/rustc-1.75.0-src/src/llvm-project/llvm/lib/Demangle -I/var/tmp/portage/dev-lang/rust-host-1.75.0-r1/work/build/x86_64-pc-linux-gnu/llvm/build/include -I/var/tmp/portage/dev-lang/rust-host-1.75.0-r1/work/rustc-1.75.0-src/src/llvm-project/llvm/include -ffunction-sections -fdata-sections -fPIC --target=x86_64-pc-linux-gnu -pipe -fprofile-use=/var/tmp/portage/dev-lang/rust-host-1.75.0-r1/work/rust-pgo-1.75.0-llvm.profdata -fvisibility-inlines-hidden -w -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG -std=c++17  -fno-exceptions -funwind-tables -fno-rtti -MD -MT lib/Demangle/CMakeFiles/LLVMDemangle.dir/MicrosoftDemangle.cpp.o -MF lib/Demangle/CMakeFiles/LLVMDemangle.dir/MicrosoftDemangle.cpp.o.d -o lib/Demangle/CMakeFiles/LLVMDemangle.dir/MicrosoftDemangle.cpp.o -c /var/tmp/portage/dev-lang/rust-host-1.75.0-r1/work/rustc-1.75.0-src/src/llvm-project/llvm/lib/Demangle/MicrosoftDemangle.cpp
F0306 00:48:52.636827    8376 main.go:174] Command failed: rpc error: code = Internal desc = /mnt/host/source/src/chromium/depot_tools/.cipd_bin/reclient/scandeps_server terminated during startup: %!w(*command.Result=&{127 3 <nil>})
[2/2503] Building CXX object lib/Demangle/CMakeFiles/LLVMDemangle.dir/MicrosoftDemangleNodes.cpp.o
FAILED: lib/Demangle/CMakeFiles/LLVMDemangle.dir/MicrosoftDemangleNodes.cpp.o 

Looks like reclient didn't like something?

Do you have any of the reproxy logs? Looks like it might be a dependency scanner issue.

I assume Raul got the those lines from this builder, so that page & linked artifacts is probably all we have?

Other CQ builders passed, but it's unclear to me if those successfully used reclient in this run, since the ebuild_logs tarball for e.g., amd64-generic-cq doesn't have a file for rust-host. I presume this extends to all host packages, which is unfortunate if so. I can add a die to check though.

Re #5, If you expand the process_remoteexec_artifacts step in that builder, you can find the reclient logs.

I0306 00:47:06.624986     762 depsscannerclient.go:500] Starting service: [/mnt/host/source/src/chromium/depot_tools/.cipd_bin/reclient/scandeps_server --server_address unix:///tmp/depscan-dev-lang_rust-host-1.75.0-r1.sock --cache_dir  --deps_cache_max_mb 128 --noenable_deps_cache]
I0306 00:47:06.629764     762 depsscannerclient.go:59] depsscannerclient stdout: 
E0306 00:47:06.630006     762 depsscannerclient.go:64] depsscannerclient stderr: /mnt/host/source/src/chromium/depot_tools/.cipd_bin/reclient/scandeps_server: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory
E0306 00:47:06.630509     762 main.go:344] Failed to initialize input processor: /mnt/host/source/src/chromium/depot_tools/.cipd_bin/reclient/scandeps_server terminated during startup: %!w(*command.Result=&{127 3 <nil>})
I0306 00:47:06.630537     762 server.go:218] Startup status: Input Processor started? false, RBE connected (or disabled)? false, Startup error? true

lol, libstd++ strikes again!

Ugh, that means that you will need to add sys-devel/gcc as BDEPEND to rust-host :`(

Yeah, that's a pretty heavy dependency :/

I'm happy to add it, but it's a bit weird to me that we're pulling these binaries in as prebuilts rather than building from source. CrOS generally defaults to building ~everything from source.

In a 64-core full build I did recently, those two packages ran almost entirely in parallel, so we'd be adding ~25 minutes to the build time for this case, minus whatever speed-up we get from building the llvm portion remotely. I don't think this is worth considering unless we can make sys-devel/gcc much faster or till we find a solution for using prebuilts for it.

Is it possible to get a version of scandeps_server that is build with libc++? Or can we build scandeps_server from source? I'm sure we can add an ebuild for it.

Looks like there are build instructions here: https://source.corp.google.com/h/team/foundry-x/re-client

It's an internal project though, so we can't easily build that unless the src was published somewhere.

Is it possible to get a version of scandeps_server that is build with libc++?

I'm actually checking how difficult that would be right now :). Also, wondering if we could just statically link the binary. Although that would also require us to build with libc++ anyways due to licensing issues.

It's an internal project though, so we can't easily build that unless the src was published somewhere.

reclient is open source. The public version can be found on github

I think building scandeps_server with libc++ may not be straightforward as we use the goma input processor, and I think that relies on libstdc++.

I have no idea what I'm looking at, but this mentions libc++: https://github.com/bazelbuild/reclient/blob/main/goma/archive_linux.sh#L20

My hack for dumping logs gave me this. I see no mention of "Using reclient to speed up builds."

Looks like USE_REMOTEEXEC is set in Chromite's BuildPackagesRunConfig, and this all is happening in UpdateSDK. So in order for this to be useful, I'll need to add USE_REMOTEEXEC logic to whatever UpdateSDK logic exists there.

The bazel builder should be setting USE_REMOTEEXEC, but it looks like the build never even got far enough to launch bazel.

Yeah, many of the builds broke due to b/328463109 (for which a fix just landed).

Do we expect non-bazel builders to also use remoteexec? If so, USE_REMOTEEXEC still needs to find its way into UpdateSDK somehow

Do we expect non-bazel builders to also use remoteexec? If so, USE_REMOTEEXEC still needs to find its way into UpdateSDK somehow

Probably a good idea so that portage and bazel are both using the same flow.

I think building scandeps_server with libc++ may not be straightforward as we use the goma input processor, and I think that relies on libstdc++.

Can we try using reclient built without goma's input processor? Reclient's README says it can be configured at build time. https://github.com/bazelbuild/reclient?tab=readme-ov-file#building

Goma is optimized for chromium which is why we use it, but I suppose if you're not building chromium (chromeos-chrome in this case), there is another version of the dependency scanner that uses llvm's clangscandeps. This is currently also built using libstdc++, but I don't expect it to be too difficult to change it to build with libc++. I'm guessing you're pulling the cipd packages here. The one's with the suffix -csd are built using clangscandeps so you'll have to use that package instead if we decide to go down that path.

For testing purposes, just to get you unblocked you can actually disable the dependency scanner by setting depsscanner_address to an empty value. Note that this is not going to work in future releases (0.132.0+ for gomaip, planned for csd) as we are removing the cgo code within reclient and there will no longer be an internal dependency scanner to fallback on.

Picking this back up again: I cut a separate bug for using reclient on more bots (b/338094621), since I don't want that distracting from the impact on the bazel effort.

I have 3 CLs out to 'fully' land reclient for the LLVM portions of cross-*/rust and dev-lang/rust-host. I skipped dev-lang/rust-bootstrap since that package is only built on the SDK builder, and most "builds" of it are "install this prebuilt." Fingers crossed that remote testing is promising; reclient with -j1000 was able to bring my local LLVM-portion-of-Rust build time down to ~150s, at least.

For the reasons in b/338094621#comment6, I plan to pause this effort for now.

Also, to write this down more discoverably in case this is picked back up: within the context of bazel, reclient did have a meaningful blocker here. Its deps scanner prebuilt depends on libstdc++, which adds a dependency edge on gcc, which takes quite a while to build & would add to the long tail.

Possible workarounds include:

• (discussed by Andus in comment #23) modifying the dep scanner's target or build
• pinning a "prebuilt" libstdc++ in the SDK (much like how we do with rust-bootstrap)
• not caring for packages sufficiently deep in the depgraph (IOW, the final platform2 package to start building probably won't be impacted by a dep on gcc)

Head of the patch stack I had, btw: https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/5503569