Status Update
Comments
gb...@google.com <gb...@google.com> #2
I believe --jobs
to x.py
, so RBE's "more parallelism" benefit doesn't really exist as it stands.
On the other hand, passing --jobs=$(get_makeopts_jobs)
applies that limit to both Rust's and LLVM's builds. cargo -j1000
isn't being a good citizen on machines that are also running make -j$(ncpu)
.
Maybe there's room for a local patch, and/or advocating upstream for --llvm-jobs
, depending on whether reclient
can be made to work with the rustc
invocations here.
rr...@google.com <rr...@google.com> #3
Hrmm,
FAILED: lib/Demangle/CMakeFiles/LLVMDemangle.dir/MicrosoftDemangle.cpp.o
/var/tmp/portage/dev-lang/rust-host-1.75.0-r1/files/reclient_compiler_launcher.sh /usr/bin/x86_64-pc-linux-gnu-clang++ -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/var/tmp/portage/dev-lang/rust-host-1.75.0-r1/work/build/x86_64-pc-linux-gnu/llvm/build/lib/Demangle -I/var/tmp/portage/dev-lang/rust-host-1.75.0-r1/work/rustc-1.75.0-src/src/llvm-project/llvm/lib/Demangle -I/var/tmp/portage/dev-lang/rust-host-1.75.0-r1/work/build/x86_64-pc-linux-gnu/llvm/build/include -I/var/tmp/portage/dev-lang/rust-host-1.75.0-r1/work/rustc-1.75.0-src/src/llvm-project/llvm/include -ffunction-sections -fdata-sections -fPIC --target=x86_64-pc-linux-gnu -pipe -fprofile-use=/var/tmp/portage/dev-lang/rust-host-1.75.0-r1/work/rust-pgo-1.75.0-llvm.profdata -fvisibility-inlines-hidden -w -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG -std=c++17 -fno-exceptions -funwind-tables -fno-rtti -MD -MT lib/Demangle/CMakeFiles/LLVMDemangle.dir/MicrosoftDemangle.cpp.o -MF lib/Demangle/CMakeFiles/LLVMDemangle.dir/MicrosoftDemangle.cpp.o.d -o lib/Demangle/CMakeFiles/LLVMDemangle.dir/MicrosoftDemangle.cpp.o -c /var/tmp/portage/dev-lang/rust-host-1.75.0-r1/work/rustc-1.75.0-src/src/llvm-project/llvm/lib/Demangle/MicrosoftDemangle.cpp
F0306 00:48:52.636827 8376 main.go:174] Command failed: rpc error: code = Internal desc = /mnt/host/source/src/chromium/depot_tools/.cipd_bin/reclient/scandeps_server terminated during startup: %!w(*command.Result=&{127 3 <nil>})
[2/2503] Building CXX object lib/Demangle/CMakeFiles/LLVMDemangle.dir/MicrosoftDemangleNodes.cpp.o
FAILED: lib/Demangle/CMakeFiles/LLVMDemangle.dir/MicrosoftDemangleNodes.cpp.o
Looks like reclient
didn't like something?
an...@google.com <an...@google.com> #4
Do you have any of the reproxy logs? Looks like it might be a dependency scanner issue.
gb...@google.com <gb...@google.com> #5
I assume Raul got the those lines from
Other CQ builders passed, but it's unclear to me if those successfully used reclient in this run, since the ebuild_logs
tarball for e.g., rust-host
. I presume this extends to all host packages, which is unfortunate if so. I can add a die
to check though.
rr...@google.com <rr...@google.com> #6
Re #5, If you expand the process_remoteexec_artifacts
step in that builder, you can find the reclient logs.
rr...@google.com <rr...@google.com> #7
I0306 00:47:06.624986 762 depsscannerclient.go:500] Starting service: [/mnt/host/source/src/chromium/depot_tools/.cipd_bin/reclient/scandeps_server --server_address unix:///tmp/depscan-dev-lang_rust-host-1.75.0-r1.sock --cache_dir --deps_cache_max_mb 128 --noenable_deps_cache]
I0306 00:47:06.629764 762 depsscannerclient.go:59] depsscannerclient stdout:
E0306 00:47:06.630006 762 depsscannerclient.go:64] depsscannerclient stderr: /mnt/host/source/src/chromium/depot_tools/.cipd_bin/reclient/scandeps_server: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory
E0306 00:47:06.630509 762 main.go:344] Failed to initialize input processor: /mnt/host/source/src/chromium/depot_tools/.cipd_bin/reclient/scandeps_server terminated during startup: %!w(*command.Result=&{127 3 <nil>})
I0306 00:47:06.630537 762 server.go:218] Startup status: Input Processor started? false, RBE connected (or disabled)? false, Startup error? true
lol, libstd++ strikes again!
rr...@google.com <rr...@google.com> #8
Ugh, that means that you will need to add sys-devel/gcc
as BDEPEND to rust-host
:`(
gb...@google.com <gb...@google.com> #9
Yeah, that's a pretty heavy dependency :/
I'm happy to add it, but it's a bit weird to me that we're pulling these binaries in as prebuilts rather than building from source. CrOS generally defaults to building ~everything from source.
tb...@google.com <tb...@google.com> #10
In a 64-core full build I did recently, those two packages ran almost entirely in parallel, so we'd be adding ~25 minutes to the build time for this case, minus whatever speed-up we get from building the llvm portion remotely. I don't think this is worth considering unless we can make sys-devel/gcc
much faster or till we find a solution for using prebuilts for it.
rr...@google.com <rr...@google.com> #11
Is it possible to get a version of scandeps_server
that is build with libc++
? Or can we build scandeps_server
from source? I'm sure we can add an ebuild for it.
rr...@google.com <rr...@google.com> #12
Looks like there are build instructions here:
It's an internal project though, so we can't easily build that unless the src was published somewhere.
an...@google.com <an...@google.com> #13
Is it possible to get a version of scandeps_server that is build with libc++?
I'm actually checking how difficult that would be right now :). Also, wondering if we could just statically link the binary. Although that would also require us to build with libc++ anyways due to licensing issues.
an...@google.com <an...@google.com> #15
I think building scandeps_server with libc++ may not be straightforward as we use the goma input processor, and I think that relies on libstdc++.
ap...@google.com <ap...@google.com> #17
Branch: main
commit 4fb27919d036b61f8f2fe7b26ca4ed9e2e25536e
Author: George Burgess IV <gbiv@google.com>
Date: Tue Mar 05 13:54:20 2024
llvm: add `exec` to reclient launcher
No need to keep these bash processes hanging around while the builds
run.
BUG=b:328275699
TEST=CQ
Change-Id: Iec77315c5bc862a73d7a26571ba188cc6f163ef3
Reviewed-on:
Auto-Submit: George Burgess <gbiv@chromium.org>
Reviewed-by: Jordan Abrahams-Whitehead <ajordanr@google.com>
Commit-Queue: Jordan Abrahams-Whitehead <ajordanr@google.com>
Tested-by: George Burgess <gbiv@chromium.org>
M sys-devel/llvm/files/reclient_compiler_launcher.sh
gb...@google.com <gb...@google.com> #18
My hack for dumping logs
Looks like USE_REMOTEEXEC
is set in Chromite's BuildPackagesRunConfig
, and this all is happening in UpdateSDK
. So in order for this to be useful, I'll need to add USE_REMOTEEXEC
logic to whatever UpdateSDK logic exists there.
rr...@google.com <rr...@google.com> #19
The bazel builder should be setting USE_REMOTEEXEC
, but it looks like the build never even got far enough to launch bazel.
gb...@google.com <gb...@google.com> #20
Yeah, many of the builds broke due to
Do we expect non-bazel builders to also use remoteexec? If so, USE_REMOTEEXEC
still needs to find its way into UpdateSDK
somehow
rr...@google.com <rr...@google.com> #21
Do we expect non-bazel builders to also use remoteexec? If so, USE_REMOTEEXEC still needs to find its way into UpdateSDK somehow
Probably a good idea so that portage and bazel are both using the same flow.
ha...@google.com <ha...@google.com> #22
I think building scandeps_server with libc++ may not be straightforward as we use the goma input processor, and I think that relies on libstdc++.
Can we try using reclient built without goma's input processor? Reclient's README says it can be configured at build time.
an...@google.com <an...@google.com> #23
Goma is optimized for chromium which is why we use it, but I suppose if you're not building chromium (chromeos-chrome in this case), there is another version of the dependency scanner that uses llvm's clangscandeps. This is currently also built using libstdc++, but I don't expect it to be too difficult to change it to build with libc++. I'm guessing you're pulling the cipd packages -csd
are built using clangscandeps so you'll have to use that package instead if we decide to go down that path.
For testing purposes, just to get you unblocked you can actually disable the dependency scanner by setting
gb...@google.com <gb...@google.com> #24
Picking this back up again: I cut a separate bug for using reclient on more bots (
I have 3 CLs out to 'fully' land reclient for the LLVM portions of cross-*/rust
and dev-lang/rust-host
. I skipped dev-lang/rust-bootstrap
since that package is only built on the SDK builder, and most "builds" of it are "install this prebuilt." Fingers crossed that remote testing is promising; reclient with -j1000
was able to bring my local LLVM-portion-of-Rust build time down to ~150s, at least.
gb...@google.com <gb...@google.com> #25
For the reasons in
gb...@google.com <gb...@google.com> #26
Also, to write this down more discoverably in case this is picked back up: within the context of bazel, reclient did have a meaningful blocker here. Its deps scanner prebuilt depends on libstdc++, which adds a dependency edge on gcc
, which takes quite a while to build & would add to the long tail.
Possible workarounds include:
- (discussed by Andus in
) modifying the dep scanner's target or buildcomment #23 - pinning a "prebuilt" libstdc++ in the SDK (much like how we do with rust-bootstrap)
- not caring for packages sufficiently deep in the depgraph (IOW, the final platform2 package to start building probably won't be impacted by a dep on gcc)
Description
In a recent Bazel discussion, it was brought up that
rust-host
takes quite a while to build. This bug tracks the investigation of whether we can use reclient for the LLVM builds embedded in builds ofrust-host
andcross-*/rust
. Methodology should be really similar to LLVM.