asan_device_setup doesn't work with user builds because of verity [63766481]

WAI

Bug

Migrated-Tools-NDK

Status Update

No update yet.

Description

gp...@gmail.com

created issue #1

Jul 17, 2017 08:40PM

Assuming one has a production, rooted of Pixel XL using the --use-su path in the asan_device_setup script will fail to correctly remount /system. When trying to remount, the mount command will succeed, however, it wont actually stay remounted as rw. In order to get remounting to work, one must turn enforcing off prior to the mount command rather than further down in the install script.
Next after setting enforcing to 0 the rw remount will not succeed with the listed device path. To validate this do the following:
adb shell "su -c set enforcing 0"
adb shell "su -c mount | grep /system | cut -d ' ' -f1"
adb shell "su -c mount rw,remount (outputOfLastADB) /system"
The error says /system is not defined in mount points. To get around this, use:
adb shell "su -c mount rw,remount /system"
This command will succeed correctly, however, the final problem is once the tty is escaped (i.e. the command completes and exits the shell) the os automatically remounts the /system command as ro.
I wasn't sure how to get around this but observed the following:
adb shell
$ > su
$ > mount rw,remount /system
$ > mount
Observe that system is still mounted rw can do whatever commands needed here
$ > exit
$ > mount
Observe that system is now mounted ro

This led me to create a script that can be run with su -c mount rw,remount /system && scriptName. This seems to work; but am still having a few problems with it. Is there a way to get the OS to not automatically switch /system to ro?

Comments

gp...@gmail.com <gp...@gmail.com> #2Jul 17, 2017 11:02PM

Fixed with the following CL (missed tagging this bug):

https://android-review.git.corp.google.com/c/platform/frameworks/support/+/2881070

There will be additional follow up work (handling new mainline versions that fix the issue), but we'll file separate bugs to improve that.

ks...@google.com <ks...@google.com> #3Jul 20, 2017 05:45PM

Assigned to ag...@google.com.

How To Get Method Tracing Working in Microbench

Options:

1) Flash API 23 through 25

2) Flash API 31 through 33, and disable wifi/network on the device

Not connecting to the network prevents mainline updates. You can validate that your device hasn't taken a problematic mainline version with the following shell command:

adb shell cmd package list packages --show-versioncode --apex-only art

You should see e.g.:

package:com.google.android.art versionCode:320000000

If the first two digits of the version code are less than 34, you're good, method tracing will work and be on by default in microbench.

3) Force Mainline Update with Fix (will be documented later)

ag...@google.com <ag...@google.com> #4Jul 20, 2017 05:53PM

I have no insight into the NDK version of ASAN, regrettably.

Why does it need to modify /system?

If it needs to, I think dm-verity is your issue here. You have to disable verity and reboot.

gp...@gmail.com <gp...@gmail.com> #5Jul 20, 2017 07:05PM

To get ASAN working, one needs the following (this is what the asan_device_setup bash file does):

1.) remount /system
2.) install libclang_rt.asan-<abi>-android.so with 644 permission to /system/lib/
3.) install app_process A wrapper that looks like this:
#/system/bin/sh-from-zygote
ASAN_OPTIONS=start_deactivated=1,malloc_context_size=0
ASAN_ACTIVATION_OPTIONS=include_if_exists=/data/local/tmp/asan.options.b
LD_PRELOAD=\:

libclang_rt.asan-arm-android.so
exec /system/bin/app_process32 $@
Which should be at /system/bin with 755 permission and u:object_r:zygote_exec:s0 as context
4.) install asanwrapper to /system/bin with 755 permission (a script that makes it easier to run any command line executable.)

Then standard stuff in modifying a NDK app to build.

Unfortunately, while I agree that disable-verity might help in this particular case. Pixel is shipped with a production build and adb disable-verity is userdebug only. I don't see any images in here:

https://developers.google.com/android/images that look like they might provide a userdebug build. If I'm understanding this correctly, one would need to get the source to O, create a new build of Android and then use adb disable-verity?

da...@google.com <da...@google.com> #6Jul 20, 2017 08:39PM

Status: Won't Fix (Intended Behavior)

(retitling because this has been true since shamu or so)

I'm going to close this in favor of

https://github.com/android-ndk/ndk/issues/380. The fact that verity blocks this is the correct behavior. The fix for this is making ASAN not require root.

> If I'm understanding this correctly, one would need to get the source to O, create a new build of Android and then use adb disable-verity?

Yeah, this would be a way to work around this issue. enh: do you know if any headway was ever made on publishing userdebug images?

ag...@google.com <ag...@google.com> #7Jul 20, 2017 08:47PM

(Note that in the next release after O [sadly we missed a critical bug in O] this is hopefully gonna work with the wrap.sh script stuff)

gp...@gmail.com <gp...@gmail.com> #8Jul 20, 2017 08:51PM

Sounds good to me. Out of curiosity, what's the bug in O that prevents wrap.sh from being run?

ag...@google.com <ag...@google.com> #9Jul 20, 2017 09:05PM

selinux policies in user builds.

gp...@gmail.com <gp...@gmail.com> #10Jul 20, 2017 09:08PM

That sounds like something that a rooted phone would be okay with handling? I submitted a patch that fixes --use-su path to llvm the other day and ported it to python so it works from Windows/osx etc. Having su is potentially doable on a lot of devices, I wonder if that might be a potential work around?

ag...@google.com <ag...@google.com> #11Jul 20, 2017 09:21PM

If "setenforce 0" works under su (I don't know whether a userbuild would have policies to forbid this :-) ), then yes, wrap.sh should work.

gp...@gmail.com <gp...@gmail.com> #12Jul 20, 2017 09:25PM

That it does (hence my thought). The only major thing that doesn't work is disable-verity and the other adb commands (root, remount etc).

I'll give it a shot and report back.

gp...@gmail.com <gp...@gmail.com> #13Jul 20, 2017 10:26PM

Yes, that does indeed work. No need to call asan_device_setup either. If you have su on an end user device; you can use wrap.sh and have ASAN work.

I'll make a blog post with example project to outline this for other devs. However, this is definitely a workable solution (so long as someone has figured out how to root a device, we can use ASAN to work on it).

en...@google.com <en...@google.com> #14Jul 20, 2017 11:54PM

alternatively/additionally, suggest changes to

https://source.android.com/devices/tech/debug/asan here, and we can work with the documentation folks to get that page updated.

gp...@gmail.com <gp...@gmail.com> #15Jul 21, 2017 12:04AM

Sure, lemme finish making the steps generic and I'll reply here with them.
Part of the process includes "root your device". I'm not certain I've ever seen a Google doc suggesting that installing su on a user device is a good idea; but hey, it works :). I'll submit my steps and example project to my blog and copy steps here when I get a chance.