IssueTracker

Just did a clean project, invalidated cache and restarted. Sam results, except I can no longer run debug builds (reported in another issue).

I used to run Android 3.0.1 with gradle 4.1 and plugin 3.0.1 and native libs were properly sized, so I compared both builds to find what was different.

Turns out the stripDebugSymbol library is the exact same size as the original, which tells me strip debug symbol is not working or disabled (?) with gradle 4.4 and above or plugin 3.1 and above.

cmake_build_command.txt are identical and every files in .externalNativeBuild and build folders, except some .log files and the stripDebugSymbol .so files (obviously).

Maybe the plugin just forgets to strip debug symbol or is it gradle that requires a special option or both?

I would really appreciate some help here.

After fiddling for quite a while on this bug, I fixed it by adding this to my gradle config:

buildTypes {
        release {
            externalNativeBuild {
                cmake {
                    arguments "-g0"

                    cFlags "-g0"
                    cppFlags "-g0"
                }
            }
        }
}

Kinda dumb to have to do that for a release build. But at least I get the APK size under control. Thanks for the help.

Nevertheless each shared library (.so) is still 500Ko bigger than before... times 6 to support most devices.

Unforunately, turns out the generated libaries are now completely unreliable.

Tested on various devices:

API 27 x86 (emulator) / arm64 (device) OK
API 26 x64 (emulator) KO
API 25 x64 (device) OK
API 24 KO
...

Not to mention how one have to launch the APK on lowest used APK and lower APK to get a successful start. That is running on a single API14 device while using classes added in higher APIs results in failed build!

Not to mention the fact that after reverting back to 3.0.1, I'm no longer able to build anything at all, only build failures with cryptic messages.

There's no way one can run a 3.1 (even less 3.2) of Android Studio alongside 3.0.1 considering gradle build files are now incompatible!

It feels like Google doesn't have a sense of urgency for this issue. I propose it be moved to a P1 priority and addressed before the next Canary release.

This bug was fixed in our internal 3.1 branch in January 23. At the least, it should be available in the next 3.1 beta. It might be available in a canary sooner but I'm not sure.

Background:
NDK16 deprecates mips/mips64 and we added hooks in Android Studio to respect this deprecation. Gradle plugin will still build deprecated ABIs but only if they are explcitly requested in abiFilters in build.gradle.
The bug was that if a deprecated ABI is requested then symbol stripping is mistakenly skipped.

In our specific case, we are not  using mips/mips64, only x86 and armeabi-v7a. However, stripping wasn't happening for us. So is the issue limited to only mips?

It looks like Andro's APK also has unstripped x86 and armeabi.

I tried the repro steps way below on canary7 and beta1 respectively. Result:
  canary7 APK has unstripped mips but stripped x86
  beta1 APK has stripped mips and stripped x86
This indicates that the fix for 72493369 made it into beta1.
Robert or Andro, can you give me some hints or repro steps to help me reproduce the issue you're seeing?

My repro steps:
(1) Create new project, check C++ checkbox
(2) in build.gradle add abiFilters:
    defaultConfig {
        ndk {
            abiFilters "x86", "armeabi-v7a", "mips"
        }
    }
(3) Create debug APK
(4) Unzip APK
(5) Run readelf -S to check for debug sections

 readelf -S libnative-lib.so
There are 29 section headers, starting at offset 0x2b27c:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .note.android.ide NOTE            00000134 000134 000098 00   A  0   0  2
  [ 2] .note.gnu.build-i NOTE            000001cc 0001cc 000024 00   A  0   0  4
  [ 3] .dynsym           DYNSYM          000001f0 0001f0 000ad0 10   A  4   1  4
  [ 4] .dynstr           STRTAB          00000cc0 000cc0 00144a 00   A  0   0  1
  [ 5] .hash             HASH            0000210c 00210c 0004c8 04   A  3   0  4
  [ 6] .gnu.version      VERSYM          000025d4 0025d4 00015a 02   A  3   0  2
  [ 7] .gnu.version_d    VERDEF          00002730 002730 00001c 00   A  4   1  4
  [ 8] .gnu.version_r    VERNEED         0000274c 00274c 000040 00   A  4   2  4
  [ 9] .rel.dyn          REL             0000278c 00278c 000ff0 08   A  3   0  4
  [10] .rel.plt          REL             0000377c 00377c 000320 08  AI  3  23  4
  [11] .plt              PROGBITS        00003aa0 003aa0 000650 04  AX  0   0 16
  [12] .text             PROGBITS        00004100 004100 01c903 00  AX  0   0 32
  [13] .gcc_except_table PROGBITS        00020a04 020a04 0003cd 00   A  0   0  4
  [14] .rodata           PROGBITS        00020de0 020de0 001b48 00   A  0   0 32
  [15] .eh_frame         PROGBITS        00022928 022928 005bfc 00   A  0   0  4
  [16] .eh_frame_hdr     PROGBITS        00028524 028524 000fbc 00   A  0   0  4
  [17] .data.rel.ro.loca PROGBITS        0002af80 029f80 0009c4 00  WA  0   0 64
  [18] .fini_array       FINI_ARRAY      0002b944 02a944 000008 04  WA  0   0  4
  [19] .init_array       INIT_ARRAY      0002b94c 02a94c 000010 04  WA  0   0  4
  [20] .data.rel.ro      PROGBITS        0002b960 02a960 000344 00  WA  0   0 32
  [21] .dynamic          DYNAMIC         0002bca4 02aca4 000120 08  WA  4   0  4
  [22] .got              PROGBITS        0002bdc4 02adc4 000070 00  WA  0   0  4
  [23] .got.plt          PROGBITS        0002be34 02ae34 00019c 00  WA  0   0  4
  [24] .data             PROGBITS        0002c000 02b000 0000c0 00  WA  0   0 32
  [25] .bss              NOBITS          0002c0c0 02b0c0 004bcc 00  WA  0   0 64
  [26] .comment          PROGBITS        00000000 02b0c0 000065 01  MS  0   0  1
  [27] .note.gnu.gold-ve NOTE            00000000 02b128 00001c 00      0   0  4
  [28] .shstrtab         STRTAB          00000000 02b144 000137 00      0   0  1
Key to Flags:

At the moment I can only share how we specify our abiFilters, which is through product flavors:


    externalNativeBuild {
        cmake {
            path '../../CMakeLists.txt'
        }
    }

    flavorDimensions "mode"

    productFlavors {
        arm {
            dimension "mode"
            ndk { abiFilters "armeabi-v7a" }
        }

        x86 {
            dimension "mode"
            ndk { abiFilters "x86" }
        }

        full {
            dimension "mode"
            ndk { abiFilters "x86", "armeabi-v7a" }
        }
    }

Maybe there's something wrong when product flavors are in the mix?

let me try that

I have an existing project and didn't try to create a test one at this time. Fiddling took me so much time.

Whether I use AS 3.1 beta 1 or AS 3.2 canary 1 with associated plugins makes no difference, NO libraries are stripped. NONE. The shared libraries are produced in different folders, one is named after 'stripped' and they all have the same size and the same is found in the produced APK. Adding the -g0 option makes all the files smaller, as opposed to AS 3.0.1 which produced a non-stripped and a striped version. See screenshot when -g0 is enabled.

Please find attached both gradle, for 32 bits and for 64 bits native modules. I have now removed MIPS to save a few MB. That said, binaries are still bigger than when produced with 3.0.1 and its associated plugin version (3.0.1).

Both libraries are based on the same source code (a java file and many C++ or C files). I use separate modules because 32 bits library built with API 23 is incompatible with older API devices. And API 23 is required for producing 64 bits native libraries.

I added -g0 in various locations to somehow reduce shared library sizes, however each are still much bigger than before.

Hope it's enough to find a solution.

#13, I'm using product flavors in app's module but not in native library.

#8, any idea if the fix will be available in next canary release of 3.2 ?

I have 3.2 canary 1 and 3.1 beta 1, so I suppose I should way for next update on any of those before trying again?

Thanks for looking into this.

Under 3.1.0-beta1, I modified my repro steps to build with productFlavors configured to build this way, but still no luck (resulting symbols are stripped). There must be something else in the respective builds that is triggering this case.
I'll poke at it to see if I can trigger the case. I can't really do much without a repro though so if either of you think of additional things that are special to your build please let me know.

Have you seen my previous post (#14), with attached gradle files? As I mentioned I used 2 modules one for 32 bits, one for 64 bits, nothing that special, however I use very specific options in those to create a runnable shared library. Maybe that prevents stripping?

Checking the command lines created by AS to run cmake, I found no differences between AS 3.1/3.2 and the working 3.0.1. Suggesting stripping is performed at a later stage, or not at all with 3.1 and above.

Re #8, when you say "This bug was fixed in our internal 3.1 branch in January 23. At the least, it should be available in the next 3.1 beta",

do you mean it should be available in 3.1 beta 2 or is that already included in beta 1?

Sounds like it's not yet available, so maybe it's actually fixed but we don't have the fix yet?

The fix that we did regarding deprecated ABIs not being stripped is in 3.1.0-beta1, that's demonstrated in #11. #1 says you're using 3.1.0-beta1 so there appears to be a distinct issue that I'm trying to figure out now

#14, it looks like this a library module so I'll look around there specifically.

@Andro, could you share the app/build.gradle of the module that consumes the library?

Here it is. I've attached the build.gradle that I use for the whole project, now running it under 3.2 canary 1.

Is the fix also included in AS 3.2 canary 1 ?

Would you mind removing the previous attachement so I can repost proper files? Or is there a way I can do that myself?

Do you mean #22? I don't think I can edit it

Found this page: https://blog.algolia.com/android-ndk-how-to-reduce-libs-size/

So I added those flags to both 32/64 bits module and rebuilt the APK:

                cFlags "-fvisibility=hidden"
                cFlags "-ffunction-sections"
                cFlags "-fdata-sections"

                cppFlags "-ffunction-sections"
                cppFlags "-fdata-sections"
                cppFlags "-fvisibility=hidden"

Add specific to release build:

                cFlags "-Os"

It saved another 200K on each library produced, but still not close to original size. And haven't verified if the library is still functionnal!

Here is a screenshot showing the various lib sizes (and compressed sizes) using 3.0.1, then 3.1.0 with gradle.build provided before and 3.1.0 with options mentioned in this post.

ARM libraries are still (with above options) nearly 600KB bigger while x86 are only 200KB bigger. The APK is still 360KB bigger (3%) overall. Or 2.3MB bigger (20%) if I don't use above options!

It seems to me all above options (and -g0) used to be part of the build process in 3.0.1, but those are no longer performed in 3.1 and above?

Again as I mentioned earlier, all .so built have the same size, in every folder I can find them: stripReleaseSymbol, mergeJniLibs, intermediate-jars or cmake. Whether I use specific options or not all .so built have the same size, is that expected or what you are seeing in your build.

@Android, are you using a different NDK version between those two builds? If so, what versions are they?
>> is that expected or what you are seeing in your build.
The only circumstance I know that will cause this is if symbols aren't built into the .so in the first place. This would, of course, happen if you've set the -g0 flag.

@Dan, are you aware of any changes in CMake-side toolchain that could affect .so size or the flags Andro mentions above?

err, s/@Android/@Andro/. Stupid muscle memory

After further fiddling and looking-up the web, found the actual option that instructs cmake to strip the library: -s
https://stackoverflow.com/questions/38675403/how-to-config-cmake-for-strip-file

Added it to arguments, cFlags and cppFlags for the release: all libraries are back to expected sizes, that is the sizes I obtained with 3.0.1 plugin without any particular options.

@jomof: Nothing I can think of. iirc the way gradle does things is that it tells CMake *not* to strip binaries and then strips them itself when packing the libraries into the APK (so you still have debug info on the host machine for the debugger to use, but the APK gets the smaller binaries). Did gradle stop stripping the binaries on install? I know that some of the code around the strip code changed to deal with the fact that there are no mips toolchains any more because it was for-eaching over the wrong ABI list. Any chance it's now for-eaching over an empty list?

#28, thanks Andro that's a big clue. I still haven't been able to repro the actual issue, but this is at least a lead to unblock me for now

Possibly https://googleplex-android-review.git.corp.google.com/c/platform/tools/base/+/3294619/5/build-system/gradle-core/src/main/java/com/android/build/gradle/internal/transforms/StripDebugSymbolTransform.java? I have no evidence (aside from unstripped binaries) that this is happening, but that patch *would* allow the plugin to silently not-strip a binary if the strip tool is not found.

Thanks for taking a look Raymond. Leave me any breadcrumbs you find and then assign back to me when you're ready

@Andro
What version of NDK are you using?  And what OS are you running on?  One possibility for this to happen is if the plugin is unable to find the executable to strip the libraries.  Under the NDK the executable is location in a path like toolchains/x86-4.9/prebuilt/windows-x86_64/bin/i686-linux-android-strip.exe.  "windows-x86_64" part of the path will change depending on what OS you are running on.  Are you able to find such executable in the NDK?

Is it possible that the gradle plugin won't find strip in a 32-bit NDK on 64-bit Windows?

It's not impossible, but it should fallback to 32-bit.  I am hoping if we have the path to the strip exe, we might be able to get some clues on why the plugin can't find it.

Here are the path of the 64-bit strip executable (from ndk bundle), for each ABI target I suppose:

.\toolchains\x86_64-4.9\prebuilt\windows-x86_64\bin\x86_64-linux-android-strip.exe
.\toolchains\x86-4.9\prebuilt\windows-x86_64\bin\i686-linux-android-strip.exe
.\toolchains\arm-linux-androideabi-4.9\prebuilt\windows-x86_64\bin\arm-linux-androideabi-strip.exe
.\toolchains\aarch64-linux-android-4.9\prebuilt\windows-x86_64\bin\aarch64-linux-android-strip.exe

I'm running Windows 10 64 bits.

Wouldn't there be some output somewhere if this executable couldn't be found? Is there a way I can enable logging/trace of every commands being run?

If that helps, NDK installed version (as shown by SDK manager): 16.1.4479499

The task to look for is transformNativeLibsWithStripDebugSymbolFor<Variant>.  If the strip command fails, it would print out a message saying "Unable to strip library '%s', packaging it as is.".  If you don't see it, most likely it means the plugin wasn't able to find the executable.  Unfortunately, the log is kind of lacking in that case.

No such message, just nothing, here is the last output with stripping:

:lib3c:transformNativeLibsWithStripDebugSymbolForRelease
:lib3c:transformNativeLibsWithIntermediateJniLibsForRelease
[187/187] Linking CXX shared library ..\..\..\..\build\intermediates\cmake\release\obj\arm64-v8a\liblib3c.so
:lib3c_x64:mergeReleaseJniLibFolders UP-TO-DATE
:lib3c_x64:transformNativeLibsWithMergeJniLibsForRelease
:lib3c_x64:transformNativeLibsWithStripDebugSymbolForRelease
:lib3c_x64:transformNativeLibsWithIntermediateJniLibsForRelease
:Toolbox:transformNativeLibsWithMergeJniLibsForPaidGoogleRelease
:Toolbox:transformNativeLibsWithStripDebugSymbolForPaidGoogleRelease
:Toolbox:packagePaidGoogleRelease
:Toolbox:assemblePaidGoogleRelease
:Toolbox:archivePaidGoogleRelease NO-SOURCE

I don't think that's the actual issue here, because if I also remove the extra flags I added, namely -Os and -g0, the native libraries size just rocket:

With -g0 -Os -s enabled in 3.1/3.2, I get the same size I used to get with 3.0.1 without those options, problem is I cannot debug any release crash, can I? :
 ARM - 976KB
 ARM64 - 1790KB
 X86 - 1804KB
 X64 - 1848KB

With only -g0 -Os:
 ARM - 1684KB
 ARM64 - 2589KB
 X86 - 2291KB
 X64 - 2535KB

With none of the 3 options:
 ARM - 4607KB
 ARM64 - 7880KB
 X86 - 6809KB
 X64 - 7885KB

It's funny to notice that without those 3 options, the release .so files are even bigger than the debug .so files! I don't have any of those 3 options in debug build !


So it's not just stripping that is no longer performed, those 3 options are no longer enabled on release builds, whereas they used to with 3.0.1 !

By the way, just received 3.2 canary 2, makes no difference. So far I cannot even go back to 3.0.1 and opened another issue because I cannot build my app with it anymore, so I'm stuck with those 3 options.

I also uninstalled NDK bundle (using SDK manager), closed/re-opened Android Studio and reinstalled NDK bundle, sync, clean, build. No difference!

This turned out to be a rather bad bug that repros only on Windows. The issue is that we failed to locate the stripping tool and silently fallback to not stripping. I fixed it for AS 3.1 beta 4 and also we will give a warning when the stripping tool isn't found. It should probably be an error instead of a warning but I didn't want to make that change so late in the 3.1 release cycle.
Thank you for reporting the bug Andros and also Raymond for debugging the issue.

Thanks, I can confirm it's fixed in 3.2 canary 4, and I could remove the extra flags I added. Now I get the full symbol output while the stripped version is included in the APK.