IssueTracker

This doesn't repro for me. Steps:
(1) Create new C++ project
(2) Run the app (this create's the binary directories
(3) Do something to cause a gradle Sync
(4) Look for files under .externalNativeBuild. They're there:
    jomof@jomof:~/projects/client/src/android/MyApplication4/app$ find "." -name "*.o"
    ./.externalNativeBuild/cmake/debug/x86/CMakeFiles/native-lib.dir/src/main/cpp/native-lib.cpp.o

Please add repro steps so I can get a better idea of what's happening.

>>  It should only delete the CMake binary directories *if and only if* parameters change in Gradle that cause cmake toolchain parameters to change (such as platform version, abi, toolchain) or parameters passed to CMake to change.
This is the way it's design to work and it is working that way for the cases I know

I realized that there were a few things we were doing that were questionable. Specifically, we set up our CMake command line params in a variable at the root build.gradle script. We were generating build dates and such in gradle and passing those down to CMake. I think things like this may be causing the CMake args to change too often, possibly resulting in this behavior. I'm in the process of eliminating a lot of this, so I'll make sure to report back the results.

So even after making my changes, I still see this behavior. My repro steps:

1. Make a change to a CMake script
2. Switch to Android Studio. I am prompted to do gradle sync, and so I do it.

On next build, a full rebuild is performed. For some reason it is not performing a delta build. Some libraries are rebuilding that should not be rebuilt. So obviously a "ninja clean" is happening somewhere. There's simply no other explanation for this.

Jomo, can you please reopen? This is still a very real problem.

Hey Robert,
From what you wrote I think this is intended behavior. If you change the CMake script then the CMake-generated Ninja project must be regenerated. Since there's a new Ninja project there needs to be a full build. There's no concept of delta build that survives changes to build command files (CMakeLists.txt or ninja build files) because any behavior difference is possible.

Most of the time, a CMakeLists.txt is modified to add, remove, or rename source files. CMake uses this list of source files as a way to know if it should regenerate to update its ninja scripts to build the new files, if new files were added.

If you do a traditional CMake build from the command line, CMake will regenerate but it will not clean your previously compiled translation units (the object files). Those remain in-place. This allows you to perform another build (e.g. execute "ninja" command) and the build tool and compiler working together will be smart enough to know which files need to be rebuilt.

From this standpoint, I disagree with your assertions. There's no functional reason AFAIC to "clean" build every time a CMake script changes. If command line parameters to CMake change in such a way that it would completely invalidate previously compiled translation units outside of the knowledge of ninja and the toolchain programs, such as changing the Android ABI via toolchain arguments, then yes you would need to clean everything and rebuild. However, beyond that, you should be relying on ninja and the compiler to perform delta builds on previously compiled translation units.

As you have it now, I am unable to do a simple task like add new CPP files without having to rebuild my entire project. This wastes about 15-20 minutes of my time (that's how long a full rebuild takes), versus less than 60 seconds to just build the new file.

I think you should aim for smarter introspection into how this mechanism works. I see two very simple scenarios:

1) A build & sync was triggered due to parameter changes from gradle (those that affect the externalNativeBuild configuration, such as native ABI changes and custom user arguments)
2) A build was triggered because CMake.exe told you that a regeneration was needed since it detected a CMakeLists.txt changed

#1 should perform a clean & rebuild
#2 should not require any special action from you; you should simply run "ninja" command as if the scripts hadn't changed at all, and let Ninja run CMake to regenerate as needed

I feel like #2 is being treated as #1 right now. Also I noticed that Android Studio wants to re-sync Gradle when a CMakeLists.txt file changed. I do not think Android Studio should be "watching" CMake scripts. It should only resync if gradle scripts changed. If a CMake script changes, let your "ninja" command handle it.

As a final note, I realize I'm probably oversimplifying this a little. I'm sure there's reasons and edge cases as to why you need that level of control over CMake. However, at some point the full clean build has to be triggered in a smarter way to avoid these giant productivity problems. You obviously know much more about how CMake integrates with Android Studio than I do; I can only offer feedback based on my direct usage of CMake outside of AS. However, I hope my feedback makes sense and is agreeable.

Just to clarify, in the logs this is the exact problem:

External native generate JSON x86Debug: - dependent build file missing or changed
External native generate JSON x86Debug: removing stale contents from 'E:\code\frontend\source\Applications\zPayService\TestTool\.externalNativeBuild\cmake\x86Debug\x86'

The "removing stale contents"... those contents aren't stale at all. A build could be executed that reuses those existing object files if all I did was modify the cmake script to add new files to compile.

Also CMake will think CMake scripts changed if you swap between git branches:

1. Go from branch A to branch B
2. Go from branch B back to A

Cmake script is still physically the same, but because you swapped branches around, the timestamp of the Cmake script changed so this triggers a full rebuild.

At this point I'd say this is our biggest issue at Ziosk and the one causing the most productivity problems. Our team of about 15 engineers is constantly complaining. There's concern on how long Google will take to address these issues, some have proposed rolling back to Eclipse + ANT. If we're a year out from any improvements I may have to consider it.

I'm prototyping a change to address this. There is a concern that CMake doesn't operate incrementally under certain circumstances. For example, changing the compiler. I think Refresh Linked C++ Projects should be enough to cover this case. I'm still looking for other cases that would be more concerning

Another kind of change to CMakeLists.txt is affected. If a build target is removed then CMake won't delete the .so and related .o files. Gradle packages *.so rather than the explicit list of .so files because build systems sometimes copy extra .so files into the output folder to be packaged.
I think I can cover this case by specific removing only unrecognized .so files during sync.

To your example: Changing compiler means changing command line arguments (e.g. `-G Ninja`) or alternatively, changing environment such as C and CXX variables. In the latter case, normal CMake does not watch for changes to environment variables, so it's the responsibility of the user (or in your case, the IDE) to monitor changes to environment and control CMake accordingly.

However, this still falls into the pattern of behavior I've outlined previously. Indirect or direct changes to Gradle or Gradle scripts that result in a different command line invocation of CMake should result in a full clean (delete binary dir + regenerate using CMake). Changes to the CMake scripts themselves shouldn't be monitored by Android Studio at all. If you have weird cases where CMake is doing work the NDK toolchain should be doing, such as lots of system introspection and runtime changes to toolchain, I'd simply document that Google does not support that usage of CMake, and all changes should go through Gradle instead (or file a bug with the NDK team and have them fix the toolchain file).

Concerning stale *.so files: What I did for my ANT integration custom targets in CMake was to delete all *.so files before running a build, that way when a build is finished, you're left with only *.so files that were copied to that destination by custom commands tied to real targets. So if a target was completely removed, its corresponding *.so file would not make it to the temporary/intermediate copy destination where you look for *.so files to package into the APK.

This solution is simple and doesn't require introspection into CMake targets. If you have more direct access to targets defined by CMake, you could keep track a list of targets found during configuration and query the OUTPUT property on each of them and correlate that to files you find in your CMake output directory. Not sure if this has much benefit over the idea above, though.

I've submitted a fix for this. Unless we find a problem the fix will appear in the next 3.1 canary

Could you explain the fix here so I (and others) can get a good understanding of how the behavior is changing as well as what the root cause (original behavior) was?

Having the same issue with Android 6.0.1 Samsung Note 4

Can you provide the below requested information to better understand the issue:

Steps to reproduce
What steps do others need to take in order to reproduce the issue themselves?
Can you please provide some applications name which has OBB files with them.

Frequency
How frequently does this issue occur? (e.g 100% of the time, 10% of the time)

- Steps to reproduce

0. Upload the app with targetSdkVersion=23 and APK Expansion files to Google Play Store via Developer Console.
1. Download the app and APK Expansion files from Google Play Store on Android 6.0
2. Launch the app and access the obb file. (XAPK validation failed error is shown to consumers)
3. Exception raised: java.io.FileNotFoundException: /storage/emulated/0/Android/obb/<package-name>/main.<expansion-version>.<packag-name>.obb: open failed: EACCES (Permission denied)
  3-1. Checking the file and directory owner, it is "root".
4. Reboot the device.
5. The file and directory owner is changed to app-specific user, e.g."u0_a131".
6. The app can access obb files correctly without runtime-permission.


- Application Name

Bumpex 2 -- (made by Edhash Games)

- Frequency
100% using Galaxy S7 and Samsung Note 4 all Android 6

Frequency here:
100% on all Android 6 devices and also on my only Android 7 device. This is a very reliable bug.

See post #11 for obb file owner/permissions before and after reboot.

Hi,

We are also experiencing this bug on both Android 6 and Android 7.

As another commenter expressed in the previous issue opened regarding this (https://issuetracker.google.com/issues/37075181), "it's a pity, because android.permission.WRITE_EXTERNAL_STORAGE considered as "Dangerous Permission" and presents scary prompt to user about photos & stuff." I think that this should be considered a very high priority bug, as it has a direct, tangible, negative impact on all apps that require expansion files.

Popping in to say that I'm also having the issue with a Samsung Galaxy S6 and several other Galaxy phones. I just spent the last 3 days troubleshooting this issue tihnking it was related to another plugin I had just installed, only to have the issue persist after rolling back to a working build that was 2 weeks old. This is a huge issue. Please.

Thank you for reporting this issue. We have shared this with our engineering team and will update this issue with more information as it becomes available.

This problem effects probably more than 80% of the apps on playstore that are larger than 100mb. We have to download the files manually after the installation or ask for read&write external permission.

After some market inspection, I can say that the lost conversion rate is around 5-10% just because of that issue. That means billions of dollars in lost revenue for both developers and the app store owners (Google).

  After 4(four) years, google realized that and decided to fix it and I find this funny, luck play a huge role in success.

  Google Play Store was my target platform for the company and now, it,s the Apple App Store.

The same issue on Samsung Galaxy Tab A (2016) SM-T585  Android 7. Build: NRD90M.T585XXU2BQE4

Permissions after install:
-rw-rw---- 1 root sdcard_rw 27515377 2017-09-04 10:13 main.4002050.com.absolutist.timegap.obb

The issue on Samsung Galaxy S7 Edge Android 7 Build:NRD90M.G9350ZHU2BQH4

The issue on following tested devices:

Samsung Galaxy S7 Edge Android 7 Build:NRD90M.G9350ZHU2BQH4

Samsung Galaxy S7 Android 6.0.1 Build:NMB29M.G930VVRU4API3

LeTV X900+ Android 6.0.1

Hi, same issue here on Samsung Galaxy S7.
Also if there is a possible workaround to downgrade target SDK level of published apk, then please let me know.
Thank you.

Issue can be reproduced on Samsung Galaxy S8 and S8+.
Nexus 6 running 7.1.1 had no issue, Nexus 6P running 8.0 (Oreo) has no issue.
None of these phones were rooted, running stock OS apart from Samsung.
Please confirm, we can reproduce this issue on S8 running 7.0 builds.
Kindly confirm this can be reproduced or not.
We can reproduce it 100% of the time.
Restarting phone solves the problem 100% of the time.
Last which was marked "fixed" and people were claiming in 7.0 solved the issue may be specific cases with phones that had no issues at all.

BUMP ! Having this issue as well.

also having this issue. S7.

I can confirm that this issue exists on the Samsung Note 8. Rebooting resets the OBB file's permissions and it can be used by the application. As none of the rest of my application requires the WRITE_EXTERNAL_STORAGE permission, this is a problem.

Also have this issue. WRITE_EXTERNAL_STORAGE permission is an ugly workaround which should not be needed for this.

We have the same problem here:
Samsung S7, Android 7.0

After installing from the Google play store (internal test channel):
at sdcard/Android/obb/com.companyname.productname:
-rw-rw---- 1 root sdcard_rw 36272558 2018-04-04 16:43 main.961.com.companyname.productname.obb

obb file cannot be accessed.

After reboot:
at sdcard/Android/obb/com.companyname.productname:
-rw-rw---- 1 u0_a947 sdcard_rw 36272558 2018-04-04 16:43 main.961.com.companyname.productname.obb

obb file can be accessed.

If I give the 'Storage' permission before starting up the app (we skip the permissions dialog) and without reboot it does work.

We'll probably go with requesting the 'Storage' permission...

The only fix that could potentially retroactively fix it for all versions, is if Google Play lifted the limit of the APK being 100mb and then we wouldn't need OBB files.

Look, Google, you want us to take the runtime permissions seriously and provide context etc.

So what is the context here?

Should I write: "We are asking for this permission because Android is a buggy mess, and Google refuses to let go of OBB files and won't let us upload single APK files as literally every other store".

How are we supposed to:
1. Provide a great experience to the users.
2. Implement permissions according to you guidelines.
3. And have the apps work properly on as many devices as possible?

Lift the APK restrictions, let OBB files die, make every developers life easier and your users happier.

I can confirm im having the exact issue and it's just destroy your mood to do games anymore.

Please take out the 100mb limit and get rid of OBB.

I've solved it asking for permissions at runtime. Android 6.
Thanks for the thread, it helped us.

Also has this.
Please take out the 100mb limit and get rid of OBB.

Also developing an Android game, and running into this issue. Asking for external write permission is al ugly fix for an ugly bug.

I guess I will just inform users to restart their phones... at least this way I don't have to ask for a dangerous permission (according to Google itself).

I don't know if this is fixed in newer Androids and that's why Google is not paying attention to it, but it's not a matter of it being fixed in the latest version.

It's a bug that affects users running Android 6 and (somewhat less frequently) Android 7. 6 and 7 are installed in approximately 50% of the available devices.

This means for the majority of Android devices out there, there is a high potential that the .obb file and folder will be set (incorrectly) as root, until the user reboots the device.

As long as these versions are around in a significant percentage (which more or less means... forever?), we are stuck asking for permissions we do not need "just in case".

It seems READ_EXTERNAL_STORAGE is needed just to be able to mount the .obb and WRITE_EXTERNAL_STORAGE is needed if we need to download the .obb ourselves.

These have scary prompts. When users ask me "why do I need to give access to my photos just to play your game", I don't know what to tell them. There is no "fixing" this in the OS, even if it's fixed in the latest version it doesn't matter, the damage is done.

Lift the limit on APKs and kill OBB files. It's the only solution and luckily it seems like a simple one.

Having this issue on Samsung Galaxy S5
Funny thing is, "force internal" (in unity build settings) doesn't solve it.
It still needs user to restart the phone, and half of the first scene's resources are not loaded. That' means there is no way to tell the user to do it.

For unity, there is a not-so-nasty workaround, to tell the user to restart his/her phone:
https://forum.unity.com/threads/apk-with-expansion-file-obb-fails-to-load-the-second-scene-until-the-phone-has-been-restarted.465714/page-2#post-4132885

Still, wish there was a better way

Please just drop obb from Android completely and raise Google Play limits to those of the App Store.... (we need 500mb+ desperately, some need 1-2gb...)
obb is an absolute mess that is falling apart, barely supported with bugs in every OEM's Android variant, heck, even the jobb tool itself is falling apart on FAT16 issues and is an unmaintained mess.

Please just save us already end the nightmare that is supporting OBB on android that is at least 75% of all problems for developers of sizeable games.

I just had the same issue on a Nokia 3 with Android 9 on it, last update july 5th 2019 (core version 3.18.119+ june 25 2019).
I recently updated a game, I package in UE4, to meet recent 64bit google requirement. I uninstalled a running version (v5) I had on my phone and installed the updated version. After making some tests on my updated version I wanted to rollback to my older v5. But now this older version does generate the XAPK Validation File error on first run. Then after rebooting the phone it works fine again.
What's really intriguing is that I perfectly remember that I did not face this issue, installing this 5th version or the previous 4 straight into my phone or using the Playstore. Always Ok on first run. This was 7 months ago, back in January 2019.

We'll die before they fix this. It's been like this for half a decade already. Clearly no intention of fixing it.

Move it, Roach!

Hello, your comments were flagged for review. We understand that you’re frustrated, but please keep your language civil. If you continue to post abusive statements to the Google Issue Tracker, we will be forced to take action by removing your ability to make any comments or changes on the Google Issue Tracker.

We are closing this bug as Obsolete. If you still think this issue is reproducible and relevant in the latest Android release then please open a new bug and add a new bug report and other relevant information and reference this bug for context.

File use for ff

Please obb resources file my gmail send me