Scanning files to index, out of memory. [122874775]

Fixed

Bug

Status Update

No update yet.

Description

an...@supercell.com

created issue #1

Jan 15, 2019 11:48AM

Trying to open large project, scanning files to index never ends and after running for a while, idea log shows out of memory error every 2 seconds.

There was also svn error, that said that my home folder was not in svn. Not sure if it tries to index my whole home folder now..

Everything works fast if I use experimental use active configuration only setting, but then I cant see my cpp files :(

Build: 3.3, AI-182.5107.16.33.5199772, 201812250239,

AI-182.5107.16.33.5199772, JRE 1.8.0_152-release-1248-b01x64 JetBrains s.r.o, OS Mac OS X(x86_64) v10.14.1 unknown, screens 2560x1440; Retina

Android Gradle Plugin: 3.3.0
Gradle: 4.10.2
NDK: from local.properties: 18.1.5063045; latest from SDK: 18.1.5063045;
LLDB: LLDB 3.1 (revision: 3.1.4508709)
CMake: from local.properties: (not specified); latest from SDK: 3.6.0-rc2; from PATH: (not found);
Source: user_sentiment_feedback

IMPORTANT: Please read

https://developer.android.com/studio/report-bugs.html carefully and supply all required information.

Comments

ma...@gmail.com <ma...@gmail.com> #2Jan 15, 2019 02:16PM

Still crashing with Compose 1.7.4.
I also tried moving sharedElement to BoxWithConstraints (rather than in its content), but did not help.
So the layout hierarchy looks like this (with more details and with "..." I replaced series of "normal" layouts):
NavHost -> ... -> AnimatedVisibility -> ... -> BoxWithConstraints with sharedElement -> some content
\> ... -> AnimatedVisibility -> ... -> LazyList -> ... -> Animated Visibility -> Selection container -> BoxWithConstraints with sharedElement -> some content

Any ideas?

ma...@gmail.com <ma...@gmail.com> #3Jan 15, 2019 04:02PM

In your BoxWithConstraints do you do any conditional composition? Things like:

if (constraintsExceedsCertainThreshold) composeSomeContent() else someOtherContent()

br...@gmail.com <br...@gmail.com> #4Jan 15, 2019 06:04PM

No, it's just a `Text` which has different style depending on constraints.
Also note `sharedElement` is put on `BoxWithContraints` now, not on the content.
Thanks!

br...@gmail.com <br...@gmail.com> #5Jan 15, 2019 06:07PM

Based on the stacktrace, it seems like the item in question is composed but never placed. Do you use custom precompose or prefetch logic in your lazy list?

jo...@google.com <jo...@google.com> #6Jan 16, 2019 04:58PM

No, nothing custom. Just the default prefetch logic.
There is also one `AnimatedContent` above the `BoxWithConstraints`. Could this cause issues?
Thank you for looking into this.

jo...@google.com <jo...@google.com> #7Jan 16, 2019 05:02PM

The AnimatedContent never skips measuring/placing children after it composes them. So I don't expect that to be the issue.

Since you mentioned the modifier is on BoxWithConstraints rather than on the children of BoxWithConstraints, my guess is it's most likely related to LazyList.

Can you clarify what type of LazyList you are using? Is it LazyColumn/Row or LazyGrid, or something else?

ex...@gmail.com <ex...@gmail.com> #8Jan 16, 2019 05:34PM

Makes sense, thanks.
It's a LazyColumn.

Message last modified on Jan 16, 2019 05:43PM

tg...@google.com <tg...@google.com> #9Jan 16, 2019 05:53PM

One interesting observation is that this stack trace did not seem to come from the ComposeView with our app's Activity content. Because I did a release with copy pasted setContent + ComposeView subclass that is catching exceptions in dispatchDraw and it did not catch this crash. Is the animations framework doing content composition outside the app's composition context? One note: at the time I tried this exception catching, sharedElement was applied to the content of BoxWithConstraints (rather than to the BoxWithConstraints itself), but I don't think this would have caused the exception catching not to work.

ma...@gmail.com <ma...@gmail.com> #10Jan 16, 2019 05:59PM

Hey, sorry to insist - any more ideas here?

uc...@google.com <uc...@google.com> Jan 16, 2019 07:08PM

Assigned to ar...@google.com.

jo...@google.com <jo...@google.com> #11Jan 16, 2019 08:15PM

I'm investigating what would cause items to be composed but not placed.

We'll submit a fix in shared element in 1.8 to account for that scenario if that's an intended behavior.

Have you tried a recent 1.8 release and see if it's reproducible?

ma...@gmail.com <ma...@gmail.com> #12Jan 16, 2019 09:17PM

Thank you!

Unfortunately we cannot release publicly with 1.8 alpha, and we never reproed this issue locally even with 1.7.x.

ma...@gmail.com <ma...@gmail.com> #13Jan 16, 2019 09:28PM

I'll prepare a fix to account for the use case where content may be composed but never participate in either measurement or placement. Keep in mind, the fix likely will only land in 1.8.

an...@supercell.com <an...@supercell.com> #14Jan 17, 2019 07:03AM

Awesome, thank you! If there's any chance to have it in 1.7.6, would be absolutely amazing, but no worries if not. Thanks a lot again for the help!

ex...@gmail.com <ex...@gmail.com> #15Jan 17, 2019 04:05PM

Project: platform/frameworks/support
Branch: androidx-main
Author: Doris Liu <tianliu@google.com>
Link: https://android-review.googlesource.com/3349070

Require shared element to draw in place before drawing in overlay

Expand for full commit details

Require shared element to draw in place before drawing in overlay 
 
This change fixes a rare issue where a shared element is composed 
in both incoming and outgoing content, but measured and placed in 
neither. The fix involves checking the shared element has been 
drawn for at least frame before drawing it in the overlay. This 
accounts for not only the requirement to measure and place shared 
elements, but also ensured that when the draw is skipped by 
a parent, shared element will not draw. 
 
Fixes: 371802474 
Test: New test added 
Change-Id: I35108ce255cf4728a9ca3ef7834d6962abcc94c0

Files:

M compose/animation/animation/src/androidInstrumentedTest/kotlin/androidx/compose/animation/SharedTransitionTest.kt
M compose/animation/animation/src/commonMain/kotlin/androidx/compose/animation/SharedContentNode.kt
M compose/animation/animation/src/commonMain/kotlin/androidx/compose/animation/SharedElement.kt

Hash: baae6769d3323d141b17e0d916323cf6487034ea
Date: Wed Nov 13 15:27:39 2024

Message last modified on Jan 17, 2019 04:06PM

jo...@google.com <jo...@google.com> #16Jan 17, 2019 04:15PM

The following release(s) address this bug.It is possible this bug has only been partially addressed:

androidx.compose.animation:animation:1.8.0-alpha07
androidx.compose.animation:animation-android:1.8.0-alpha07
androidx.compose.animation:animation-jvmstubs:1.8.0-alpha07
androidx.compose.animation:animation-linuxx64stubs:1.8.0-alpha07

ar...@google.com <ar...@google.com> Jan 17, 2019 05:04PM

Reassigned to tg...@google.com.

np...@gmail.com <np...@gmail.com> #17Jan 17, 2019 08:57PM

Same situation and same problem with 3.3: we've got cpp files outside of (C:\programs\projectName\src) our Android Studio project structure (C:\programs\projectName\build\android). Android Studio indexes the entire drive and runs out of memory while "loading symbols" and crashes.

In the app.iml file, it looks like Android Studio has added C:\ as a sources root. Manually changing that to C:\programs\projectName and restarting Android Studio causes the first round of indexing to complete quickly and behave normally for a few seconds, after which it overwrites the file almost immediately with the same C:\ sources root again, after which it enters the same indexing loop forever.

tg...@google.com <tg...@google.com> #18Jan 17, 2019 09:10PM

Hi Nicholas, do you use cmake or ndk-build for your native code? Can you provide more information on the structure of your project? Specifically how are the cpp files referenced? Thank you!

np...@gmail.com <np...@gmail.com> #19Jan 18, 2019 01:16AM

We use ndk-build. This is on a Windows 8.1 machine. Here are the (lightly sanitized) gradle and mk files:

https://www.dropbox.com/s/wl826flzufuzldq/androidBuild.zip?dl=0 Importing those from scratch caused the same problem several times in a row.

There are a couple absolute paths in there. I worried that might have been causing the "C:\" sources root, but even making them relative paths (and importing from scratch again) didn't fix it.

[Deleted User] <[Deleted User]> #20Jan 20, 2019 08:26AM

Same here, but on macOS High Sierra, Android Gradle Plugin 3.3.0, Gradle: 4.10.1, NDK 18.1.5063045, using ndk-build.

Scanning files to index was taking hours, so I left it overnight. In the morning I find the Android Studio process had read 185.35 GB from the disk and is now asking for more RAM. It either tries to read my whole disk (which is about 400 GB), or reads the same data over and over again. I'm not going to give this thing more RAM to do more of what it does now.

I never imagined that it can get any worse than Android Studio 3.2, but 3.3 is plain unusable.

an...@supercell.com <an...@supercell.com> #21Jan 21, 2019 11:30AM

I tried to create 5 levels of empty directories before the actual project directory, but it did not help. Still stuck on scanning files.

But, I discovered that creating a disk image (MacOS with Diskutility), copied project there and opened. Now the scanning phase completed. Tested with two projects that got stuck when working under home direcotry. Both worked this way.

jo...@google.com <jo...@google.com> #22Jan 21, 2019 05:29PM

Thank you for the repro case Nicholas it was extremely helpful. I have a change in flight for 3.3.1 that should fix it.
I don't have a workaround for you but this happens in the case that ndk-build is used and a target builds sources from NDK's android_app_glue along with sources in project folder. In this case, Android Studio incorrectly finds the common ancestor folder, C:\. There may be other ways to hit the bug but that's what's happening in your case.

an...@supercell.com <an...@supercell.com> #23Jan 23, 2019 11:01AM

Getting this stack trace also. Seems to be related.

java.lang.Throwable: The file '/' is not under content entry root '/Applications/Android Studio.app/Contents/bin'
at com.intellij.openapi.diagnostic.Logger.error(Logger.java:126)
at com.intellij.openapi.roots.impl.ContentEntryImpl.assertFolderUnderMe(ContentEntryImpl.java:378)
at com.intellij.openapi.roots.impl.ContentEntryImpl.assertCanAddFolder(ContentEntryImpl.java:290)
at com.intellij.openapi.roots.impl.ContentEntryImpl.assertCanAddFolder(ContentEntryImpl.java:285)
at com.intellij.openapi.roots.impl.ContentEntryImpl.addSourceFolder(ContentEntryImpl.java:209)
at com.intellij.openapi.roots.impl.ContentEntryImpl.addSourceFolder(ContentEntryImpl.java:216)
at com.intellij.openapi.roots.impl.ContentEntryImpl.addSourceFolder(ContentEntryImpl.java:202)
at com.intellij.openapi.roots.impl.ContentEntryImpl.addSourceFolder(ContentEntryImpl.java:195)
at com.android.tools.ndk.GradleWorkspace.lambda$updateContentRoots$7(GradleWorkspace.java:399)
at com.intellij.openapi.roots.ModuleRootModificationUtil.updateModel(ModuleRootModificationUtil.java:143)
at com.android.tools.ndk.GradleWorkspace.updateContentRoots(GradleWorkspace.java:374)
at com.android.tools.ndk.GradleWorkspace.lambda$null$0(GradleWorkspace.java:302)
at com.intellij.openapi.roots.impl.ProjectRootManagerImpl.mergeRootsChangesDuring(ProjectRootManagerImpl.java:295)
at com.android.tools.ndk.GradleWorkspace.lambda$null$2(GradleWorkspace.java:301)
at com.intellij.openapi.application.impl.ApplicationImpl.runWriteAction(ApplicationImpl.java:1038)
at com.android.tools.ndk.GradleWorkspace.lambda$updateGradleWorkspace$3(GradleWorkspace.java:297)
at com.intellij.openapi.application.TransactionGuardImpl$2.run(TransactionGuardImpl.java:315)
at com.intellij.openapi.application.impl.LaterInvocator$FlushQueue.doRun(LaterInvocator.java:447)
at com.intellij.openapi.application.impl.LaterInvocator$FlushQueue.runNextEvent(LaterInvocator.java:431)
at com.intellij.openapi.application.impl.LaterInvocator$FlushQueue.run(LaterInvocator.java:415)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:762)
at java.awt.EventQueue.access$500(EventQueue.java:98)
at java.awt.EventQueue$3.run(EventQueue.java:715)
at java.awt.EventQueue$3.run(EventQueue.java:709)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:80)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:732)
at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:817)
at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:758)
at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:394)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)

74...@gmail.com <74...@gmail.com> #24Jan 23, 2019 04:15PM

Seems related to this issue

https://issuetracker.google.com/123131649

Somehow the home directory is being added as a source root:

<component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_7">
<output url="file://$MODULE_DIR$/build/intermediates/javac/debug/compileDebugJavaWithJavac/classes" />
<output-test url="file://$MODULE_DIR$/build/intermediates/javac/debugUnitTest/compileDebugUnitTestJavaWithJavac/classes" />
<exclude-output />
<content url="file://$USER_HOME$">
<sourceFolder url="file://$USER_HOME$" isTestSource="false" /> <------ Bad
....

Which forces the entire contents of the user's home directory to be scanned and indexed. We are using C++ libraries outside of the project's root - namely prebuilt Conan packages located in ~/.conan. Is there any workaround?

qu...@gmail.com <qu...@gmail.com> #25Jan 29, 2019 07:00AM

A workaround that I have found is that when I see that is indexing "C:\" in the "Project Source Files" pane. I go to the app.iml file and remove the content tag that has the "C:\" and save the file. That does that it stops the indexation of "C:\" and continue with the correct source directories.

But that makes that when the indexation "ends", it gets "stuck" (actually looks like is loading something, but never ends), in order to solve that I go to the Gradle pane and refresh the gradle project. When it is finally done, the indexing also ends and Android Studio allows me to build the project.

tg...@google.com <tg...@google.com> Jan 29, 2019 05:43PM

Reassigned to jo...@google.com.

jo...@google.com <jo...@google.com> Feb 2, 2019 02:59AM

Marked as fixed.

ma...@gmail.com <ma...@gmail.com> #26Feb 6, 2019 10:13PM

This particular issue is confirmed as fixed in 3.3.1. The scanning and indexing is still an order of magnitude slower than 3.2.1 (as is Android Studio overall), but at least now it completes in minutes vs. hours and doesn't crash or run out of memory anymore as per this ticket.

ar...@google.com <ar...@google.com> #27Feb 6, 2019 11:22PM

Thanks for verifying!
Can you provide more context around the scanning and indexing slowness you observed with 3.3.1 compared to 3.2.1?
Is it the same as

https://issuetracker.google.com/issues/121340427 or something else?
Thanks!

ro...@gmail.com <ro...@gmail.com> #28Apr 30, 2019 02:56PM

Although the bug is fixed, I still experiment the same problem in Android Studio 3.2.2 and 3.4.

When opening a project with a large amount of .h files (webrtc library), It takes around 10 mins in the indexing phase (long time but ok). The problem comes in the Building Symbols stage, it eats all the RAM avaiable and ends with a Low memory warning that makes the IDE unusable.
You can see in the attached screenshot where I configured the IDE to use 8Gb of RAM. If you increase the value it doesn't solve anything, it just does not stop consiming RAM until it hits the limit, no matter how high it is.

3.2 works ""ok"" (at least it doesn't freeze, but no code completion, syntax highlighting... )

Is there any workaround that could mitigate the issue?

Screenshot 2019-04-30 at 13.07.56.png

23 KB

View

Download

ar...@google.com <ar...@google.com> #29Apr 30, 2019 07:58PM

Hi, can you give us some idea about the scale of the number of .h files in your library?
And if it's open source, where we can try to repro it ourselves, that would be even better.

[Deleted User] <[Deleted User]> #30May 2, 2019 11:37PM

This bug is still showing in 3.4......

ro...@gmail.com <ro...@gmail.com> #31May 3, 2019 09:14AM

comment #29

Thanks for answering!

Our project has around 60.000 .h/.hpp files.
Unfortunately our project is not open source, however, our biggest dependency is webrtc which is.

I would say that if you create a project that depends on webrtc, you will see the same problem.

tg...@google.com <tg...@google.com> #32May 3, 2019 07:18PM

Hi, I tried downloading the WebRTC source code following instructions from

https://webrtc.org/native-code/android/. There seems to be no CMakeLists.txt in the project and there are ~2300 headers in it. Do you create custom CMakeLists in order to include WebRTC as a dependency for your project?

Also, do you by any chance use Android Studio to open the generated gradle file in step

https://webrtc.org/native-code/android/#using-android-studio?

ho...@gmail.com <ho...@gmail.com> #33Jun 5, 2019 11:07PM

Hello, I am experiencing the same problem on 3.4. My project has around 2,000 .h, .c, and .cpp files. They are located across many different paths on my hard drive. Is there any progress on this issue? Thank you.

ar...@gmail.com <ar...@gmail.com> #34Jul 15, 2019 12:03PM

Maybe this issue related to this one

https://issuetracker.google.com/issues/131854856 ?

Message last modified on Jul 15, 2019 12:03PM

vi...@gmail.com <vi...@gmail.com> #35Oct 26, 2020 05:35PM

This bug is not fixed on Android Studio 4.1.
Same behaviour. Same setup. OS X, c++

Found SO question that describes this as well on 4.1.

https://stackoverflow.com/questions/64401439/

an...@supercell.com <an...@supercell.com> #36Oct 27, 2020 08:04AM

Yes, I have exactly the same issue happening also again with 4.1. With 4.2 it works again.

mi...@gmail.com <mi...@gmail.com> #37Oct 30, 2020 08:22AM

Exact same issue. It is paralyzing for all ndk c++ game developers. AS 4.1. Thanks

jo...@google.com <jo...@google.com> #38Feb 17, 2021 09:44PM

This is fixed in 4.2 and later

ph...@googlemail.com <ph...@googlemail.com> #39Jan 28, 2022 12:33AM

Comment has been deleted.

Message last modified on Jan 29, 2022 12:30AM

ka...@gmail.com <ka...@gmail.com> Jul 5, 2023 07:33AM

deleted

Restricted+

0 B