IssueTracker

For the statement an app that mostly works in background does not mean its unimportant for the user, actually if it's important to the user, it should try to stay in the foreground, i.e., using a foreground service. We're squeezing the capability of running arbitrary amount of time for background apps. Look at the app restrictions since Android O, i.e., background services will be stopped by the system after 60s. So if an app is really offering critical functions to the user, it should try to stay in the foreground.

For the phantom processes, it's really a grey area, where the system does allow this kind of processes, but it's not formally supported, and it's not tracked by the system previously - it's going to hurt the overall system performance, as in resource-limited environment, other processes could be starving because of it. Thus we can't let the number of such kind of processes grow arbitrarily. The 32 may or may not be a good number, at least it's in par with the max cached processes, it's hard to justify what number is the perfect one though. I'd also be un-favor with the idea of introducing the per-package limit - from the system health point of view, it doesn't make sense to me.

For the battery optimizations disabled, this is orthogonal to the problem here I think. The battery optimization is more about the controls about background activities like Jobs/Alarms which could impact the battery negatively, while the problem here is more about other system resources such as CPU time and RAM.

For the cases like McAfee Security app, as you also show, the order of the killing is basically based on the parent process's oom adj score - if you app is killed before McAfee, that means it has a better oom adj score.

it should try to stay in the foreground, i.e., using a foreground service.

By background, I meant not visible like a social media app, and minimized to notifications. The TermuxService already is a foreground service and still its processes are getting killed. It normally has an oom adj of 200, while McAfee app also is a foreground service, but is usually at 100 oom adj, likely because of additional connections, and hence it survives, while Termux does not. Normally the heuristic of giving priority to McAfee would be fine-ish, but the heuristic is still just a guess at what is more important. What the OS's guess is of what's more important is not always what's more important for the user. An OS can't decide such a case. I ask a user, what he would rather have killed?, the Termux app that has important on-going work for him, which has session data that will be lost if its killed and can't be reproduced or like a backup operation going on that will have to restarted from scratch, costing with more CPU + battery + network bandwidth usage later, or would he rather have the McAfee app killed, that can just be restarted later and continue its monitoring again, without permanent data loss, since scans are reproducible. Even if loss was permanent for the later case, it may not be important to the user. I'm pretty sure the answer will be Termux for lot of users, if not most. I don't think one would hear many complaints of "Oh no, my antivirus got restarted and its scan data got lost!". Wouldn't it be better that the user gets to decide what the OS obviously can't decide. Moreover, as an app developer, the only choice would be to create useless connections to try to lower the oom odj to compete with MacAfee app, which would then waste OS resources and even that would be playing russian roulette.

The 32 may or may not be a good number, at least it's in par with the max cached processes, it's hard to justify what number is the perfect one though. I'd also be un-favor with the idea of introducing the per-package limit - from the system health point of view, it doesn't make sense to me.

How about, since 32 may not always be a good number, or will be different for each device or even each user, add a configurable option in android settings, or perhaps developer settings, like it is for cached processes, with a disable completely option as well for power users. Although, this doesn't solve the excessive usage killing issue.

it's going to hurt the overall system performance, as in resource-limited environment

I have a LG G5, with Snapdragon 820 and 4GB RAM + 2GB swap in /data partition, so its an old device but it runs apps relatively fine even today, with shell processes running through out the day. Some shell processes do take extra CPU and RAM usage, like daily compression+encryption backups, but those last a few minutes, and I as a user, I am completely fine with getting slow downs for a few minutes and don't want OS to kill such processes. I also have longer backups to the cloud, that run for over an hour during the night, with even more CPU usage, and they run without causing any user experience loss since I am not even using the phone and its on charging so battery is not an issue either. With the new design such things are not possible reliably. One would not want to wake up and find out that the OS decided to randomly kill their backup process during the night to save resources. I also use Activity State Changes Tasker Project that monitors every activity transition I do (in user space, not OS), and which happen quite often. Transitions are being monitored by multiple logcat processes and each transition runs dumpsys command as well to check fullscreen mode. I have never noticed any noticeable lag because of all that, in fact I have to put delays of like 1-2s because the tasks are much faster than required. The Tasker app barely even shows in battery usages in android settings. Other users on r/tasker have hundreds of event based tasks running all the time without any noticeable performance loss or battery usage.

So the argument that phantom processes will harm user experience is not always true. The r/tasker and r/termux users have actively been using such processes without issue for years. Moreover, every user may not even want an even faster OS/device, but a reliable one that is actually useful for his daily work, instead of just being faster. Devices are fast enough today, with batteries large enough too.

I am also not sure how many android OS devs experiment with such stuff in their daily life, who can provide more usable stats on this, even I don't get much time now. Do you guys have any such stats collected from such devs or done as part of development itself, like how killing normal phantom processes (not rogue) of apps like Termux and Tasker actually noticeably improved user experience and system performance for other apps. Those would be interesting to look at.

Yes, I completely agree that phantom processes can decrease system performance in some cases, specially like the case above with McAfee, but the user should get to decide if he is fine with a certain app causing a performance loss, hence the request for an exemption. Moreover, the current design in fact trimmed rogue logcat processes from 100+ to 32, but it also killed the bash process that wasn't even using much CPU and was useful to the user and the design prevented termux from actually running anything, so from a user perspective, the collateral damage is too high.

The battery optimization is more about the controls about background activities like Jobs/Alarms which could impact the battery negatively, while the problem here is more about other system resources such as CPU time and RAM.

Okay, fair enough. But from a user point of view, more CPU usage would be proportional to battery usage and it would have been easier to use the same permission. But if you want to introduce another permission "Disable resource optimizations", that would be acceptable too. Moreover, the apps that already had battery optimization disabled would automatically also get to use CPU unconditionally and bypass phantom process restriction if same permission was used, so that is probably not something one may want and a separate permission is likely more ideal.

The permission check could be added to ActivityManagerService.checkExcessivePowerUsage() before monitoring CPU usage and before bookeeping a phantom process in PhantomProcessList.getOrCreatePhantomProcessIfNeededLocked(). It would be really great if such an exemption was added for specific apps. The power users would be really grateful that they get to live another day and enjoy their device like they want to.

In terms of the oom adj score McAfee vs. Termux, I guess McAfee has a binding from a higher priority processes, such as the system. As you know, the Android OS doesn't have the intelligence to determine which one is important to the user - it has no idea what this app is for, an user could say the Termux is more important to them, while the other user could say McAfee is more important. It's not scalable for the Android OS to ask user in terms of the importance for every single app. Remember, user perception/interaction is a resource too.

The number 32 is AOSP definition, OEMs could alter this number if they like, there is no any hard requirement on this. I'd say we could explore the option to add a configuration in the developer options for it. But again, this kind of thing is never formally supported as an Android feature, I'm not sure if this Settings makes sense in this context.

However I'd disagree to associate it with the battery optimization settings, as aforementioned, they're orthogonal problems. Actually for most of the users, they don't have an idea about what the turning off battery optimization would actually mean to them. Apps without battery restrictions, don't mean they could do whatever they want. All of the Android OS users, should be protected from buggy or malicious apps - we do have seen buggy apps with native executables eating up the system resources. I'd also disagree with the statement Devices are fast enough today, with batteries large enough too, it's never our goal of stopping making faster device and longer battery life.

has a binding from a higher priority processes

Yes, that's what I was referring to, check the ProcessRecord entries which were posted in linked issue.

McAfee ProcessRecord

$ adb shell "/system/bin/dumpsys activity processes | grep -E -A 50 'ProcessRecord\{.*com.wsandroid.suite'"
  *APP* UID 10164 ProcessRecord{24e34c1 3461:com.wsandroid.suite/u0a164}
    user #0 uid=10164 gids={3002, 3003, 3001, 50164, 20164, 9997}
    mRequiredAbi=arm64-v8a instructionSet=arm64
    class=com.wsandroid.suite.MMSApplication
    manageSpaceActivityName=com.wavesecure.activities.ManageSpaceActivity
    dir=/data/app/~~iWO3mRIXmB_uMRsr0gJvYw==/com.wsandroid.suite-IQCqZvqHzO_TB7Dqs4PPsw==/base.apk publicDir=/data/app/~~iWO3mRIXmB_uMRsr0gJvYw==/com.wsandroid.suite-IQCqZvqHzO_TB7Dqs4PPsw==/base.apk data=/data/user/0/com.wsandroid.suite
    packageList={com.wsandroid.suite}
    packageDependencies={com.google.android.webview}
    compat={440dpi}
    thread=android.app.IApplicationThread$Stub$Proxy@2d1b295
    pid=3461
    lastActivityTime=-1m3s99ms
    startSeq=41
    mountMode=DEFAULT
    lastPssTime=-1m14s752ms pssProcState=4 pssStatType=1 nextPssTime=+58m44s597ms
    lastPss=210MB lastSwapPss=41MB lastCachedPss=0.00 lastCachedSwapPss=0.00 lastRss=220MB
    trimMemoryLevel=0
    procStateMemTracker: best=1 (1=1 Infinityx)
    lastRequestedGc=-15h6m1s363ms lastLowMemory=-15h10m20s35ms reportLowMemory=false
    reportedInteraction=false fgInteractionTime=-58s379ms
    adjSeq=647074 lruSeq=194598
    oom adj: max=1001 curRaw=100 setRaw=100 cur=100 set=100
    mCurSchedGroup=2 setSchedGroup=2 systemNoUi=false
    curProcState=4 mRepProcState=4 setProcState=4 lastStateTime=-2d8h39m55s697ms
    curCapability=LCMN setCapability=LCMN
    allowStartFgsState=4
    hasShownUi=true pendingUiClean=true
    cached=false empty=true
    notCachedSinceIdle=true initialIdlePss=81684
    lastTopTime=-58s378ms
    lastInvisibleTime=2021-11-01 20:53:45.546 (-58s385ms)
    hasStartedServices=true
    mHasForegroundServices=true forcingToImportant=null
    Services:
      - ServiceRecord{15396e2 u0 com.wsandroid.suite/com.mcafee.monitor.MMSAccessibilityService}
      - ServiceRecord{45bf48f u0 com.wsandroid.suite/com.mcafee.messaging.google.GoogleFCMMessageReceiver}
      - ServiceRecord{a1e649e u0 com.wsandroid.suite/com.mcafee.sdk.wp.core.siteadvisor.service.SiteAdvisorService}
      - ServiceRecord{c0ba211 u0 com.wsandroid.suite/com.mcafee.broadcast.MMSSystemBroadcastListencerService}
    mConnections:
      - ConnectionRecord{1036c33 u0 CR com.wsandroid.suite/org.chromium.content.app.SandboxedProcessService0:0:@1ff27a2}
      - ConnectionRecord{109c957 u0 CR WACT CAPS com.google.android.gms/.chimera.PersistentDirectBootAwareApiService:@305f2d6}
      - ConnectionRecord{6cc1bee u0 CR WPRI com.wsandroid.suite/org.chromium.content.app.SandboxedProcessService0:0:@ba02569}
      - ConnectionRecord{b9567ee u0 CR IMP com.wsandroid.suite/com.mcafee.messaging.google.GoogleFCMMessageReceiver:@3cf2169}
    Published Providers:
      - androidx.lifecycle.ProcessLifecycleOwnerInitializer
        -> ContentProviderRecord{bf09faa u0 com.wsandroid.suite/androidx.lifecycle.ProcessLifecycleOwnerInitializer}
      - com.mcafee.core.provider.KeyValueDataProvider
        -> ContentProviderRecord{d95f79b u0 com.wsandroid.suite/com.mcafee.core.provider.KeyValueDataProvider}
      - com.mcafee.core.provider.JSONDataContentProvider
        -> ContentProviderRecord{72dd738 u0 com.wsandroid.suite/com.mcafee.core.provider.JSONDataContentProvider}
      - com.moengage.core.internal.storage.database.MoEProvider

Termux ProcessRecord

$ adb shell "/system/bin/dumpsys activity processes | grep -E -A 50 'ProcessRecord\{.*com.termux'"
  *APP* UID 10383 ProcessRecord{f26c83c 18144:com.termux/u0a383}
    user #0 uid=10383 gids={3003, 50383, 20383, 9997}
    mRequiredAbi=arm64-v8a instructionSet=arm64
    class=com.termux.app.TermuxApplication
    dir=/data/app/~~gQtDuMCK77ed6DPdCmeWAA==/com.termux-MfgXPeqANrvwr29oJKqfNA==/base.apk publicDir=/data/app/~~gQtDuMCK77ed6DPdCmeWAA==/com.termux-MfgXPeqANrvwr29oJKqfNA==/base.apk data=/data/user/0/com.termux
    packageList={com.termux}
    compat={440dpi}
    thread=android.app.IApplicationThread$Stub$Proxy@f4fbbd
    pid=18144
    lastActivityTime=-53s621ms
    startSeq=4028
    mountMode=DEFAULT
    lastPssTime=-59s212ms pssProcState=4 pssStatType=1 nextPssTime=+4m4s71ms
    lastPss=20MB lastSwapPss=379KB lastCachedPss=0.00 lastCachedSwapPss=0.00 lastRss=57MB
    trimMemoryLevel=0
    procStateMemTracker: best=1 (1=1 7.59375x)
    lastRequestedGc=-9m30s398ms lastLowMemory=-9m30s398ms reportLowMemory=false
    reportedInteraction=false fgInteractionTime=-50s903ms
    adjSeq=647062 lruSeq=194591
    oom adj: max=1001 curRaw=200 setRaw=200 cur=200 set=200
    mCurSchedGroup=2 setSchedGroup=2 systemNoUi=false
    curProcState=4 mRepProcState=4 setProcState=4 lastStateTime=-9m30s394ms
    curCapability=-CMN setCapability=-CMN
    allowStartFgsState=4
    hasShownUi=true pendingUiClean=true
    cached=false empty=true
    lastTopTime=-50s903ms
    lastInvisibleTime=2021-11-01 20:53:37.304 (-50s903ms)
    hasStartedServices=true
    mHasForegroundServices=true forcingToImportant=null
    Services:
      - ServiceRecord{ecab86c u0 com.termux/.app.TermuxService}
    mConnections:
      - ConnectionRecord{44ab2ca u0 com.termux/.app.TermuxService:@99e35}
    Published Providers:
      - com.termux.filepicker.TermuxDocumentsProvider
        -> ContentProviderRecord{e30f5b2 u0 com.termux/.filepicker.TermuxDocumentsProvider}
      - com.termux.app.TermuxOpenReceiver$ContentProvider
        -> ContentProviderRecord{f610903 u0 com.termux/.app.TermuxOpenReceiver$ContentProvider}
    Connected Providers:
      - 8f1e68c/com.android.providers.settings/.SettingsProvider->18144:com.termux/u0a383 s1/1 u0/0 +9m30s337ms
    lastCompactTime=0 lastCompactAction=0
    isFreezeExempt=false isPendingFreeze=false isFrozen=false
    Activities:
      - ActivityRecord{aa40b10 u0 com.termux/.app.TermuxActivity t270}
    Recent Tasks:
      - Task{eb62c28 #270 type=standard A=10383:com.termux U=0 visible=false mode=fullscreen translucent=true sz=1}
    BoundClientUids:[10383]
     Configuration={1.0 310mcc150mnc [en_US,es_US,nb_NO] ldltr sw392dp w392dp h767dp 440dpi nrml long port night finger -keyb/v/h -nav/h winConfig={ mBounds=Rect(0, 0 - 1080, 2220) mAppBounds=Rect(0, 0 - 1080, 2176) mMaxBounds=Rect(0, 0 - 1080, 2220) mWindowingMode=fullscreen mDisplayWindowingMode=fullscreen mActivityType=undefined mAlwaysOnTop=undefined mRotation=ROTATION_0} as.2 s.6012 fontWeightAdjustment=0}
     OverrideConfiguration={1.0 310mcc150mnc [en_US,es_US,nb_NO] ldltr sw392dp w392dp h767dp 440dpi nrml long port night finger -keyb/v/h -nav/h winConfig={ mBounds=Rect(0, 0 - 1080, 2220) mAppBounds=Rect(0, 0 - 1080, 2176) mMaxBounds=Rect(0, 0 - 1080, 2220) mWindowingMode=fullscreen mDisplayWindowingMode=fullscreen mActivityType=standard mAlwaysOnTop=undefined mRotation=ROTATION_0} as.2 s.1 fontWeightAdjustment=0}
     mLastReportedConfiguration={1.0 310mcc150mnc [en_US,es_US,nb_NO] ldltr sw392dp w392dp h767dp 440dpi nrml long port night finger -keyb/v/h -nav/h winConfig={ mBounds=Rect(0, 0 - 1080, 2220) mAppBounds=Rect(0, 0 - 1080, 2176) mMaxBounds=Rect(0, 0 - 1080, 2220) mWindowingMode=fullscreen mDisplayWindowingMode=fullscreen mActivityType=undefined mAlwaysOnTop=undefined mRotation=ROTATION_0} as.2 s.6012 fontWeightAdjustment=0}

an user could say the Termux is more important to them, while the other user could say McAfee is more important. I

Yes, exactly. That is exactly the situation a permission would help in. One would grant permission to Termux, the other to McAfee. User decides for android on who to prioritize. If both have permission, then oom adj can be used as fallback. Of course, that situation wouldn't occur in context of phantom processes, since phantom process records wouldn't be tracked if permission was granted.

It's not scalable for the Android OS to ask user in terms of the importance for every single app.

Android does not need to ask. Only a runtime setting needs to exist and apps that are phantom process oriented can ask the user to grant them. Battery optimization permission exists too after all and so does draw over apps and these are already being used by such apps without complaints from users, other than OEM killers of course.

I'd say we could explore the option to add a configuration in the developer options for it.

That would be great and at least would be something. However, a permission exemption would still be better a way in my opinion because if its based on permission, then only specific apps will get to run phantom processes that were explicitly granted to do so and possibly trusted to be reliable. The other apps would still be restricted to the 32 limit and issue of rogue/buggy processes wasting system would be handled as well, which will improve user experience and performance, which is apparently your desired goal. If a configurable limit is used, then it would apply to all apps, and the rogue apps would be allowed to waste resources and users increasing the limit will suffer. But if the permission is not going to be added, then a configuration settings would be greatly appreciated, since at least adb or root will not be required, which would be better for users and devs.

Note that com.google.android.gms overrides the device config received from google servers after BOOT_COMPLETED event, as mentioned in the Why device config may get reset? section, so that case would need to be handled too, so that user setting is not overridden. Currently, that is also the issue with setting max_phantom_processes to 2147483647, since config gets overridden at boot, and user needs to set max_phantom_processes after that happens on every boot or has to disable config sync, which is of course not advisable for normal users. Moreover, even max_cached_processes value set in developer options does not survive reboots, since mOverrideMaxCachedProcesses gets set to -1 at boot. So asking user to set a value on every boot from developer options is not good from a usability perspective.

But again, this kind of thing is never formally supported as an Android feature

Background processes and daemons are part of linux kernel, like the execvp. If android still doesn't support them, maybe it is time it did. CPU performance and memory has greatly increased over last 10 years that it may be time do so if explicitly allowed by the user. Nothing wrong with opening things up sometimes, instead of locking them up.

Apps without battery restrictions, don't mean they could do whatever they want.

Sure, but killing processes like in case of phantom processes when there is no significant CPU usage or memory pressure is not good either. Killing them under memory pressure based on their oom adj when the time comes is perfectly fine. But before that, if the user wants them to, let them keep running in background and use the CPU as needed.

it's never our goal of stopping making faster device and longer battery life.

Yeah definitely, don't stop improving. I completely agree. I meant in the context that things are fast enough that a user may be satisfied with current levels of performance. If something gets done currently in 100ms and after optimizations it gets done in 80ms, he may still be fine with not having that extra boost, if he has to give up customizability and freedom for it. Users are often also not upgrading their devices every year to newer models, since a lot of them are fine with the performance of their device.

BTW, as a workaround, can you try this command, it effectively bump the limit to 1024

adb shell device_config put activity_manager max_phantom_processes 1024

and use this command to check the result

adb shell device_config get activity_manager max_phantom_processes 

Okay I saw you're aware of the above command already, this at least eliminate the limit 32 here. And for the killings, as if the app has a foreground service, it won't be killed unless under severe memory pressure.

Android runtime permission is designed for privacy concerns specifically, here it's not the case apparently. Battery optimization is NOT a permission.

The argument like 100ms to 80ms doesn't make sense to me, at least there are some scenarios where every millisecond matters. I totally understand your point that, if a user does want to do this, just allow it. All the cases here are specifically for the advanced users IIUC, that's why I was proposing a developer option settings.

I am aware of that. It's already listed in the How to disable the phantom processes killing? section at https://github.com/termux/termux-app/issues/2366#issuecomment-961305970. I used Integer.MAX_VALUE (2147483647) to completely disable the killing of phantom processes since the outer if (mService.mConstants.MAX_PHANTOM_PROCESSES < mPhantomProcesses.size()) condition will always fail in PhantomProcessList.trimPhantomProcessesIfNecessary(). It was confirmed to be working by the issue poster V-no-A.

So far, Termux has been working as expected. I have ran a Python script multiple times in the last 24 hours for anywhere between 20 to 90 minutes at a time with no issues so far. Even just now looking through the antivirus app while running Termux hasn't caused it to get killed So far, the setting is still 2147483647

I also asked in first comment if disabling it completely has any harms that I am not seeing, other than "useless" bookeeping going on for processes that will never be killed. The user experience should be consistent with android < 12 in that case, right?

But as mentioned above, that setting is overridden by gms after BOOT_COMPLETED unless adb shell "/system/bin/device_config set_sync_disabled_for_tests persistent is run to disable config sync. Denying gms the WRITE_DEVICE_CONFIG permission is not a possibility like its done by opengapps since its not a runtime or development permission. Hence, any developer settings option will also need to consider gms overrides.

Sorry, saw your comment after I already commented.

if the app has a foreground service, it won't be killed unless under severe memory pressure.

Yes, the service is not getting killed, but the terminal login shell process gets killed even though its the child of the app process, so can't really use the app with the login shell (bash) dead. Check the picture https://github.com/termux/termux-app/issues/2366#issuecomment-955066569.

Battery optimization is NOT a permission.

Yeah, I understand the context in which you say android.permission.REQUEST_IGNORE_BATTERY_OPTIMIZATIONS is really not a permission and so "Disable resource optimizations" can also not be called a permission ;) Although, that doesn't appear to be on the table, so its fine.

specifically for the advanced users IIUC, that's why I was proposing a developer option settings.

I personally am fine with that as long it survives reboots and is not overridden by gms config update and there is a completely disable setting, although 1024 seems high enough. This would be great for power users. Thanks

Yes it's disabled via the adb command above, it should be behaving the same as earlier Android OS.

BTW, it'll never reach 2147483647, there is a max process limit typically, you should be able to get it via ulimit -a

I'm not an expert of the gms config sync, so not sure why it's overwritten somehow. If it does happen, I'd reckon that could be a bug somewhere.

Re #9

If I read the github comment correctly, that login process was killed due to trimming, while when the config works, it shouldn't happen IMO?

it should be behaving the same as earlier Android OS.

Cool.

BTW, it'll never reach 2147483647, there is a max process limit typically, you should be able to get it via ulimit -a

Ah, I didn't think of the process limit itself. Thanks for pointing that out.

I'm not an expert of the gms config sync, so not sure why it's overwritten somehow. If it does happen, I'd reckon that could be a bug somewhere.

Check the Why device config may get reset? section or search for updateFromConfigurations. It details how it happens. Gms just replaces the current settings with the custom one it receives.

On Pixel 5, even max_cached_processes is set to 64 via update, even though in OS source, default is 32. Check the What are default cache settings? section.

At this point I am regretting not posting my comment in a github gist instead, would have been much easier to link sections. Sorry, about that.

If I read the github comment correctly, that login process was killed due to trimming, while when the config works, it shouldn't happen IMO?

It gets killed with the default 32 value since the buggy McAfee app is taking all the slots since its actually running 100s of logcat processes (found when limit was removed). If value is set to 2147483647, then bash and other processes are not killed.

while when the config works

Not sure what you mean by config here though.

Are you saying the max_cached_processes is being reset, or the max_phantom_processes is being reset? I'm not surprised by the former one, but the latter one shouldn't happen.

The DeviceConfig.setProperties() is called by gms which overrides the properties for the namespaces and removes any properties whose keys are not part of the properties objects passed. The gms decompiled source code is also posted in the gms classes collapsed section.

All configs in all namespaces, not just activity_manager are completely overwritten, or at least activity_manager and media are. Most namespaces don't have any properties set by default so I tested only 2. Happens on android 11 too.

# Initial at boot
$ adb shell "/system/bin/device_config list activity_manager"
max_phantom_processes=2147483647
max_phantom_processes1=1
push_messaging_over_quota_behavior=0

# After few minutes
$ adb shell "/system/bin/device_config list activity_manager"
push_messaging_over_quota_behavior=0

Are you saying the max_cached_processes is being reset, or the max_phantom_processes is being reset?

Both are removed on emulator.

On pixel 3a after restart in which max_phantom_processes=2147483647 was already set before reboot, following is returned. Note max_cached_processes=64 value.

$ adb shell "/system/bin/device_config list activity_manager"
compact_full_delta_rss_throttle_kb=8000
compact_full_rss_throttle_kb=12000
compact_proc_state_throttle=11
compact_statsd_sample_rate=0.0
max_cached_processes=64
oom_re_ranking_lru_weight=0.35
oom_re_ranking_number_to_re_rank=8
oom_re_ranking_rss_weight=0.15
oom_re_ranking_uses_weight=0.5
push_messaging_over_quota_behavior=0
use_compaction=true
use_oom_re_ranking=false

And considering that it doesn't look like the permission exemption will be added for apps, it would be great if developer options adds an additional toggle for "do not kill processes using extension cpu". It would allow apps like termux and tasker to run processes that use CPU for more than a few minutes, like even common linux commands like zip, xz, gpg, rsync, rclone which are CPU extensive by nature. Such commands are common among our users and other power users. Such processes would still get killed under memory pressure but that's understandable.

If a permission were to be added, then only specific apps could have been allowed to use extensive cpu, but since its not, a global toogle would be great so that power users can get old android experience.

Also note that some apps like bank apps refuse to run if a user has enabled developer options, and don't consider such users/developers "normal human beings", so that will be a concern too. Custom settings like max_cached_processes are reset to default if developer options are disabled, so "power users" won't be able to run such apps including bank apps if settings are added to developer options, instead of a permission.

Additional excessive CPU usage use cases

So I completely forgot some other important use cases that needs excessive CPU and memory. Termux supports on-device code compilation from source code, including building linux packages with our build environment. Termux also supports a full linux Desktop environment, like xfce4, etc for on our own rootfs/$PREFIX. Both use excessive CPU and memory, specially the former.

The termux app provides 1000+ packages from our main repo and even more from other repos (science, game, root, x11). But since we have limited storage space, bandwidth and maintainers/contributors available, we have a strict package approval requirements. We only accept packages that are popular, open source, not installable through other package mangers (like cpan, gem, npm or pip), do not take too much disk space, do not serve duplicated functionality and not hacking related. Moreover, we only keep the latest version of each package on our servers so that older version don't use storage.

So all the packages/code that fall out of these conditions are compiled locally by users. This may include python modules and other language dependencies, older versions, penetration and network testing tools due to no hacking packages policy, packages that are not accepted in our repos, etc.

Users also compile other open source codes available on the internet or their own code written for the many programming languages we support, or run entire servers. Lot of users learn programming languages on their phones, considering a huge population around the world can't actually afford PCs/Laptops, at least early in their lives. So for people who say that mobiles aren't meant for compiling code and stuff and people should use PCs should get informed about the conditions around the world. Instead of locking these people further down into ignorance and poverty, they should be helped out any way they can be.

Termux recently also started providing openjdk-17, kotlin and gradle and users can now also compile android apps including native code.

The maintainers and contributors often compile code locally on their device, specially the ones that don't have PCs, before they send package update pull requests, which are being upgraded on a regular/daily basis.

Someone may reply and say that android does not support executing compiled code from app private data directory for targetSdkVersion > 28 due to W^X violation, so let me answer that before its asked. Technically such locally compiled code can be packaged as an APK as a native lib, installed, extracted, symlinked and executed. So that's kinda irrelevant to excessive CPU usage itself. Moreover, this proposed solution by google/android to pack executables as native lib has so many issues for something as large and complex as termux/debian packaging system and would be such a usability and maintenance nightmare, that no maintainer actually wants to implement it and we have been sticking to targetSdkVersion 28. But that discussion would be off-topic for this issue.

Some people may also claim that the phantom process trimming and excessive CPU restrictions will only effect a relatively few people, so who cares. So let me provide some stats. Tasker alone (not other automation apps) have 1M+ downloads on playstore, but since Tasker is a paid app, its actually cracked A Lot. So users for it should be few hundred thousand at least (likely more), considering download count != active users. The Termux app has 10M+ downloads on playstore and we have been providing it on F-Droid and Github for years too. But that's just APK installs.

Our primary packages server was being donated 64TB month bandwidth by Bintray, before it got shutdown, and actual usage is near it if lot of package updates are done. Now our primary server/bandwidth is donated thanks to Fosshost, but that's only the primary one, we have multiple mirrors in different countries, including multiple universities in China. Some mirrors are load balanced. We can't provide stats for our Fosshost server, since we don't keep download logs. But on 30 Oct, we changed our primary server in pkg script from Fosshost to one backed by Cloudflare and users who have manually upgraded to the latest pkg script (termux-tools) version till now may be using it instead of Fosshost one and other mirrors based on weighting. Since cloudflare does keep stats (check attachments), it shows a spike in its usage and is currently at 30K unique visitors/day and 733K requests/day. For last 30 days, it had 526K unique visitors and 21M total requests, even though it wasn't even in the pkg script and would have to be manually set by users as the repo in sources.list, like via termux-change-repo. Note that users have to manually update packages and often don't do it until really needed, possibly weeks, months, so stats are only for who are actually updating.

To provide another data point/angle from grimler's mirror, the libllvm (compiler) has been the top 10 of most downloaded packages every day for the last 14 days, with 2645 average downloads per day with 47100 average unique visitors per day. Based on its weight of 17, we should have 15500 libllvm downloads per day and 280K unique visitors in total on all 5 repos in pkg script, not including chinese, which is a whole other world. Of course, these estimates may not be accurate.

So that is a whole lot of users who are compiling code, who would be affected by excessive CPU restrictions, not just a few. Compilations can take from a few minutes to many hours. And such users have been compiling code for years on android without complaints. And now that compilation has become even faster due to faster CPU/memory and larger memory, putting restrictions in place without exemptions is actually counterproductive for such users. Why improve performance if you can't actually use it? Slow downs during such compilation are expected by the user. When a developer compiles code on PC, like android source or apps, he pretty much expects that PC may slow down to a halt and he may barely be able to use it, with windows hanging and all, but that doesn't mean that OS should just kill the compilation process to free resources. That's not what the user wants!

So exemption for both trimming and excessive CPU usage must be provided and are equally important.

Some people have also responded to me (which I'm relaying) that a "developer" != "power user" and are not it support of a developer option and prefer a permission. Android may be partially treating them the same over the years, and it may probably be too late in the game to change it now, but putting such an option in developer option will have issues like apps checking if developer options is enabled and OEMs not providing the option in their fork or not allowing config to be changed even over adb. So such things need to be considered.

Phantom Processes List

I also had another thought. If the goal of android is to actually improve performance by tracking rogue processes that may be wasting resources, I think it would be beneficial if the currently tracked phantom processes, recently trimmed processes, and excessive CPU usage processes are shown in the processes list in developer options. The list should probably include info of uid, pid, process and app oom adj, and full commandline. The commandline should probably be trimmed, cause it may take lot of memory and cause crashes (activity limits), cause it can be 128KB or more, defined by ARG_MAX. Maybe copy list support too since lot of records may exist if limit is increased.

Firstly, from a developer perspective, considering not everybody knows command line and don't know how to use adb and ps, etc to find which processes their apps are running, it would be beneficial to see them in the GUI and know if they are being killed. Even developers who know command line probably don't check such processes. So this should help developers debug problems in their code, instead of android just silently killing them without anyone knowing, and bug continuing on an on.

Secondly, from a user perspective, it would be more clear to a user what processes such apps are running in the background and how android is managing them. It could also help users report bugs to devs more easily too, like the case of McAfee or for user or device specific issues.

Move of phantom processes info

I have also moved the detailed comment at https://github.com/termux/termux-app/issues/2366#issuecomment-961305970 to a gist at https://gist.github.com/agnostic-apollo/dc7e47991c512755ff26bd2d31e72ca8 for easier navigation.

I just want to say I agree that there should be a way for power users to grant freedom to certain apps to bypass this limitation. it's good to trim the fat but you should recognize that there are a lot of users whose setups would be crippled by this, you can/should allow us to bypass the limit for the apps we want to keep using.

I totally agree with this.

Re #22,

As mentioned above, this is never formally supported as an Android feature, I don't think there is a need to mention changes for something we've never formally supported.

The RunTime class was added in the initial android contribution in 2008 and is officially listed in the android developer docs and hence supported since the start. It has been maintained (pre android 7) over the years and exec tested (check References of RunTime class, like OldRuntimeTest). The exec call is not marked as UnsupportedAppUsage, unlike the loadLibrary call which was deprecated and removed in 2016. The android process docs does not says anything about it being unsupported either.

Android AOSP officially ships with binaries in /system/bin and some in /system/vendor/bin by individual vendors, which have been updated in each android version over the years, including Android 12, mainly via toybox updates and some BSD stuff like grep. Android also provides lot of android specific binaries including logcat, am, pm, cmd, settings, etc and even has special permissions like READ_LOGS and WRITE_SECURE_SETTINGS, etc as official development permissions. Android google-api's avd even ships with su binary. How are all these "features" unsupported or supposed to be called without forking processes?

The ulimit (max user processes) is being shown as 14348 on my Android 7 and 6653 on Android 12, so android is not even limiting that either. Android is based on linux kernel so one can't really say child processes have been unsupported by Android considering its core part of the linux kernel.

Child processes being discouraged by android devs in forums since it can waste resources by bugs and poor implementation is not really an official stance or good enough reason to block/limit forking for all apps without allowing exemptions to be granted by the user. It only takes one process to waste resources in badly written apps and I am sure we have all seen such apps.

The Runtime.exec is still working as of today, even Android 12. This API is more a legacy thing from the initial Java API, which was to allow some native utilities to run in early days, but many of them were being replaced with Android implementations later. These native child processes are something not managed by the Android system, from the life-cycle to resource management to security models. Android developers are discouraged from using these as you also mentioned. I don't know if there is a plan to deprecate it, but I wish so. The android process docs in #24 won't mention it because Android system (not Linux kernel) wasn't aware of the presence at all.

The bug here is more about to the number of maximum child processes forked from this. There was no such a document about how many child processes the Android is allowing at all. The ulimit number in Android 7 and Android 12 is mentioned nowhere, OEMs are free to make changes on their own, the termux happens to working well before because nobody limited it before. All these other utilities like am, pm are supposed to be used from adb shell, otherwise many of their functions would be broken because it's not running with the correct user credentials.

As mentioned above, I'm adding a developer options to allow turn the monitoring mechanism off. Stay tuned.

security models.

SeLinux policies and android permission models do apply to such processes. pm etc ran via cmd give failed transaction errors.

but I wish so?

Isn't the system implemented now to monitor such processes and kill excessive ones good enough (apart for 32), why do you wish for such things to be completely disabled, considering millions of users around the world have different use cases for them and have been using them for years, even if you don't? Note that I'm in full support of monitoring, killing and increased security, as long as exemptions can be allowed by user.

As mentioned above, I'm adding a developer options to allow turn the monitoring mechanism off. Stay tuned.

Thanks! That's some great news!

Will options be provided for both max phantom processes and excessive CPU usage? A device config option could be added to disable excessive CPU monitoring considering a permission is not being added.

Will those settings be reset if developer options are disabled after custom values are set, considering issues with bank apps and gaming apps checking if developer options are enabled. If yes, then maybe two settings should be created for each. One in activity manager settings and one in device config, like mOverrideMaxCachedProcesses is in activity manager settings and set via BackgroundProcessLimitPreferenceController and which gets reset when developer options are disabled, but device config max_cached_processes does not. Such users who don't want or can't enable developer settings could set device config variants over adb instead.

https://gist.github.com/agnostic-apollo/dc7e47991c512755ff26bd2d31e72ca8#cache-settings

https://gist.github.com/agnostic-apollo/dc7e47991c512755ff26bd2d31e72ca8#how-to-check-cache-settings

How will gms config update issues be handled? u/DutchOfBurdock has notified me that config update was sent after 3-4 days after reboot and custom max_phantom_processes value was reset. When value is updated, updateMaxPhantomProcesses() is called and it immediately calls trimPhantomProcessesIfNecessary(), which would result in any processes greater than 32 to get killed. So processes would randomly be killed whenever gms decides to send an update or update the values. For such cases, disabling device config sync is the only way to prevent killing, which requires adb, unless a developer option setting is added as well or google app adds an option to not update values itself. Disabling device config sync is likely not advisable since needed by android rescue.

Thanks a lot Jing Ji for https://android-review.googlesource.com/c/platform/frameworks/base/+/1920354. Really really appreciate all the help in this thread and for implementing this and of course the discussions. :) Thanks to @MishaalRahman as well for notifying me of the commit.

Based on the commit, trimming of phantom processes and excessive CPU usage of phantom processes will not be done if flag is off, but later will still be done for main app process. Monitoring of excessive CPU usage of main app process should not be an issue for Termux since it itself doesn't use excessive CPU, like for compilation, etc and its consistent with old android version behaviour.

The flag should also not reset to default when dev options are disabled, which is great since it solves the mentioned issue of bank/games apps checking if dev options was enabled. I confirmed by testing current flags in android 12 and they are not reset.

Will this likely be merged for Android 12L or any expected timeline? Even if its merged for android 12L, it would still be a concern for devices that are shipped with android 12 who won't be getting an update anytime soon or possibly at all for 12L. Or which were released with old versions and will max receive android 12, or earlier revisions.

And Thank You again for listening to the concerns of the community!

Thanks for reporting this. I've merged the CL, it should be able to make into Android 12L.

You are welcome. That's great. Thanks again.

Re #30. Sorry I don't have this information, you'd probably need to check with the individual OEMs.

Re #35, the CL mentioned in #27 missed the cutoff window of the recent Android 12L beta release build, it should be included in the next beta release though. So stay tuned with the https://developer.android.com/about/versions/12/12L to upgrade your devices if there is a newer release.

To the #34, Android 12L is actually a new Android release, it's just smaller in comparison to other releases. IMO, OEMs would have two options: a) cherry-pick this specific CL to their Android 12 maintenance release build, if there is any; b) upgrade it to Android 12L.

As already commented here and detailed in the How to disable the phantom processes killing? section of the gist referenced in the comment above, run following command with adb once to prevent phantom processing trimming on Android 12 (at your own risk).

adb shell "/system/bin/device_config set_sync_disabled_for_tests persistent; /system/bin/device_config put activity_manager max_phantom_processes 2147483647"

Disabling device config sync is necessary since gms may reset settings randomly even after boot when update comes. I need to update the gist.

Even with the workaround to prevent trimming, processes using excessive cpu will be killed. So users will have to limit max cpu usage for commands, like compilation commands. Those issues should be solvable for people on android 12L or 13 by disabling the settings_enable_monitor_phantom_procs feature flag in developer options, assuming above mentioned commit is merged in aosp release and vendor forks and flag is available. Users with root on android 12 should also be able to disable it with xposed/riru modules.

https://unix.stackexchange.com/questions/151883/limiting-processes-to-not-exceed-more-than-10-of-cpu-usage

And users (or trolls) should stop spamming this thread and giving threats, screaming here won't help, fix can't be released by google and all vendors instantaneously, updates take time, that's just how android works, other bug and issue fixes exist too that will need to wait. What's done is done, be thankful that an option will be provided in future. Contact your individual vendors, likely on their support thread and ask them to prioritize merging, like already done for samsung.

However, I do agree that, were it not a silent change, it could have been caught earlier and option available in android 12 release. So any future changes to phantom processes should be public, considering the amount of users and apps around the world that rely on them.

The settings_enable_monitor_phantom_procs flag is available in Pixel Android 12L beta 3 releases (which have been pulled). Unfortunately, feature flags menu won't be available in user build production devices, so users will have to run adb shell settings put global settings_enable_monitor_phantom_procs false over adb or run setprop persist.sys.fflag.override.settings_enable_monitor_phantom_procs false with root to disable phantom process trimming and killing of processes using excessive cpu. This is not really user friendly, but it is what it is.

https://twitter.com/MishaalRahman/status/1491487491026833413

https://twitter.com/MishaalRahman/status/1491489385170227205

RE #55: Yes, settings will be preserve for both commands. settings put global value will take precedence over setprop one. And welcome.

The flag still has value because its safer to use than disabling device sync since that also disables device recovery (RescueParty). Moreover, the flag also disables excessive cpu usage killing so those kinda commands (compilation, GUI, syncs) won't be killed either. But yeah, I agree, developer option should have been available for user builds. Running commands over adb won't be easy for all users, specially non-termux users.

Another issue is that how would a user even know that a feature flag is supported if feature flags list is not even shown in production devices? getprop won't show props with default values, and settings global key would be unset too by default.

The FeatureFlagUtils.getAllFeatureFlags() is not being called anywhere else other than dev options and tests, so at least there is no loop for something dumpsys and so apparently there is not shell API either to get the supported flags.

Vendors not supporting is definitely a concern since they would not have any obligation to support a feature flag, specially considering their history of aggressiveness against background processes. Hopefully, it will be available in all 12L and 13, but how would a user know it? Usually, normal settings have some UI showing their state, but this one won't. Since there is no API either as far as I can see, users would actually have to try to trigger the phantom process killer by starting > 32 processes and see if the settings command actually had an affect or not. Not all apps have a terminal or shell APIs either to test this.

However, I have added support in termux app with 68cdbd6f to get the DEFAULT_FLAGS and the feature values with reflection from FeatureFlagUtils and show the value for settings_enable_monitor_phantom_procs in termux settings about page, but that actually requires bypassing android hidden API restrictions, which is not something normal apps can be recommended to do.

Considering all this is not really user friendly, Jing Ji can the flag toggle be shown outside feature flags page or even in it for user builds? It would require some additional code but it would really improve user experience. Thanks

#Initial Commit!!!

This may be related -- it appears killing the web view process brings down the entire app, even when a foreground service is active with a notification, even when recording from the mic. This is breaking the Android API.

getHistoricalProcessExitReasons only returns a record for the web view.

ApplicationExitInfo(timestamp=2022-04-06, 9:08 a.m. pid=20757 realUid=99039 packageUid=10283 definingUid=10283 user=0 process=com.google.android.webview:sandboxed_process0:org.chromium.content.app.SandboxedProcessService0:0 reason=13 (OTHER KILLS BY SYSTEM) subreason=17 (ISOLATED NOT NEEDED) status=0 importance=300 pss=28MB rss=105MB description=isolated not needed state=empty trace=null

However, the entire app was terminated.

Example device that is doing this:

Board: msmnile
Brand: samsung
CPU ABI: [arm64-v8a, armeabi-v7a, armeabi]
Device: beyond1q
Manufacturer: samsung
Model: SM-G973W
Product: beyond1qltecs
Version codename: REL
Version incremental: G973WVLS6HVA1
Version release: 12
Version code: 31

#74

That's a different issue of webview package becoming a dependency of the app package till app process death if the app opens a WebView even once. Check https://www.reddit.com/r/tasker/comments/sz2kg7/investigation_accessibility_services_getting for details. It's an old android issue and fixed in Android 12L with ec8cff62. For older android versions, the only fix as far as I see is to never use a WebView.

#75 Thanks!

Hmm it seems it doesn't totally fix the issue. It will still kill and restart the services if I'm understanding that commit correctly. Unfortunately this would still break things like a foreground service recording from the mic. Hmm, it looks like battery saving mode also wouldn't affect this?

Webview package becoming a dependency would be your issue and yes, for your use case, the fix wouldn't work since recording will stop, only services will get restarted. Not using WebView is the only way probably. Disabling battery optimizations shouldn't affect it since its a dependency issue. Maybe jj can recommend something else.

Updated instructions for disabling killing of phantom process and excessive cpu usage processes has been uploaded to https://github.com/agnostic-apollo/Android-Docs/blob/master/docs/apps/processes/phantom-cached-and-empty-processes.md#how-to-disable-the-phantom-processes-killing for Android 12 and 12L/13+. I suggest users read it before asking further questions and filling up the issue with repeat comments.

The gist at https://gist.github.com/agnostic-apollo/dc7e47991c512755ff26bd2d31e72ca8 now just provides redirection links.

The fix is already available in Android 12L and 13 in AOSP with the settings_enable_monitor_phantom_procs settings flag added in 09dcdad. OEMs and device manufacturers should also have included the fix in their own release. If they have not, then its not google's fault, ask your device manufacturer. If you are currently using Termux app github builds, you can open termux app and open left drawer -> Settings icon -> About and it should show the current value of MONITOR_PHANTOM_PROCS under Software info section on Android 12+ devices and will be marked <unsupported> if its not supported in current android build. It should also show in next F-Droid release v0.119.0.

Please check the Android 14 beta release, there is a toggle in Settings-> Developer Options -> Disable child process restrictions

Thanks a lot for the addition!

It sets persist.sys.fflag.override.settings_enable_monitor_phantom_procs to false, which normally requires root. Current state can be checked with getprop persist.sys.fflag.override.settings_enable_monitor_phantom_procs. Note that disabling Developer Options will enable restrictions again as toggle state will be reverted. Will update the docs when I get time.

https://github.com/agnostic-apollo/Android-Docs/blob/master/en/docs/apps/processes/phantom-cached-and-empty-processes.md#android-12l-13-and-higher-1

Can always run command in docs over adb or root once for such cases.

=

This change has far more implications than annoying hobbyist/poweruser or making poor children in Uganda unable to learn how to program. In my opinion, an exemption in developer settings is not enough, let alone the fact that enabling developer settings makes many daily-used apps like banking apps refuse to work.

Like, it could possibly break VPN apps as well, because VPN apps may also launch native processes to handle the traffic.

Just take Shadowsocks-Android as an example.

In case you are not familiar with what Shadowsocks is, you may check the Outline VPN project (hint: Jigsaw is a company started by Google): https://en.wikipedia.org/wiki/Jigsaw_(company)#Outline

(tldr: Shadowsocks is a prominent censorship circumvention project, and the protocol of Shadowsocks was later adopted by Jigsaw as the Outline VPN project)

Look at the code, Shadowsocks-android is launching its critical traffic handling process in the exact "phantom" way:

https://github.com/shadowsocks/shadowsocks-android/blob/5a10078d23e930bb6f50de259d4e99ea00459bef/core/src/main/java/com/github/shadowsocks/bg/GuardedProcessPool.kt#L53-L55

        fun start() {
            process = ProcessBuilder(cmd).directory(Core.deviceStorage.noBackupFilesDir).start()
        }

To me, this change essentially attempts to drive native processes/executables away from Android ecosystem, which breaks many legitimate usages, which might be far beyond VPN apps or Termux.

Please reconsider this.