Phantom Processes

Android 12 via 15755084 and updated via 5706277f has added the mechanism to monitor forked child processes started by apps and kills them if more than the default 32 are found running.

The app or phantom processes may also be killed if they use excessive CPU, which was done previously too, but only for app process.

Track the child processes that are forked by app processes

Apps could use Runtime.exec() to spawn child process and framework will have no idea about its lifecycle. Now track those processes whenever we find them - currently during the cpu stats sampling they could be spotted. If it's consuming too much CPU while its parent app process are also in the background, kill it.

By default we allow up to 32 such processes; the process with the worst oom adj score of their parents will be killed if there are too many of them.

This 32 process limit is for all apps combined and not per app.

Check PhantomProcessList commit history here.

I am one of the maintainers for the termux-app and the new changes have completely broken our app's core functionality and affects other similar apps as well, like Tasker. The termux-app#2366 issue was opened by a Pixel 3a user in which I found out about this after going through android source. This change is neither listed in android 12 changes list nor behavior changes and was done silently.

The problem is that if more than 32 processes are running, then processes belonging to termux app greater than the limit will get trimmed even if battery optimizations are disabled for the app. Moreover, since the limit is for all apps combined, if termux tries to run just 1 process and is in background and another (buggy) app has >= 32 processes running, and at the time of killing, the other app's app process has a lower oom adj than termux's app process (not the oom adj of the child processes themselves), the single process of termux would get killed first by PhantomProcessList.trimPhantomProcessesIfNecessary().

If there are buggy apps running, like McAfee Security app currently, it will not let apps like Termux run any processes at all for long.

Foreground oriented apps will always be given higher priority, but an app that mostly works in background does not mean its unimportant for the user..

Current design can also be abused by malicious apps by spawning unneeded processes.

I have also written a very detailed comment about this and how it affects apps trying to run background commands. The main sections to check would be the McAfee app gone crazy, The "unfair" advantage for foreground oriented apps, Buggy and malicious apps and Improvements in heuristics and exemptions for apps. Some other sections from it are referred below too. I didn't copy all that info again here since its too long and Github has better markdown support, considering collapse sections are not working here. By I can do it if it's requested.

Steps to reproduce the problem

Install the termux apps from Github or F-Droid as per installation instructions. F-Droid direct link is https://f-droid.org/en/packages/com.termux. After installing the app, open it and let bootstrap installation to finish.

Run the following command and put termux in background and connect or disconnect the charger.

for i in $(seq 40); do
sha256sum /dev/zero &
done

You can monitor cpu usage with top command and run killall sha256sum to kill the processes started.

What happened

When you return to Termux app, it should say [Process completed (signal 9) - press Enter] because the main login bash shell should have gotten killed and any other older processes above 32. Sometimes bash gets killed as soon as one runs the loop while Termux is still in foreground, sometimes after switching to another app and come back, and sometimes takes a few minutes depending on when phantom process killing is scheduled. The connecting or disconnecting the charger works instantly, as per Battery power changes trigger mentioned in the How phantom process killing gets scheduled? section.

Note that bash was started with pid 5683 and logcat logs the killing. The sha256sum commands below are CPU intensive but only their count matters for the killing. The bash process would barely use any CPU but still get killed first since its the oldest. Check the How are phantom processes killed? section for more info on how killing is prioritized.

I ActivityManager: Killing PhantomProcessRecord {9ccd0ab 5683:5641:bash/u0a147}: Trimming phantom processes
I libprocessgroup: Successfully killed process cgroup uid 10147 pid 5683 in 0ms
I ActivityManager: Process PhantomProcessRecord {9ccd0ab 5683:5641:bash/u0a147} died

Transcript to start `1` bash process + `40` sha256sum processes

~ $ ps -efww
UID        PID  PPID  C STIME TTY          TIME CMD
u0_a147   5641   340  5  1970 ?        00:00:02 com.termux
u0_a147   5683  5641  0  1970 pts/0    00:00:00 /data/data/com.termux/files/usr/bin/bash
u0_a147   5728  5683  1  1970 pts/0    00:00:00 ps -ef
~ $ for i in $(seq 40); do sha256sum /dev/zero & done
[1] 5731
[2] 5732
[3] 5733
[4] 5734
[5] 5735
[6] 5736
[7] 5737
[8] 5738
[9] 5739
[10] 5740
[11] 5741
[12] 5742
[13] 5743
[14] 5744
[15] 5745
[16] 5746
[17] 5747
[18] 5748
[19] 5749
[20] 5750
[21] 5751
[22] 5752
[23] 5753
[24] 5754
[25] 5755
[26] 5756
[27] 5757
[28] 5758
[29] 5759
[30] 5760
[31] 5761
[32] 5762
[33] 5763
[34] 5764
[35] 5765
[36] 5766
[37] 5767
[38] 5768
[39] 5769
[40] 5770
~ $
[Process completed (signal 9) - press Enter]

Logcat showing `9` phantom processes trimmed

I ActivityManager: Killing PhantomProcessRecord {9ccd0ab 5683:5641:bash/u0a147}: Trimming phantom processes
I ActivityManager: Killing PhantomProcessRecord {30b2308 5731:5641:sha256sum/u0a147}: Trimming phantom processes
I ActivityManager: Killing PhantomProcessRecord {9a8aa1 5732:5641:sha256sum/u0a147}: Trimming phantom processes
I libprocessgroup: Successfully killed process cgroup uid 10147 pid 5683 in 0ms
I ActivityManager: Killing PhantomProcessRecord {19044c6 5733:5641:sha256sum/u0a147}: Trimming phantom processes
I ActivityManager: Killing PhantomProcessRecord {685cc87 5734:5641:sha256sum/u0a147}: Trimming phantom processes
I libprocessgroup: Successfully killed process cgroup uid 10147 pid 5731 in 0ms
I ActivityManager: Killing PhantomProcessRecord {2b367b4 5735:5641:sha256sum/u0a147}: Trimming phantom processes
I libprocessgroup: Successfully killed process cgroup uid 10147 pid 5732 in 0ms
I ActivityManager: Killing PhantomProcessRecord {9342fdd 5736:5641:sha256sum/u0a147}: Trimming phantom processes
I libprocessgroup: Successfully killed process cgroup uid 10147 pid 5733 in 0ms
I ActivityManager: Killing PhantomProcessRecord {3bbe752 5737:5641:sha256sum/u0a147}: Trimming phantom processes
I libprocessgroup: Successfully killed process cgroup uid 10147 pid 5734 in 0ms
I ActivityManager: Killing PhantomProcessRecord {2e8aa23 5738:5641:sha256sum/u0a147}: Trimming phantom processes
I libprocessgroup: Successfully killed process cgroup uid 10147 pid 5735 in 0ms
I libprocessgroup: Successfully killed process cgroup uid 10147 pid 5736 in 0ms
I libprocessgroup: Successfully killed process cgroup uid 10147 pid 5737 in 0ms
I libprocessgroup: Successfully killed process cgroup uid 10147 pid 5738 in 0ms
I ActivityManager: Process PhantomProcessRecord {9ccd0ab 5683:5641:bash/u0a147} died
I ActivityManager: Process PhantomProcessRecord {2e8aa23 5738:5641:sha256sum/u0a147} died
I init    : Untracked pid 5738 received signal 9
I ActivityManager: Process PhantomProcessRecord {9342fdd 5736:5641:sha256sum/u0a147} died
I ActivityManager: Process PhantomProcessRecord {685cc87 5734:5641:sha256sum/u0a147} died
I ActivityManager: Process PhantomProcessRecord {19044c6 5733:5641:sha256sum/u0a147} died
I init    : Untracked pid 5734 received signal 9
I init    : Untracked pid 5736 received signal 9
I init    : Untracked pid 5733 received signal 9
I ActivityManager: Process PhantomProcessRecord {30b2308 5731:5641:sha256sum/u0a147} died
I init    : Untracked pid 5731 received signal 9
I ActivityManager: Process PhantomProcessRecord {9a8aa1 5732:5641:sha256sum/u0a147} died
I ActivityManager: Process PhantomProcessRecord {2b367b4 5735:5641:sha256sum/u0a147} died
I ActivityManager: Process PhantomProcessRecord {3bbe752 5737:5641:sha256sum/u0a147} died
I init    : Untracked pid 5732 received signal 9
I init    : Untracked pid 5735 received signal 9
I init    : Untracked pid 5737 received signal 9

Remaining `32` tracked phantom processes

  All Active App Child Processes:
    proc #0: PhantomProcessRecord {c1b29f4 5739:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5739 ppid=5641 knownSince=-56s278ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #1: PhantomProcessRecord {511651d 5740:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5740 ppid=5641 knownSince=-56s278ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #2: PhantomProcessRecord {6a51392 5741:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5741 ppid=5641 knownSince=-56s278ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #3: PhantomProcessRecord {52c0163 5742:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5742 ppid=5641 knownSince=-56s277ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #4: PhantomProcessRecord {1baf160 5743:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5743 ppid=5641 knownSince=-56s277ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #5: PhantomProcessRecord {2643619 5744:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5744 ppid=5641 knownSince=-56s277ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #6: PhantomProcessRecord {d7cd6de 5745:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5745 ppid=5641 knownSince=-56s277ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #7: PhantomProcessRecord {55350bf 5746:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5746 ppid=5641 knownSince=-56s277ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #8: PhantomProcessRecord {9cdc38c 5747:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5747 ppid=5641 knownSince=-56s277ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #9: PhantomProcessRecord {8791ad5 5748:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5748 ppid=5641 knownSince=-56s277ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #10: PhantomProcessRecord {77b82ea 5749:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5749 ppid=5641 knownSince=-56s277ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #11: PhantomProcessRecord {c7639db 5750:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5750 ppid=5641 knownSince=-56s277ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #12: PhantomProcessRecord {e598c78 5751:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5751 ppid=5641 knownSince=-56s277ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #13: PhantomProcessRecord {1bd8f51 5752:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5752 ppid=5641 knownSince=-56s277ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #14: PhantomProcessRecord {906e3b6 5753:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5753 ppid=5641 knownSince=-56s277ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #15: PhantomProcessRecord {61498b7 5754:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5754 ppid=5641 knownSince=-56s276ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #16: PhantomProcessRecord {1d6f824 5755:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5755 ppid=5641 knownSince=-56s276ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #17: PhantomProcessRecord {7facf8d 5756:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5756 ppid=5641 knownSince=-56s276ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #18: PhantomProcessRecord {5158542 5757:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5757 ppid=5641 knownSince=-56s276ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #19: PhantomProcessRecord {bd00953 5758:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5758 ppid=5641 knownSince=-56s276ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #20: PhantomProcessRecord {39d7290 5759:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5759 ppid=5641 knownSince=-56s276ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #21: PhantomProcessRecord {e51d789 5760:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5760 ppid=5641 knownSince=-56s276ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #22: PhantomProcessRecord {57ab38e 5761:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5761 ppid=5641 knownSince=-56s276ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #23: PhantomProcessRecord {5c7e7af 5762:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5762 ppid=5641 knownSince=-56s276ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #24: PhantomProcessRecord {f0f27bc 5763:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5763 ppid=5641 knownSince=-56s276ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #25: PhantomProcessRecord {ef76345 5764:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5764 ppid=5641 knownSince=-56s276ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #26: PhantomProcessRecord {bf27a9a 5765:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5765 ppid=5641 knownSince=-56s276ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #27: PhantomProcessRecord {4b54fcb 5766:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5766 ppid=5641 knownSince=-56s275ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #28: PhantomProcessRecord {18503a8 5767:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5767 ppid=5641 knownSince=-56s275ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #29: PhantomProcessRecord {8afeec1 5768:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5768 ppid=5641 knownSince=-56s275ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #30: PhantomProcessRecord {7eda666 5769:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5769 ppid=5641 knownSince=-56s275ms killed=false
      lastCpuTime=0 oom adj=0 seq=13
    proc #31: PhantomProcessRecord {1c71da7 5770:5641:sha256sum/u0a147}
      user #0 uid=10147 pid=5770 ppid=5641 knownSince=-56s275ms killed=false
      lastCpuTime=0 oom adj=0 seq=13

What you think the correct behavior should be

Copied the Improvements in heuristics and exemptions for apps section...

Neither trimming nor excessive CPU killing, nor empty process killing (including possibly daemon, like sshd, termux-app/#2015, termux-boot/#5) should be done for apps that have battery optimizations disabled. A user explicitly granted an app the permission to "use battery", android shouldn't be killing its processes, unless under memory pressure, or at the very least shouldn't prioritize it. That should be like the whole point of disabling battery optimizations. Adding this to android OS should solve most of the concerns.

And if trimming needs to be done, the heuristics should improve. It should have some minimum allowed limit of phantom processes for each app below which they are all ignored and also a max allowed limit per app that should be something like max_phantom_processes/3 (or whatever the android overlords think is ideal) above which trimming of the phantom processes of that specific app is prioritized before starting to kill processes of other apps, so that foreground oriented, malicious and buggy apps don't affect other apps too much.

If I hadn't asked OP for the logs, we wouldn't have known that McAfee was causing so much trouble for other apps and who knows how long it has been doing it. And OP was actually capable of reporting and providing logs, but an average user wouldn't be, and from their perspective, their apps would just be getting killed because had knowingly or blindly installed a buggy app from playstore. And even if they reported (without logs), devs wouldn't be able to reproduce the issue on their device, since they wouldn't likely have a buggy app installed, so this kinda issues will be harder to solve for devs and will have nothing to do with their code. The OS should protect users from that, even without them having to check logcat and dumpsys output.

Moreover, total 32 processes limit of all apps combined is too low. That can easily be hit during what one would consider normal conditions and not excessive. Just running 2-3 terminal sessions, each with optional terminal multiplexer (tmux) and then actually running some commands in them or just a script that divides work with background jobs (command &), plus some processes started incrementally as or for plugins, like termux-api or termux-tasker and you are easily looking at more than 15 to 30 processes for just Termux. Now add some user running some servers in Termux or add Tasker with its profiles and tasks, plus any other app, and 32 limit is hit instantly and processes getting trimmed, and some important work or data being lost.

Questions

Based on the info provided and the breakage of multiple apps or at least the core functionality of our app, does android plan on changing the behavior so that phantom processes and "excessive CPU usage" processes of apps with battery optimizations disabled are not killed. The adb way to disable the killing can't be expected to be done by all our users. Moreover, completely disabling phantom killing will allow apps going crazy with background processes to consume lot of battery and android should provide a per app exemption instead, so that only specific apps can be exempted and phantom killing still applies to other apps. This way user experience of (power) users is not harmed.
Will disabling it completely do any additional harm. I "tried" to answer that in the What are the risks with disabling phantom process killing? section, but best ask the android devs that question.

Thanks,

agnostic-apollo

Termux App Info

APP_NAME: Termux
PACKAGE_NAME: com.termux
VERSION_NAME: 0.117
VERSION_CODE: 117
UID: 10147
TARGET_SDK: 28
IS_DEBUGGABLE_BUILD: true
APK_RELEASE: Github
SIGNING_CERTIFICATE_SHA256_DIGEST: B6DA01480EEFD5FBF2CD3771B8D1021EC791304BDD6C4BF41D3FAABAD48EE5E1

Device Software Info

OS_VERSION: 5.10.43-android12-9-00031-g02d62d5cece1-ab7792588
SDK_INT: 31
RELEASE: 12
ID: SE1A.211012.001
DISPLAY: sdk_gphone64_x86_64-userdebug 12 SE1A.211012.001 7818354 dev-keys
INCREMENTAL: 7818354
SECURITY_PATCH: 2021-11-05
IS_DEBUGGABLE: 1
IS_EMULATOR: 1
IS_TREBLE_ENABLED: true
TYPE: userdebug
TAGS: dev-keys

Device Hardware Info

MANUFACTURER: Google
BRAND: google
MODEL: sdk_gphone64_x86_64
PRODUCT: sdk_gphone64_x86_64
BOARD: goldfish_x86_64
HARDWARE: ranchu
DEVICE: emulator64_x86_64_arm64
SUPPORTED_ABIS: x86_64, arm64-v8a