IssueTracker

10 is the limit for Go instances; I'm not sure what it is for other runtimes.

this limit should be determined dynamically, based on past performance. making it configurable is a second-best solution.

> 10 is the limit for Go instances; I'm not sure what it is for other runtimes.

According to Amy:
https://groups.google.com/d/msg/google-appengine/Wvco3AQR-AM/N1Y7S2HfukkJ

10 - Java
10 - Go
8  - Python

> this limit should be determined dynamically, based on past performance. making it configurable is a second-best solution.

That's seems reasonable. Actually, based on new information I'm wondering if the limited couldn't be removed entirely.

According to @proppy's answer on stackoverflow:
http://stackoverflow.com/a/11882719/236564

There are 4 factors that go into determining, whether or not an instance may take on an additional request.

1. The current number of requests is < 10 (8 with Python27)
2. Instance Memory utilization < soft Memory limit
3. Instance CPU utilization < class CPU limit
4. The instance is warming up

The original justification for the N=10 request limit was to prevent instance from out-of-memory death:
https://groups.google.com/forum/#!msg/google-appengine/y-LnZ2WYJ5Q/j_w13F4oSSkJ%5B1-25%5D

Since instance memory & CPU utilization are now determining factors; could the N=10 limit be removed? Are there other considerations? If the soft memory limit doesn't provide an adequate buffer - is there a way to determine average memory usage per request? If so, maybe that could be used to adjust the buffer.

Thoughts?

Is this limit documented anywhere? I couldn't find one.

Escalated the issue to the engineering team.

We have seen more than 50 requests/second on instances during load tests. So is this still an issue?

This limit is on the number of concurrent requests. As long as your average request runs for less than 160ms (Python) or 200ms (Go or Java), your result is consistent with the explanation above. This limit effects apps that require little compute time relative to the time they spend waiting for dependent requests, such as those to the datastore API, to complete.

@Ingo It's not necessarily an indication that it's fixed. The issue is how many concurrent requests an instance is allowed to handle.

Does that mean that GAE will put more than 10 concurrent requests on an instance if the expected latency is <= 200 ms? We have seen >70 requests/s for requests with 70 ms latency but only 5-7 requests/s for requests with 700 ms latency (see http://stackoverflow.com/questions/16925653/how-to-reduce-the-number-of-front-end-instances-launched-by-gae).

Where do those numbers come from for Python and Java? Are they documented anywhere?

Has this been fixed yet? My instances are not being used as efficiently as they could be because of this limit - if you have to do a load of requests with something blocking like a urlfetch but not CPU or Mem intensive then you should be able to have loads of concurrent requests on a single instance.

Currently I'm having to code around this (batch requests up so that each request spawns it's own background threads), but this is something that I shouldn't have to do.

I don't see any mention of this in 1.8.6 pre-release either. We have a tuned server that can process most responses in less than 20ms (most <17ms).
I'd like to set at 15 or 20 instead of spinning up new instances which then service 1 request each and shut down because the first and longest running instance is happily servicing further requests.
Alternatively if Google is improving the instance startup to prevent this issue, perhaps using past instance/server performance metrics (thus making this configuration option redundant) I'd like to know.

[Comment deleted]

I see for the GAE/J SDK version 1.8.7 dated 5th November 2013, one entry in the release notes is:

  "The max_concurrent_requests setting is now configurable per version/module. [link to this issue]"

I cannot see any mention of this new configuration either in my GAE/J Administration Console application settings or in the GAE/J documentation. Either it's there and I have missed it, or it isn't there yet.

I trust that, if missing, access to this new setting will be added and documented.

This was fixed with 1.8.7:
https://developers.google.com/appengine/docs/go/modules/#Go_Configuration
https://developers.google.com/appengine/docs/java/modules/#Java_Configuration
https://developers.google.com/appengine/docs/python/modules/#Python_Configuration

For the record, this issue is not fixed for me, but since it is marked as "fixed" then that is probably the end of the matter.

The Java configuration documentation link above concerns app modules, not apps in general, and in particular a new "max-concurrent-requests" tag.

I still cannot see a setting for my app in the Administration Console, nor can I see anything in the Java application (non-module) configuration documentation.

If by "old configuration" you mean "Java Application Configuration with appengine-web.xml", then according to the GAE/J documentation no mention is made that this configuration method is depreciated. (Do you know where it states in the documentation that GAE/J apps must use modules?)

Also, again according to the GAE/J documentation, modules and their configuration are still in preview, so it would seem a little strange to me that an existing configuration method is depreciated before its replacement is out of preview.

#19 you should be able to add this following configuration directive to your main application appengine-web.xml:
  <automatic-scaling>
    <max-concurrent-requests>50</max-concurrent-requests>
  </automatic-scaling>

What it will actually do is configure the 'default module' of your application. Note that those settings can't coexist with the Administration Console Performance settings, so once you do the switch: you will have to configure performance settings using appengine-web.xml.

I haven't used Java but if it is anything like the python one then it uses modules regardless of whether you specify them or not (if you don't it just creates one module called default for you) and thus by not using the new syntax all you are doing is depriving yourself of additional configuration.

Yeah. Just tried this with our java hdr app on the default module. Although my appengine-web.xml config tells GAE to allow 100 QPS per instance, it still levels all my instances out at 10 QPS. It seems like this setting is ignored for frontend instances.

Reply to #20:
I agree with that. The severity and priority of this issue should be increased.

we should also open a ticket or issue at Microsoft if the error is from their side

By the way, probably you guys are dealing with this with your own ways, but in my case the only way I found to keep working was using my real phone for testing the apps (through adb), and using WSL2 in the pc

Reply to #18:
Worked for me also!  THANKS!

Reply to #18:
Could you share patched qemu-system-x86_64.exe, please?

Thanks a lot.

Reply to #27:
Thank you, I can confirm that this patched version perfectly works with all W11 Hyper-V & Core Isolation features turned on.

Reply to #27:

Thanks for the binaries finally the emulator is working on AMD Ryzen 5950 with WSL2 under Windows 11.

Note: They require to update Android Emulator (30.9.5) first which is not automatically offered when using the Android SDK Manager of Visual Studio

Reply to #29 (from author of #18) -

Just a note that I hope is helpful:  You can get the 30.9.5 update for the Emulator with the Visual Studio Android SDK Manager by clicking the gear icon, selecting "Repository" -> "Full List (Unsupported)".  Once you do that, you should see available updates for the Android Emulator component to get from 30.8.4 to 30.9.5.

I'm glad to see that this change is also working for people with AMD CPUs...  I don't have one so I couldn't test it.  :-)

Awesome, thanks #27 and #18 !!
Now works fine for me in Windows 11, AMD Ryzen 7 5800H

thanks #27 and #18，Works in Windows 11, AMD Ryzen 7 5800X

While I applaud and think it's fantastic that this issue was resolved by #27 and #18 (hence by the community and thanks to open source!), I think it would be better if Google would be able to make the same fix and distribute an official patched emulator to us.

Google, any updates on this issue from your side? Can you apply the same fix and ship it through Android Studio?

Update:
Microsoft confirmed the issue was related to xsave. And we are working on emulator update based on Microsoft's suggestions.

thanks #27 and #18 works on Ryzen 5950 win11

Thanks #29, #27 and #18. Your suggestions worked on a Windows 11 with AMD 5800H. For Visual Studio 2019 / Xamarin users: the files kindly shared by #27 must be placed in the following folder:

C:\Program Files (x86)\Android\android-sdk\emulator\qemu\windows-x86_64

Please, do that *after* updating your emulator to 30.9.5, since the installer might try to "fix" your installation and get rid of any manually copied files.

#38 indeed, how they're putting android emulator being borked on the latest Windows (Ryzen, Hyper V is common and common among developers respectively) on P2 is beyond me. Shouldn't this be their first priority?

The fix from #27 also worked for me, interestingly enough on my i9-11980HK laptop running Windows 11 and WSL 2 w/Ubuntu 20. Every time I would try to start an emulator in Android Studio, qemu would just freeze. I downloaded the files and extracted them to the android sdk folder, which in my case was:
C:\Users\USERNAME\AppData\Local\Android\Sdk\emulator\qemu\windows-x86_64

It immediately started working. Thank you so much, hopefully this fix will get baked in at some point.

#40 perhaps you aren't using HAXM on intel? afaik HAXM isn't affected by this issue.

Hi,

Anyone can try emulator 31.1.2 and see if it fixes the issue. The fix is basically the same as #18 but only remove "disable xsave" on Windows 11.

(apologies for repost, I was in the wrong Google account)

Hi, I have been running this with a build from the repo code at 2c6ba8d (a few commits after the bump to v31.1.2) for the last 5 days, approx 2-3 hours a day, and it has worked well on my i7-1185g7 on Windows 11.

Is there an internal/nightly build you would like me to test instead?  Or would you like me to rebuild at the bump commit (f6a220d) ?

You can email me directly if that is easier.  I'm happy to help, just let me know...

Thanks,
Daniel
(author of #18)

Hi, Daniel,

Thanks for offering help. It would be perfect to run at 2c6ba8d. Actually to my surprise (I am not the one working on releases), I just found that our emulator binary 31.1.2 might not be compiled at exactly the bump commit but rather a few commits after. So your build is actually very close to emulator binary 31.1.2.
It does seem my commit can fix the problem from your test results.

Thanks!

Can someone provide the link to version 31.1.2?  Or, provide a way to get a list of all available from Android Studio?  Thanks!!

re #46,
try to use canary channel in studio's sdk manager, 31.1.2 should show up.

Reply to #46:
windows: https://dl.google.com/android/repository/emulator-windows_x64-7869535.zip
linux: https://dl.google.com/android/repository/emulator-linux_x64-7869535.zip
macosx: https://dl.google.com/android/repository/emulator-darwin_x64-7869535.zip

thanks #48

Thanks #18 for fixing!

+1 to #18

This has not resolved my issue.

Edition Windows 11 Pro
Version 21H2
Installed on    ‎10/‎5/‎2021
OS build        22000.318
Experience      Windows Feature Experience Pack 1000.22000.318.0

Hi #53 - your error appears to be a different cause than the one that was fixed.  Note in the initial log posted with this issue it shows that WHPX fails to initialize.  

From your log it appears WHPX initializes successfully:

WHPX on Windows 10.0.22000 detected.
Windows Hypervisor Platform accelerator is operational

Further down we see:

ERROR   | It seems too many emulator instances are running on this machine. Aborting.

I believe this is caused by an issue opening the network ports for ADB connections.  

In my logs, I will have this where yours fails:

VERBOSE | control console listening on port 5554, ADB on port 5555

If you don't have something using the ports already, you may need to configure your firewall rule or something similar.  Google search the "too many emulator instances" error message text and you should be able to find more information...

Hope this helps.

This fixed my issue and emulator did start and run (on windows 11Build 22000.346 with hyper-v) but then I killed the emulator uing ctrl c and on a restart I got errror below. Running again with -wipe-data fixed the issue. Restarts fine if I exit by closing the window

PS C:\Users\mlwp\Software\Android\emulator>  .\emulator.exe  -avd Pixel_C_API_30  -show-kernel
INFO    | Android emulator version 31.1.2.0 (build_id 7869535) (CL:N/A)
WARNING | unexpected system image feature string, emulator might not function correctly, please try updating the emulator.
INFO    | added library vulkan-1.dll
INFO    | configAndStartRenderer: setting vsync to 60 hz
ERROR   | Failed to open /qemu.conf, err: 2
WHPX on Windows 10.0.22000 detected.
Windows Hypervisor Platform accelerator is operational
INFO    | Started GRPC server at 127.0.0.1:8554, security: Local
INFO    | Advertising in: C:\Users\mlwp\AppData\Local\Temp\avd\running\pid_12832.ini
[ 5497.725387] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 5497.727452] rcu:     1-...!: (0 ticks this GP) idle=ffc/0/0x0 softirq=62892/62892 fqs=1 last_accelerate: d2d8/d3ec, Nonlazy posted: ..D
[ 5497.727452]  (detected by 0, t=21015 jiffies, g=160085, q=10509)
[ 5497.727452] Sending NMI from CPU 0 to CPUs 1:
[ 5497.755413] NMI backtrace for cpu 1 skipped: idling at default_idle+0x23/0x40
[ 5497.727452] rcu: rcu_preempt kthread starved for 15723 jiffies! g160085 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1
[ 5497.758981] rcu: RCU grace-period kthread stack dump:
[ 5497.758981] rcu_preempt     I14680    10      2 0x80004000
[ 5497.758981] Call Trace:
[ 5497.758981]  __schedule+0x3f4/0x6f0
[ 5497.758981]  schedule+0x81/0xb0
[ 5497.758981]  schedule_timeout+0xb5/0x150
[ 5497.758981]  ? __prandom_timer+0x8/0x8
[ 5497.758981]  rcu_gp_kthread+0x69d/0xe70
[ 5497.758981]  ? usb_stor_control_thread+0x8/0x8
[ 5497.758981]  kthread+0x180/0x190
[ 5497.758981]  ? usb_stor_control_thread+0x8/0x8
[ 5497.758981]  ? change_clocksource+0x8/0x8
[ 5497.758981]  ret_from_fork+0x35/0x40
[ 5518.810673] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 5518.814623] rcu:     1-...!: (0 ticks this GP) idle=048/0/0x0 softirq=62892/62892 fqs=0 last_accelerate: e014/e884, Nonlazy posted: ..D
[ 5518.814623]  (detected by 3, t=5257 jiffies, g=160089, q=10499)
[ 5518.827577] Sending NMI from CPU 3 to CPUs 1:
[ 5518.834136] NMI backtrace for cpu 1 skipped: idling at default_idle+0x23/0x40

Reply to #55, I think that's a different issue. You can file a new bug. One of our kernel developers should be able to take a look. Thanks!

Does this hypervisor driver for AMD worked with WSL2 now? I need both of them, my env is Ryzen 3700X with Windows11

Just want to thank #18 and #27 for the fix - picked up a Surface Pro 8 with Windows 11 and had to resort to compiling on my phone (not helpful when people are calling you on it!). Fixed my issue and happily coding away, thank you!

Custom download at #48 fix the problem on Windows 11 + AMD Ryzen and Hyper-V enabled!

Reply to #57, the error is benign. The driver is for people who do not use hyper-V (and WSL2 depends on hyper-V).

Reply to #18

Didn't work for me.

#27 worked perfect on AMD 5900x Windows 11. Can finally run Android emulator and use WSL2 without having to keep turning off and on HyperVisor.

Can Windows 10 use this fix?

Reply to #64, the fix in official emulator releases only applies to Windows 11.

Please, can we have Windows 10 fix, too?

thanks to #18 for fixing.

#48 worked for me. Thank you so much.

I've spent days on this and can't seem to get it to work.  I don't see any recent posts, so I expected that the latest builds had incorporated the fix, but doesn't work for me.
Data:
1) Windows 11, Intel CPU, with WSL 2 installed and working (Hyper-V)
2) Tried Android emulator-31-3-14 and it runs slow (no acceleration)
3) Run command "emulator-check.exe accel" and the result is:

accel:
0
Please disable Hyper-V before using the Android Emulator.  Start a command prompt as Administrator, run 'bcdedit /set hypervisorlaunchtype off', reboot.WHPX (10.0.22621) is installed and usable.
accel

4) Renamed the folder
FROM: C:\Program Files (x86)\Android\android-sdk\emulator
TO: C:\Program Files (x86)\Android\android-sdk\emulator.ORIG-31-3-14

5) Downloaded the Windows.ZIP from message #48 above, unziped into the folder:
C:\Program Files (x86)\Android\android-sdk\emulator

6) Reboot
7) Run command: "emulator-check.exe accel" and the result is unchanged:

C:\Program Files (x86)\Android\android-sdk\emulator>emulator-check.exe accel
accel:
0
Please disable Hyper-V before using the Android Emulator.  Start a command prompt as Administrator, run 'bcdedit /set hypervisorlaunchtype off', reboot.WHPX (10.0.22621) is installed and usable.
accel

I've tried many things, but the result is the same.  I'm out of ideas.  By the lack of posts, I suspect that it is working for everyone else in the current version.  Why not on my system?

-B

Hi, perhaps, we should clean up the output from accel-check.
"Please disable Hyper-V before using the Android Emulator.  Start a command prompt as Administrator, run 'bcdedit /set hypervisorlaunchtype off', reboot." --------- This sentence comes from old days when only Intel HAXM is supported.

Nowadays, WHPX+HyperV may be a little bit slower than Intel HAXM but should not differ too much.

So the problem is why emulator runs slow. Can you tell me your CPU model?
And run "emulator" with "-verbose -show-kernel" options and paste the command line output?