Can't Repro
Status Update
No update yet.
Comments
jm...@google.com
We are experiencing a higher-than-normal volume in the Issue Tracker at the moment, and so our response time is slightly delayed. Thank you for your patience as we get to each issue.
mi...@gmail.com
I second this.
vo...@gmail.com
Thanks for supporting this request. Don't forget to toggle the little star at the top of the page. Feature requests with higher number of stars are more likely to be noticed by Google.
jm...@google.com
I have notified the engineering team of this request and they are currently investigating a solution. There is no ETA at this time for the implementation.
- Note that you may be able to override the indexing of the _ah_SESSION kind's '_expires' property via a custom datastore-indexes.xml configuration file [1].
[1]https://cloud.google.com/appengine/docs/standard/java/config/indexref
- Note that you may be able to override the indexing of the _ah_SESSION kind's '_expires' property via a custom datastore-indexes.xml configuration file [1].
[1]
vo...@gmail.com
Thanks for replying!
I fail to see how datastore-indexes.xml can be used to disable a single property index. As far as I know, this file deals exclusively with declaring custom indexes. Or am I mistaken? Could you please provide us with an actual datastore-indexes.xml entry which would prevent the _expires property from being indexed?
I fail to see how datastore-indexes.xml can be used to disable a single property index. As far as I know, this file deals exclusively with declaring custom indexes. Or am I mistaken? Could you please provide us with an actual datastore-indexes.xml entry which would prevent the _expires property from being indexed?
vo...@gmail.com
I take the second part of my request back. As it turns out, the expiration timestamp rewrites are only performed on sessions which are least 25% expired. That's according to the current implementation of SessionManager:
https://github.com/GoogleCloudPlatform/appengine-java-vm-runtime/blob/master/appengine-jetty-managed-runtime/src/main/java/com/google/apphosting/runtime/jetty9/SessionManager.java
Since the 25%-rule reduces datastore access significantly, the risk of contention caused by indexing the _expires property seems to be minimal. At least for "read-mainly" applications which were my original concern. However, having an option to disable indexing of the _expires property still seems like necessity for applications which update session data more frequently.
Since the 25%-rule reduces datastore access significantly, the risk of contention caused by indexing the _expires property seems to be minimal. At least for "read-mainly" applications which were my original concern. However, having an option to disable indexing of the _expires property still seems like necessity for applications which update session data more frequently.
rm...@google.com
Hello everyone. I'm the SME for Datastore on our Cloud Support team.
I believe that this contention error may actually be innocuous.
If the entity is validated for the 25% expiry rule, it's definitely possible at high QPS that more than one end user request will prompt update the entity _expires property at the same time, which will exceed the 1qps limit on Datastore entity groups. As long as one of these succeeds, the session is freshened, that's all that really matters.
The good news is that this will be solved anyway with the migration of Datastore customers to Firestore, which eliminates the 1qps write rate limit. [1]
It's worth pointing out here, that contention and hotspotting are two very different things. An even distribution of values (not monotonically increasing ones) is used to avoid hotspotting, where one tablet server has too much of the "interesting" data you are trying to work with. Datastore scales by splitting the keyspace(index) to avoid concentrating reads and writes on a single server (which holds a range of keys).
Contention ("Too much contention on these datastore entities" ) is caused by attempting to write to an entity group too often.
You will not receive explicit errors for hotspotting. It will simply appear as poor write throughput and higher read latency, including timeouts.
[1]https://cloud.google.com/datastore/docs/upgrade-to-firestore
I believe that this contention error may actually be innocuous.
If the entity is validated for the 25% expiry rule, it's definitely possible at high QPS that more than one end user request will prompt update the entity _expires property at the same time, which will exceed the 1qps limit on Datastore entity groups. As long as one of these succeeds, the session is freshened, that's all that really matters.
The good news is that this will be solved anyway with the migration of Datastore customers to Firestore, which eliminates the 1qps write rate limit. [1]
It's worth pointing out here, that contention and hotspotting are two very different things. An even distribution of values (not monotonically increasing ones) is used to avoid hotspotting, where one tablet server has too much of the "interesting" data you are trying to work with. Datastore scales by splitting the keyspace(index) to avoid concentrating reads and writes on a single server (which holds a range of keys).
Contention ("Too much contention on these datastore entities" ) is caused by attempting to write to an entity group too often.
You will not receive explicit errors for hotspotting. It will simply appear as poor write throughput and higher read latency, including timeouts.
[1]
rm...@google.com
On the topic of the issue, we actually need the property to be indexed as part of our Jetty session management code. We don't really see any evidence of hotspotting as we usually won't see write rates to new entities and this property that would manifest that.
ji...@collavate.com
We had a support case on this issue , [Case#16360521] java.util.ConcurrentModificationException with _ah_SESSION [ref:_00D00VNwG._500f21FDZPi:ref]
You can find evidence of hotspotting there. Regardless, I will be moving the api with this issue to a different module without sessions. that seems to be the only solution so far.
You can find evidence of hotspotting there. Regardless, I will be moving the api with this issue to a different module without sessions. that seems to be the only solution so far.
vo...@gmail.com
Thanks for clarifying the difference between contention and hotspotting. Every time I used the term 'contention', I actually meant 'hotspotting'.
My concern is with overloading computers responsible for storing index records of the _expires property. Not with overloading computers which store the actual _ah_SESSION entities. I didn't know there were two distinct terms for these things. Apologies for being a bit misleading.
As mentioned in your post, an even distribution of values (not monotonically increasing ones) is used to avoid hotspotting. But since the _expires property DOES contain monotonically increasing values AND is being indexed, its index records will NOT be evenly distributed. I was never concerned with exceeding the 1 write/s limit for some particular entity group. But rather with overloading a single tablet server. Again, sorry for misleading you by using the word 'contention'.
Anyway, thanks for giving this a thought. Cheers!
PS: Like others, I ended up disabling the GAE sessions and using signed cookies and stuff like that to do things a bit more efficiently.
My concern is with overloading computers responsible for storing index records of the _expires property. Not with overloading computers which store the actual _ah_SESSION entities. I didn't know there were two distinct terms for these things. Apologies for being a bit misleading.
As mentioned in your post, an even distribution of values (not monotonically increasing ones) is used to avoid hotspotting. But since the _expires property DOES contain monotonically increasing values AND is being indexed, its index records will NOT be evenly distributed. I was never concerned with exceeding the 1 write/s limit for some particular entity group. But rather with overloading a single tablet server. Again, sorry for misleading you by using the word 'contention'.
Anyway, thanks for giving this a thought. Cheers!
PS: Like others, I ended up disabling the GAE sessions and using signed cookies and stuff like that to do things a bit more efficiently.
Description
Suggested solution:
Allow developers to choose whether they want to index the _expires property. Also, allow developers to configure App Engine to update _ah_SESSION entities only when data in the session change. These decisions could be ideally declared by adding some rules to appengine-web.xml, for example:
<session-index-expiration>false</session-index-expiration>
<session-write-when-needed>true</session-write-when-needed>
Benefits:
No contention risk and giving developers possibility to make significant performance improvements. Updating _ah_SESSION entity will cost only 2 writes and not having to store it after every request will save time as well.
Downsides:
Developers who choose not to index the _expires property will have to crawl through all _ah_SESSION entities to find and remove expired sessions. This, however, seems to be a fair price to pay for avoiding contention.
Not updating the expiration timestamp of _ah_SESSION after every request can lead to session unexpectedly expiring in between requests. Developers could, however, easily enforce expiration timestamp update by making some dummy change to session data. It would be up to the developer to decide which requests should trigger the expiration timestamp update. This would allow optimization choices like: It's ok for a user to get unexpectedly logged out during casual browsing of Wikipedia articles.
PS: Implementing this feature solves (or at least closely relates) to the following feature requests and issues risen on Stack Overflow:
Btw, according to the following feature request, the _expires field was originally UNindexed. And back in 2010, there seemed to be no plans to change that: