Status Update
No update yet.
Comments
ya...@google.com
Hello Kévin
Thank you for contacting Google Cloud Platform Support. I understand you are having an issue with internal load balancing and I will do my best to advise you.
Internal load balancers are regional resources, the scope of an internal TCP/UDP load balancer is regional, not global. This means that an internal TCP/UDP load balancer cannot span multiple regions. Within a single region, the load balancer services all zones. [1]
A feature request exists to enable an internal load balancer which can handle requests from multiple regions[2]. But I can neither provide you with an ETA for its implementation, nor a guarantee that it will ever be supported. If you would like to create your own feature request, follow the link to Google issue tracker to search or create feature request [3].
Your VMs are able to talk to each other from one region to another as they are connected through the same VPC. Resources within a VPC network can communicate with one another using internal (private) IPv4 addresses, subject to applicable network firewall rules.[4] However, for ILB the behaviour is handled differently due to the regional nature of the ILB(s).
The reason VM instances attached to a network with a specific address mask getting assigned IP with /32 mask is a design requirement of GCP. However, let me explain why VM instances are assigned with the /32 mask on Google Compute Engine (GCE). Please note that the /32 is an artificial construct. The instance talks to the software defined network, which creates and manages the "real" subnets. So, it is really a link between the single address and the gateway for the subnet. As long as the link layer is there, communication is established, and everything works.
In addition to that, network masks are enforced at the network layer, not the host layer. This helps avoid generation of unnecessary broadcast traffic (which underlying network wouldn't distribute anyway), this might further clarify why VM instances are assigned with a /32 netmask, when they are attached to a subnet, for example, /24 mask.
I hope that makes things clearer for you. If you have any other questions or concerns about your issue, please do not hesitate to contact me by replying to this message. I will be more than happy to help at that time.
[1]ILB Scope:https://cloud.google.com/load-balancing/docs/internal/#protocols_scheme_and_scope
[2]Feature Request for multi-region ILB:https://issuetracker.google.com/111021512
[3]Create and Search feature requests:https://cloud.google.com/support/docs/issue-trackers#search_and_create_feature_requests_by_product
[4]VPC Specifications:https://cloud.google.com/vpc/docs/vpc#specifications
Thank you for contacting Google Cloud Platform Support. I understand you are having an issue with internal load balancing and I will do my best to advise you.
Internal load balancers are regional resources, the scope of an internal TCP/UDP load balancer is regional, not global. This means that an internal TCP/UDP load balancer cannot span multiple regions. Within a single region, the load balancer services all zones. [1]
A feature request exists to enable an internal load balancer which can handle requests from multiple regions[2]. But I can neither provide you with an ETA for its implementation, nor a guarantee that it will ever be supported. If you would like to create your own feature request, follow the link to Google issue tracker to search or create feature request [3].
Your VMs are able to talk to each other from one region to another as they are connected through the same VPC. Resources within a VPC network can communicate with one another using internal (private) IPv4 addresses, subject to applicable network firewall rules.[4] However, for ILB the behaviour is handled differently due to the regional nature of the ILB(s).
The reason VM instances attached to a network with a specific address mask getting assigned IP with /32 mask is a design requirement of GCP. However, let me explain why VM instances are assigned with the /32 mask on Google Compute Engine (GCE). Please note that the /32 is an artificial construct. The instance talks to the software defined network, which creates and manages the "real" subnets. So, it is really a link between the single address and the gateway for the subnet. As long as the link layer is there, communication is established, and everything works.
In addition to that, network masks are enforced at the network layer, not the host layer. This helps avoid generation of unnecessary broadcast traffic (which underlying network wouldn't distribute anyway), this might further clarify why VM instances are assigned with a /32 netmask, when they are attached to a subnet, for example, /24 mask.
I hope that makes things clearer for you. If you have any other questions or concerns about your issue, please do not hesitate to contact me by replying to this message. I will be more than happy to help at that time.
[1]ILB Scope:
[2]Feature Request for multi-region ILB:
[3]Create and Search feature requests:
[4]VPC Specifications:
yu...@google.com
Thanks for your quick reply on ILB and thank you also for details on networking on gcp.
I hope that this feature requesthttps://issuetracker.google.com/111021512 will be considered :).
I am surprised that it has been designed like this because it's a pretty big drawbacks compared to AWS for example.
Do you know if there is another solution for me to interconnect local service to another region except create loadbalancer in a dedicated instance ?
Thanks in advance,
Kévin
I hope that this feature request
I am surprised that it has been designed like this because it's a pretty big drawbacks compared to AWS for example.
Do you know if there is another solution for me to interconnect local service to another region except create loadbalancer in a dedicated instance ?
Thanks in advance,
Kévin
ra...@google.com
At the time of writing this, creating a dedicated instance which will serve as a load balancer between regions, is the best workable option for this type of scenario.
cl...@ck-na.com
I have the exact same feature request to access ILB from multi regions. It would be great if Google can prioritize this feature. Thanks.
cl...@ck-na.com
Hi, please take a look at my update to the public feature request:
https://b.corp.google.com/issues/111021512#comment37
You'll be pleased to note that GKE 1.16 allows you to annotate a service of type LoadBalancer such that it is both internal and globally accessible. We can close out this issue now.
You'll be pleased to note that GKE 1.16 allows you to annotate a service of type LoadBalancer such that it is both internal and globally accessible. We can close out this issue now.
ya...@google.com
@raging Honestly I didn't find any suspicious log. But from the behavior, could it be that ARCVM didn't launch automatically (until user clicks)?
ul...@google.com
NOTE: One of our edu customers just reported this issue is affecting him when trying to use com.contentkeeper.cloudexpresstp android app as force installed from Admin Console.
yu...@google.com
@hidehiko Do you think this is related to the deferral here?
2025-02-28T18:53:22.136251Z VERBOSE1 chrome[1373:1373]: [arc_session_manager.cc(1060)] ARC opt-in. Starting ARC session.
2025-02-28T18:53:22.136804Z VERBOSE1 chrome[1373:1373]: [arc_session_manager.cc(1348)] ARC activation is deferred until user sesssion start up tasks are completed
hi...@google.com
Looking at the log. Indeed, ARC is not activated after deferring. That means, in the user session, (not limited to ARC's) session start up task is not completed at all. I was giving it a try to enter into managed session by using my testing account, and giving it a try the steps in #1, but it does not reproduce. So, I think this is happening under some particular conditions for EDU (or some commercial specific ones).
Do we have any repro environment we can use for testing/investigation?
hi...@google.com
Note: deferring was enabled on M124 (crrev.com/5363013), so something else is updated, i guess.
ro...@gmail.com
Hello,
We are monitoring this ticket, which is affecting many of our customers around the world. The condition is that the Android app is missing so our tamperproofing mechanisms do not allow students to use the internet until the Playstore is restored and the application loads. The impact on our education customers is that they cannot participate in classroom activities or take state tests.
I see the issue has been raised to P1, thanks! Do we know which component update in v132 has caused this and is there a way to remove it and release a new client to solve the issue? This is affecting hundreds of thousands of students and hundreds of customers around the world.
Rex Corkin
VP of Operations
Ativion
C: 530.282.6270
E: rcorkin@ativion.com
We are monitoring this ticket, which is affecting many of our customers around the world. The condition is that the Android app is missing so our tamperproofing mechanisms do not allow students to use the internet until the Playstore is restored and the application loads. The impact on our education customers is that they cannot participate in classroom activities or take state tests.
I see the issue has been raised to P1, thanks! Do we know which component update in v132 has caused this and is there a way to remove it and release a new client to solve the issue? This is affecting hundreds of thousands of students and hundreds of customers around the world.
Rex Corkin
VP of Operations
Ativion
C: 530.282.6270
E: rcorkin@ativion.com
ob...@google.com
Hello
- Do you have a list of the affected users so far?
- And times when they started experiencing this issue?
ro...@gmail.com
Yes,
I have 52 school districts who have reported it. Here is a couple Google Support tickets which have been opened but they are telling the customers to powerwash or roll back which is extremely disruptive for a school system, some with 150,000 students. Some customers are having trouble rolling back as well.
G-56938618
G-57205582
Here are a few of the school districts of the 52 that have reported it. They have been reporting the problem since the release of v132 which your team confirmed other customers were reporting in Google ticket 57205582 - See attached image.
Buckingham County Public Schools (VA30100469)
Redondo Beach Unified (CA30019529)
Morenci Unified District (AZ30053965)
Waxahachie Independent School District (TX30020144)
Hamilton School District (WI30123700)
Katy Independent School District (TX30128896)
Bennington Public Schools (US30099878)
Buckingham County Public Schools
Scarsdale Public Schools
Savannah-Chatham County Public Schools (GA30052344)
Greenwood School District
Any help you can provide would be appreciated. Our customers are expecting us to fix this but our hands are tied and they have no solution if they can't roll the OS back.
Rex Corkin
VP of Operations
Ativion
C: 530.282.6270
E: rcorkin@ativion.com
I have 52 school districts who have reported it. Here is a couple Google Support tickets which have been opened but they are telling the customers to powerwash or roll back which is extremely disruptive for a school system, some with 150,000 students. Some customers are having trouble rolling back as well.
G-56938618
G-57205582
Here are a few of the school districts of the 52 that have reported it. They have been reporting the problem since the release of v132 which your team confirmed other customers were reporting in Google ticket 57205582 - See attached image.
Buckingham County Public Schools (VA30100469)
Redondo Beach Unified (CA30019529)
Morenci Unified District (AZ30053965)
Waxahachie Independent School District (TX30020144)
Hamilton School District (WI30123700)
Katy Independent School District (TX30128896)
Bennington Public Schools (US30099878)
Buckingham County Public Schools
Scarsdale Public Schools
Savannah-Chatham County Public Schools (GA30052344)
Greenwood School District
Any help you can provide would be appreciated. Our customers are expecting us to fix this but our hands are tied and they have no solution if they can't roll the OS back.
Rex Corkin
VP of Operations
Ativion
C: 530.282.6270
E: rcorkin@ativion.com
gr...@google.com
Support, can we please get more data about schools reporting they cannot perform an OS rollback? Has this issue been validated in the field?
ro...@gmail.com
Hello,
For clarity, a few have reported that rolling back does not work but many are unable to rollback due to the impact on students. The challenge we have is that they are saying they should not have to roll back because our product is not working.
The best case scenario is that you are able to identify the bug, correct it in a new version and make it available. Can you let me know the probability of this?
Rex Corkin
VP of Operations
Ativion
C: 530.282.6270
E: rcorkin@ativion.com
For clarity, a few have reported that rolling back does not work but many are unable to rollback due to the impact on students. The challenge we have is that they are saying they should not have to roll back because our product is not working.
The best case scenario is that you are able to identify the bug, correct it in a new version and make it available. Can you let me know the probability of this?
Rex Corkin
VP of Operations
Ativion
C: 530.282.6270
E: rcorkin@ativion.com
ro...@gmail.com
Here is some feedback from a customer in California - Atascadero
Vincent Regalado
Mar 17, 2025, 11:55 AM EDT
Hi Rex, thanks for reaching out. Rolling back the updates to version 130 at the High School has been hit or miss on resolving the issues. Some kids still have to open the play store and wait for it to download, others it has fixed the problem though.
Vincent Regalado
Mar 17, 2025, 11:55 AM EDT
Hi Rex, thanks for reaching out. Rolling back the updates to version 130 at the High School has been hit or miss on resolving the issues. Some kids still have to open the play store and wait for it to download, others it has fixed the problem though.
of...@google.com
Thanks for the clarification Rex.
We are investigating the issue. As we are working with multiple customers, we need to limit access to this bug. A separate bug will be open internally for this issue.
If you are experiencing this issue, please ensure that you have a support case open [1], and mention
Description
It's also helpful if you submit feedback logs through the use of "alt + shift + i" and utilize a unique identifying #key so that we can easily identify this and look up your system logs for further debugging.
Feel free to also attach any screenshots or logs directly here as well.
================================
Chrome Version: <From about:version: Google Chrome x.x.x.x> 133.0.6943.146
Chrome OS Version: <From about:version: Platform x.x.x.x> 16151.53.0
Chrome OS Device : <Make/model of computer running Chrome OS> Varies
ARC Version: <Android 7/9/11/etc> 13017017
Network info: <network, encryption type, router model (if known)>
Widely Varies - Between thousands of affected devices this parameter is not applicable
Steps To Reproduce:
(1) Using a managed Chromebook(Google Workspace) that has deployed Apps from Google Play Store:
(2) Upgrade or Powerwash a Chromebook to ChromeOS 132+
(3) Applications will not auto-launch until the Play Store is *manually* launched from the user
Expected Result:
Upon upgrade or powerwash (to ChromeOS 132+), Google Play should download automatically install, autolaunch, then subsequently install managed applications.
Actual Result:
Google Play appears to download but does not actually install until manually selected by the user. Other managed Applications appear installed (icons) but are uncached between restarts unless the Google Play Store is manually launched at least once.
How frequently does this problem reproduce? (Always, sometimes, hard to reproduce?)
Sometimes
What is the impact to the user, and is there a workaround? If so, what is it?
Customers using Google Workspace for Education depend on managed applications to function for their larger student population. Even a percentage of those users results in thousands of affected devices.
I've attached 3 support logs as a reference:
All are post powerwash on ChromeOS 133
- PostPowerwash-NonFunctioning Capture-
-- General non functioning reference captured right after the powerwash
-- manually opened the com.contentkeeper.cloudexpresstp application
-- com.contentkeeper.cloudexpresstp begins initiating and launching as normal
- AfterReboot-PostPowerwash-NonFunctioning Capture-
-- Rebooted Chromebook
-- com.contentkeeper.cloudexpresstp should be cached locally and autolaunch if called from another process
---this doesn't occur due to the apps uncached status
-OpenPlayStore-NonFuncitoning-to-Functioning Capture-
--Opened Google Play Store after a reboot
-- Google Play Appears to Install as if uncached despite being deployed to the device in the previous captures
-- com.contentkeeper.cloudexpresstp application also installs in the background
-- subsequent restarts behave as intended with applications remaining cached and allowed to autolaunch