IssueTracker

Thank you for the report.

When the compile time fluctuate this much, it could be caused by the JVM getting close to max memory and the GC portion of the compile time goes up.

As you have a dump, then you can easily reproduce the same build with different settings using tools/compiledump.py. tools/compiledump.py use the gsutil to download the compiler from our build repository. If you don't have that tool, you can get the compiler with curl -O https://storage.googleapis.com/r8-releases/raw/8.3.35/r8lib.jar (for version 8.3.35), and then add --r8-jar r8lib.jar to the commands below. The commands all pass -da which disables assertions. tools/compiledump.py enabled assertions by default, but that makes compilation even slower, and AGP does not do that.

1. Are you seeing the same loooooong compile time locally when running tools/compiledump.py -da -d <dump>?
2. If so you can try to pass the -verbose:gc to the JVM when running, should be doable like this: tools/compiledump.py -da -J=-verbose:gc -d <dump>. Seems like the output is not streamed so you will have to wait for the command to finish.
3. If that shows massive GC, you can try different heap sizes, to see if changing that affects compilation time, e.g. like this: tools/compiledump.py -da --xmx 8G -J=-verbose:gc -d <dump>

If possible you can share the dump privately with sgjesse@google.com and ricow@google.com, then we can also try to see if we can reproduce the issue.

The CI calculates the values for xmx etc... depending on the current machine. The one use here should have enough resources to build the app without issues.

This is what it calculated for this particular machine, which has 32 GB of RAM (properties written inside of ~/.gradle/gradle.proprties).

org.gradle.jvmargs = -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -XX:MaxMetaspaceSize=1g -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -Xmx16854m -Xms16854m
kotlin.daemon.jvmargs = -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -XX:MaxMetaspaceSize=1g -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -Xmx7223m -Xms7223m

I let gradle and kotlin use 75% of the total memory, 70% of that is for gradle, the remaining 30% for kotlin.

I once ran the build with -Dcom.android.tools.r8.printmemory, but the CI stopped after it reached its limit of 3.5 hours. This is what I saw:

R8 is running with total memory:17683185664
R8 is running with free memory:14891000672
R8 is running with max memory:17683185664

I have multiple zips inside the directory of dumpinputtodirectory and compiledump.py takes roughly 10 minutes to process the largest one. Even when I pass --Xmx 8G.

I'll try to play a bit with the memory settings and see if that's the problem, as you suspect. I could also try to run R8 in a separate process as mentioned here.

I'll try to can figure this out and I'll see if I can share the dump in case I can't find anything.

Thank you for the prompt response.

I think you were right about this being a memory issue. I added the following and even after multiple tries, the minification step took 10 minutes consistently.

android {
    execution {
        profiles {
            standard {
                r8 {
                    jvmOptions += ["-Xms2048m", "-Xmx8192m", "-XX:+HeapDumpOnOutOfMemoryError"]
                    runInSeparateProcess = true
                }
            }
        }

        defaultProfile = "standard"
    }
}

I don't know if something has changed, I thought 16GB were enough for gradle and R8 combined, considering that a build without R8 requires less than 8GB.

Anyway, thank you for the pointer, I think you can close this now.

Thank you for the feedback, good to hear that you found a solution.

I had already written what is below, so sharing this anyway.

Each dump has the file build.properties in the root. The one which has tool=R8 is the R8 invocation.

One other tip is to unzip the dump unzip dump.zip -d /tmp/dump, and add --temp /tmp/dump to running tools/compiledump.py. Then the command printed by tools/compiledump.py (shuld be like the one below) can be easily re-run with JVM different options without foing through the tools/compiledump.py wrapper.

/usr/local/google/home/sgjesse/prj/r8/ws1/third_party/openjdk/jdk-11/linux/bin/javac /usr/local/google/home/sgjesse/prj/r8/ws1/src/main/java/com/android/tools/r8/utils/CompileDumpCompatR8.java /usr/local/google/home/sgjesse/prj/r8/ws1/src/main/java/com/android/tools/r8/utils/CompileDumpBase.java -d /tmp/dump -cp r8lib.jar
Running: /usr/local/google/home/sgjesse/prj/r8/ws1/third_party/openjdk/jdk-11/linux/bin/java -verbose:gc -cp /tmp/dump:r8lib.jar com.android.tools.r8.utils.CompileDumpCompatR8 --release /tmp/dump/program.jar --output /tmp/dump/out.jar --lib /tmp/dump/library.jar --classpath /tmp/dump/classpath.jar --desugared-lib /tmp/dump/desugared-library.json --desugared-lib-pg-conf-output /tmp/dump/desugared-library-keep-rules.config --pg-conf /tmp/dump/proguard.config --pg-map-output /tmp/dump/out.jar.map --min-api 23 --enable-missing-library-api-modeling

Thank you for waiting!

We have verified and logged this issue internally. Please note that we cannot give you any timelines, but you can star the issue to get notifications.

About the navigation issue, a ticket is already opened here: https://issuetracker.google.com/issues/217936671

About the missing blue lines, it is indeed caused by a change in the tolerance about how far panoramas can be from a road line on the map. They lowered it from ~45 meters to 10 meters and it was a disaster. There is not mystery there, the only solution is to revert this change. I do not even see why this change was made in the first place, there is no need for it. This only caused trouble for everyone, notably API users who pay for a broken service.

Especially, 10 meters is absurdly low. Many roads are wider than this yet still represented by a single road line on the map. This means a Google car driving on the outer edge of a large road may be already too far from the road line to have its pictures attached to it, causing coverage to be removed.

Attached is an image showing an example with a T-junction where cars can turn on the right. The Google car came from the northeastern road and turned right towards west. Since the car used the rightmost lane to turn, it went away from any existing road line up to 19 meters. This created a gap in the blue line, blocking any navigation as you can experience here: https://goo.gl/maps/dAwF5vRgcACDUuJo9

This is a small example but it's big enough to block navigation, causing trouble for applications or Google Maps users. Overall we are talking about tens of thousands of similar cases all over the world causing the loss of thousands of kilometers, all of this because of a simple but useless change.

Please revert this change. It really makes no sense considering how imprecise Google Maps is in so many places.

Wait... I remember the threshold value being lowered to 9. That doesn't really fix the issue, as it only removes all the past imagery from the September 2021 update, including all early Gen 4 content circa March to September 2017.

Seriously? We're still not finished. :(

Thank you for the additional information!

To further check the issue, it appears that we need to reopen this issue and consult this to our specialists. That said, may I ask the following from you:

1. What are the steps on how to reproduce the issue?
2. What coordinates (latitude, longitude) where the broken street views are visible?
3. Adding screenshots would also be helpful.

The broken coverage situation affects almost the entire world. (See comment #4 for details.) To say this once more, past imagery from the September 2021 update also disappeared in the US and Japan when on March 1 Google apparently managed to reduce the threshold value from 10 to 9, which is not effective. I would personally roll this removal back and restore those images when the bug gets fixed later on.

Screenshots exhibiting this behavior are found here: https://support.google.com/maps/thread/132766700/since-september-street-view-is-broken-everywhere-and-this-is-quite-problematic?hl=en

And on my end, what I mean by 'past imagery from the September 2021 update', it's these panos, most in Gen 4 quality: https://goo.gl/maps/JSPg2nm4C76MAVqv7 (dated March 2017), https://goo.gl/maps/3FojuB2m3pt6Qf768 (dated April 2017), https://goo.gl/maps/3PPasyHL8FBvTNT69 (dated May 2017), https://goo.gl/maps/3rJYZGw52YSUsLRv8 (dated June 2017), https://goo.gl/maps/rTTLhR9VL37hGjxe6 (dated July 2017), https://goo.gl/maps/ZW6imaxm7ZkkBnhH9 (dated August 2017) and https://goo.gl/maps/cTmaoVP7i3vrHVZx5 ^ (dated September 2017), plus https://goo.gl/maps/gQDs69TJDNkTyM6T8 ^ (dated February 2020 in Gen 3 quality) as well as https://goo.gl/maps/TVJcGuBt9PxQ7m7x8 (dated December 2015 in Gen 3 quality), along with https://goo.gl/maps/4R3XL3sic1QZMrVu7 ^ (dated May 2017 in Gen 4 quality). Their pano IDs are still valid, but as of March 1 they're no longer accessible from the 'Time Machine' on desktop Maps. It also really disappoints me that some areas needing higher-quality or new coverage lost these images.

^The imagery's pano IDs have expired and therefore no longer available for viewing.

However, if you do click on one of these links, you'll usually be taken to the actual image which indeed exists. But if you go to the Time Machine and choose another image (excluding related captures of the same quality) or leap to another lane or the other side of the roadway (unless you're really careful), they disappear and you can never go back (but thankfully, you can still click the 'Back' button on your browser to jump to the previously-viewed pano).

Illustrating such removal of past imagery that was current: Here in the screenshot we're at this particular roadway in the Angeles National Forest, CA. For many years its imagery was never updated, and it didn't see any update until September 2021, when Google released new pictures that also included more past imagery. Initially, this particular roadway had only two revisions: Dec 2007 and Jan 2009. When the update came in September 2021, it gained Gen 4-quality pictures with the images being dated May 2017, thus making it a total of 3 revisions. Unfortunately, however, this addition did not last long. In mid-March the Gen 4-quallity images were deleted from the roadway entirely along with the other older images that were included in the 9/10/2021 update and the Time Machine once again now only shows two revisions, in rubbish Gen 1 and Gen 2 quality. So, you can't access the newer images anymore.

Quoting from comment #16: ”It's very highly likely a separate issue.”

And I’ve been trying to point the issue out myself, but sadly, they don’t seem to believe me or understand. It's all because I spoke too soon on issue 214122362, which I created myself, when I should've known that the pictures actually existed. Images from the September 2021 update with captures between the years 2008 and 2021 should have never disappeared like that.

We'll be consulting these issues to our specialists and will get back to you for updates. Thank you for your understanding.

I'm hoping the missing imagery problem gets fixed. I expect to see the data within the coming months, probably including the 9/10/2021 data.

Add-on to comment #14: This is not a complete list. Other major US cities such as Phoenix, Chicago, Washington DC, Sacramento, Seattle, Denver, San Antonio, and possibly New York City could have also gained more past imagery from the 9/10/2021 update. The most particular are additional Gen 4 content from 2017, and even the remainder of Gen 3 imagery from 2019 and 2020.

Someone reported on issue 35826204 that some older imagery is missing on a roadway in Billinge, England--specifically Upholland Road.

Also, it seems to be stated on issue 228201902 that certain 2017 and 2018 imagery on the island of Curaçao became extinct for unknown reasons. (I'm not quite sure about this, but it is a really big problem for the most part.) Oh, and it also seems that, according to issue 213477924, some parts of the United Kingdom (especially Chippenham) lost coverage.

Hmm... going back to the 9/10/2021 update, I seem to have discovered more past imagery from that update elsewhere around the world that was also affected by the March 1 removal. Here's a quick list.

• The update included 'the very first coverage in Japan, all in blurry Gen 1.' Example: https://goo.gl/maps/WqzuNcTGUXCEipg8A. The imagery had been previously removed for a long while back then because the vehicles' cameras 'were too high', enough to be a privacy concern for everyone. And yet after being restored, they were removed again on March 1, 2022.
• In San Marino, Italy, many parts of the area received some new coverage. Example: https://goo.gl/maps/RCbeA5ikVSmYGMXK7. If you close Street View after viewing the linked image, you will not be able to go back to that image as there are obviously no blue lines anymore, as illustrated in the image attached. (This image's pano ID has expired, and the link now leads you to a user-contributed photosphere.)
• Dhaka, Bangladesh also received some updates, mainly from 2021 with captures being done with a third-party camera--the same camera system that was used to update parts of Alaska in 2019. Example: https://goo.gl/maps/9vzt84opYPQqwurx5. If you go to the Time Machine and select the oldest revision, the Time Machine becomes disabled and you cannot go back to the newer images (except if you're clicking the browser's Back button). There's also small pieces of new coverage in the same area that had existed but both the images and blue lines were taken down as well: https://goo.gl/maps/U4Mf73n2E19Lt9jU6. Unfortunately, their pano IDs expired and were replaced with current 2013 imagery.

I can confirm that this is obviously a bug that Google needs to sort out, because we never used to have a faulty processing system for new imagery and the past data from that update should have never been removed in the first place.

(Add-on to comment #23) A very small number of towns in Germany also had new coverage which were affected by the March 1 removal as well. Example: https://goo.gl/maps/AJRzqNkGtoxDiw7dA. Again, exiting Street View also means you cannot go back to the imagery anymore due to the absence of blue lines. (Sadly, this image's pano ID also expired, so going there will just take you to a blank page.) Screenshot attached showing this behavior.

With Street View's 15th anniversary already arriving, it's just going to be an instant party foul because of the issues we've been facing lately with the service ever since September 2021. I wish Google can respond and fix the coverage problem right away... :'(

Any update at the moment? I'd be happy to hear if Google is working hard to restore the 9/10/2021 data.

Anything yet?

I'm quite worried that if this issue suddenly closes as a "Won't Fix (Infeasible)", then the Street View service will be left in jeopardy and the 9/10/2021 past data will become a curse for good.

Anything new yet? I'm still begging for the missing data to return, and it still hasn't happened.

Please hurry up and get the missing-data problem fixed! I already had a while.

Thank you for waiting!

We have verified and logged this issue internally. Please note that we cannot give you any timelines, but you can star the issue to get notifications.

I'm not 100% sure if Google has also acknowledged the situation on my end, because this is another thing that needs to be resolved as well. This kind of removal is totally unexpected.

I've been looking at the links I provided to the 9/10/2021 data; blue lines and associated imagery are still yet to be readded. I'll keep checking, but to be honest, it's probably going to take a very long while for Google to restore the data. Looking forward to see it again, however.

Update? Would love to hear if the data is being recovered.

(Replying to comment #36) Umm... your issue would be different. I would refer here: https://issuetracker.google.com/issues/217936671

Will this get fixed anytime soon? If the 9/10/2021 data is already being recovered, I'd be happy to hear that.

Anything new at the moment? I'm still waiting for the missing data to be recovered.

The folks at Google should be aware of this. We saw removals in some countries on September 10 last year, with more occurring in March this year. This is especially true and unfair because it took forever to upload those extra images when they were first captured and unreleased until September 2021, and now they had to be removed? That sounds very heartbreaking. :'(

Quoting from comment #43: "It has been nearly two months since I first made this."

It's also been a month since the Google experts confirmed this. I have still not seen any return of blue lines leading to the 9/10/2021 data, despite recent Street View being added.

I'm hoping Google hasn't been caught off-guard. Whoever's responsible for assigning/CCing themselves to this issue needs to update us. Whether they are working on it or found the issue to be 'Infeasible' to fix, who knows at this point of time? :\

The 9/10/2021 data's pano IDs are expiring as new imagery is being released. Google needs to take action right now because if the data is not recovered/restored, they will forever be lost.

We greatly appreciate your patience.

We believe this issue is now fixed, hence marking it as such.

Thanks for working with us!

‘Fixed’? What makes you think it has been fixed? I’m still seeing no return of blue lines leading to the 9/10/2021 data on Google Maps. Unless you have managed to find and recover the missing data (which may take a while for them to appear), there is no sign of truth at this point of time.

Thank you for the fix.

I can confirm many blue lines recovered, I see you increased the threshold from 10 meters to around ~20 meters. While is it not as good as before (~50 meters), it is already enough for most roads (though I would probably increase to around 25 meters as some areas lack of good / recent satellite imagery to correctly trace roads).

The remaining broken roads should directly be corrected on the maps because if they are still broken, it means they are too misplaced. Normally that's what Google is supposed to do with the Street View data so hopefully the map will be progressively improved.

Unfortunately, the fix did not bring back any imagery from the 9/10/2021 update. So, I created issue 241306068 as a follow-up to this.

(Replying to comment #56) And that's something I did, so take a look at issue 241306068. But honestly, the mass deletion of 9/10/2021 Street View data has been totally unexpected, and I would still count that as a part of this bug because whenever new Street View is released, it causes breaks in coverage.

Unfortunately, one user is reporting on issue 204903434 that a section of Butterfield Rd--which is residential--in San Anselmo, CA had its Street View blue lines removed (screenshot attached). So, I don't think this issue is fully fixed just yet. Speaking of deleted coverage, I'm hoping that the folks at Google are able to resolve issue 241306068.

Then what else are we supposed to do? I’ve been trying to explain that the 9/10/2021 update was not the main source of the broken coverage bug, but it rather has to do with the Street View API itself because the Street View cars rely on it for the processing and uploading of imagery. The code could possibly contain errors in it.

(P.S.: I am proactively monitoring issue 266309501 since I feel like there is no necessity of removing the Gen 4 data from 2017, because it intended to provide some quality updates to outdated coverage.)

(Replying to comment #61) Wait... so this is what we're seeing right now? Whenever we open a new ticket about the broken coverage bug, it gets closed as a 'duplicate' or takes forever for the assignee to follow back with an update? Not good. This is probably why the Maps and Street View services got reduced quality, because it makes sense that Google doesn't seem to realize what's happening right now. Maybe they're fixated on something else? :\

Quoting from comment #64: "Everyone else around here but you has basically arrived at the correct understanding that Google doesn't care anymore about this product […]"

Umm... you mean the Street View service is in a totally unsupported state? Then that doesn't sound very good. They should do something about it. :(