Workaround for JDK-8272564 seems to be required on API level 28-30 [271408544]

Fixed

Bug

Status Update

No update yet.

Description

sg...@google.com

created issue #1

Mar 2, 2023 01:15PM

From the issue https://github.com/android/android-test/issues/1642 reported on Espresso 3.5.1 it looks as if the rewrite for JDK-8272564 need to be applied to API level 28-30.

This was first applied before 26, but after b/231450655 it was also applied on API level 26 and 27. However the Espresso issue indicate that this can also cause issues on API level 28-30.

The failure is

java.lang.IncompatibleClassChangeError: Class 'org.hamcrest.StringDescription' does not implement interface 'java.lang.Iterable' in call to 'java.util.Iterator java.lang.Iterable.iterator()' (declaration of 'org.hamcrest.BaseDescription' appears in /data/app/~~aq_hVVuOv1vc2xlxRbeSpw==/gmk57.testespresso.test-gmCldUGeO36ETzXPU6Zi0g==/base.apk)
at org.hamcrest.BaseDescription.appendValueList(BaseDescription.java:76)
at androidx.test.espresso.matcher.ViewMatchers.getMismatchDescriptionString(ViewMatchers.java:3)
at androidx.test.espresso.matcher.ViewMatchers.assertThat(ViewMatchers.java:5)
...

The code as androidx.test.espresso.matcher.ViewMatchers.getMismatchDescriptionString(ViewMatchers.java:3) is

    .line 3
    invoke-interface {v0}, Lorg/hamcrest/Description;->toString()Ljava/lang/String;

    move-result-object p1

    invoke-virtual {p1}, Ljava/lang/String;->trim()Ljava/lang/String;

    move-result-object p1

    .line 4

How that ends up in org.hamcrest.BaseDescription.appendValueList(BaseDescription.java:76) I cannot explain.

.method public appendValueList(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/Iterable;)Lorg/ham
crest/Description;
    .locals 1
    .param p1, "start"    # Ljava/lang/String;
    .param p2, "separator"    # Ljava/lang/String;
    .param p3, "end"    # Ljava/lang/String;
    .annotation system Ldalvik/annotation/Signature;
        value = {
            "<T:",
            "Ljava/lang/Object;",
            ">(",
            "Ljava/lang/String;",
            "Ljava/lang/String;",
            "Ljava/lang/String;",
            "Ljava/lang/Iterable<",
            "TT;>;)",
            "Lorg/hamcrest/Description;"
        }
    .end annotation

    .line 76
    .local p4, "values":Ljava/lang/Iterable;, "Ljava/lang/Iterable<TT;>;"
    invoke-interface {p4}, Ljava/lang/Iterable;->iterator()Ljava/util/Iterator;

If the JDK-8272564 rewrite is applied the java.lang.IncompatibleClassChangeError does not happen.

Comments

sg...@google.com <sg...@google.com> #2Mar 2, 2023 01:45PM

Forgot to mention that all this only happens in the cloud, dev server works fine.

ap...@google.com <ap...@google.com> #3Mar 2, 2023 03:10PM

Marked as fixed.

Ah, u'=C5=BCar=C3=B3wka1111111111111111'.decode('quotedprintable') breaks on my local
python 2.5 as well. 2.6 works fine, so nothing appengine specific here. It can be
worked around by doing .encode('ascii') on the unicode string first.

ap...@google.com <ap...@google.com> #4Mar 3, 2023 07:42AM

Project: r8
Branch: main

commit 9e5692c8ce91b3ea3f4199740b41d57c097c9947
Author: Søren Gjesse <sgjesse@google.com>
Date: Fri Mar 03 08:40:44 2023

Fix test after changing API level for rewriting of JDK-8272564

Bug:

b/271408544

Change-Id: I67552a7fb13657ac9b4fb706592754d74d1b0653

M src/test/java/com/android/tools/r8/desugar/jdk8272564/Jdk8272564Test.java

https://r8-review.googlesource.com/76885

ap...@google.com <ap...@google.com> #5Mar 3, 2023 11:19AM

[Comment deleted]

ap...@google.com <ap...@google.com> #6Mar 3, 2023 11:19AM

This problem happens with additional parameters (to HTTP-POST), and it's very hard to work around because it encodes to base64 only if the string contains international characters.

ap...@google.com <ap...@google.com> #7Mar 3, 2023 11:19AM

Project: r8
Branch: 8.0

commit 2a7c539ec35668c77b0eacb27612d7a5d8a2d11a
Author: Søren Gjesse <sgjesse@google.com>
Date: Fri Mar 03 08:43:21 2023

Also apply the rewriting for JDK-8272564 up to Andorid R

Bug:

b/271408544

Change-Id: I240de7d301677668b9c13b3b49c7b7951e6434b6

M src/main/java/com/android/tools/r8/utils/InternalOptions.java

https://r8-review.googlesource.com/76886

ap...@google.com <ap...@google.com> #8Mar 3, 2023 11:19AM

Project: r8
Branch: 4.0

commit a53fc28f4b588c1e7c0df8c36faf9559292b1d0f
Author: Søren Gjesse <sgjesse@google.com>
Date: Fri Mar 03 08:44:54 2023

Version 4.0.55

Bug:

b/271408544

Change-Id: I9dceedf684646380d8cff6f7af2eb3bcbd610a72

M src/main/java/com/android/tools/r8/Version.java

https://r8-review.googlesource.com/76891

ap...@google.com <ap...@google.com> #9Mar 3, 2023 11:19AM

Here's a workaround if the page is UTF-8:

# Upload page, starts blobstore session
class MainHandler(webapp.RequestHandler):
def get(self):
upload_url = blobstore.create_upload_url('/upload')
self.response.out.write('<html>')
self.response.out.write("""<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>""")
self.response.out.write('<body>')
self.response.out.write('<form action="%s" method="POST" enctype="multipart/form-data">' % upload_url)
self.response.out.write("""Upload File: <input type="file" name="file"><br>
Some value: <input type="text" name="value" value="value"/>
<input type="submit" name="submit" value="Submit"> </form></body></html>""")

# Upload Handler, decodes and forces into correct format
class UploadHandler(blobstore_handlers.BlobstoreUploadHandler):
def post(self):
upload_files = self.get_uploads('file') # 'file' is file upload field in the form
value = self.request.get("value")
value = unicode(base64.b64decode(value), "utf-8")
blob_info = upload_files[0]
self.redirect('/show/%s' % blob_info.key())

ap...@google.com <ap...@google.com> #10Mar 3, 2023 11:19AM

This workaround results in a UnicodeError for me.

UnicodeDecodeError: 'utf8' codec can't decode byte 0xa0 in position 2: unexpected code byte

(The site is using utf-8)

ap...@google.com <ap...@google.com> #11Mar 3, 2023 11:19AM

Any updates??? The workaround is not working for me also. I am trying to post a form with text input and value 'Čučoriedka'...getting

UnicodeDecodeError: 'utf8' codec can't decode byte 0xcb in position 1: unexpected end of data

BTW 'Čučoriedka' mingled into =C8u=E8oriedka

ap...@google.com <ap...@google.com> #12Mar 3, 2023 11:19AM

Was disappointed that this was not fixed in last release. Basically this makes it impossible to include non-English text with a file upload. I would say that this is a pretty serious limitation of the Python runtime. It is more than a year since this bug was reported. Please fix it, or at least provide a monkey patch.

ap...@google.com <ap...@google.com> #13Mar 3, 2023 11:19AM

The workaround that I have successfully implemented is to encode all values in base64 using JavaScript and decode them on the server. Far from ideal, but it got the job done.

ap...@google.com <ap...@google.com> #14Mar 3, 2023 11:20AM

Hate to be spamming everyone, but could you please shed some light on these mysterious labels (squash-1 log-3482676)? They sound hopeful, but it would be nice to hear a definite answer. Should I be working around this problem or is it going to be fixed Really Soon Now?

ap...@google.com <ap...@google.com> #15Mar 3, 2023 11:20AM

I believe I've found a pretty simple solution to this. Please try adding accept-charset="utf-8" as an attribute to your blobstore <form> tag. This will force the user's browser to encode all string values in the form data as utf-8.

I've successfully tested this for a blobstore app of mine that I could break with unicode characters.

ap...@google.com <ap...@google.com> #16Mar 3, 2023 11:20AM

To be more clear, in a JSP it would look something like the following:
<form action="<%= blobstoreService.createUploadUrl("/upload") %>" method="post" enctype="multipart/form-data" accept-charset="utf-8">

and for Python it would look something like the following:
<form action="%s" method="POST" enctype="multipart/form-data" accept-charset="utf-8">' % upload_url

ap...@google.com <ap...@google.com> #17Mar 3, 2023 11:20AM

accept-charset will still return base64 encoded strings. Sometimes these string can't be decoded back to the user's input.

ap...@google.com <ap...@google.com> #18Mar 3, 2023 11:22AM

Sorry to have you waited so long.

Short version:

Try attached appengine_config.py if you're using webapp framework.

A bit longer version:

The root cause of this issue is that the webob module bundled in app engine SDK just ignores charset and transfer encoding setting specified in a header part of each multipart field. The attached file intends to override this behavior by monkey patching.

Could you try attached file and tell us how it goes?

The longest version:
I'm still drafting...

ap...@google.com <ap...@google.com> #19Mar 3, 2023 11:22AM

Thank you very much. I will try the monkey patch and report back.

sg...@google.com <sg...@google.com> #20Mar 3, 2023 12:03PM

issue 4265

related?

tt...@gmail.com <tt...@gmail.com> #21Mar 9, 2023 02:25AM

I have this issue in latest SDK with russian letters :(

Solution from comment 21
Works for me

ro...@ebay.com <ro...@ebay.com> #22Jun 14, 2023 10:36PM

I solved this problem by using chardet.

import chardet

def blob_decode(str):
encoding = chardet.detect(str)['encoding']
return str.decode(encoding)

text = blob_decode(base64.b64decode(text))

Using chardet, I could detect encoding of original text and decode properly.

sg...@google.com <sg...@google.com> #23Jun 20, 2023 09:32AM

I tried the appengine_config.py patch and it does appear to work except there appears to be a bug. The indentation for "obj.add(

field.name, field_value)" causes field_value to be set from a prior loop when field.filename is present.

Issue 271408544

Description

Issue summary

Comments

sg...@google.com <sg...@google.com> #2Mar 2, 2023 01:45PM

ap...@google.com <ap...@google.com> #3Mar 2, 2023 03:10PM

ap...@google.com <ap...@google.com> #4Mar 3, 2023 07:42AM

ap...@google.com <ap...@google.com> #5Mar 3, 2023 11:19AM

ap...@google.com <ap...@google.com> #6Mar 3, 2023 11:19AM

ap...@google.com <ap...@google.com> #7Mar 3, 2023 11:19AM

ap...@google.com <ap...@google.com> #8Mar 3, 2023 11:19AM

ap...@google.com <ap...@google.com> #9Mar 3, 2023 11:19AM

ap...@google.com <ap...@google.com> #10Mar 3, 2023 11:19AM

ap...@google.com <ap...@google.com> #11Mar 3, 2023 11:19AM

ap...@google.com <ap...@google.com> #12Mar 3, 2023 11:19AM

ap...@google.com <ap...@google.com> #13Mar 3, 2023 11:19AM

ap...@google.com <ap...@google.com> #14Mar 3, 2023 11:20AM

ap...@google.com <ap...@google.com> #15Mar 3, 2023 11:20AM

ap...@google.com <ap...@google.com> #16Mar 3, 2023 11:20AM

ap...@google.com <ap...@google.com> #17Mar 3, 2023 11:20AM

ap...@google.com <ap...@google.com> #18Mar 3, 2023 11:22AM

ap...@google.com <ap...@google.com> #19Mar 3, 2023 11:22AM

sg...@google.com <sg...@google.com> #20Mar 3, 2023 12:03PM

tt...@gmail.com <tt...@gmail.com> #21Mar 9, 2023 02:25AM

ro...@ebay.com <ro...@ebay.com> #22Jun 14, 2023 10:36PM

sg...@google.com <sg...@google.com> #23Jun 20, 2023 09:32AM

Add comment

Issue metadata