IssueTracker

I've also chosen "Accept all SSL certificates"/"SSL (Accept all certificates)" but it makes no difference.

Hello Magnus-

It would be helpful if you could provide some actual server addresses so that we can try similar connections (not your Email address or credentials, just the server itself).  Both for Exchange and for IMAP (they may be different).

If you are not comfortable putting that information into this bug, please send them to me directly at stadler@android.com.

Thank you,
Andy

I assume the server at mail.liu.se. I can reproduce the problem in the browser and with a small command line client. Looking at a tcpdump, the client disconnects upon receiving a valid response from the server. The problem seems to be with SSL_MODE_SMALL_BUFFERS and specifically that ssl3_read_n is not resizing the buffer if the necessary number of bytes for the next packet doesn't fit into the current buffer (near the comment /* does not happen */)

While this SSL_MODE_SMALL_BUFFERS code is not new in Gingerbread, it was reworked for openssl 1.0.0 which is new (vs 0.9.8), which probably is when the problem was introduced. Its not clear to me yet why it works with www but not mail, except that how the SSL messages are broken into packets seems different, I might guess that www is a different server version, or even entirely different OS and web server, where I'm guessing that mail is a Windows box for Exchange.

I did compare the certificate chains, mail and www do not have exactly the same server certificate chain, since www includes the root CA, which technical is optional and I would recommend to remove. However, that is not relevant to this issue.
$ openssl s_client -connect mail.liu.se:443
---
Certificate chain
 0 s:/C=SE/O=Link\xC3\xB6pings universitet/OU=LiU-IT/CN=mail.liu.se
   i:/C=NL/O=TERENA/CN=TERENA SSL CA
 1 s:/C=NL/O=TERENA/CN=TERENA SSL CA
   i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware
 2 s:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
---
$ openssl s_client -connect www.liu.se:443
---
Certificate chain
 0 s:/C=SE/O=Link\xC3\xB6pings universitet/CN=www.liu.se
   i:/C=NL/O=TERENA/CN=TERENA SSL CA
 1 s:/C=NL/O=TERENA/CN=TERENA SSL CA
   i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware
 2 s:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
 3 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
---
$

In any case, I just wanted to update this while I'm investigating to make it clear this problem is not an email specific issue.

Thank you for your fast reply.

> I assume the server at mail.liu.se.

That's correct; I'm using mail.liu.se for both ActiveSync and IMAPS.

> I might guess that www is a different server version, or even entirely different
> OS and web server, where I'm guessing that mail is a Windows box for Exchange.

Also correct; www is an Apache 2.2 on an a couple of months old customized Linux 2.6, while mail is an Exchange 2007 on Windows Server 2008 SP2.

I was able to debug this last night and the fix is pretty small and obvious in hindsight. I assume it doesn't help you as you don't seem to be running a custom build, but I'll mention it here for those that might be.

diff --git a/patches/small_records.patch b/patches/small_records.patch
index f5f2286..a2ea51c 100644
--- a/patches/small_records.patch
+++ b/patches/small_records.patch
@@ -157,7 +157,7 @@
 +              /* If we receive a valid record larger than the current buffer size,
 +               * allocate some memory for it.
 +               */
-+              if (rr->length > s->s3->rbuf.len - SSL3_RT_HEADER_LENGTH)
++              if (rr->length > s->s3->rbuf.len - SSL3_RT_HEADER_LENGTH - align)
 +                      {
 +                      if ((p=OPENSSL_realloc(s->s3->rbuf.buf, rr->length + SSL3_RT_HEADER_LENGTH + align))==NULL)
 +                              {
diff --git a/ssl/s3_pkt.c b/ssl/s3_pkt.c
index 51f21e0..c9ef2cd 100644
--- a/ssl/s3_pkt.c
+++ b/ssl/s3_pkt.c
@@ -359,7 +359,7 @@ fprintf(stderr, "Record type=%d, Length=%d\n", rr->type, rr->length);
                /* If we receive a valid record larger than the current buffer size,
                 * allocate some memory for it.
                 */
-               if (rr->length > s->s3->rbuf.len - SSL3_RT_HEADER_LENGTH)
+               if (rr->length > s->s3->rbuf.len - SSL3_RT_HEADER_LENGTH - align)
                        {
                        if ((p=OPENSSL_realloc(s->s3->rbuf.buf, rr->length + SSL3_RT_HEADER_LENGTH + align))==NULL)
                                {

> I assume it doesn't help you as you don't seem to be running a custom build

I havn't really had a good reason to until now. Guess I'll have to see how much it bothers me in the coming days. :-)  Hopefully there will be a 2.3.4 release in the not-too-distant future...

Thanks!

Just to add weight to this issue, we are seeing issues with Gingerbread.  Our software no longer can connect successfully to the Mac via HTTPS and yet we don't encounter the issue when connecting to a Windows machine.

We suspect there is something different in the timing/negotiation between the two platforms that causes the bug in one place but not the other.  We are looking for clues and wonder if this issue what we are seeing.

From the client (device's) standpoint the call to connect() never returns and the server (Mac) never gets a valid socket.

bto...@markspace.com,

This issue was specifically after the TCP connect, during SSL handshaking. With an SSLSocket connection and SSL/TLS handshaking are a bit hard to separate. When you are talking about the server not getting a valid socket, are you saying it never does a TCP handshake or that it never gets an SSLSocket?

this issue definitely was server specific, which is why it worked on one server and not another for the original reporter. it also is sensitive to the size of handshake messages which can include data that varies from server to server.

if you had a TCP dump or a public server I could connect to, I could take a look to see if its the same issue. What device are you on?

I've got the same problem with Exchange 2010 and a stock Nexus One running 2.3.3. So this is to be fixed in a future release? I cannot get either the native client or Nitrodesk Touchdown to work.

johnd...@gmail.com, yes, future release of Gingerbread. I generally mark them as such when they are committed internally but I can't comment on future releases timelines.

I would like to know if there is a supported way I can downgrade the O/S. I know there are ways to install custom ROMs and all that, but I do not wish to go that direction. That being said, Exchange support is very important to me. Is ther a supported method or any known workarounds?

johnd...@gmail.com, I honestly don't know about support for downgrading. I bet there are unsupported ways to be found on the web. Basically instead of installing a custom ROM, you'd be basically installing an older official ROM. You could try contacting Nexus One support for a more official answer: http://www.htc.com/www/support/nexusone/ 

The problem with workarounds is that even third party apps are likely to use the system libssl.so library where the problem is found. I'm not aware of any custom ROMs that have picked up this fix, so even just trying to grab a fixed libssl.so from them wouldn't be an option. Doing your own AOSP build with the above fix and installing just the libssl.so probably would work fine, but I don't know how it would affect a later OTA of the official fix, you'd probably need  to keep the original file to swap back in.

I received an OTA update to 2.3.4 (GRJ22) this morning and it now works flawlessly.

As noted by magnus.g...@gmail.com, this was released in 2.3.4.

As noted by others, my N1 is now working flawlessly. Finally. Thank you, Google!

Hello,

    I believe I am currently experiencing the above issue in Python 2.7, SSL and OpenSSL 0.9.8.

However I am having trouble reproducing this issue in our QA lab, it occurs in production when a few clients connect over a satellite connection to our server.

The small command line that created the issue, can it be shared? It would help me understand the problem and ensure our patch resolves the issue.

Thanks
Michael Gale
gale.michael@gmail.com

gale.michael, This is an issue with the Android small_records.patch for OpenSSL 1.0.0. I'm not sure what this has todo with your Python 2.7 SSL problem.

Hey,

    That is too bad, from the post I thought the issue was caused by changes
added in OpenSSL 0.9.8.

I am receiving the following and the symptoms appeared to match:
write(2, "ERROR:root:Exception in I/O handler for fd 9533
Traceback (most recent call last):
  File \"/hub/switchboard/lib/ioloop.py\", line 253, in start
    self._handlers[fd](fd, events)
  File \"/hub/switchboard/lib/stack_context.py\", line 127, in wrapped
    callback(*args, **kwargs)
  File \"/hub/switchboard/lib/iostream.py\", line 168, in _handle_events
    self._handle_read()
  File \"/hub/switchboard/lib/iostream.py\", line 360, in _handle_read
    super(SSLIOStream, self)._handle_read()
  File \"/hub/switchboard/lib/iostream.py\", line 210, in _handle_read
    chunk = self.socket.recv(self.read_chunk_size)
  File \"/hub/apps/Python-2.7/lib/python2.7/ssl.py\", line 217, in recv
    return self.read(buflen)
  File \"/hub/apps/Python-2.7/lib/python2.7/ssl.py\", line 138, in read
    return self._sslobj.read(len)
SSLError: [Errno 1] _ssl.c:1347: error:14095044:SSL
routines:SSL3_READ_N:internal error\n", 896) = 896

Michael

Even Android's OpenSSL 0.9.8 with this patch did not have this bug as I understand it. It was introduced in porting the small_records.patch to 1.0.0. If this is reproducible, you should report it to whoever provides your openssl implementation (such as your linux distribution) or perhaps to openssl-users@openssl.org. Internal errors can mean a bug which could be a security issue and should be investigated.

I am getting this issue still using K9 with a SSL smtp connection over port 465, and like others have mentioned it is specifically with a Mac email server. I have recently installed the OTA 2.3.4 update, which did not fix the issue. Does K9 use the built-in OpenSSL or include their own?

Any insights would be helpful.

mmcwill...@gmail.com, a quick look at k-9 seems to show that they are just using Java SSLSockets. However, I recently diagnosed an email SSL connectivity issue on CM where they were not using the OpenSSL implementation for SSLSocket, but you seem to be implying you are running an Google provided 2.3.4 on a Nexus device? If you tell me the server name, I try to diagnose and see if there are any SSL handshake problems with the server.

Any chance of getting this fix for Honeycomb?

My Asus Transformer is currently running 3.0.1 and I am running into the same issue with the standard mail client :/

AndSandA...@googlemail.com, the fix is in 3.1.

After getting the 3.1 upgrade today I now receive the following error, suggesting there is still something odd about SSL connections and the according library.

You will see the new error message which occurs in the same situation as the original error message reported above.

AndSandA...@googlemail.com, looking at your message, openssl doesn't seem to think its talking to an SSL/TLS server, at least not one running a version it understands. Is it possible that port 25 is only offering unencrypted service and that you need to use something like port 465? (see http://en.wikipedia.org/wiki/SMTPS) If you would provide your server name, I could take a look.

I agree that you should use port 465 or select "TLS" rather than SSL.

I have specified port 25 because the server does not use the standard port 465 it says
"Unable to open connection to server." if I try to connect using prt 465.
Also Windows Live Mail is able to connect to the server  ( smtp-fb3.informatik.uni-bremen.de ) using SSL encryption on port 25 so I assumed it would work here too.

From comment @29, I'm am guessing that SSL = SSMTP = raw SSL wrapped around SMTP, where the TLS option means to use STARTTLS, which will talk to the SMTP server to discover if TLS is supporeted, and upgrade if available?

Surprisingly enough just choosing TLS as security type on its standard port using the same server allows me to finish the setup without a failure.

Thanks I guess.

That's not surprising, although I guess the "documentation" could be better.   SSL is used to mean an SSL-encrypted session; TLS is used to mean "an SSL session wrapped in an unencrypted session using the STARTTLS command".

Alright, thank you two for the clarification and for the hint.
Using TLS I can now use the same account on my Galaxy S 2 as well though it only runs 2.3.3 atm which does not contain the original fix.

AndSandA...@googlemail.com, I'm not surprised, the original problem did not affect handshaking with most servers. If it hadt, there would have been a lot more reports than just from magnus.g...@gmail.com and johnd...@gmail.com. :)

I am having this same issue.  We use various varieties of DROID phones with our Exchange Server and 1 of the DROID X units was pushed this 2.3.3 update over the weekend which has broken the ActiveSync on that phone only.  The only support I have received from Verizon was "reset the phone to factory default."  

I have contacted Verizon multiple timesand referenced this issue number.  Everyone I talk to says that they cannot send any updates to the phone.  Then I get transferred to Motorola who says that they cannot help.  I received a replacement DROID X which was supposed to have "the latest version" loaded, but it does not.  I am stuck at version 2.3.3.  Someone please help!

Who do I talk to for getting the 2.3.4 version for the DROID X?  Verizon says they do not have it.

@39: Verizon and Motorola need to release the update.

How do we get Verizon and Motorola to release the update to repair something that had been working for 9 months?!

b...@google.com Thanks for following up. My issue wasn't actually SSL related. When I was at home it worked and when I was out of the house it didn't. I tracked it down in my mail logs to my phone providers IP address (the shared address used by my 3G data) that was on a RBL. I have whitelisted my device and everything is working great now. :-) Thanks again.

hi, I'm getting the same issue as comment 27 on an asus transformer with 3.1

Unfortunately, in my case I can't disable tls, so I'm stuck with the problem.

For info, I can use the same server in K9 without any problem, but that is not really tablet friendly :-)

I'm having the same ActiveSync issues now that I have a droid 3 with 2.3.4. My original droid worked fine via touchdown, and now the new droid does not and it complains of ActiveSync not being supported, which I need...

If anyone has any updates on when a fix might be released, sounds like there are lots of people here that are interested in that.. I'm currently unable to view corporate email via my droid 3, and corporate email is the reason I got it!

To follow up on comment #44, stevesch...@gmail.com also reported the issue as http://code.google.com/p/android/issues/detail?id=18367. In that issue thread, he note the issue was resolved and the root cause was his company had inadvertently locked his device from synchronizing.

[Comment deleted]

[Comment deleted]

[Comment deleted]

sadik.al...@gmail.com, I don't recall any specific issues in 2.3.4. The last report I had, the problem ended up being a exchange server configuration issue, where the device had to be added to a white list. can you connect to the server in the browser without error with either name? I'd be happy to take a quick look at the SSL handshaking if you can provide the server names.

[Comment deleted]

sadik.al...@gmail.com,

connecting on the desktop is no indication of a correctly configured server. I could cite many other reports of people connecting on the desktop to servers that were missing intermediate CAs certs in the server configuration. these usually work on the desktop because desktop browsers will dynamic add intermediate CAs to their trusted list, where mobile browers usually do not.

so when I asked to connect with the browser, I mean on the device with the stock android browser. a misconfigured server can usually be diagnosed with the server name remotely, either with "openssl s_client" and examining the certificate chain returned or with one of the CA configuration checking tools.

[Comment deleted]

[Comment deleted]

sadik.al...@gmail.com, sorry I missed that part in comment 50. send it to b/d/c@google.com without the slashes. however, given that you can connect in the browser and not the exchange app, perhaps its what this person experienced: http://code.google.com/p/android/issues/detail?id=18367

[Comment deleted]

I'm not having any trouble connecting to your server at the SSL level and the certs look fine, but I'm not surprised given that you can connect in the browser.

http://code.google.com/p/android/issues/detail?id=10784#c3 has some notes on how to enable some email specific logging, but its not something I'm familiar with.

I'd recommend filing a new issue with sanitized log info if you can. Mention it here and I'll try to make sure it gets triaged to the email team. I would contact your IT department as well to see if they do have any restrictions based on device type. I don't know how this type of restriction works with exchange.

You might also try an alternative email client like K-9 in the mean while:
https://market.android.com/details?id=com.fsck.k9
http://code.google.com/p/k9mail/ 

Please note the context of my comments 49/51/54/56 where in response to comments that have since been deleted by the user. The user's problem was indeed that the server was not allowing his type of device to connect. Some more useful information in what the user found in the logs that led to this conclusion in case it might help others:

[22:17:16] AbstractSyncService| Testing EAS: SERVER.SERVER.COM, DOMAIN\USERNAME, ssl = 1
[22:17:24] AbstractSyncService| Validation (OPTIONS) response: 500
[22:17:24] AbstractSyncService| Validation failed, reporting I/O error: 500

It negotiated SSL, provided credentials, and bombed with an error 500 (server error, not auth error.)

The USER that was denied sync would have bombed with 403 (the thread you linked to.) 500 was auth succeeded, device forbidden.

Android OS 2.3.4 as well as latest available emulator cannot connect to this url:

https://travel01.home2sc.net:49150


works fine in 2.2.

Error is:

 SSL handshake aborted: ssl=0x35a8e8: Failure in SSL library, usually a protocol error
error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure (external/openssl/ssl/s23_clnt.c:658 0xad12959f:0x00000000)

SSL handshake aborted: ssl=0x35a8e8: Failure in SSL library, usually a protocol error
error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure (external/openssl/ssl/s23_clnt.c:658 0xad12959f:0x00000000)
javax.net.ssl.SSLProtocolException: SSL handshake aborted: ssl=0x35a8e8: Failure in SSL library, usually a protocol error
error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure (external/openssl/ssl/s23_clnt.c:658 0xad12959f:0x00000000)
        at org.apache.harmony.xnet.provider.jsse.NativeCrypto.SSL_do_handshake(Native Method)
        at org.apache.harmony.xnet.provider.jsse.OpenSSLSocketImpl.startHandshake(OpenSSLSocketImpl.java:474)
        at org.apache.harmony.xnet.provider.jsse.OpenSSLSocketImpl$SSLInputStream.<init>(OpenSSLSocketImpl.java:750)
        at org.apache.harmony.xnet.provider.jsse.OpenSSLSocketImpl.getInputStream(OpenSSLSocketImpl.java:692)
        at org.apache.http.impl.io.SocketInputBuffer.<init>(SocketInputBuffer.java:93)
        at org.apache.http.impl.SocketHttpClientConnection.createSessionInputBuffer(SocketHttpClientConnection.java:83)
        at org.apache.http.impl.conn.DefaultClientConnection.createSessionInputBuffer(DefaultClientConnection.java:170)
        at org.apache.http.impl.SocketHttpClientConnection.bind(SocketHttpClientConnection.java:106)
        at org.apache.http.impl.conn.DefaultClientConnection.openCompleted(DefaultClientConnection.java:129)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:171)
        at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:164)
        at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:119)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:359)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:555)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:487)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:465)

Is this a known issue

Known?  Yes.

Fixed?  Nothing I've seen on the horizon.  

RickGill...@gmail.com,

I'm guessing the problem with https://travel01.home2sc.net:49150 is that it only supports the old SSLv3 handshake protocol.

Note that openssl on my desktop won't connect to it with the default options:

$ openssl s_client -connect travel01.home2sc.net:49150
CONNECTED(00000003)
8322:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:188:
$ openssl s_client -ssl3 -connect travel01.home2sc.net:49150
CONNECTED(00000003)
---
Certificate chain
 0 s:/O=travel01.home2sc.net/OU=Domain Control Validated/CN=travel01.home2sc.net
   i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure Certification Authority/serialNumber=07969287
 1 s:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure Certification Authority/serialNumber=07969287
   i:/C=US/O=The Go Daddy Group, Inc./OU=Go Daddy Class 2 Certification Authority
 2 s:/C=US/O=The Go Daddy Group, Inc./OU=Go Daddy Class 2 Certification Authority
   i:/C=US/O=The Go Daddy Group, Inc./OU=Go Daddy Class 2 Certification Authority
---
$ openssl s_client -tls1 -connect travel01.home2sc.net:49150
CONNECTED(00000003)
8381:error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number:s3_pkt.c:284:
$

There was a renegotiation vulnerability discovered in SSLv3 (see http://en.wikipedia.org/wiki/Transport_Layer_Security) which is why a lot of things are phasing out support for it. Any modern server should support TLS in addition to the older SSL handshaking (they really are just an evolution).

b...@google.com,

Thanks for the investigation.

If you don't mind, can you help me understand how to react to this?  

Should I be waiting for a fix in the OS to be rolled out, or are you saying that Android as a platform will no longer support SSLv3 ?

I'm a little concerned if its the second one, cause these upgrades are pushed OTA by the carriers and end users don't usually expect for things to stop working when that happens.  We can't always control the server support.

RickGill...@gmail.com, there won't be a 2.3 update for this. I believe the issue is that the default SSL options have become more conservative and don't have SSLv3 enabled for handshake by default. an individual app can request it still, but browser  in 2.3. did not.

the browser in 3.0 has a different https stack and it does work there.

You could try Firefox on the device on 2.3, it has its own SSL stack entirely.

.....  an individual app can request it still,.....

That sounds interesting, do you have any links or more info on how an app would do that?

javax.net.ssl.SSLSocket.setEnabledProtocols

http://developer.android.com/reference/javax/net/ssl/SSLSocket.html#setEnabledProtocols(java.lang.String[])

I tried enabling the SSLv3 protocol and found that it was already enabled by default, and the connection still failed in the same way.

Should I be tracking this under a different problem?

Tested on 2.2 and found the same set of enabled protocols ( by using the debugger), but the call to getEnabledProtocols() throws an exception.

RickGill...@gmail.com, yes, this sounds like a different issue. Please file a new one. Feel free to reference it here so I can assign it to myself. Please include the exception you mentioned.

Opened issue 36932621

http://code.google.com/p/android/issues/detail?id=19659

Does anybody know if HTC Sensation based on Android OS 2.3.4 has this fix? I heard that sometimes vendors selectively include OS fixes into their firmware updates, so I'm wondering if somebody can verify if HTC included the fix for SSL problem into its update.

I'm using TouchDown software which syncs with Exchange server OK via WiFi, but fails to do this when connected through 3G. Communicated with the vendor and my IT support, and they suggest that the reason may cover in OS's SSL protocol implementation...

After 2.3.3 got pushed out to any android device all phones lost connection to the 2010 exchange server. I just tested my Bionic running 2.3.4 and still fails to even talk , gives a login in failure.

Wondering if you can help see what we need to have them turn on...
owa is email.med.umich.edu 

Thanks
Socks

2.3.4 should have fixed this. I just tried that server with my 2.3.6 Nexus One in the browser and it connected fine. Have you tried using owa in the browser, or just using the Email app.

i'm facing the same issue with an SSL Connection (exactly HTTPS) on 2.3.3, unfotunately I have no update for the Device.

Is there any workaround to prevent this problem on either Server or Client?
The only solution that I see at moment is to disable SSL, are there any other possibilities?

bast.ralf, is your device a Nexus One? If so you should be newer than 2.3.4. If you are stuck on 2.3.3 for some reason, I can dig up a resource for moving forward.

Unfortunately there are no real workarounds that I recall, its been a while since I worked on this issue. If I recall correctly, one factor was the large number of optional attributes, extensions, etc which made the server certificate chain larger, but I could be mistaking this for another 2.3 era issue.