Google Cloud Run decodes url search part before reaching the app [212674416]

Assigned

Bug

Status Update

No update yet.

Description

gg...@fgribreau.com

created issue #1

Dec 31, 2021 09:26AM

Context

I was trying to understand WHY 😵‍💫 my app (Image-Charts.com) works on Google Kubernetes Engine (GKE) but the same container does not work on Cloud-Run.

How to reproduce

To be precise, why the signed URL (HMAC) below works on GKE and not on Cloud-Run

https://image-charts.com/chart?chbr=8&chd=t%3A10%2C15%2C25%2C30%2C40%2C80&chf=b0%2Clg%2C90%2C05B142%2C1%2C0CE858%2C0.2&chl=%7C%7C%7C%7C%2033%25%20%21%7Cx2%20&chma=0%2C0%2C10%2C10&chs=700x450&cht=bvs&chtt=Revenue%20per%20month&chxl=0%3A%7CJan%7CFev%7CMar%7CAvr%7CMay&chxs=1N%2AcUSD0sz%2A%2C000000%2C14&chxt=x%2Cy&icac=fgribreau&ichm=d92a75886c0657013da32eda4d82b8db3af39d355b3419caebd5952bc827990d

Note that HMAC signature (ichm query parameter) is computed like this:

ichm=sign(url.search, shared_secret)

My main intuition was that cloud-run had some internal reverse proxies that changes part of the URL on the fly.

If that's true, let's find what encoded characters Cloud-Run would automatically decode in the URL query part before reaching your app:

new Array(155).join('-').split('-').map((x, i) => '%'+i.toString(16).toUpperCase()).join('-_-')

// output:
// %0-_-%1-_-%2-_-%3-_-%4-_-%5-_-%6-_-%7-_-%8-_-%9-_-%A-_-%B-_-%C-_-%D-_-%E-_-%F-_-%10-_-%11-_-%12-_-%13-_-%14-_-%15-_-%16-_-%17-_-%18-_-%19-_-%1A-_-%1B-_-%1C-_-%1D-_-%1E-_-%1F-_-%20-_-%21-_-%22-_-%23-_-%24-_-%25-_-%26-_-%27-_-%28-_-%29-_-%2A-_-%2B-_-%2C-_-%2D-_-%2E-_-%2F-_-%30-_-%31-_-%32-_-%33-_-%34-_-%35-_-%36-_-%37-_-%38-_-%39-_-%3A-_-%3B-_-%3C-_-%3D-_-%3E-_-%3F-_-%40-_-%41-_-%42-_-%43-_-%44-_-%45-_-%46-_-%47-_-%48-_-%49-_-%4A-_-%4B-_-%4C-_-%4D-_-%4E-_-%4F-_-%50-_-%51-_-%52-_-%53-_-%54-_-%55-_-%56-_-%57-_-%58-_-%59-_-%5A-_-%5B-_-%5C-_-%5D-_-%5E-_-%5F-_-%60-_-%61-_-%62-_-%63-_-%64-_-%65-_-%66-_-%67-_-%68-_-%69-_-%6A-_-%6B-_-%6C-_-%6D-_-%6E-_-%6F-_-%70-_-%71-_-%72-_-%73-_-%74-_-%75-_-%76-_-%77-_-%78-_-%79-_-%7A-_-%7B-_-%7C-_-%7D-_-%7E-_-%7F-_-%80-_-%81-_-%82-_-%83-_-%84-_-%85-_-%86-_-%87-_-%88-_-%89-_-%8A-_-%8B-_-%8C-_-%8D-_-%8E-_-%8F-_-%90-_-%91-_-%92-_-%93-_-%94-_-%95-_-%96-_-%97-_-%98-_-%99-_-%9A

To run this experiment I've set a special route on Image-Charts that outputs what Image-Charts http server got as part of url.search.

On Google Kubernetes Engine (GKE), Image Charts API got an unmodified query string:

%0-_-%1-_-%2-_-%3-_-%4-_-%5-_-%6-_-%7-_-%8-_-%9-_-%A-_-%B-_-%C-_-%D-_-%E-_-%F-_-%10-_-%11-_-%12-_-
%13-_-%14-_-%15-_-%16-_-%17-_-%18-_-%19-_-%1A-_-%1B-_-%1C-_-%1D-_-%1E-_-%1F-_-%20-_-%21-_-%22-_-
%23-_-%24-_-%25-_-%26-_-%27-_-%28-_-%29-_-%2A-_-%2B-_-%2C-_-%2D-_-%2E-_-%2F-_-%30-_-%31-_-%32-_-
%33-_-%34-_-%35-_-%36-_-%37-_-%38-_-%39-_-%3A-_-%3B-_-%3C-_-%3D-_-%3E-_-%3F-_-%40-_-%41-_-%42-_-
%43-_-%44-_-%45-_-%46-_-%47-_-%48-_-%49-_-%4A-_-%4B-_-%4C-_-%4D-_-%4E-_-%4F-_-%50-_-%51-_-%52-_-
%53-_-%54-_-%55-_-%56-_-%57-_-%58-_-%59-_-%5A-_-%5B-_-%5C-_-%5D-_-%5E-_-%5F-_-%60-_-%61-_-%62-_-
%63-_-%64-_-%65-_-%66-_-%67-_-%68-_-%69-_-%6A-_-%6B-_-%6C-_-%6D-_-%6E-_-%6F-_-%70-_-%71-_-%72-_-
%73-_-%74-_-%75-_-%76-_-%77-_-%78-_-%79-_-%7A-_-%7B-_-%7C-_-%7D-_-%7E-_-%7F-_-%80-_-%81-_-%82-_-
%83-_-%84-_-%85-_-%86-_-%87-_-%88-_-%89-_-%8A-_-%8B-_-%8C-_-%8D-_-%8E-_-%8F-_-%90-_-%91-_-%92-_-
%93-_-%94-_-%95-_-%96-_-%97-_-%98-_-%99-_-%9A

On Google Cloud Run however, things are different:

%0-_-%1-_-%2-_-%3-_-%4-_-%5-_-%6-_-%7-_-%8-_-%9-_-%A-_-%B-_-%C-_-%D-_-%E-_-%F-_-%10-_-%11-_-%12-_-
%13-_-%14-_-%15-_-%16-_-%17-_-%18-_-%19-_-%1A-_-%1B-_-%1C-_-%1D-_-%1E-_-%1F-_-%20-_-!-_-%22-_-
%23-_-%24-_-%25-_-%26-_-%27-_-(-_-)-_-*-_-%2B-_-%2C-_---_-.-_-%2F-_-0-_-1-_-2-_-
3-_-4-_-5-_-6-_-7-_-8-_-9-_-%3A-_-%3B-_-%3C-_-%3D-_-%3E-_-%3F-_-%40-_-A-_-B-_-
C-_-D-_-E-_-F-_-G-_-H-_-I-_-J-_-K-_-L-_-M-_-N-_-O-_-P-_-Q-_-R-_-
S-_-T-_-U-_-V-_-W-_-X-_-Y-_-Z-_-%5B-_-%5C-_-%5D-_-%5E-_-_-_-%60-_-a-_-b-_-
c-_-d-_-e-_-f-_-g-_-h-_-i-_-j-_-k-_-l-_-m-_-n-_-o-_-p-_-q-_-r-_-
s-_-t-_-u-_-v-_-w-_-x-_-y-_-z-_-%7B-_-%7C-_-%7D-_-~-_-%7F-_-%80-_-%81-_-%82-_-
%83-_-%84-_-%85-_-%86-_-%87-_-%88-_-%89-_-%8A-_-%8B-_-%8C-_-%8D-_-%8E-_-%8F-_-%90-_-%91-_-%92-_-
%93-_-%94-_-%95-_-%96-_-%97-_-%98-_-%99-_-%9A

Yep. Characters like ")", "(", "a", "~" were converted on the fly by some Cloud Run internal proxy.

Now lets filter this for better readability:

const search_raw = new Array(155).join('-').split('-').map((x, i) => '%'+i.toString(16).toUpperCase());
const search_processed_by_cloud_run = `%0-_-%1-_-%2-_-%3-_-%4-_-%5-_-%6-_-%7-_-%8-_-%9-_-%A-_-%B-_-%C-_-%D-_-%E-_-%F-_-%10-_-%11-_-%12-_-%13-_-%14-_-%15-_-%16-_-%17-_-%18-_-%19-_-%1A-_-%1B-_-%1C-_-%1D-_-%1E-_-%1F-_-%20-_-!-_-%22-_-%23-_-%24-_-%25-_-%26-_-%27-_-(-_-)-_-*-_-%2B-_-%2C-_---_-.-_-%2F-_-0-_-1-_-2-_-3-_-4-_-5-_-6-_-7-_-8-_-9-_-%3A-_-%3B-_-%3C-_-%3D-_-%3E-_-%3F-_-%40-_-A-_-B-_-C-_-D-_-E-_-F-_-G-_-H-_-I-_-J-_-K-_-L-_-M-_-N-_-O-_-P-_-Q-_-R-_-S-_-T-_-U-_-V-_-W-_-X-_-Y-_-Z-_-%5B-_-%5C-_-%5D-_-%5E-_-_-_-%60-_-a-_-b-_-c-_-d-_-e-_-f-_-g-_-h-_-i-_-j-_-k-_-l-_-m-_-n-_-o-_-p-_-q-_-r-_-s-_-t-_-u-_-v-_-w-_-x-_-y-_-z-_-%7B-_-%7C-_-%7D-_-~-_-%7F-_-%80-_-%81-_-%82-_-%83-_-%84-_-%85-_-%86-_-%87-_-%88-_-%89-_-%8A-_-%8B-_-%8C-_-%8D-_-%8E-_-%8F-_-%90-_-%91-_-%92-_-%93-_-%94-_-%95-_-%96-_-%97-_-%98-_-%99-_-%9A`.split('-_-');

search_raw.reduce((m, encoded, i) => encoded === search_processed_by_cloud_run[i] ? m : m.concat([encoded, search_processed_by_cloud_run[i]].join(' => ')), []).join('\n')
    
// output
%21 => !
%28 => (
%29 => )
%2A => *
%2D => -
%2E => .
%30 => 0
%31 => 1
%32 => 2
%33 => 3
%34 => 4
%35 => 5
%36 => 6
%37 => 7
%38 => 8
%39 => 9
%41 => A
%42 => B
%43 => C
%44 => D
%45 => E
%46 => F
%47 => G
%48 => H
%49 => I
%4A => J
%4B => K
%4C => L
%4D => M
%4E => N
%4F => O
%50 => P
%51 => Q
%52 => R
%53 => S
%54 => T
%55 => U
%56 => V
%57 => W
%58 => X
%59 => Y
%5A => Z
%5F => _
%61 => a
%62 => b
%63 => c
%64 => d
%65 => e
%66 => f
%67 => g
%68 => h
%69 => i
%6A => j
%6B => k
%6C => l
%6D => m
%6E => n
%6F => o
%70 => p
%71 => q
%72 => r
%73 => s
%74 => t
%75 => u
%76 => v
%77 => w
%78 => x
%79 => y
%7A => z
%7E => ~

These are the characters decoded on the fly by cloud-run internal proxy thus breaking HMAC and sadly its not reversible.

Other information (workarounds you have tried, documentation consulted, etc):

I could not code a workaround because the change done by the intermediate cloud run proxy is not reversible (it has information loss).

Impact

Every potentiel Cloud Run customer that relies on HMAC based on the query string (used as raw text) will have the same issue as me :)

Comments

sh...@google.com <sh...@google.com> #2Jan 3, 2022 08:34AM

Assigned to gc...@google.com.

Hello,

Thank you for reaching out.

Google Cloud Product team is aware of this issue. There is no ETA at this time, however, you can keep track of the status by following this thread.

br...@gmail.com <br...@gmail.com> #3Apr 6, 2022 04:33PM

Hello,

I'm blocked by the same problem.
A simple url like :

https://myservice.com/api?name=NA%27ME is replaced by

https://myservice.com/api?name=NA'ME
(here %27 is decoded to ')

As the issue creator, i'm trying to sign my urls ... but since the url is modified by Cloud run, i'm stuck.

I see that the issue is assigned.
However, in the meantime, is it possible to have an exhaustive list of all the characters that are decoded so i can workaround this issue ?

Thanks

Message last modified on Apr 6, 2022 05:28PM

[Deleted User] <[Deleted User]> #4Feb 23, 2023 08:29AM

We're experiencing the same problem with Twilio webhooks. Such a shame

[Deleted User] <[Deleted User]> #5Feb 23, 2023 08:30AM

We had to write our own "url undecoder proxy" as a nasty workaround

va...@google.com <va...@google.com> Aug 1, 2023 06:09AM

Reassigned to ku...@google.com.

ku...@google.com <ku...@google.com> Aug 4, 2023 06:54AM

Reassigned to gc...@google.com.

jo...@pingu-solutions.com <jo...@pingu-solutions.com> #6Feb 11, 2025 03:07PM

I guess this is related, as I am not able to run a certain Docker container on Cloud Run.

The internal urls contain characters like %5B inside the url. Instead of finding the asset, i'll receive a 404 error.
This does not happen with other providers. I did not test it on GKE though.

I am commenting, as this still seems to be an issue. At least in my case.