IssueTracker

Hello Edward!

To properly reproduce your issue here, we would need the code for your application with method /test, that dumps all the request headers.

It is of value to also have a sample from the corresponding entries in the requests logs, to see how it looks on your side.

Thank you.

I have created a test App Engine application at https://referer-161821.appspot.com to debug this issue. Code attached.

I went to http://www.msn.com in Chrome, opened the Javascript console, and ran:

location.href = 'https://referer-161821.appspot.com/test?ref=' + encodeURIComponent(location.href);

Repeated for https://github.com/features

In both cases, the /test method dumped the correct referrer path in the response, first { "Referer": "http://www.msn.com/" } and then { "Referer": "https://github.com/features" }.

The corresponding request logs are attached. In both cases, the referer in the logs is “-”.

—

P.S. I have located the source of the bug, see next comment.

I started noticing that the referer was missing from the request logs for requests I made from my installed Chrome (56.0.2924.87), but was not missing for requests I made from my installed Internet Explorer (11.0.9600.18538).

When I made requests from Python and set the “Referer” header manually, the referer was turning up in the GAE request logs.

So then I copied the headers from the Chrome request for which the referer was not being reported, and sent the same exact (literal) headers from Python.

It turns out that Chrome sends “referer”, not “Referer”… and GAE doesn’t handle it properly, as far as request logs are concerned.

# Summary

In a GAE application, although e.g. in Flask, request.headers['Referer'] will return the correct referer, it seems the internal GAE code that writes the referer to the request log expects the header field name to be “Referer” with a capital R; if the request header is sent as “referer”, with small r, the referer will be reported as “-” in the request logs. Since HTTP header names should be case-insenstive, this is a bug.

# Reproducing the bug

## Server

@app.route('/test')
def test():
    return jsonify(Referer=request.headers.get('Referer'))

## Test program

import requests
print requests.get('https://referer-161821.appspot.com/test?r', headers={'referer': 'https://example.com'}).text
print requests.get('https://referer-161821.appspot.com/test?R', headers={'Referer': 'https://example.com'}).text

## Test program output

{"Referer": "https://example.com"}
{"Referer": "https://example.com"}

## Corresponding request_logs
0.0.0.0 - - [18/Mar/2017:02:44:42 -0700] "GET /test?r HTTP/1.1" 200 91 - "python-requests/2.12.4"
0.0.0.0 - - [18/Mar/2017:02:44:43 -0700] "GET /test?R HTTP/1.1" 200 91 "https://example.com" "python-requests/2.12.4"

Seems like it has already been reported as issue 35896891 too.

I can confirm the behavior you describe.  As there is no mention of this header being stripped[0] and it is indeed available to applications receiving requests, I believe this is not intended behavior.

I have forwarded this report to the engineering team.  Any questions, comments or updates they may have will be posted here.  Thank you for bringing this to our attention.

[0]: https://cloud.google.com/appengine/docs/standard/python/how-requests-are-handled#request-headers

Thank you.