Status Update
No update yet.
Comments
al...@google.com
ma...@gmail.com
Thanks for making this public.
Interestingly enough, when querying:
https://dns.google.com/query?name=www.fb.com&type=A&dnssec=false&ecs=200.14.96.0%2F23
everything works fine, although the result has even longer CNAME chain and the problematicwww.facebook.com CNAME is included.
www.fb.com IN CNAME www.facebook.com .
www.facebook.com IN CNAME star-mini.c10r.facebook.com .
star-mini.c10r.facebook.com . IN A ......
Similarly:
https://dns.google.com/query?name=de-de.facebook.com&type=A&dnssec=false&ecs=200.14.96.0%2F23
works fine, and the CNAME chain is:
de-de.facebook.com IN CNAME star.facebook.com .
star.facebook.com IN CNAME star.c10r.facebook.com .
star.c10r.facebook.com . IN A ......
The above two are clearly crossing zones as well, and even though seem to work fine. Alsowww.facebook.com works fine with IPv6 ECS.
Isn't it possible that the reason is different?
It's clear that Google DNS servers get very high number of requests forwww.facebook.com over IPv4
and much less requests over IPv6 and for the other domains.
Isn't this some kind of cache overflow (out of resources) issue resulting in automatic disabling of ECS forwww.facebook.com/IPv4 ?
The returned scope /0 with incorrect results is suspect.
Interestingly enough, when querying:
everything works fine, although the result has even longer CNAME chain and the problematic
Similarly:
works fine, and the CNAME chain is:
The above two are clearly crossing zones as well, and even though seem to work fine. Also
Isn't it possible that the reason is different?
It's clear that Google DNS servers get very high number of requests for
and much less requests over IPv6 and for the other domains.
Isn't this some kind of cache overflow (out of resources) issue resulting in automatic disabling of ECS for
The returned scope /0 with incorrect results is suspect.
lu...@google.com
Hey Bolek, do you still observe that Google Public DNS inconsistently handles CNAME chains for ECS-enabled domains, which results in incorrect replies for www.facebook.com ?
ma...@gmail.com
As far as I can see, it works correctly now. Thanks.
bk...@gmail.com
Hi,
The results are correct now, thank you!
When exactly was the fix deployed?
Thanks,
Bolek
The results are correct now, thank you!
When exactly was the fix deployed?
Thanks,
Bolek
ma...@gmail.com
Hi,
problem has reappeared on another domain --www.microsoft.com
When trying to resolve it using "digwww.microsoft.com ":
Correct answer (fresh, not from cache):
;; ANSWER SECTION:
www.microsoft.com . 1634 IN CNAME www.microsoft.com-c-3.edgekey.net .
www.microsoft.com-c-3.edgekey.net . 20777 IN CNAME www.microsoft.com-c-3.edgekey.net.globalredir.akadns.net .
www.microsoft.com-c-3.edgekey.net.globalredir.akadns.net . 899 IN CNAME e13678.dspb.akamaiedge.net .
e13678.dspb.akamaiedge.net . 19 IN A 2.18.70.63
Wrong answers from cache:
;; ANSWER SECTION:
www.microsoft.com . 2706 IN CNAME www.microsoft.com-c-3.edgekey.net .
www.microsoft.com-c-3.edgekey.net . 10773 IN CNAME www.microsoft.com-c-3.edgekey.net.globalredir.akadns.net .
www.microsoft.com-c-3.edgekey.net.globalredir.akadns.net . 585 IN CNAME e13678.dspb.akamaiedge.net .
e13678.dspb.akamaiedge.net . 5 IN A 23.53.173.48
;; ANSWER SECTION:
www.microsoft.com . 1674 IN CNAME www.microsoft.com-c-3.edgekey.net .
www.microsoft.com-c-3.edgekey.net . 10641 IN CNAME www.microsoft.com-c-3.edgekey.net.globalredir.akadns.net .
www.microsoft.com-c-3.edgekey.net.globalredir.akadns.net . 257 IN CNAME e13678.dspb.akamaiedge.net .
e13678.dspb.akamaiedge.net . 1 IN A 23.53.169.93
When querying "dige13678.dspb.akamaiedge.net ", correct answer is always returned (even from cache):
;; ANSWER SECTION:
e13678.dspb.akamaiedge.net . 17 IN A 2.18.70.63
;; ANSWER SECTION:
e13678.dspb.akamaiedge.net . 9 IN A 2.18.70.63
;; ANSWER SECTION:
e13678.dspb.akamaiedge.net . 15 IN A 2.18.70.63
problem has reappeared on another domain --
When trying to resolve it using "dig
Correct answer (fresh, not from cache):
;; ANSWER SECTION:
Wrong answers from cache:
;; ANSWER SECTION:
;; ANSWER SECTION:
When querying "dig
;; ANSWER SECTION:
;; ANSWER SECTION:
;; ANSWER SECTION:
ma...@gmail.com
Well, seems that www.microsoft.com returns the first CNAME record with scope=0, which confuses the DNS process.
Actually facebook seems to have workarounded this bug by setting scope of the initial CNAME records to 24, which however means that the whole CNAME chain is being cached separately for each /24.
According to RFC7871, it is valid to return CNAME with scope=0 to indicate that this CNAME is valid for all IP addresses. This way CNAME could be only cached once, and only the final A records are cached separately according to actual scope (which might be much larger e.g. /16 etc). So facebook's workaround - setting scope of all CNAMEs to 24 actually wastes a lot of memory at all resolvers.
Any timeline for the real bugfix?
Actually facebook seems to have workarounded this bug by setting scope of the initial CNAME records to 24, which however means that the whole CNAME chain is being cached separately for each /24.
According to RFC7871, it is valid to return CNAME with scope=0 to indicate that this CNAME is valid for all IP addresses. This way CNAME could be only cached once, and only the final A records are cached separately according to actual scope (which might be much larger e.g. /16 etc). So facebook's workaround - setting scope of all CNAMEs to 24 actually wastes a lot of memory at all resolvers.
Any timeline for the real bugfix?
al...@google.com
ch...@gmail.com
Facebook is not working around anything by setting a scope. At the time this was fixed we were not setting anything.
The issue could as well had been legitimately fixed and a regression could have been introduced later.
The issue could as well had been legitimately fixed and a regression could have been introduced later.
ma...@gmail.com
Facebook's action - setting scope of the initial CNAME records to fixed /24 for IPv4 and fixed /48 for IPv6 is inconsistent with their whole content delivery system, which uses dynamic prefix lengths. That's why I suspect it's some kind of workaround.
Anyway, guidelines for ECS (https://developers.google.com/speed/public-dns/docs/ecs ) suggest:
5. Authoritative name servers returning ECS-enabled CNAME responses SHOULD only include the first CNAME in the chain, and the final target of the CNAME chain should be ECS-enabled to the same scope prefix length.
Why the same scope length for initial CNAME and final A record? I believe it's perfectly valid to have initial CNAME with scope /0 and final A with scope /16, i.e. completely different scopes on those two.
Anyway, guidelines for ECS (
5. Authoritative name servers returning ECS-enabled CNAME responses SHOULD only include the first CNAME in the chain, and the final target of the CNAME chain should be ECS-enabled to the same scope prefix length.
Why the same scope length for initial CNAME and final A record? I believe it's perfectly valid to have initial CNAME with scope /0 and final A with scope /16, i.e. completely different scopes on those two.
ch...@gmail.com
> I believe it's perfectly valid to have initial CNAME with scope /0 and final A with scope /16, i.e. completely different scopes on those two.
agreed and I did not mean to say it was not.
agreed and I did not mean to say it was not.
bv...@gmail.com
ch...@gmail.com
Facebook should not (anymore) return /24 by default for CNAMEs like www.facebook.com
du...@gmail.com
Incorrect ECS-based results for CNAME chains crossing zones
al...@google.com
deleted
al...@google.com
deleted
ma...@gmail.com
I can still see the problem also with www.facebook.com :
The correct, fresh answer:
digwww.facebook.com @8.8.8.8
; <<>> DiG 9.10.6 <<>>www.facebook.com @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60874
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;www.facebook.com . IN A
;; ANSWER SECTION:
www.facebook.com . 3002 IN CNAME star-mini.c10r.facebook.com .
star-mini.c10r.facebook.com . 59 IN A 31.13.84.36
;; Query time: 18 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Oct 02 19:19:46 CEST 2019
;; MSG SIZE rcvd: 90
Wrong answer from cache:
digwww.facebook.com @8.8.8.8
; <<>> DiG 9.10.6 <<>>www.facebook.com @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35865
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;www.facebook.com . IN A
;; ANSWER SECTION:
www.facebook.com . 2026 IN CNAME star-mini.c10r.facebook.com .
star-mini.c10r.facebook.com . 39 IN A 185.60.216.35
;; Query time: 7 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Oct 02 19:19:42 CEST 2019
;; MSG SIZE rcvd: 90
Same withhttps://dns.google.com
The correct, fresh answer:
dig
; <<>> DiG 9.10.6 <<>>
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60874
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;
;; ANSWER SECTION:
;; Query time: 18 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Oct 02 19:19:46 CEST 2019
;; MSG SIZE rcvd: 90
Wrong answer from cache:
dig
; <<>> DiG 9.10.6 <<>>
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35865
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;
;; ANSWER SECTION:
;; Query time: 7 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Oct 02 19:19:42 CEST 2019
;; MSG SIZE rcvd: 90
Same with
iz...@gmail.com
Comment has been deleted.
be...@gmail.com
Comment has been deleted.
ro...@gmail.com
Comment has been deleted.
ro...@gmail.com
Comment has been deleted.
pu...@google.com
We have made some fixes recently for ECS handling. Can you check if the problem is still happening?
ma...@gmail.com
For the vast majority of answers (including those from cache) it works correctly now.
However, there are still rare cases, wherehttps://dns.google/resolve?name=www.facebook.com&type=A&edns_client_subnet=x.x.x.x/23 returns wrong answer with scope /0
However, there are still rare cases, where
so...@gmail.com
Apakah kasus ini mendapatkan penyelesaian ??
je...@gmail.com
Ayuda
ms...@gmail.com
ms...@gmail.com
an...@gmail.com
ms...@gmail.com
نعم
مرسل من هاتف Samsung Galaxy الذكي.
-------- الرسالة الأصلية --------من: buganizer-system@google.com التاريخ: ١٥/١/٢٠٢٢ ٣:٥٨ م (GMT+03:00) إلى: b-system+2115813771@google.com نسخة: mstfyshyla1982@gmail.com الموضوع: Re:
Replying to this email means your email address will be shared with the team that works on this product.
https://issuetracker.google.com/issues/68051640
Changed
an...@gmail.com added
190192
ในวันที่ อา. 12 ธ.ค. 2021 09:02 น. <buganizer-system@google.com> เขียนว่า:
مرسل من هاتف Samsung Galaxy الذكي.
-------- الرسالة الأصلية --------من: buganizer-system@google.com التاريخ: ١٥/١/٢٠٢٢ ٣:٥٨ م (GMT+03:00) إلى: b-system+2115813771@google.com نسخة: mstfyshyla1982@gmail.com الموضوع: Re:
Replying to this email means your email address will be shared with the team that works on this product.
Changed
an...@gmail.com added
190192
ในวันที่ อา. 12 ธ.ค. 2021 09:02 น. <buganizer-system@google.com> เขียนว่า:
_______________________________
Reference Info: 68051640 Incorrect ECS-based results for CNAME chains crossing zones
component: Public Trackers > Public DNS
status: Assigned
reporter: bk...@gmail.com
assignee: lu...@google.com
cc: bk...@gmail.com, go...@google.com, pu...@google.com, and 1 more
type: Bug
priority: P2
severity: S2
blocked by: 36433722, 70838535, 112930095, 158077304
blocking: 157597995, 195968607
duplicate issue: 36595953, 37553232, 67561349, 70161656, 117811335, 128641688
hotlist: adexe s nau, PublicDNS-Cache, PublicDNS-Client-Subnet, PublicDNS-GeoIP, PublicDNS-KnownIssue
retention: Component default
Affected: Domain
Service: UDP
Generated by Google IssueTracker notification system
You're receiving this email because you are subscribed to updates on Google IssueTracker
Unsubscribe from this issue.
c....@gmail.com
The issue is still there. The response is not deterministic(see CLIENT-SUBNET).
~ % digwww.facebook.com @8.8.8.8 +subnet=196.41.48.0/24
; <<>> DiG 9.10.6 <<>>www.facebook.com @8.8.8.8 +subnet=196.41.48.0/24
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14025
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; CLIENT-SUBNET:196.41.48.0/24/0
;; QUESTION SECTION:
;www.facebook.com . IN A
;; ANSWER SECTION:
www.facebook.com . 1046 IN CNAME star-mini.c10r.facebook.com .
star-mini.c10r.facebook.com . 2 IN A 157.240.28.35
;; Query time: 8 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Mar 14 16:19:57 PDT 2022
;; MSG SIZE rcvd: 101
~ % digwww.facebook.com @8.8.8.8 +subnet=196.41.48.0/24
; <<>> DiG 9.10.6 <<>>www.facebook.com @8.8.8.8 +subnet=196.41.48.0/24
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22959
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; CLIENT-SUBNET:196.41.48.0/24/22
;; QUESTION SECTION:
;www.facebook.com . IN A
;; ANSWER SECTION:
www.facebook.com . 1719 IN CNAME star-mini.c10r.facebook.com .
star-mini.c10r.facebook.com . 60 IN A 102.132.96.35
;; Query time: 25 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Mar 14 16:19:58 PDT 2022
;; MSG SIZE rcvd: 101
~ % dig
; <<>> DiG 9.10.6 <<>>
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14025
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; CLIENT-SUBNET:
;; QUESTION SECTION:
;
;; ANSWER SECTION:
;; Query time: 8 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Mar 14 16:19:57 PDT 2022
;; MSG SIZE rcvd: 101
~ % dig
; <<>> DiG 9.10.6 <<>>
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22959
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; CLIENT-SUBNET:
;; QUESTION SECTION:
;
;; ANSWER SECTION:
;; Query time: 25 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Mar 14 16:19:58 PDT 2022
;; MSG SIZE rcvd: 101
ch...@gmail.com
Comment has been deleted.
pa...@gmail.com
Comment has been deleted.
al...@gmail.com
Comment has been deleted.
na...@gmail.com
##39;
ta...@gmail.com
#1
lu...@google.com
pu...@google.com
This has been fully fixed for a few months now. Please let us know if you are still seeing a problem.
Otherwise I will close it out in a week.
Otherwise I will close it out in a week.
ma...@gmail.com
Well, the bug is still present, I'm currently getting about 50 % of incorrect answers:
https://dns.google/resolve?name=www.facebook.com&type=A&edns_client_subnet=196.41.48.0/23
{"Status":0,"TC":false,"RD":true,"RA":true,"AD":false,"CD":false,"Question":[{"name":"www.facebook.com .","type":1}],"Answer":[{"name":"www.facebook.com .","type":5,"TTL":2165,"data":"star-mini.c10r.facebook.com ."},{"name":"star-mini.c10r.facebook.com .","type":1,"TTL":60,"data":"102.132.96.35"}],"edns_client_subnet":"196.41.48.0/22 ","Comment":"Response from 129.134.30.12."}
{"Status":0,"TC":false,"RD":true,"RA":true,"AD":false,"CD":false,"Question":[{"name":"www.facebook.com .","type":1}],"Answer":[{"name":"www.facebook.com .","type":5,"TTL":2119,"data":"star-mini.c10r.facebook.com ."},{"name":"star-mini.c10r.facebook.com .","type":1,"TTL":18,"data":"157.240.0.35"}],"edns_client_subnet":"196.41.48.0/0 "}
But this one always works correctly:
https://dns.google/resolve?name=www.fb.com&type=A&edns_client_subnet=196.41.48.0/23
{"Status":0,"TC":false,"RD":true,"RA":true,"AD":false,"CD":false,"Question":[{"name":"www.fb.com .","type":1}],"Answer":[{"name":"www.fb.com .","type":5,"TTL":3799,"data":"www.facebook.com ."},{"name":"www.facebook.com .","type":5,"TTL":1428,"data":"star-mini.c10r.facebook.com ."},{"name":"star-mini.c10r.facebook.com .","type":1,"TTL":35,"data":"102.132.96.35"}],"edns_client_subnet":"196.41.48.0/22 "}
{"Status":0,"TC":false,"RD":true,"RA":true,"AD":false,"CD":false,"Question":[{"name":"
{"Status":0,"TC":false,"RD":true,"RA":true,"AD":false,"CD":false,"Question":[{"name":"
But this one always works correctly:
{"Status":0,"TC":false,"RD":true,"RA":true,"AD":false,"CD":false,"Question":[{"name":"
al...@gmail.com
bi...@gmail.com
Comment has been deleted.
pe...@flextock.com
Comment has been deleted.
cb...@gmail.com
Comment has been deleted.
Description
It looks like the behavior depends on whether a CNAME chain crosses zones or not. I provide two examples below.
[CASE 1] Different zones:
Using client-subnet
But using the same client-subnet and querying for
[CASE 2] Same zone: [REDACTED1].
Querying both domains with client-subnet
RFC 7871 mentions caching CNAME chains at the longest prefix-length (“For message-focused resolvers, rather than RRset-focused ones, this will mean caching the entire CNAME chain at the longest PREFIX-LENGTH of any RRset in the chain”) so I believe the behavior in CASE 1 should be similar to CASE 2.
[EDIT: Contact info removed for move from Restricted subcomponent to Public DNS]