Assigned
Status Update
No update yet.
Comments
mk...@google.com
Information redacted by Android Beta Feedback.
ki...@google.com
Please provide the requested information to proceed further. Unfortunately the issue will be closed within 7 days if there is no further update.
ma...@deliveryhero.com
The name of the given attribute has a length of 16 characters and the topic is owned by a different team in a different department. So we could in theory ask them to use a non-human readable encoding with just one character, but that should ideally not be needed. Also we would still need 21 characters per country check and between each check 4 characters for the " OR ". So in total only 7 checks would be possible, if I'm not wrong.
jp...@umich.edu
+1
Our use case:
Front end publishes requests to a "gpu_infernece" topic with `attribute.model_name = my_modelname`. To reduce hardware requirements and encapsulate complexity, the backend partitions all models which must be served across different deployments. These partitions (i.e. mapping between model_names and deployments) can change over time. Since the deployments know the model_names they support, they automatically create a subscription to the gpu_inference topic with a filter such as: `attribute.model_name = foo OR attribute.model_name = bar OR ...`. The 256 byte limit combined with the inexpressive query language make this design infeasible.
TL;DR we run into the same blocker as ma...@deliveryhero.com:
> So in total only 7 checks would be possible
Our use case:
Front end publishes requests to a "gpu_infernece" topic with `attribute.model_name = my_modelname`. To reduce hardware requirements and encapsulate complexity, the backend partitions all models which must be served across different deployments. These partitions (i.e. mapping between model_names and deployments) can change over time. Since the deployments know the model_names they support, they automatically create a subscription to the gpu_inference topic with a filter such as: `attribute.model_name = foo OR attribute.model_name = bar OR ...`. The 256 byte limit combined with the inexpressive query language make this design infeasible.
TL;DR we run into the same blocker as ma...@deliveryhero.com:
> So in total only 7 checks would be possible
se...@google.com
Hello,
Thank you for your patience.
I have forwarded this information to the product team. One there is any update, it will be shared in this thread.
Kind Regards
se...@google.com
so...@google.com
+1 for this feature
My customer Manticore Games currently is trying to filter their events and this is what their filter looks like :
# bq-export-acdf-20210609
hasPrefix(attributes.event_type, "A") OR hasPrefix(attributes.event_type, "C") OR hasPrefix(attributes.event_type, "D") OR hasPrefix(attributes.event_type, "F")
They can only filter by 4 characters in a single filter and are not happy with the 180 character limit. What is the work around if Pubsub subscription filter limit of 180 characters is insufficient?
My customer Manticore Games currently is trying to filter their events and this is what their filter looks like :
# bq-export-acdf-20210609
hasPrefix(attributes.event_type, "A") OR hasPrefix(attributes.event_type, "C") OR hasPrefix(attributes.event_type, "D") OR hasPrefix(attributes.event_type, "F")
They can only filter by 4 characters in a single filter and are not happy with the 180 character limit. What is the work around if Pubsub subscription filter limit of 180 characters is insufficient?
so...@google.com
Also, if the subscription filters support regex it will allow the customers to make the best use of 180 characters.
[Deleted User] <[Deleted User]>
Any updates on this feature? My team would also find it beneficial to have these limits increased.
jo...@tempus.com
+1! We would also benefit from expanding the 256 byte limit.
ss...@league.com
Having a similar issue, are there any updates on the limit for the subscription filter ?
mj...@league.com
Is there any update on the limit for the subscription filter ?
nr...@google.com
la...@google.com
[Deleted User] <[Deleted User]>
Hi guys, I'm following this thread and I have a sub-question, do we have documentation references that the filter limits are 256 bytes?
Thanks!
Thanks!
va...@google.com
va...@google.com
ni...@virtahealth.com
A 2x increase (or more) to the filter limit would be great, given the limitations of the query syntax.
Our use case is for topic for FHIR store CDC events. A common use case is filtering for certain resource types. Given the current limitations (24 characters are eaten up by each attributes.resourceType=), we can filter 6-7 resources max. This is very low considering the number of FHIR resources.
As stated in the original post, this would be greatly alleviated by more dense notation support. However, an increase in the bytes limit would be sufficient in the short term.
g....@saltosystems.com
Same use case here, CDC and one topic for multiple type of events. Currently we applied a workaround by grouping events by services and filtering out those in event processors if they don't apply. Not ideal and we're "processing" / discarding events when not needed, but does the trick for now.
pu...@google.com
iv...@hm.com
We have the same problem...we have a central PubSub topic, and use filtered subscriptions based on this concept:
https://medium.com/google-cloud/how-to-chain-and-observe-interdependent-data-products-on-google-cloud-610336839695
However, the 256 characters limit greatly impacts us when we try to set up a more complex chaining.
However, the 256 characters limit greatly impacts us when we try to set up a more complex chaining.
ra...@gmail.com
facing similar issue where we want filter based on more than one attribute value . we are having possibility of 10 values in attribute but can 't do this due to size limitation . feature like in(val1.val2, ..... val3) needed in filter instead of using OR again and again with repeating attribute.<name> each time.
be...@ext.ons.gov.uk
an in() / contains() feature would be useful.
Or even if you could use "attr." or "a." rather than "attributes." would help a lot.
Or even if you could use "attr." or "a." rather than "attributes." would help a lot.
Description
we are having problems to use the filter possibility on subscriptions. One of our topics is providing messages for many different countries. On the subscription side we want to filter for a couple of countries. It is impossible for us to fit that into the allowed size of 256 bytes for filters. According to the documentation even the name of an attribute could contain already 256 bytes, how should one be able to write a filter in this case?
Our proposal would be to:
1. Increase limit of 256 bytes for subscription filters
2. Offer IN operator instead of only equal check for more dense notation
3. Offer regular expressions for even more compact filters
We hope that at least one of the suggestions could be possibly added.
Best
Marc