IssueTracker

This is still work in progress.

We just went through a whole ordeal with Google Support to get insight into if and when this feature (Logical Decoding) will become available, but in the end ended up being redirected to post our thoughts here and wait for the Cloud SQL Product Team to respond. This is after having been in contact with Google in early 2019 and been told to expect an update on the roadmap for this feature in Q3 2019...

Google Support asked us to repost our concerns and urgency here, for the Product Team to respond, so here we go:

use case

Our use-case is that we need to be able to extract ALL changes made to our data in near-real-time, without overhead to the client making the changes and in a maintainable fashion. We need to extract ALL changes made to our databases to keep external systems in sync, in a near real-time manner.

Our current approach is using INSERT/UPDATE/DELETE triggers on each table, but this incurs way too much overhead, causing unacceptable client slowdown. Doing this on the application level isn't an option either, as it's unmaintainable

what we need

So we want to handle this on a database-level and with PostgreSQL this would be using their Logical Decoding feature. However, settings and privileges' needed to use Logical Decoding on Cloud SQL for PostgreSQL are currently not enabled:

• wal_level parameter is set to 'replica' (probably because its needed for Create Replica), but for Logical Decoding it needs to be set to logical
• ability to control the max_replication_slots setting, as a replication slot is per database on a server and we run servers with way more databases that the default value (10) for max_replication_slots (probably also requires the max_wal_senders setting to be configurable
• ability to create (and drop) replications slots, see Logical Decoding Examples
• ability to create PUBLICATIONS, needed when using the pgoutput plugin icw Logical Decoding
• optionally support output plugins like wal2json Logical Decoding Plugins

urgency

As for the urgency: we're currently running into the limitations of the trigger-based approach, from a performance perspective. We can probably get away with it for a few more months (among others by disabling some features), but giving the lead times of the possible solutions, we need to set the wheels in motion right now.

As we see it we have 2 options:

1. Wait for Cloud SQL for PostgreSQL to add support for Logical Decoding
2. Migrate to any other hosted PostgreSQL offering, as pretty mush all other major and minor managed PostgreSQL offerings offer support for Logical Decoding

what we want to know

Given that setting in motion and executing a migration to another managed PostgreSQL provider has a lead time from somewhere between 2 to 9 months, we would like to know the roadmap for Logical Decoding on Cloud SQL for PostgreSQL, as we don't want to set the wheels in motion, only to learn 3 or 6 months from now that support was added for Logical Decoding, voiding all our migration efforts, as we're otherwise very happy with Cloud SQL and it's our preference to remain on it.

So, in the end, what we are after if for Cloud SQL for PostgreSQL to come up to the level of all other major and minor PostgreSQL hosting providers and make it possible to use Logical Decoding and create Logical Replication slots, so tools like Debezium can be used.

Would love to see Cloud SQL for PostgreSQL be added to this list: Setting Up PostgreSQL

Hoping the above clarifies our needs and urgency and hoping for a speedy response, so we can decide which wheels to set in motion.