IssueTracker

Problem you have encountered:

HTTP push subscription sending invalid GZIP data to HTTP endpoint.

What you expected to happen:

Valid GZIP data to be sent to the HTTP endpoint. I have tested this with the production Pub/Sub service and it sends the data correctly.

Steps to reproduce:

1. Run the local Pub/Sub emulator in docker.
2. Run a local HTTP server that logs the buffer data of incoming messages.
3. Create a topic in the emulator
4. Create a push subscription for the topic sending messages to the local HTTP. The push subscription should be configured not to wrap the messages and send the metadata as headers.
5. Send a buffer of compressed data to the topic.

import zlib from 'zlib';
import { PubSub } from '@google-cloud/pubsub';

/**
 *
 * @param {string} topicName
 */
async function sendCompressedTestPayloadToTopic(topicName) {
  const payload = JSON.stringify({
    internalTopic: 'test',
    payload: { something: 12345, isCompressed: true },
  });

  const compressedBuffer = zlib.gzipSync(Buffer.from(payload), {
    level: zlib.constants.Z_BEST_COMPRESSION,
  });

  const topic = new PubSub({ projectId: process.env.PUBSUB_PROJECT_ID }).topic(topicName);

  await topic.publishMessage({
    data: compressedBuffer,
    attributes: {
      'internal-topic': 'test',
      'Content-Type': 'application/json',
      'Content-Encoding': 'gzip',
    },
  });

  console.log('compressedBuffer', compressedBuffer);
  console.log('utf8 decoded gzip data', Buffer.from(compressedBuffer.toString('utf-8')));
}

Other information (workarounds you have tried, documentation consulted, etc):

I took a tcpdump of sending the buffer to the emulator and it was sent correctly.

I took a tcpdump of the message sent from the emulator to my HTTP server and the data was not correct.

compressedBuffer <Buffer 1f 8b 08 ...>
utf8 decoded gzip data <Buffer 1f ef bf bd 08 ...>

The correct magic number should be 1f8b however the incoming data is actually 1fef.

It would appear the emulator is decoding the data in the message as utf-8 then sending a buffer of the decoded utf-8 data to the HTTP endpoint.