Implement compression for file sync [150827486]

Duplicate

of 223614029

Feature Request

Status Update

No update yet.

Description

jm...@google.com

created issue #1

Mar 5, 2020 04:55AM

People are using `adb shell tar -cfvz - ... | tar xfvz` to improve sync performance (e.g.

https://android-review.googlesource.com/c/platform/development/+/1132497): we should just build that into adb.

I implemented brotli compression in 3 out of 4 directions (adbd can push/pull, adb can only pull) in

https://android-review.googlesource.com/c/platform/system/core/+/1250677

At the default compression level, it's *slow*: a 1GB file of zeroes takes over a minute to compress *on my workstation*. At compression level 1, it's actually a marginal improvement over localhost, so it's probably a reasonable gain over a slow transport, but I just bricked the walleye on my workstation somehow, so I don't have any actual numbers yet.

TODOs:
implement the remaining direction
add a flag to turn it on/off
train a dictionary from adb sync?
look into other algorithms?

Comments

as...@google.com <as...@google.com> #2Mar 24, 2020 06:45PM

Hi Josh, friendly ping for an update on progress

en...@google.com <en...@google.com> #3Mar 24, 2020 07:52PM

i think the TL;DR is that this is stuck behind investigating the current lab outages, which are more urgent.

pending CL is https://android-review.googlesource.com/c/platform/system/core/+/1250677.

do...@google.com <do...@google.com> #4Mar 31, 2020 12:53PM

Have you consider snappy or zstd? From my experience snappy compresses at around 200~300MB/s with reasonable compression rate ~30%.

lo...@google.com <lo...@google.com> #5Mar 31, 2020 01:04PM

For brotli, try quality level 1, this can give around 150-200MB/s with a much better compression ration than snappy of over 60%. Brotli quality 0 is even faster for still over 50% compression.

jm...@google.com <jm...@google.com> #6Apr 13, 2020 09:23PM

Marked as fixed.

Hmm, gerrit-watcher didn't post the CLs on this bug, so I guess I'll do so manually:

https://android.googlesource.com/platform/system/core/+/939fc19aee7f6a19ce0f4fa91557bbbe9cd974f5 added brotli support (with quality level 1)

https://android.googlesource.com/platform/system/core/+/ec44d35fde60d20d9ab1c6fb9ac7ee844335f1eb added LZ4

Throughput numbers from a device:

           none    brotli    lz4
USB 3.0     120       110    190
USB 2.0      38        75     63

I'm seeing identical throughput with brotli quality 0 and 1 (110MB/s end to end), although there's some giant low hanging fruit to compress multiple files at once.

zstd isn't in the tree yet (I started the ball rolling on that, but no idea when it'll end up getting merged), but I'll take a look at it eventually, probably when I get around to implementing compression for generic streams in adb (b/153901735).

This is probably good enough to call this fixed for now.

sh...@google.com <sh...@google.com> Dec 6, 2022 12:43AM

Status: Duplicate of

223614029

Issue 150827486