IssueTracker

Here is an example illustrating why the current bulkloader download_data design should be improved.  I was downloading an entity kind with 22 million records.  Here is the breakdown of the time consumed by this task:

1) downloading the data to create a 10 GB bulkloader-results.sql3 file: 30 hours.

2) intermediate step creating a 25 GB temp file (described in Issue 4524): 1.5 hours.

3) writing out the resulting CSV file from the temporary view: 7 hours

I believe that steps 2 and 3 are unnecessary. It would be much better to be writing the transformed output in parallel with the download.

(In my example, this would save 25% of the overall time and save 200% of the disk I/O operations).

This feature would solve http://code.google.com/p/googleappengine/issues/detail?id=4524