Bug P2
Status Update
Comments
ch...@gcresell.deloitte.co.uk <ch...@gcresell.deloitte.co.uk> #2
SFDC cases associated with this issue have been notified.
ch...@gcresell.deloitte.co.uk <ch...@gcresell.deloitte.co.uk> #3
Hello,
Thanks for the feature request!
This has been forwarded to the Cloud Composer Engineering Team so that they may evaluate it. Note that there are no ETA's or guarantees of implementation for feature requests. All communication regarding this feature request is to be done here.
Description
*NOTE: This template is ONLY used for reporting bugs. Please see [Get Help](
# Steps to Reproduce
*NOTE: If the bug was triggered with JavaScript code in the Code Editor, use the [Get Link](
*NOTE: If the bug was triggered using the Python client library, create a [Colab notebook](
*NOTE: Make sure that any datasets needed to reproduce the bug are publicly shared. If the datasets are sensitive and cannot be made public, replace them with other public datasets.*
*NOTE: If this is a user interface bug, include a screenshot.*
1.Go to this url,
2. Unzip to reveal complex folder structure: eg ..../terr50_cesh_gb/data/hp/hp40_OST50CONT_20240531/
In each of these folders is data for 2 shapefiles:
Line data(contours)
Point data(High points)
Across the whole file structure there are:
2858 line shapefiles
2795 point shapefiles
3. Attempt upload as follows:
ALL line-shapefiles in this folder structure into a single table in EarthEngine, via table_manifest
ALL point-shapefiles in this folder structure into a single table in EarthEngine, via table_manifest
# Current Behavior
* When we attempt to upload via table_manifest & the following code:
import ee
import uuid
ee.Initialize(project=project_id)
ee.Authenticate()
ee.data.startTableIngestion(f"table-upload-{uuid.uuid4()}", manifest, allow_overwrite=True)
we see the following error msg:
Error: Execution failed; out of memory. (Error code: 8)
# Expected Behavior
* We expect all point files to upload to a single table and all line files to upload to a single table
# Environment Details
*Include the Python or JavaScript Client Library Version (if applicable)*
# Other details about the situations or use cases affected by this bug
* The table manifests for each of the 2 long list of shapefile urls is <0.5MB in size, so much less that the 10MB limit specified here:
The number of rows in each of the the 2 final tables is not expected to exceed 100million rows - the table-size limit specified here:
* This error happens when we try to upload either the full set of point or the full set of line shapefiles in one table_manifest.
The shapefiles that fail to upload each time are not the same ones - it can be different shapefiles failing on each attempt.
* This error does not happen when we try to upload a small subset of this data, <10 shapefiles, containing shapefiles that have failed on previous full-set upload attempts, in one table_manifest.