IssueTracker

This will create a feature request which anybody can view and comment on.

Please describe your requested enhancement. Good feature requests will solve common problems or enable new use cases.

What you would like to accomplish:

We currently process individual csv files from cloud storage via an event driven pipeline which uses Google Cloud Functions (python 3.x). We read these files in, apply some changes, and write them back to a seperate cloud storage bucket.

Rarely (in about 0.1% of cases), these files are too large to fit within the 32GB memory limit of the function we are using for processing, so we have to chunk the reads/writes as per https://stackoverflow.com/questions/25962114/how-do-i-read-a-large-csv-file-with-pandas. We would prefer to not have to do this, as it splits one input file into multiple output files, and means that we have to union them when we process downstream in BigQuery.


How this might work:

Provide a 64GB memory allocation option (as was done with 32GB, which is currently in preview as per https://cloud.google.com/blog/products/serverless/cloud-functions-2nd-generation-now-generally-available)

If applicable, reasons why alternative solutions are not sufficient:

1. Current process writes one files to multiple different files and causes additional effort downstream
2. Cloud Functions is fit for purpose for 99.9% of our use cases so implementing workarounds for these edge cases using Compute Engine etc. would introduce additional operations effort and risk

Other information (workarounds you have tried, documentation consulted, etc):

Currently chunking the reads/writes as per https://stackoverflow.com/questions/25962114/how-do-i-read-a-large-csv-file-with-pandas