What you would like to accomplish: Currently it is not allowed to specify an existing output table in BigQuery as the destination of the prediction results.
The suggested feature is to include the possibility of selecting an existing table as destination and allow overwriting of that table or appending the new prediction to it.
How this might work:
Allowing the path bq://project_id.dataset.table with an existing table.
Adding a parameter in the API that controls the BigQuery write disposition, e.g. WRITE_APPEND to append the data or WRITE_TRUNCATE to overwrite the table.
This might be useful if we are only interested in the latest batch prediction. Then there is no need to filter by timestamp the tables in the dataset to get the latest one in case this is just a step in a larger workflow of the data.
If applicable, reasons why alternative solutions are not sufficient: Alternatives involve writing a custom script that first submits the prediction job, removes the older predictions table after or concatenates the tables if the desired operation was to append the data.
Adding the possibility of overwriting or appending data to an existing table makes this process cleaner and easier.
Description
What you would like to accomplish: Currently it is not allowed to specify an existing output table in BigQuery as the destination of the prediction results.
The suggested feature is to include the possibility of selecting an existing table as destination and allow overwriting of that table or appending the new prediction to it.
How this might work:
bq://project_id.dataset.tablewith an existing table.WRITE_APPENDto append the data orWRITE_TRUNCATEto overwrite the table.This might be useful if we are only interested in the latest batch prediction. Then there is no need to filter by timestamp the tables in the dataset to get the latest one in case this is just a step in a larger workflow of the data.
If applicable, reasons why alternative solutions are not sufficient: Alternatives involve writing a custom script that first submits the prediction job, removes the older predictions table after or concatenates the tables if the desired operation was to append the data.
Adding the possibility of overwriting or appending data to an existing table makes this process cleaner and easier.