IssueTracker

I want to predicting using Batch Prediction in Vertex AI. My script runs normally before and has no issue, but somehow a few days ago, without any changes in my script, the batch prediction elapsed time longer than my usual prediction.

I've run the batch prediction before for 7,503 images, it worked flawlessly for only ~ 32 mins elapsed time.
While predicting 20 image, few days ago, somehow, the batch prediction longer with around 1 hour 21 mins elapsed time so I need to cancel the job due to duration prediction that will be impacting to cost.

I've check the VIEW DETAILS and the View Logs it gave me nothing. Do someone has facing the same issue? or know the solution?

My data input for batch prediction, already looked like this documentation here:
https://cloud.google.com/vertex-ai/docs/image-data/object-detection/get-predictions

Input .jsonl
{"content": "gs://bucket_name/folder/sub_folder/JPEG_01.jpg", "mimeType": "image/jpeg"}
{"content": "gs://bucket_name/folder/sub_folder/JPEG_02.jpg", "mimeType": "image/jpeg"}

My script for batch prediction
def create_batch_prediction_job(job_display_name, data_input_path, result_prediction_path, model_resource_name):

    aiplatform.init(project=project_id, location=region_id)
    model = aiplatform.Model(model_resource_name)

    batch_prediction_job = model.batch_predict(
        job_display_name = job_display_name,
        gcs_source = data_input_path,
        gcs_destination_prefix = result_prediction_path,
        sync = True,
    )
   
    batch_prediction_job.wait()
   
    print(f"Batch Prediction Job Name: {job_display_name}")
    print(f"Batch Prediction Job : {batch_prediction_job.resource_name}")
    print(f"\nState")
    print(batch_prediction_job.state)
   
    return batch_prediction_job