Status Update
No update yet.
Comments
[Deleted User] <[Deleted User]>
ge...@gmail.com
ds...@google.com
ds...@google.com
in...@nplan.io
As the issue has been resolved, can you please elaborate how we can request extra disk space? Thank you
ma...@flyrlabs.com
ds...@google.com
Hello,
This has been forwarded to the Vertex AI Engineering Team so that they may evaluate it. Note that there are no ETAs or guarantees of implementation for feature requests. All communication regarding this feature request is to be done here .
je...@ml6.eu
Really in need of this functionality. The current workaround using create_custom_training_job_op_from_component works, but is cumbersome.
li...@snyk.io
Also in need of this. The bigger GPU instances are pointless with such small defaults disk volume
li...@snyk.io
I don't know if Google never plan on looking into this or is this perceived to be messy since kfp package only has set_cpu_limit, set_gpu_limit, set_memory_limit, but not disk limit. (You can guess these API are designed by Google since they never need to write to local disk ... ). So adding a new setter method would require work from both kfp open source and Vertex AI.
Just in case it helps Vertex AI team: the existing add_node_selector_constraint appears to be a potentially a good place to implement this on Vertex AI side. The spirit of this API is to allow arbitrary constraints passed to workers. Vertex AI already use this picking GPU, which eventually arrives at workerPoolSpecs, where diskSpec also resides. So the implementation might not be that hard.
li...@snyk.io
Like others mentioned, create_custom_training_job_op_from_component works but is a bit problematic. Here's my hack (in case anyone on this thread still uses Vertex AI despite such issues)
my_comp = create_custom_training_job_from_component(adhoc_comp)(
project=PROJECT_ID, location=REGION, timeout="1209600s",
command=CLI_COMMAND)
my_comp.set_display_name("ad hoc")
worker_pool_spec = my_comp.component_spec.inputs["worker_pool_specs"].default[0]
worker_pool_spec["disk_spec"]["boot_disk_size_gb"] = 500
adhoc_comp is my kfp component and command is my component's parameter.
Description
It would be really nice/convenient if customers are able to request the disk space for a python function based component, just like requests for CPU, Memory and GPUs when Specifying machine types for a Vertex AI pipeline[1].
How this might work:
Be able to specify Disk size just like specifying CUP, Memory when Specifying machine types for a Vertex AI pipeline[1].
If applicable, reasons why alternative solutions are not sufficient:
Currently the boot_disk_size_gb option[2] could help increase the disk size, however, it is a little bit inconvenient. Specifying machine types for a Vertex AI pipeline[1] with disk option available is more straightforward.
Other information (workarounds you have tried, documentation consulted, etc):
[1]
[2]