AI notebooks is a very useful tool with growing adoption. Recently we have encountered two issues related to the performance of AI notebooks.
We wish to provide some feedback to the product team.

case 1. When running LightGBM on a pandas.DataFrame object that includes ~800K rows and ~400 columns the kernel restarts. while trying to solve the issue we have encountered several points that we'd like to raise
a. AI notebooks run a jupyter server container that sends logs to the console. The logs are not sent to stackdriver, and when there is an error it's hard to monitor the problem. It is possible to view the logs by fetching the logs from the serial port of the underlying VM. We suggest a few options:
a.1 add the ability to send these logs to stackdriver (with cost in mind)
a.2 Improve the troubleshooting docs to mention that these logs can be retrieved from the console serial port.

b. Add lightgbm for the base package - lightgbm is a broadly used library, however it looks like the AI notebooks not only that don't come with it installed, but also are not optimized for it.

c. Check performance issues.
We have created a benchmark for AI notebooks vs running on a mac and running on a custom made VM with jupyter/data-science container.
The macbook and the VM showed almost 10X better performance for training lightGBM and XGBoost.

Case 2: The notebook doesn't use all cores for high load tasks. When running a python stress test https://danielflannery.ie/simulate-cpu-load-with-python/ it seemed like the notebook has a limitation for using more than 32 cores (see GCP ticket 24603358). Often data scientists take the machine to the edge for experimenting. I would add to the process of releasing versions of this service stress tests and maybe limit(?) the machines that can be deployed to the service up to the limit that shows reasonable performance.

That been said, we understand that eventually it's a hard product to maintain since it is a managed service, but it lets users run any code they want.

Issue 163845419

Mentioned issues (2)

Links (8)

Issue metadata