When Preemptible secondary workers are decommissioned or removed, not all YARN logs are being aggregated from the worker before it is completely decommissioned.
We ask to add an option to ensure the log aggregation is done before the yarn app finishes.
This causes the following:
Some YARN logs are not accessible; when clicking the application ID link, the page shows the Log Aggregation Status of "TIME_OUT";
clicking the "TIME_OUT" link shows at least one node with status TIME_OUT;
clicking App Attempt "Logs" link leads to an HTTP ERROR 500 page with "Server Error"
Description
When Preemptible secondary workers are decommissioned or removed, not all YARN logs are being aggregated from the worker before it is completely decommissioned.
We ask to add an option to ensure the log aggregation is done before the yarn app finishes.
This causes the following: