Note: This will create a public issue which anybody can view and comment on.
This issue report has been forwarded to the PRODUCT engineering team so that they may investigate it, but there is no ETA for a resolution at this time. Future updates regarding this issue will be provided here.
Please provide as much information as possible. At least, this should include a description of your issue and steps to reproduce the problem. If possible please provide a summary of what steps or workarounds you have already tried, and any docs or articles you found (un)helpful.
Problem you have encountered: The YARN NodeManager is not able to start due to a BindException as the address was already in use. Therefore, systemd tries to restart the process, failing with java.net.BindException: Address already in use. The log messages:
ERROR 2023-06-28T09:52:49.218Z Error starting NodeManager
INFO 2023-06-28T09:52:49.220Z SHUTDOWN_MSG:
DEFAULT 2023-06-28T09:52:49.341857067Z java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) [...]
DEFAULT 2023-06-28T09:52:49.346775021Z org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.net.BindException: Problem binding to [0.0.0.0:8026] java.net.BindException: Address already in use; For more details see: http://wiki.apache.org/hadoop/BindException at org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:138) at org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:65) [...]
...
DEFAULT 2023-06-28T09:52:49.349799076Z /************************************************************
DEFAULT 2023-06-28T09:52:49.350117867Z SHUTDOWN_MSG: Shutting down NodeManager at *****-w-1/10.**.**.**
DEFAULT 2023-06-28T09:52:49.350352747Z ************************************************************/
What you expected to happen: The YARN NodeManager should not face the BindException and it should be able to start and operate normally. The workaround has been provided; however, a permanent fix is needed.
Steps to reproduce: No reproduction steps available at the moment.
Other information (workarounds you have tried, documentation consulted, etc): The Product team has provided a script as a workaround to solve the issue. It needs to be added to the cluster as initialization action:
cat << 'END' > /kill-port-8026.sh
#!/bin/bash
set -e
ROLE=$(/usr/share/google/get_metadata_value attributes/dataproc-role)
if [[ "${ROLE}" == 'Master' ]]; then
# Don't need to run on the Master.
exit 0
fi
PID=$(sudo netstat -ano -p tcp | grep LISTEN | grep 8026 | awk -F "[ :]+" '{print $7}' | awk -F "[/]" '{print $1}')
if [ ! -z $PID ] ; then
echo "Kill pid=[$PID] to free up port 8026 for yarn NodeManager."
sudo kill $PID
fi
END
chmod +x /kill-port-8026.sh
echo 'KillMode=control-group' >> /etc/systemd/system/hadoop-yarn-nodemanager.service.d/worker-restart.conf
echo 'ExecStartPre=/kill-port-8026.sh' >> /etc/systemd/system/hadoop-yarn-nodemanager.service.d/worker-restart.conf
systemctl daemon-reload
Description
Note: This will create a public issue which anybody can view and comment on.
This issue report has been forwarded to the PRODUCT engineering team so that they may investigate it, but there is no ETA for a resolution at this time. Future updates regarding this issue will be provided here.
Please provide as much information as possible. At least, this should include a description of your issue and steps to reproduce the problem. If possible please provide a summary of what steps or workarounds you have already tried, and any docs or articles you found (un)helpful.
Problem you have encountered: The YARN NodeManager is not able to start due to a
BindExceptionas the address was already in use. Therefore, systemd tries to restart the process, failing withjava.net.BindException: Address already in use. The log messages:What you expected to happen: The YARN NodeManager should not face the
BindExceptionand it should be able to start and operate normally. The workaround has been provided; however, a permanent fix is needed.Steps to reproduce: No reproduction steps available at the moment.
Other information (workarounds you have tried, documentation consulted, etc): The Product team has provided a script as a workaround to solve the issue. It needs to be added to the cluster as initialization action: