Describe the issue

google-cloud-ops-agent-fluent-bit.service fails to start. A 'segmentation fault' error is shown in journalctl logs.

Steps to reproduce the behavior:

Start a GCE VM with Debian 10 image.
Install MongoDB Community Edition 5.0.11
Install Ops Agent 2.32.0
Apply the example configuration provided in the guide about Operations Suite for MongoDB.

In our case, it worked fine for some hours. Then the fluent-bit.service failed and was unable to start again.

Expected behavior:

All Agent Ops services working correctly, or at least an informative error message about what's misconfigured.

Environment:

Project ID pulse-api-370218
VM ID 7971989526812200110
VM distro / OS: Debian 10
Ops Agent version 2.32.0
Ops Agent configured following the MongoDB operations suite guide for installation on individual VMs and then the MongoDB integration

Output of journalctl -u google-cloud-ops-agent*:

May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]: 2023/05/23 16:46:26 Built-in config:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]: logging:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   receivers:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     syslog:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: files
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       include_paths:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       - /var/log/messages
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       - /var/log/syslog
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   service:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     pipelines:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       default_pipeline:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         receivers: [syslog]
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]: metrics:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   receivers:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     hostmetrics:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: hostmetrics
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       collection_interval: 60s
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   processors:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     metrics_filter:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: exclude_metrics
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       metrics_pattern: []
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   service:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     pipelines:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       default_pipeline:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         receivers: [hostmetrics]
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         processors: [metrics_filter]
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]: 2023/05/23 16:46:26 Merged config:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]: logging:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   receivers:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     mongodb:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: mongodb
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     syslog:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: files
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       include_paths:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       - /var/log/messages
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       - /var/log/syslog
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   service:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     pipelines:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       default_pipeline:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         receivers: [syslog]
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       mongo:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         receivers: [mongodb]
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]: metrics:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   receivers:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     hostmetrics:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: hostmetrics
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       collection_interval: 60s
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     mongodb:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: mongodb
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       insecure: true
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       insecure_skip_verify: null
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       cert_file: ""
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       key_file: ""
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       ca_file: ""
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       collection_interval: ""
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   processors:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     metrics_filter:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: exclude_metrics
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       metrics_pattern: []
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   service:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     pipelines:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       default_pipeline:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         receivers: [hostmetrics]
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         processors: [metrics_filter]
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       mongo:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         receivers: [mongodb]
May 23 16:46:26 mongodb-dev-1 systemd[1]: Started Google Cloud Ops Agent - Logging Agent.
May 23 16:46:27 mongodb-dev-1 google_cloud_ops_agent_wrapper[17343]: 2023/05/23 16:46:27 signal: segmentation fault
May 23 16:46:27 mongodb-dev-1 systemd[1]: google-cloud-ops-agent-fluent-bit.service: Main process exited, code=exited, status=255/EXCEPTION
May 23 16:46:27 mongodb-dev-1 systemd[1]: google-cloud-ops-agent-fluent-bit.service: Failed with result 'exit-code'.
May 23 16:46:27 mongodb-dev-1 systemd[1]: google-cloud-ops-agent-fluent-bit.service: Service RestartSec=100ms expired, scheduling restart.
May 23 16:46:27 mongodb-dev-1 systemd[1]: google-cloud-ops-agent-fluent-bit.service: Scheduled restart job, restart counter is at 4.
May 23 16:46:27 mongodb-dev-1 systemd[1]: Stopped Google Cloud Ops Agent - Logging Agent.

Additional context

We are running MongoDB as a standalone replica, with authentication enabled. Monitoring works fine, we are experiencing issues with logging only. We opened an issue at the ops-agent repository on github.

Up until this moment we have tried everything stated on the Troubleshoot the Ops Agent guide without avail.

I attach here the output of the diagnostics tool as mentioned in the answer to the issue we opened in github. The output of the tool is in the the diagnositic-tool.zip and the last 10000 of our mongodb logs in the mongod.log file

Issue 284274549

Links (5)

Issue metadata