IssueTracker

Describe the issue

google-cloud-ops-agent-fluent-bit.service fails to start. A 'segmentation fault' error is shown in journalctl logs.

Steps to reproduce the behavior:

• Start a GCE VM with Debian 10 image.
• Install MongoDB Community Edition 5.0.11
• Install Ops Agent 2.32.0
• Apply the example configuration provided in the guide about Operations Suite for MongoDB.

In our case, it worked fine for some hours. Then the fluent-bit.service failed and was unable to start again.

Expected behavior:

All Agent Ops services working correctly, or at least an informative error message about what's misconfigured.

Environment:

• Project ID pulse-api-370218
• VM ID 7971989526812200110
• VM distro / OS: Debian 10
• Ops Agent version 2.32.0
• Ops Agent configured following the MongoDB operations suite guide for installation on individual VMs and then the MongoDB integration

Output of journalctl -u google-cloud-ops-agent*:

May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]: 2023/05/23 16:46:26 Built-in config:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]: logging:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   receivers:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     syslog:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: files
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       include_paths:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       - /var/log/messages
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       - /var/log/syslog
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   service:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     pipelines:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       default_pipeline:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         receivers: [syslog]
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]: metrics:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   receivers:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     hostmetrics:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: hostmetrics
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       collection_interval: 60s
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   processors:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     metrics_filter:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: exclude_metrics
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       metrics_pattern: []
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   service:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     pipelines:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       default_pipeline:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         receivers: [hostmetrics]
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         processors: [metrics_filter]
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]: 2023/05/23 16:46:26 Merged config:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]: logging:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   receivers:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     mongodb:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: mongodb
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     syslog:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: files
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       include_paths:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       - /var/log/messages
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       - /var/log/syslog
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   service:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     pipelines:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       default_pipeline:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         receivers: [syslog]
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       mongo:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         receivers: [mongodb]
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]: metrics:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   receivers:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     hostmetrics:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: hostmetrics
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       collection_interval: 60s
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     mongodb:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: mongodb
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       insecure: true
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       insecure_skip_verify: null
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       cert_file: ""
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       key_file: ""
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       ca_file: ""
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       collection_interval: ""
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   processors:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     metrics_filter:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: exclude_metrics
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       metrics_pattern: []
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   service:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     pipelines:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       default_pipeline:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         receivers: [hostmetrics]
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         processors: [metrics_filter]
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       mongo:
May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         receivers: [mongodb]
May 23 16:46:26 mongodb-dev-1 systemd[1]: Started Google Cloud Ops Agent - Logging Agent.
May 23 16:46:27 mongodb-dev-1 google_cloud_ops_agent_wrapper[17343]: 2023/05/23 16:46:27 signal: segmentation fault
May 23 16:46:27 mongodb-dev-1 systemd[1]: google-cloud-ops-agent-fluent-bit.service: Main process exited, code=exited, status=255/EXCEPTION
May 23 16:46:27 mongodb-dev-1 systemd[1]: google-cloud-ops-agent-fluent-bit.service: Failed with result 'exit-code'.
May 23 16:46:27 mongodb-dev-1 systemd[1]: google-cloud-ops-agent-fluent-bit.service: Service RestartSec=100ms expired, scheduling restart.
May 23 16:46:27 mongodb-dev-1 systemd[1]: google-cloud-ops-agent-fluent-bit.service: Scheduled restart job, restart counter is at 4.
May 23 16:46:27 mongodb-dev-1 systemd[1]: Stopped Google Cloud Ops Agent - Logging Agent.

Additional context

We are running MongoDB as a standalone replica, with authentication enabled. Monitoring works fine, we are experiencing issues with logging only. We opened an issue at the ops-agent repository on github.

Up until this moment we have tried everything stated on the Troubleshoot the Ops Agent guide without avail.

I attach here the output of the diagnostics tool as mentioned in the answer to the issue we opened in github. The output of the tool is in the the diagnositic-tool.zip and the last 10000 of our mongodb logs in the mongod.log file