IssueTracker

I’m experiencing an intermittent authentication issue with the Google Cloud Vision API when running in production. The service, which uses a service account to access the API via Python’s ImageAnnotatorClient, works fine on both localhost and the development server. However, in production, it occasionally fails with an authentication error.

Details:

Service Environment: FastAPI service running on a production server.
Problem: The API sometimes returns authentication errors. The issue is inconsistent but appears only in the production environment.
Working Conditions: The same service account JSON file works without issues on localhost and dev servers.
Troubleshooting Steps Taken:
Verified that the service account JSON file is correctly configured.
Attempted to resolve the issue by creating multiple service accounts, but the problem persists.

Error Traceback:

debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:142.250.181.74:443: Failed to connect to remote host: FD Shutdown", grpc_status:14, created_time:"2024-08-18T07:15:34.844327353+05:00"}"
The above exception was the direct cause of the following exception:
Traceback (most recent call last):  
  File "<frozen myocr.app>", line 118, in card_ocr  
  File "<frozen jpocr.card_processing>", line 261, in __call__  
  File "<frozen myocr.visionocr>", line 42, in extract_text_google  
  File "/home/myocr_env/lib/python3.10/site-packages/google/cloud/vision_helpers/decorators.py", line 112, in inner    
    response = self.annotate_image(  
  File "/home/myocr_env/lib/python3.10/site-packages/google/cloud/vision_helpers/__init__.py", line 76, in annotate_image    
    r = self.batch_annotate_images(  
  File "/home/myocr_env/lib/python3.10/site-packages/google/cloud/vision_v1/services/image_annotator/client.py", line 566, in batch_annotate_images    
    response = rpc(  
  File "/home/myocr_env/lib/python3.10/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__    
    return wrapped_func(*args, **kwargs)  
  File "/home/myocr_env/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 81, in error_remapped_callable    
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.ServiceUnavailable: 503 failed to connect to all addresses; last error: UNKNOWN: ipv4:142.250.181.74:443: Failed to connect to remote host: FD Shutdown

2024-08-18 07:16:35,315 - uvicorn - ERROR - Response Message: 401 Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project. [reason: "ACCESS_TOKEN_EXPIRED" domain: "googleapis.com" metadata {   key: "service"   value: "vision.googleapis.com" } metadata {   key: "method"   value: "google.cloud.vision.v1.ImageAnnotator.BatchAnnotateImages"
18
Additional Context:

The issue only occurs intermittently and exclusively in the production environment.
I have validated that the service account credentials and API configurations are correct.
Network issues have been ruled out as the server is capable of making other network requests reliably.