[Speech-to-Text V2] CXR Phrase is not working well with long model [328499488]

Assigned

Customer Issue

Status Update

No update yet.

Description

na...@google.com

created issue #1

Mar 6, 2024 11:07PM

Issue Summary

The CXR is utilizing the Speech-to-Text V2 API but not getting correct or expected transcription.
This was their request parameter [1]

Expected transcription:

"certifié Labelix"

Actual Transcription

Output Transcription: confidence = 0.832697 transcript = certifié la belle X

Model

telephony long

Other Details

API Method: StreamingRecognize
API Version: V2
Client Library: use gRPC in C++

[1]

*StreamingRecognizeRequest request;*

* auto* streaming_config = request.mutable_streaming_config();*

* auto* config = streaming_config->mutable_config();*


* config->set_model("long");*

* config->add_language_codes("fr-FR");*


* ExplicitDecodingConfig* decoding_config =
config->mutable_explicit_decoding_config();*

*
decoding_config->set_encoding(ExplicitDecodingConfig_AudioEncoding_LINEAR16);*

* decoding_config->set_sample_rate_hertz(8000);*

* decoding_config->set_audio_channel_count(1);*


* StreamingRecognitionFeatures* streaming_feature =
streaming_config->mutable_streaming_features();*

* streaming_feature->set_interim_results(true);*

* request.set_recognizer(*
"projects/customer_project_id/locations/global/recognizers/_"*);*


* RecognitionFeatures* features = config->mutable_features();*

* features->set_enable_automatic_punctuation(true);*


* SpeechAdaptation* speech_adaptation = config->mutable_adaptation();*

* auto *adaptation_phrase_set = speech_adaptation->add_phrase_sets();*

* auto *phrase_set = adaptation_phrase_set->mutable_inline_phrase_set();*

* auto *phrase = phrase_set->add_phrases();*

* phrase->set_value("Labelix");*

* phrase->set_boost(20);*

* phrase = phrase_set->add_phrases();*

* phrase->set_value("certifié Labelix");*

* phrase->set_boost(20);*

Comments

na...@google.com <na...@google.com> #2Mar 6, 2024 11:12PM

I have informed our engineering team of this feature request. There is currently no ETA for its implementation.

A current workaround would be to check the returned "boundingPoly" [1] "vertices" for the returned "textAnnotations". If the calculated rectangle's heights > widths, than your image is sideways.

[1]

https://cloud.google.com/vision/reference/rest/v1/images/annotate#boundingpoly

to...@labvoice.ai <to...@labvoice.ai> #3Mar 21, 2024 11:05PM

I also need this problem solved :)

ja...@carescribe.io <ja...@carescribe.io> #4Apr 15, 2024 03:54PM

same :D

ke...@planeenglish.net <ke...@planeenglish.net> #5Apr 16, 2024 07:40PM

ke...@planeenglish.net <ke...@planeenglish.net> #6Apr 18, 2024 04:55PM

em...@google.com <em...@google.com> #7Jun 19, 2024 03:24PM

This needs more attention. It's not just a display issue as described in the report. The co-ordinates returned in 'boundingPoly' are incorrect if the image was taken on a phone. All the x points should be y and vice versa.

The workaround does not make sense as "boundingPoly" [1] "vertices" for "textAnnotations" does not indicate the image dimensions - it indicates the dimensions of the relevant text block inside the image.

em...@google.com <em...@google.com> #8Jul 16, 2024 02:09PM

Message last modified on Jul 16, 2024 02:09PM

ph...@gmail.com <ph...@gmail.com> #9Aug 27, 2024 05:48AM

Would be great if this could be implemented.