Detect Web entities and pages - inconsistent results with same request [301393789]

Bug

Status Update

No update yet.

Description

co...@marcinstecki.pl

created issue #1

Sep 21, 2023 04:07PM

Problem we have encountered:

Our product is using https://cloud.google.com/vision/docs/detecting-web to moderate user content and detect it's presence over the web. Last 4-6 weeks we are observing inconsistent results from the api. Sending the same exact image twice can yield different results in regards to web presence.

What you expected to happen: We expected the api we pay for to behave in deterministic manner. We don't mind getting an incorrect response or a false positive in web detection, but random results are unacceptable.

Steps to reproduce:

go to https://cloud.google.com/vision/docs/detecting-web
scroll down to 'try it' section at the bottom of the page
send following request using the api explorer

{
  "requests": [
    {
      "image": {
        "source": {
         "imageUri": "https://lh3.googleusercontent.com/drive-viewer/AITFw-wbbMaLtYHBjkS_DSPxxw1u61JBmNroG0ZIx-KQtoAn-icjLsoiibcvLfi_XcmIUEh9hoGcFR0jt0WIR2in2IGKiY-0Ug=s2560"
        }
      },
      "features": [
        {
          "type": "WEB_DETECTION",
          "maxResults": 10
        }
      ],
    }
  ]
}

repeat last step 10 times and observe how the response is changing. Roughly 5 times out of 10 you will get pagesWithMatchingImages field in the response. For the rest of the requests you wont see that field in the response, only visuallySimilarImages will be there.

Type 1 of response you will get (correct, detects images):

{
  "responses": [
    {
      "webDetection": {
        "webEntities": [
          {
            "entityId": "/m/06ng1",
            "score": 0.2747,
            "description": "Sex"
          },
          {
            "entityId": "/m/0b2334",
            "score": 0.2491,
            "description": "Reddit"
          },
          {
            "entityId": "/m/0rz56gw",
            "score": 0.2491
          },
          {
            "entityId": "/m/02js9",
            "score": 0.2309,
            "description": "Erotica"
          },
          {
            "entityId": "/g/11h01s_tpf",
            "score": 0.2226,
            "description": "OnlyFans"
          },
          {
            "entityId": "/m/0fc1fy",
            "score": 0.22173013,
            "description": "Black hair"
          },
          {
            "entityId": "/m/03q69",
            "score": 0.2003,
            "description": "Hair"
          },
          {
            "entityId": "/m/05k21",
            "score": 0.2001,
            "description": "Nudity"
          },
          {
            "entityId": "/t/24mypdx4svpvn",
            "score": 0.1907
          },
          {
            "entityId": "/m/03gpl8",
            "score": 0.1423,
            "description": "Model M keyboard"
          }
        ],
        "partialMatchingImages": [
          {
            "url": "https://preview.redd.it/i1xu2dj1r4b21.jpg?width=617&auto=webp&s=a7471b760a494017782d6c6b25e01ab4fae4307d"
          },
          {
            "url": "https://tse3.mm.bing.net/th?id=OIP.tP1SUnbjHnA1Y6MUjk-NZgHaJN&pid=15.1"
          }
        ],
        "pagesWithMatchingImages": [
          {
            "url": "https://www.hotzxgirl.com/ana/ana-marie-zabala.htm",
            "pageTitle": "Ana Marie Zabala | Hot Sex Picture",
            "partialMatchingImages": [
              {
                "url": "https://tse3.mm.bing.net/th?id=OIP.tP1SUnbjHnA1Y6MUjk-NZgHaJN&pid=15.1"
              }
            ]
          },
          {
            "url": "https://pornopics.topsexgames.xyz/porno/ana-zabala-reddit/100206/",
            "pageTitle": "29++ Ana Zabala Reddit Download - Porno Pics",
            "partialMatchingImages": [
              {
                "url": "https://preview.redd.it/i1xu2dj1r4b21.jpg?width=617&auto=webp&s=a7471b760a494017782d6c6b25e01ab4fae4307d"
              }
            ]
          }
        ],
        "visuallySimilarImages": [
          {
            "url": "https://hips.hearstapps.com/rbk.h-cdn.co/assets/17/01/640x959/adore-me-norelle-bra.jpg?resize=980:*"
          },
          {
            "url": "https://img.freepik.com/premium-photo/perfect-woman-beautiful-seductive-burgundy-lingerie_133748-1871.jpg?w=2000"
          },
          {
            "url": "https://www.shutterstock.com/image-photo/beautiful-young-asian-girl-dark-600w-28042864.jpg"
          },
          {
            "url": "https://img.freepik.com/premium-photo/sexy-asian-woman-black-lingerie-isolated-gray-background_900706-21574.jpg?w=2000"
          },
          {
            "url": "https://beisat.com/377808-medium_default/plus-rib-lingerie-set.jpg"
          },
          {
            "url": "https://www.donnashape.com/59122-large_default/black-and-pink-lace-lingerie.jpg"
          },
          {
            "url": "https://i.ebayimg.com/images/g/2X8AAOSwTN5h7fvR/s-l1200.webp"
          },
          {
            "url": "https://img.freepik.com/premium-photo/beautiful-petite-filipino-woman-black-lingerie-black-stockings_900706-25586.jpg"
          },
          {
            "url": "https://st.focusedcollection.com/9163412/i/1800/focused_528433558-stock-photo-confident-asian-female-black-lingerie.jpg"
          },
          {
            "url": "https://image.made-in-china.com/202f0j00kuVoNCUBrjqH/Lingerie-Lace-Babydoll-Sexy-Bra-and-Panty-Sets-Stripper-Outfit-Womens-Sexy-Lingerie-Women-Bra-Underwear-Set.jpg"
          }
        ],
        "bestGuessLabels": [
          {
            "label": "lingerie"
          }
        ]
      }
    }
  ]
}

Type 2 (incorrect no pagesWithMatchingImages field - no images detected)

{
  "responses": [
    {
      "webDetection": {
        "webEntities": [
          {
            "entityId": "/g/11c3wtg652",
            "score": 0.4792,
            "description": "Hair M"
          },
          {
            "entityId": "/m/01dvt1",
            "score": 0.47189686,
            "description": "Joint"
          },
          {
            "entityId": "/m/0fc1fy",
            "score": 0.3617702,
            "description": "Black hair"
          },
          {
            "entityId": "/m/03q69",
            "score": 0.3603,
            "description": "Hair"
          },
          {
            "entityId": "/m/02pd__z",
            "score": 0.3500381,
            "description": "Long hair"
          },
          {
            "entityId": "/m/02js9",
            "score": 0.3062,
            "description": "Erotica"
          },
          {
            "entityId": "/m/03gpl8",
            "score": 0.3032,
            "description": "Model M keyboard"
          },
          {
            "entityId": "/m/016m0f",
            "description": "Human skeleton"
          },
          {
            "entityId": "/m/01540",
            "description": "Biology"
          },
          {
            "entityId": "/m/06mq7",
            "description": "Science"
          }
        ],
        "visuallySimilarImages": [
          {
            "url": "https://beisat.com/377808-large_default/plus-rib-lingerie-set.jpg"
          },
          {
            "url": "https://image.made-in-china.com/202f0j00kuVoNCUBrjqH/Lingerie-Lace-Babydoll-Sexy-Bra-and-Panty-Sets-Stripper-Outfit-Womens-Sexy-Lingerie-Women-Bra-Underwear-Set.jpg"
          },
          {
            "url": "http://www.lumiere18.com/cdn/shop/files/MaisieLingerieSetGIF.gif?v=1689576296"
          },
          {
            "url": "https://img.freepik.com/premium-photo/perfect-woman-beautiful-seductive-burgundy-lingerie_133748-1871.jpg?w=2000"
          },
          {
            "url": "https://oneempire.com.au/cdn/shop/products/Women_sLingerie_IntimatesOneEmpire_16_3c06a9b0-6ead-40f9-a0dc-509b6421ecff.jpg?v=1680310974"
          },
          {
            "url": "https://www.lisecharmel.com/media/wysiwyg/Lise-CHarmel-Eclat-de-Soie-C80-Splendeur-Aurore-M.jpg"
          },
          {
            "url": "https://img.joomcdn.net/4ed88aabe577bac4a9985e000f7dea7b20c52403_682_1024.jpeg"
          },
          {
            "url": "https://as2.ftcdn.net/v2/jpg/04/48/54/77/1000_F_448547794_oSEj4SZuK96ztmlIR2jiBuvAwtrY6IyR.jpg"
          },
          {
            "url": "https://img.kwcdn.com/product/Fancyalgo/VirtualModelMatting/2c76c2ce804ad92087065a63486a1b35.jpg?imageMogr2/auto-orient%7CimageView2/2/w/1300/q/80/format/webp"
          },
          {
            "url": "https://images.meesho.com/images/products/76610195/miwuv_512.webp"
          }
        ],
        "bestGuessLabels": [
          {
            "label": "lingerie"
          }
        ]
      }
    }
  ]
}

Comments

nr...@google.com <nr...@google.com> Sep 22, 2023 05:21PM

Assigned to nr...@google.com.

nr...@google.com <nr...@google.com> #2Sep 25, 2023 04:32AM

Reassigned to gc...@google.com.

I have informed our engineering team of this feature request. There is currently no ETA for its implementation.

A current workaround would be to check the returned "boundingPoly" [1] "vertices" for the returned "textAnnotations". If the calculated rectangle's heights > widths, than your image is sideways.

[1]

https://cloud.google.com/vision/reference/rest/v1/images/annotate#boundingpoly

mi...@google.com <mi...@google.com> #3Sep 26, 2023 03:12PM

I also need this problem solved :)

co...@marcinstecki.pl <co...@marcinstecki.pl> #4Oct 4, 2023 08:42AM

same :D

Message last modified on Oct 4, 2023 08:42AM

aa...@google.com <aa...@google.com> Dec 21, 2023 03:24PM

Status: New

ni...@copyrightagent.com <ni...@copyrightagent.com> #5Feb 26, 2024 12:42PM

co...@marcinstecki.pl <co...@marcinstecki.pl> #6Feb 27, 2024 12:15PM

jo...@gmail.com <jo...@gmail.com> #7Aug 27, 2024 09:11AM

This needs more attention. It's not just a display issue as described in the report. The co-ordinates returned in 'boundingPoly' are incorrect if the image was taken on a phone. All the x points should be y and vice versa.

The workaround does not make sense as "boundingPoly" [1] "vertices" for "textAnnotations" does not indicate the image dimensions - it indicates the dimensions of the relevant text block inside the image.

IssueTracker