IssueTracker

Problem you have encountered

When using nested objects and data structures, as in this sample notebook, function calls that used to work with Gemini 1.5 now return "unknown" when using the same schema.

A user also reported a similar issue in GitHub.

Example function declaration

create_product_listing = FunctionDeclaration(
    name="create_product_listing",
    description="Create a product listing using the details provided by the user",
    parameters={
        "type": "object",
        "properties": {
            "product": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "brand": {"type": "string"},
                    "price": {"type": "number"},
                    "category": {"type": "string"},
                    "description": {"type": "string"},
                    "colors": {
                        "type": "object",
                        "properties": {
                            "color": {"type": "number"},
                        },
                    },
                },
            }
        },
    },
)

Working output

I get the following with gemini-1.0-pro-002, and I also used to get this response with gemini-1.5-pro-001 and gemini-1.5-pro-002:

[name: "create_product_listing"
 args {
   fields {
     key: "product"
     value {
       struct_value {
         fields {
           key: "price"
           value {
             number_value: 149.99
           }
         }
         fields {
           key: "name"
           value {
             string_value: "Noise-Canceling Headphones"
           }
         }
         fields {
           key: "description"
           value {
             string_value: "Create a distraction-free environment with these noise-canceling headphones."
           }
         }
         fields {
           key: "colors"
           value {
             struct_value {
               fields {
                 key: "color"
                 value {
                   list_value {
                     values {
                       string_value: "black"
                     }
                     values {
                       string_value: "white"
                     }
                     values {
                       string_value: "red"
                     }
                   }
                 }
               }
             }
           }
         }
         fields {
           key: "category"
           value {
             string_value: "Electronics"
           }
         }
       }
     }
   }
 }]

Unknown output

I get the following with gemini-1.5-pro-002:

[name: "create_product_listing"
 args {
   fields {
     key: "product"
     value {
       string_value: "unknown"
     }
   }
 }]

Steps to reproduce

1. Open the notebook https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/function_calling_data_structures.ipynb.
2. Navigate to the final example in that notebook (Example: Nested parameters and data structures).
3. In that code example, change the two occurrences of array to object (I'll update the sample notebook to fix this part).
4. Run the cells and see the output with "unknown"
5. Change to gemini-1.0-pro-002.
6. Rerun the cells to see the expected output with nested data structures.