IssueTracker

Problem you have encountered:

When going through the regression tests for LangChain.js, running the function calling test for Gemini 2.0 would consistently fail while the identical test passes with the Gemini 1.5 models.

Here is the code. If necessary, I can see exactly what REST is being sent:

  test("function", async () => {
    const tools: GeminiTool[] = [
      {
        functionDeclarations: [
          {
            name: "test",
            description:
              "Run a test with a specific name and get if it passed or failed",
            parameters: {
              type: "object",
              properties: {
                testName: {
                  type: "string",
                  description: "The name of the test that should be run.",
                },
              },
              required: ["testName"],
            },
          },
        ],
      },
    ];
    const model = new ChatVertexAI({
      modelName,
    }).bind({
      tools,
    });
    const result = await model.invoke("Run a test on the cobalt project");
    expect(result).toHaveProperty("content");
    expect(result.content).toBe("");
    const args = result?.lc_kwargs?.additional_kwargs;
    expect(args).toBeDefined();
    expect(args).toHaveProperty("tool_calls");
    expect(Array.isArray(args.tool_calls)).toBeTruthy();
    expect(args.tool_calls).toHaveLength(1);
    const call = args.tool_calls[0];
    expect(call).toHaveProperty("type");
    expect(call.type).toBe("function");
    expect(call).toHaveProperty("function");
    const func = call.function;
    expect(func).toBeDefined();
    expect(func).toHaveProperty("name");
    expect(func.name).toBe("test");
    expect(func).toHaveProperty("arguments");
    expect(typeof func.arguments).toBe("string");
    expect(func.arguments.replaceAll("\n", "")).toBe('{"testName":"cobalt"}');
  });

What you expected to happen:

The Gemini 1.5 models properly respond with function calling information with the parameter "testName" set to "cobalt". (As verified by the last line in the test.)

In general, I expected relatively vague phrases or human-like references to be handled well. What is most surprising is that these work well in the 1.5 models, but less well in 2.0.

What happened:

The test fails because there is text content saying "I need the name of the test that you want to run.". Attempting other variants on the prompt (such as "Run the cobalt test") gave similar results.

It isn't until I give it a very very specific prompt ("Run a test named cobalt") that it works.

Other information (workarounds you have tried, documentation consulted, etc):