Vertex AI API . projects . locations . evaluationRuns

cancel(name, body=None, x__xgafv=None)

Cancels an Evaluation Run. Attempts to cancel a running Evaluation Run asynchronously. Status of run can be checked via GetEvaluationRun.

Args:
  name: string, Required. The name of the EvaluationRun resource to be cancelled. Format: `projects/{project}/locations/{location}/evaluationRuns/{evaluation_run}` (required)
  body: object, The request body.
    The object takes the form of:

{ # Request message for EvaluationManagementService.CancelEvaluationRun.
}

  x__xgafv: string, V1 error format.
    Allowed values
      1 - v1 error format
      2 - v2 error format

Returns:
  An object of the form:

    { # A generic empty message that you can re-use to avoid defining duplicated empty messages in your APIs. A typical example is to use it as the request or the response type of an API method. For instance: service Foo { rpc Bar(google.protobuf.Empty) returns (google.protobuf.Empty); }
}

close()

Close httplib2 connections.

create(parent, body=None, x__xgafv=None)

Creates an Evaluation Run.

Args:
  parent: string, Required. The resource name of the Location to create the Evaluation Run in. Format: `projects/{project}/locations/{location}` (required)
  body: object, The request body.
    The object takes the form of:

{ # EvaluationRun is a resource that represents a single evaluation run, which includes a set of prompts, model responses, evaluation configuration and the resulting metrics.
  "completionTime": "A String", # Output only. Time when the evaluation run was completed.
  "createTime": "A String", # Output only. Time when the evaluation run was created.
  "dataSource": { # The data source for the evaluation run. # Required. The data source for the evaluation run.
    "bigqueryRequestSet": { # The request set for the evaluation run. # Evaluation data in bigquery.
      "candidateResponseColumns": { # Optional. Map of candidate name to candidate response column name. The column will be in evaluation_item.CandidateResponse format.
        "a_key": "A String",
      },
      "promptColumn": "A String", # Optional. The name of the column that contains the requests to evaluate. This will be in evaluation_item.EvalPrompt format.
      "rubricsColumn": "A String", # Optional. The name of the column that contains the rubrics. This will be in evaluation_rubric.RubricGroup format (cl/762595858).
      "samplingConfig": { # The sampling config. # Optional. The sampling config for the bigquery resource.
        "samplingCount": 42, # Optional. The total number of logged data to import. If available data is less than the sampling count, all data will be imported. Default is 100.
        "samplingDuration": "A String", # Optional. How long to wait before sampling data from the BigQuery table. If not specified, defaults to 0.
        "samplingMethod": "A String", # Optional. The sampling method to use.
      },
      "uri": "A String", # Required. The URI of a BigQuery table. e.g. bq://projectId.bqDatasetId.bqTableId
    },
    "evaluationSet": "A String", # The EvaluationSet resource name. Format: `projects/{project}/locations/{location}/evaluationSets/{evaluation_set}`
  },
  "displayName": "A String", # Required. The display name of the Evaluation Run.
  "error": { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors). # Output only. Only populated when the evaluation run's state is FAILED or CANCELLED.
    "code": 42, # The status code, which should be an enum value of google.rpc.Code.
    "details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
      {
        "a_key": "", # Properties of the object. Contains field @type with type URL.
      },
    ],
    "message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
  },
  "evaluationConfig": { # The Evalution configuration used for the evaluation run. # Required. The configuration used for the evaluation.
    "autoraterConfig": { # The autorater config used for the evaluation run. # Optional. The autorater config for the evaluation run.
      "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
      "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
        "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
        "candidateCount": 42, # Optional. Number of candidates to generate.
        "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
        "frequencyPenalty": 3.14, # Optional. Frequency penalties.
        "logprobs": 42, # Optional. Logit probabilities.
        "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
        "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
        "modelConfig": { # Config for model selection. # Optional. Config for model selection.
          "featureSelectionPreference": "A String", # Required. Feature selection preference.
        },
        "presencePenalty": 3.14, # Optional. Positive penalties.
        "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
        "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
        "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
        "responseModalities": [ # Optional. The modalities of the response.
          "A String",
        ],
        "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
          "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
          "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
            # Object with schema name: GoogleCloudAiplatformV1beta1Schema
          ],
          "default": "", # Optional. Default value of the data.
          "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
            "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
          },
          "description": "A String", # Optional. The description of the data.
          "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
            "A String",
          ],
          "example": "", # Optional. Example of the object. Will only populated when the object is the root.
          "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
          "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
          "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
          "maxLength": "A String", # Optional. Maximum length of the Type.STRING
          "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
          "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
          "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
          "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
          "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
          "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
          "nullable": True or False, # Optional. Indicates if the value may be null.
          "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
          "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
            "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
          },
          "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
            "A String",
          ],
          "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
          "required": [ # Optional. Required properties of Type.OBJECT.
            "A String",
          ],
          "title": "A String", # Optional. The title of the Schema.
          "type": "A String", # Optional. The type of the data.
        },
        "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
          "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
            "modelRoutingPreference": "A String", # The model routing preference.
          },
          "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
            "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
          },
        },
        "seed": 42, # Optional. Seed.
        "speechConfig": { # The speech generation config. # Optional. The speech generation config.
          "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
          "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
            "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
              "voiceName": "A String", # The name of the preset voice to use.
            },
          },
        },
        "stopSequences": [ # Optional. Stop sequences.
          "A String",
        ],
        "temperature": 3.14, # Optional. Controls the randomness of predictions.
        "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
          "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
          "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
        },
        "topK": 3.14, # Optional. If specified, top-k sampling will be used.
        "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
      },
      "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
    },
    "metrics": [ # Required. The metrics to be calculated in the evaluation run.
      { # The metric used for evaluation runs.
        "llmBasedMetricSpec": { # Specification for an LLM based metric. # Spec for an LLM based metric.
          "additionalConfig": { # Optional. Optional additional configuration for the metric.
            "a_key": "", # Properties of the object.
          },
          "judgeAutoraterConfig": { # The autorater config used for the evaluation run. # Optional. Optional configuration for the judge LLM (Autorater).
            "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
            "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
              "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
              "candidateCount": 42, # Optional. Number of candidates to generate.
              "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
              "frequencyPenalty": 3.14, # Optional. Frequency penalties.
              "logprobs": 42, # Optional. Logit probabilities.
              "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
              "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
              "modelConfig": { # Config for model selection. # Optional. Config for model selection.
                "featureSelectionPreference": "A String", # Required. Feature selection preference.
              },
              "presencePenalty": 3.14, # Optional. Positive penalties.
              "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
              "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
              "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
              "responseModalities": [ # Optional. The modalities of the response.
                "A String",
              ],
              "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
                "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
                "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
                  # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                ],
                "default": "", # Optional. Default value of the data.
                "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
                  "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                },
                "description": "A String", # Optional. The description of the data.
                "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
                  "A String",
                ],
                "example": "", # Optional. Example of the object. Will only populated when the object is the root.
                "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
                "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
                "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
                "maxLength": "A String", # Optional. Maximum length of the Type.STRING
                "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
                "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
                "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
                "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
                "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
                "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
                "nullable": True or False, # Optional. Indicates if the value may be null.
                "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
                "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
                  "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                },
                "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
                  "A String",
                ],
                "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
                "required": [ # Optional. Required properties of Type.OBJECT.
                  "A String",
                ],
                "title": "A String", # Optional. The title of the Schema.
                "type": "A String", # Optional. The type of the data.
              },
              "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
                "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
                  "modelRoutingPreference": "A String", # The model routing preference.
                },
                "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
                  "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
                },
              },
              "seed": 42, # Optional. Seed.
              "speechConfig": { # The speech generation config. # Optional. The speech generation config.
                "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
                "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
                  "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
                    "voiceName": "A String", # The name of the preset voice to use.
                  },
                },
              },
              "stopSequences": [ # Optional. Stop sequences.
                "A String",
              ],
              "temperature": 3.14, # Optional. Controls the randomness of predictions.
              "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
                "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
                "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
              },
              "topK": 3.14, # Optional. If specified, top-k sampling will be used.
              "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
            },
            "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
          },
          "metricPromptTemplate": "A String", # Required. Template for the prompt sent to the judge model.
          "predefinedRubricGenerationSpec": { # Specification for a pre-defined metric. # Dynamically generate rubrics using a predefined spec.
            "metricSpecName": "A String", # Required. The name of a pre-defined metric, such as "instruction_following_v1" or "text_quality_v1".
            "parameters": { # Optional. The parameters needed to run the pre-defined metric.
              "a_key": "", # Properties of the object.
            },
          },
          "rubricGenerationSpec": { # Specification for how rubrics should be generated. # Dynamically generate rubrics using this specification.
            "modelConfig": { # The autorater config used for the evaluation run. # Optional. Configuration for the model used in rubric generation. Configs including sampling count and base model can be specified here. Flipping is not supported for rubric generation.
              "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
              "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
                "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
                "candidateCount": 42, # Optional. Number of candidates to generate.
                "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
                "frequencyPenalty": 3.14, # Optional. Frequency penalties.
                "logprobs": 42, # Optional. Logit probabilities.
                "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
                "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
                "modelConfig": { # Config for model selection. # Optional. Config for model selection.
                  "featureSelectionPreference": "A String", # Required. Feature selection preference.
                },
                "presencePenalty": 3.14, # Optional. Positive penalties.
                "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
                "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
                "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
                "responseModalities": [ # Optional. The modalities of the response.
                  "A String",
                ],
                "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
                  "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
                  "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
                    # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                  ],
                  "default": "", # Optional. Default value of the data.
                  "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
                    "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                  },
                  "description": "A String", # Optional. The description of the data.
                  "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
                    "A String",
                  ],
                  "example": "", # Optional. Example of the object. Will only populated when the object is the root.
                  "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
                  "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
                  "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
                  "maxLength": "A String", # Optional. Maximum length of the Type.STRING
                  "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
                  "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
                  "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
                  "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
                  "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
                  "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
                  "nullable": True or False, # Optional. Indicates if the value may be null.
                  "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
                  "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
                    "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                  },
                  "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
                    "A String",
                  ],
                  "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
                  "required": [ # Optional. Required properties of Type.OBJECT.
                    "A String",
                  ],
                  "title": "A String", # Optional. The title of the Schema.
                  "type": "A String", # Optional. The type of the data.
                },
                "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
                  "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
                    "modelRoutingPreference": "A String", # The model routing preference.
                  },
                  "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
                    "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
                  },
                },
                "seed": 42, # Optional. Seed.
                "speechConfig": { # The speech generation config. # Optional. The speech generation config.
                  "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
                  "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
                    "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
                      "voiceName": "A String", # The name of the preset voice to use.
                    },
                  },
                },
                "stopSequences": [ # Optional. Stop sequences.
                  "A String",
                ],
                "temperature": 3.14, # Optional. Controls the randomness of predictions.
                "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
                  "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
                  "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
                },
                "topK": 3.14, # Optional. If specified, top-k sampling will be used.
                "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
              },
              "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
            },
            "promptTemplate": "A String", # Optional. Template for the prompt used to generate rubrics. The details should be updated based on the most-recent recipe requirements.
            "rubricContentType": "A String", # Optional. The type of rubric content to be generated.
            "rubricTypeOntology": [ # Optional. An optional, pre-defined list of allowed types for generated rubrics. If this field is provided, it implies `include_rubric_type` should be true, and the generated rubric types should be chosen from this ontology.
              "A String",
            ],
          },
          "rubricGroupKey": "A String", # Use a pre-defined group of rubrics associated with the input. Refers to a key in the rubric_groups map of EvaluationInstance.
          "systemInstruction": "A String", # Optional. System instructions for the judge model.
        },
        "metric": "A String", # Required. The name of the metric.
        "predefinedMetricSpec": { # Specification for a pre-defined metric. # Spec for a pre-defined metric.
          "metricSpecName": "A String", # Required. The name of a pre-defined metric, such as "instruction_following_v1" or "text_quality_v1".
          "parameters": { # Optional. The parameters needed to run the pre-defined metric.
            "a_key": "", # Properties of the object.
          },
        },
        "rubricBasedMetricSpec": { # Specification for a metric that is based on rubrics. # Spec for rubric based metric.
          "inlineRubrics": { # Defines a list of rubrics, used when providing rubrics inline. # Use rubrics provided directly in the spec.
            "rubrics": [ # The list of rubrics.
              { # Message representing a single testable criterion for evaluation. One input prompt could have multiple rubrics.
                "content": { # Content of the rubric, defining the testable criteria. # Required. The actual testable criteria for the rubric.
                  "property": { # Defines criteria based on a specific property. # Evaluation criteria based on a specific property.
                    "description": "A String", # Description of the property being evaluated. Example: "The model's response is grammatically correct."
                  },
                },
                "importance": "A String", # Optional. The relative importance of this rubric.
                "rubricId": "A String", # Unique identifier for the rubric. This ID is used to refer to this rubric, e.g., in RubricVerdict.
                "type": "A String", # Optional. A type designator for the rubric, which can inform how it's evaluated or interpreted by systems or users. It's recommended to use consistent, well-defined, upper snake_case strings. Examples: "SUMMARIZATION_QUALITY", "SAFETY_HARMFUL_CONTENT", "INSTRUCTION_ADHERENCE".
              },
            ],
          },
          "judgeAutoraterConfig": { # The autorater config used for the evaluation run. # Optional. Optional configuration for the judge LLM (Autorater). The definition of AutoraterConfig needs to be provided.
            "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
            "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
              "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
              "candidateCount": 42, # Optional. Number of candidates to generate.
              "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
              "frequencyPenalty": 3.14, # Optional. Frequency penalties.
              "logprobs": 42, # Optional. Logit probabilities.
              "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
              "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
              "modelConfig": { # Config for model selection. # Optional. Config for model selection.
                "featureSelectionPreference": "A String", # Required. Feature selection preference.
              },
              "presencePenalty": 3.14, # Optional. Positive penalties.
              "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
              "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
              "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
              "responseModalities": [ # Optional. The modalities of the response.
                "A String",
              ],
              "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
                "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
                "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
                  # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                ],
                "default": "", # Optional. Default value of the data.
                "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
                  "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                },
                "description": "A String", # Optional. The description of the data.
                "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
                  "A String",
                ],
                "example": "", # Optional. Example of the object. Will only populated when the object is the root.
                "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
                "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
                "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
                "maxLength": "A String", # Optional. Maximum length of the Type.STRING
                "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
                "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
                "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
                "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
                "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
                "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
                "nullable": True or False, # Optional. Indicates if the value may be null.
                "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
                "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
                  "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                },
                "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
                  "A String",
                ],
                "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
                "required": [ # Optional. Required properties of Type.OBJECT.
                  "A String",
                ],
                "title": "A String", # Optional. The title of the Schema.
                "type": "A String", # Optional. The type of the data.
              },
              "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
                "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
                  "modelRoutingPreference": "A String", # The model routing preference.
                },
                "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
                  "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
                },
              },
              "seed": 42, # Optional. Seed.
              "speechConfig": { # The speech generation config. # Optional. The speech generation config.
                "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
                "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
                  "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
                    "voiceName": "A String", # The name of the preset voice to use.
                  },
                },
              },
              "stopSequences": [ # Optional. Stop sequences.
                "A String",
              ],
              "temperature": 3.14, # Optional. Controls the randomness of predictions.
              "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
                "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
                "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
              },
              "topK": 3.14, # Optional. If specified, top-k sampling will be used.
              "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
            },
            "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
          },
          "metricPromptTemplate": "A String", # Optional. Template for the prompt used by the judge model to evaluate against rubrics.
          "rubricGenerationSpec": { # Specification for how rubrics should be generated. # Dynamically generate rubrics for evaluation using this specification.
            "modelConfig": { # The autorater config used for the evaluation run. # Optional. Configuration for the model used in rubric generation. Configs including sampling count and base model can be specified here. Flipping is not supported for rubric generation.
              "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
              "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
                "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
                "candidateCount": 42, # Optional. Number of candidates to generate.
                "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
                "frequencyPenalty": 3.14, # Optional. Frequency penalties.
                "logprobs": 42, # Optional. Logit probabilities.
                "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
                "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
                "modelConfig": { # Config for model selection. # Optional. Config for model selection.
                  "featureSelectionPreference": "A String", # Required. Feature selection preference.
                },
                "presencePenalty": 3.14, # Optional. Positive penalties.
                "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
                "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
                "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
                "responseModalities": [ # Optional. The modalities of the response.
                  "A String",
                ],
                "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
                  "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
                  "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
                    # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                  ],
                  "default": "", # Optional. Default value of the data.
                  "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
                    "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                  },
                  "description": "A String", # Optional. The description of the data.
                  "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
                    "A String",
                  ],
                  "example": "", # Optional. Example of the object. Will only populated when the object is the root.
                  "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
                  "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
                  "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
                  "maxLength": "A String", # Optional. Maximum length of the Type.STRING
                  "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
                  "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
                  "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
                  "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
                  "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
                  "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
                  "nullable": True or False, # Optional. Indicates if the value may be null.
                  "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
                  "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
                    "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                  },
                  "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
                    "A String",
                  ],
                  "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
                  "required": [ # Optional. Required properties of Type.OBJECT.
                    "A String",
                  ],
                  "title": "A String", # Optional. The title of the Schema.
                  "type": "A String", # Optional. The type of the data.
                },
                "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
                  "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
                    "modelRoutingPreference": "A String", # The model routing preference.
                  },
                  "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
                    "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
                  },
                },
                "seed": 42, # Optional. Seed.
                "speechConfig": { # The speech generation config. # Optional. The speech generation config.
                  "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
                  "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
                    "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
                      "voiceName": "A String", # The name of the preset voice to use.
                    },
                  },
                },
                "stopSequences": [ # Optional. Stop sequences.
                  "A String",
                ],
                "temperature": 3.14, # Optional. Controls the randomness of predictions.
                "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
                  "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
                  "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
                },
                "topK": 3.14, # Optional. If specified, top-k sampling will be used.
                "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
              },
              "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
            },
            "promptTemplate": "A String", # Optional. Template for the prompt used to generate rubrics. The details should be updated based on the most-recent recipe requirements.
            "rubricContentType": "A String", # Optional. The type of rubric content to be generated.
            "rubricTypeOntology": [ # Optional. An optional, pre-defined list of allowed types for generated rubrics. If this field is provided, it implies `include_rubric_type` should be true, and the generated rubric types should be chosen from this ontology.
              "A String",
            ],
          },
          "rubricGroupKey": "A String", # Use a pre-defined group of rubrics associated with the input content. This refers to a key in the `rubric_groups` map of `RubricEnhancedContents`.
        },
      },
    ],
    "outputConfig": { # The output config for the evaluation run. # Optional. The output config for the evaluation run.
      "bigqueryDestination": { # The BigQuery location for the output content. # BigQuery destination for evaluation output.
        "outputUri": "A String", # Required. BigQuery URI to a project or table, up to 2000 characters long. When only the project is specified, the Dataset and Table is created. When the full table reference is specified, the Dataset must exist and table must not exist. Accepted forms: * BigQuery path. For example: `bq://projectId` or `bq://projectId.bqDatasetId` or `bq://projectId.bqDatasetId.bqTableId`.
      },
      "gcsDestination": { # The Google Cloud Storage location where the output is to be written to. # Cloud Storage destination for evaluation output.
        "outputUriPrefix": "A String", # Required. Google Cloud Storage URI to output directory. If the uri doesn't end with '/', a '/' will be automatically appended. The directory is created if it doesn't exist.
      },
    },
    "promptTemplate": { # Prompt template used for inference. # The prompt template used for inference. The values for variables in the prompt template are defined in EvaluationItem.EvaluationPrompt.PromptTemplateData.values.
      "gcsUri": "A String", # Prompt template stored in Cloud Storage. Format: "gs://my-bucket/file-name.txt".
      "promptTemplate": "A String", # Inline prompt template. Template variables should be in the format "{var_name}". Example: "Translate the following from {source_lang} to {target_lang}: {text}"
    },
    "rubricConfigs": [ # Optional. The rubric configs for the evaluation run. They are used to generate rubrics which can be used by rubric-based metrics. Multiple rubric configs can be specified for rubric generation but only one rubric config can be used for a rubric-based metric. If more than one rubric config is provided, the evaluation metric must specify a rubric group key. Note that if a generation spec is specified on both a rubric config and an evaluation metric, the rubrics generated for the metric will be used for evaluation.
      { # Configuration for a rubric group to be generated/saved for evaluation.
        "predefinedRubricGenerationSpec": { # Specification for a pre-defined metric. # Dynamically generate rubrics using a predefined spec.
          "metricSpecName": "A String", # Required. The name of a pre-defined metric, such as "instruction_following_v1" or "text_quality_v1".
          "parameters": { # Optional. The parameters needed to run the pre-defined metric.
            "a_key": "", # Properties of the object.
          },
        },
        "rubricGenerationSpec": { # Specification for how rubrics should be generated. # Dynamically generate rubrics using this specification.
          "modelConfig": { # The autorater config used for the evaluation run. # Optional. Configuration for the model used in rubric generation. Configs including sampling count and base model can be specified here. Flipping is not supported for rubric generation.
            "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
            "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
              "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
              "candidateCount": 42, # Optional. Number of candidates to generate.
              "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
              "frequencyPenalty": 3.14, # Optional. Frequency penalties.
              "logprobs": 42, # Optional. Logit probabilities.
              "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
              "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
              "modelConfig": { # Config for model selection. # Optional. Config for model selection.
                "featureSelectionPreference": "A String", # Required. Feature selection preference.
              },
              "presencePenalty": 3.14, # Optional. Positive penalties.
              "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
              "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
              "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
              "responseModalities": [ # Optional. The modalities of the response.
                "A String",
              ],
              "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
                "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
                "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
                  # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                ],
                "default": "", # Optional. Default value of the data.
                "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
                  "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                },
                "description": "A String", # Optional. The description of the data.
                "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
                  "A String",
                ],
                "example": "", # Optional. Example of the object. Will only populated when the object is the root.
                "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
                "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
                "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
                "maxLength": "A String", # Optional. Maximum length of the Type.STRING
                "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
                "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
                "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
                "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
                "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
                "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
                "nullable": True or False, # Optional. Indicates if the value may be null.
                "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
                "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
                  "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                },
                "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
                  "A String",
                ],
                "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
                "required": [ # Optional. Required properties of Type.OBJECT.
                  "A String",
                ],
                "title": "A String", # Optional. The title of the Schema.
                "type": "A String", # Optional. The type of the data.
              },
              "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
                "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
                  "modelRoutingPreference": "A String", # The model routing preference.
                },
                "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
                  "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
                },
              },
              "seed": 42, # Optional. Seed.
              "speechConfig": { # The speech generation config. # Optional. The speech generation config.
                "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
                "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
                  "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
                    "voiceName": "A String", # The name of the preset voice to use.
                  },
                },
              },
              "stopSequences": [ # Optional. Stop sequences.
                "A String",
              ],
              "temperature": 3.14, # Optional. Controls the randomness of predictions.
              "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
                "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
                "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
              },
              "topK": 3.14, # Optional. If specified, top-k sampling will be used.
              "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
            },
            "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
          },
          "promptTemplate": "A String", # Optional. Template for the prompt used to generate rubrics. The details should be updated based on the most-recent recipe requirements.
          "rubricContentType": "A String", # Optional. The type of rubric content to be generated.
          "rubricTypeOntology": [ # Optional. An optional, pre-defined list of allowed types for generated rubrics. If this field is provided, it implies `include_rubric_type` should be true, and the generated rubric types should be chosen from this ontology.
            "A String",
          ],
        },
        "rubricGroupKey": "A String", # Required. The key used to save the generated rubrics. If a generation spec is provided, this key will be used for the name of the generated rubric group. Otherwise, this key will be used to look up the existing rubric group on the evaluation item. Note that if a rubric group key is specified on both a rubric config and an evaluation metric, the key from the metric will be used to select the rubrics for evaluation.
      },
    ],
  },
  "evaluationResults": { # The results of the evaluation run. # Output only. The results of the evaluation run. Only populated when the evaluation run's state is SUCCEEDED.
    "evaluationSet": "A String", # The evaluation set where item level results are stored.
    "summaryMetrics": { # The summary metrics for the evaluation run. # Optional. The summary metrics for the evaluation run.
      "failedItems": 42, # Optional. The number of items that failed to be evaluated.
      "metrics": { # Optional. Map of metric name to metric value.
        "a_key": "",
      },
      "totalItems": 42, # Optional. The total number of items that were evaluated.
    },
  },
  "evaluationSetSnapshot": "A String", # Output only. The specific evaluation set of the evaluation run. For runs with an evaluation set input, this will be that same set. For runs with BigQuery input, it's the sampled BigQuery dataset.
  "inferenceConfigs": { # Optional. The candidate to inference config map for the evaluation run. The candidate can be up to 128 characters long and can consist of any UTF-8 characters.
    "a_key": { # An inference config used for model inference during the evaluation run.
      "generationConfig": { # Generation config. # Optional. Generation config.
        "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
        "candidateCount": 42, # Optional. Number of candidates to generate.
        "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
        "frequencyPenalty": 3.14, # Optional. Frequency penalties.
        "logprobs": 42, # Optional. Logit probabilities.
        "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
        "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
        "modelConfig": { # Config for model selection. # Optional. Config for model selection.
          "featureSelectionPreference": "A String", # Required. Feature selection preference.
        },
        "presencePenalty": 3.14, # Optional. Positive penalties.
        "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
        "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
        "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
        "responseModalities": [ # Optional. The modalities of the response.
          "A String",
        ],
        "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
          "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
          "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
            # Object with schema name: GoogleCloudAiplatformV1beta1Schema
          ],
          "default": "", # Optional. Default value of the data.
          "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
            "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
          },
          "description": "A String", # Optional. The description of the data.
          "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
            "A String",
          ],
          "example": "", # Optional. Example of the object. Will only populated when the object is the root.
          "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
          "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
          "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
          "maxLength": "A String", # Optional. Maximum length of the Type.STRING
          "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
          "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
          "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
          "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
          "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
          "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
          "nullable": True or False, # Optional. Indicates if the value may be null.
          "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
          "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
            "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
          },
          "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
            "A String",
          ],
          "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
          "required": [ # Optional. Required properties of Type.OBJECT.
            "A String",
          ],
          "title": "A String", # Optional. The title of the Schema.
          "type": "A String", # Optional. The type of the data.
        },
        "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
          "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
            "modelRoutingPreference": "A String", # The model routing preference.
          },
          "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
            "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
          },
        },
        "seed": 42, # Optional. Seed.
        "speechConfig": { # The speech generation config. # Optional. The speech generation config.
          "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
          "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
            "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
              "voiceName": "A String", # The name of the preset voice to use.
            },
          },
        },
        "stopSequences": [ # Optional. Stop sequences.
          "A String",
        ],
        "temperature": 3.14, # Optional. Controls the randomness of predictions.
        "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
          "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
          "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
        },
        "topK": 3.14, # Optional. If specified, top-k sampling will be used.
        "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
      },
      "model": "A String", # Required. The fully qualified name of the publisher model or endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
    },
  },
  "labels": { # Optional. Labels for the evaluation run.
    "a_key": "A String",
  },
  "metadata": "", # Optional. Metadata about the evaluation run, can be used by the caller to store additional tracking information about the evaluation run.
  "name": "A String", # Identifier. The resource name of the EvaluationRun. This is a unique identifier. Format: `projects/{project}/locations/{location}/evaluationRuns/{evaluation_run}`
  "state": "A String", # Output only. The state of the evaluation run.
}

  x__xgafv: string, V1 error format.
    Allowed values
      1 - v1 error format
      2 - v2 error format

Returns:
  An object of the form:

    { # EvaluationRun is a resource that represents a single evaluation run, which includes a set of prompts, model responses, evaluation configuration and the resulting metrics.
  "completionTime": "A String", # Output only. Time when the evaluation run was completed.
  "createTime": "A String", # Output only. Time when the evaluation run was created.
  "dataSource": { # The data source for the evaluation run. # Required. The data source for the evaluation run.
    "bigqueryRequestSet": { # The request set for the evaluation run. # Evaluation data in bigquery.
      "candidateResponseColumns": { # Optional. Map of candidate name to candidate response column name. The column will be in evaluation_item.CandidateResponse format.
        "a_key": "A String",
      },
      "promptColumn": "A String", # Optional. The name of the column that contains the requests to evaluate. This will be in evaluation_item.EvalPrompt format.
      "rubricsColumn": "A String", # Optional. The name of the column that contains the rubrics. This will be in evaluation_rubric.RubricGroup format (cl/762595858).
      "samplingConfig": { # The sampling config. # Optional. The sampling config for the bigquery resource.
        "samplingCount": 42, # Optional. The total number of logged data to import. If available data is less than the sampling count, all data will be imported. Default is 100.
        "samplingDuration": "A String", # Optional. How long to wait before sampling data from the BigQuery table. If not specified, defaults to 0.
        "samplingMethod": "A String", # Optional. The sampling method to use.
      },
      "uri": "A String", # Required. The URI of a BigQuery table. e.g. bq://projectId.bqDatasetId.bqTableId
    },
    "evaluationSet": "A String", # The EvaluationSet resource name. Format: `projects/{project}/locations/{location}/evaluationSets/{evaluation_set}`
  },
  "displayName": "A String", # Required. The display name of the Evaluation Run.
  "error": { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors). # Output only. Only populated when the evaluation run's state is FAILED or CANCELLED.
    "code": 42, # The status code, which should be an enum value of google.rpc.Code.
    "details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
      {
        "a_key": "", # Properties of the object. Contains field @type with type URL.
      },
    ],
    "message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
  },
  "evaluationConfig": { # The Evalution configuration used for the evaluation run. # Required. The configuration used for the evaluation.
    "autoraterConfig": { # The autorater config used for the evaluation run. # Optional. The autorater config for the evaluation run.
      "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
      "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
        "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
        "candidateCount": 42, # Optional. Number of candidates to generate.
        "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
        "frequencyPenalty": 3.14, # Optional. Frequency penalties.
        "logprobs": 42, # Optional. Logit probabilities.
        "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
        "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
        "modelConfig": { # Config for model selection. # Optional. Config for model selection.
          "featureSelectionPreference": "A String", # Required. Feature selection preference.
        },
        "presencePenalty": 3.14, # Optional. Positive penalties.
        "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
        "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
        "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
        "responseModalities": [ # Optional. The modalities of the response.
          "A String",
        ],
        "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
          "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
          "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
            # Object with schema name: GoogleCloudAiplatformV1beta1Schema
          ],
          "default": "", # Optional. Default value of the data.
          "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
            "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
          },
          "description": "A String", # Optional. The description of the data.
          "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
            "A String",
          ],
          "example": "", # Optional. Example of the object. Will only populated when the object is the root.
          "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
          "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
          "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
          "maxLength": "A String", # Optional. Maximum length of the Type.STRING
          "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
          "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
          "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
          "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
          "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
          "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
          "nullable": True or False, # Optional. Indicates if the value may be null.
          "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
          "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
            "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
          },
          "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
            "A String",
          ],
          "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
          "required": [ # Optional. Required properties of Type.OBJECT.
            "A String",
          ],
          "title": "A String", # Optional. The title of the Schema.
          "type": "A String", # Optional. The type of the data.
        },
        "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
          "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
            "modelRoutingPreference": "A String", # The model routing preference.
          },
          "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
            "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
          },
        },
        "seed": 42, # Optional. Seed.
        "speechConfig": { # The speech generation config. # Optional. The speech generation config.
          "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
          "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
            "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
              "voiceName": "A String", # The name of the preset voice to use.
            },
          },
        },
        "stopSequences": [ # Optional. Stop sequences.
          "A String",
        ],
        "temperature": 3.14, # Optional. Controls the randomness of predictions.
        "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
          "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
          "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
        },
        "topK": 3.14, # Optional. If specified, top-k sampling will be used.
        "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
      },
      "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
    },
    "metrics": [ # Required. The metrics to be calculated in the evaluation run.
      { # The metric used for evaluation runs.
        "llmBasedMetricSpec": { # Specification for an LLM based metric. # Spec for an LLM based metric.
          "additionalConfig": { # Optional. Optional additional configuration for the metric.
            "a_key": "", # Properties of the object.
          },
          "judgeAutoraterConfig": { # The autorater config used for the evaluation run. # Optional. Optional configuration for the judge LLM (Autorater).
            "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
            "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
              "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
              "candidateCount": 42, # Optional. Number of candidates to generate.
              "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
              "frequencyPenalty": 3.14, # Optional. Frequency penalties.
              "logprobs": 42, # Optional. Logit probabilities.
              "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
              "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
              "modelConfig": { # Config for model selection. # Optional. Config for model selection.
                "featureSelectionPreference": "A String", # Required. Feature selection preference.
              },
              "presencePenalty": 3.14, # Optional. Positive penalties.
              "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
              "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
              "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
              "responseModalities": [ # Optional. The modalities of the response.
                "A String",
              ],
              "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
                "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
                "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
                  # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                ],
                "default": "", # Optional. Default value of the data.
                "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
                  "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                },
                "description": "A String", # Optional. The description of the data.
                "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
                  "A String",
                ],
                "example": "", # Optional. Example of the object. Will only populated when the object is the root.
                "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
                "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
                "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
                "maxLength": "A String", # Optional. Maximum length of the Type.STRING
                "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
                "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
                "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
                "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
                "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
                "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
                "nullable": True or False, # Optional. Indicates if the value may be null.
                "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
                "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
                  "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                },
                "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
                  "A String",
                ],
                "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
                "required": [ # Optional. Required properties of Type.OBJECT.
                  "A String",
                ],
                "title": "A String", # Optional. The title of the Schema.
                "type": "A String", # Optional. The type of the data.
              },
              "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
                "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
                  "modelRoutingPreference": "A String", # The model routing preference.
                },
                "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
                  "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
                },
              },
              "seed": 42, # Optional. Seed.
              "speechConfig": { # The speech generation config. # Optional. The speech generation config.
                "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
                "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
                  "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
                    "voiceName": "A String", # The name of the preset voice to use.
                  },
                },
              },
              "stopSequences": [ # Optional. Stop sequences.
                "A String",
              ],
              "temperature": 3.14, # Optional. Controls the randomness of predictions.
              "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
                "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
                "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
              },
              "topK": 3.14, # Optional. If specified, top-k sampling will be used.
              "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
            },
            "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
          },
          "metricPromptTemplate": "A String", # Required. Template for the prompt sent to the judge model.
          "predefinedRubricGenerationSpec": { # Specification for a pre-defined metric. # Dynamically generate rubrics using a predefined spec.
            "metricSpecName": "A String", # Required. The name of a pre-defined metric, such as "instruction_following_v1" or "text_quality_v1".
            "parameters": { # Optional. The parameters needed to run the pre-defined metric.
              "a_key": "", # Properties of the object.
            },
          },
          "rubricGenerationSpec": { # Specification for how rubrics should be generated. # Dynamically generate rubrics using this specification.
            "modelConfig": { # The autorater config used for the evaluation run. # Optional. Configuration for the model used in rubric generation. Configs including sampling count and base model can be specified here. Flipping is not supported for rubric generation.
              "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
              "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
                "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
                "candidateCount": 42, # Optional. Number of candidates to generate.
                "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
                "frequencyPenalty": 3.14, # Optional. Frequency penalties.
                "logprobs": 42, # Optional. Logit probabilities.
                "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
                "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
                "modelConfig": { # Config for model selection. # Optional. Config for model selection.
                  "featureSelectionPreference": "A String", # Required. Feature selection preference.
                },
                "presencePenalty": 3.14, # Optional. Positive penalties.
                "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
                "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
                "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
                "responseModalities": [ # Optional. The modalities of the response.
                  "A String",
                ],
                "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
                  "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
                  "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
                    # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                  ],
                  "default": "", # Optional. Default value of the data.
                  "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
                    "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                  },
                  "description": "A String", # Optional. The description of the data.
                  "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
                    "A String",
                  ],
                  "example": "", # Optional. Example of the object. Will only populated when the object is the root.
                  "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
                  "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
                  "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
                  "maxLength": "A String", # Optional. Maximum length of the Type.STRING
                  "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
                  "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
                  "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
                  "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
                  "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
                  "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
                  "nullable": True or False, # Optional. Indicates if the value may be null.
                  "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
                  "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
                    "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                  },
                  "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
                    "A String",
                  ],
                  "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
                  "required": [ # Optional. Required properties of Type.OBJECT.
                    "A String",
                  ],
                  "title": "A String", # Optional. The title of the Schema.
                  "type": "A String", # Optional. The type of the data.
                },
                "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
                  "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
                    "modelRoutingPreference": "A String", # The model routing preference.
                  },
                  "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
                    "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
                  },
                },
                "seed": 42, # Optional. Seed.
                "speechConfig": { # The speech generation config. # Optional. The speech generation config.
                  "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
                  "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
                    "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
                      "voiceName": "A String", # The name of the preset voice to use.
                    },
                  },
                },
                "stopSequences": [ # Optional. Stop sequences.
                  "A String",
                ],
                "temperature": 3.14, # Optional. Controls the randomness of predictions.
                "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
                  "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
                  "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
                },
                "topK": 3.14, # Optional. If specified, top-k sampling will be used.
                "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
              },
              "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
            },
            "promptTemplate": "A String", # Optional. Template for the prompt used to generate rubrics. The details should be updated based on the most-recent recipe requirements.
            "rubricContentType": "A String", # Optional. The type of rubric content to be generated.
            "rubricTypeOntology": [ # Optional. An optional, pre-defined list of allowed types for generated rubrics. If this field is provided, it implies `include_rubric_type` should be true, and the generated rubric types should be chosen from this ontology.
              "A String",
            ],
          },
          "rubricGroupKey": "A String", # Use a pre-defined group of rubrics associated with the input. Refers to a key in the rubric_groups map of EvaluationInstance.
          "systemInstruction": "A String", # Optional. System instructions for the judge model.
        },
        "metric": "A String", # Required. The name of the metric.
        "predefinedMetricSpec": { # Specification for a pre-defined metric. # Spec for a pre-defined metric.
          "metricSpecName": "A String", # Required. The name of a pre-defined metric, such as "instruction_following_v1" or "text_quality_v1".
          "parameters": { # Optional. The parameters needed to run the pre-defined metric.
            "a_key": "", # Properties of the object.
          },
        },
        "rubricBasedMetricSpec": { # Specification for a metric that is based on rubrics. # Spec for rubric based metric.
          "inlineRubrics": { # Defines a list of rubrics, used when providing rubrics inline. # Use rubrics provided directly in the spec.
            "rubrics": [ # The list of rubrics.
              { # Message representing a single testable criterion for evaluation. One input prompt could have multiple rubrics.
                "content": { # Content of the rubric, defining the testable criteria. # Required. The actual testable criteria for the rubric.
                  "property": { # Defines criteria based on a specific property. # Evaluation criteria based on a specific property.
                    "description": "A String", # Description of the property being evaluated. Example: "The model's response is grammatically correct."
                  },
                },
                "importance": "A String", # Optional. The relative importance of this rubric.
                "rubricId": "A String", # Unique identifier for the rubric. This ID is used to refer to this rubric, e.g., in RubricVerdict.
                "type": "A String", # Optional. A type designator for the rubric, which can inform how it's evaluated or interpreted by systems or users. It's recommended to use consistent, well-defined, upper snake_case strings. Examples: "SUMMARIZATION_QUALITY", "SAFETY_HARMFUL_CONTENT", "INSTRUCTION_ADHERENCE".
              },
            ],
          },
          "judgeAutoraterConfig": { # The autorater config used for the evaluation run. # Optional. Optional configuration for the judge LLM (Autorater). The definition of AutoraterConfig needs to be provided.
            "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
            "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
              "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
              "candidateCount": 42, # Optional. Number of candidates to generate.
              "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
              "frequencyPenalty": 3.14, # Optional. Frequency penalties.
              "logprobs": 42, # Optional. Logit probabilities.
              "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
              "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
              "modelConfig": { # Config for model selection. # Optional. Config for model selection.
                "featureSelectionPreference": "A String", # Required. Feature selection preference.
              },
              "presencePenalty": 3.14, # Optional. Positive penalties.
              "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
              "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
              "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
              "responseModalities": [ # Optional. The modalities of the response.
                "A String",
              ],
              "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
                "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
                "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
                  # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                ],
                "default": "", # Optional. Default value of the data.
                "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
                  "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                },
                "description": "A String", # Optional. The description of the data.
                "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
                  "A String",
                ],
                "example": "", # Optional. Example of the object. Will only populated when the object is the root.
                "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
                "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
                "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
                "maxLength": "A String", # Optional. Maximum length of the Type.STRING
                "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
                "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
                "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
                "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
                "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
                "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
                "nullable": True or False, # Optional. Indicates if the value may be null.
                "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
                "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
                  "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                },
                "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
                  "A String",
                ],
                "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
                "required": [ # Optional. Required properties of Type.OBJECT.
                  "A String",
                ],
                "title": "A String", # Optional. The title of the Schema.
                "type": "A String", # Optional. The type of the data.
              },
              "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
                "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
                  "modelRoutingPreference": "A String", # The model routing preference.
                },
                "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
                  "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
                },
              },
              "seed": 42, # Optional. Seed.
              "speechConfig": { # The speech generation config. # Optional. The speech generation config.
                "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
                "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
                  "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
                    "voiceName": "A String", # The name of the preset voice to use.
                  },
                },
              },
              "stopSequences": [ # Optional. Stop sequences.
                "A String",
              ],
              "temperature": 3.14, # Optional. Controls the randomness of predictions.
              "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
                "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
                "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
              },
              "topK": 3.14, # Optional. If specified, top-k sampling will be used.
              "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
            },
            "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
          },
          "metricPromptTemplate": "A String", # Optional. Template for the prompt used by the judge model to evaluate against rubrics.
          "rubricGenerationSpec": { # Specification for how rubrics should be generated. # Dynamically generate rubrics for evaluation using this specification.
            "modelConfig": { # The autorater config used for the evaluation run. # Optional. Configuration for the model used in rubric generation. Configs including sampling count and base model can be specified here. Flipping is not supported for rubric generation.
              "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
              "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
                "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
                "candidateCount": 42, # Optional. Number of candidates to generate.
                "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
                "frequencyPenalty": 3.14, # Optional. Frequency penalties.
                "logprobs": 42, # Optional. Logit probabilities.
                "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
                "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
                "modelConfig": { # Config for model selection. # Optional. Config for model selection.
                  "featureSelectionPreference": "A String", # Required. Feature selection preference.
                },
                "presencePenalty": 3.14, # Optional. Positive penalties.
                "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
                "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
                "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
                "responseModalities": [ # Optional. The modalities of the response.
                  "A String",
                ],
                "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
                  "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
                  "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
                    # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                  ],
                  "default": "", # Optional. Default value of the data.
                  "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
                    "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                  },
                  "description": "A String", # Optional. The description of the data.
                  "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
                    "A String",
                  ],
                  "example": "", # Optional. Example of the object. Will only populated when the object is the root.
                  "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
                  "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
                  "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
                  "maxLength": "A String", # Optional. Maximum length of the Type.STRING
                  "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
                  "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
                  "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
                  "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
                  "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
                  "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
                  "nullable": True or False, # Optional. Indicates if the value may be null.
                  "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
                  "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
                    "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                  },
                  "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
                    "A String",
                  ],
                  "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
                  "required": [ # Optional. Required properties of Type.OBJECT.
                    "A String",
                  ],
                  "title": "A String", # Optional. The title of the Schema.
                  "type": "A String", # Optional. The type of the data.
                },
                "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
                  "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
                    "modelRoutingPreference": "A String", # The model routing preference.
                  },
                  "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
                    "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
                  },
                },
                "seed": 42, # Optional. Seed.
                "speechConfig": { # The speech generation config. # Optional. The speech generation config.
                  "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
                  "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
                    "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
                      "voiceName": "A String", # The name of the preset voice to use.
                    },
                  },
                },
                "stopSequences": [ # Optional. Stop sequences.
                  "A String",
                ],
                "temperature": 3.14, # Optional. Controls the randomness of predictions.
                "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
                  "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
                  "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
                },
                "topK": 3.14, # Optional. If specified, top-k sampling will be used.
                "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
              },
              "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
            },
            "promptTemplate": "A String", # Optional. Template for the prompt used to generate rubrics. The details should be updated based on the most-recent recipe requirements.
            "rubricContentType": "A String", # Optional. The type of rubric content to be generated.
            "rubricTypeOntology": [ # Optional. An optional, pre-defined list of allowed types for generated rubrics. If this field is provided, it implies `include_rubric_type` should be true, and the generated rubric types should be chosen from this ontology.
              "A String",
            ],
          },
          "rubricGroupKey": "A String", # Use a pre-defined group of rubrics associated with the input content. This refers to a key in the `rubric_groups` map of `RubricEnhancedContents`.
        },
      },
    ],
    "outputConfig": { # The output config for the evaluation run. # Optional. The output config for the evaluation run.
      "bigqueryDestination": { # The BigQuery location for the output content. # BigQuery destination for evaluation output.
        "outputUri": "A String", # Required. BigQuery URI to a project or table, up to 2000 characters long. When only the project is specified, the Dataset and Table is created. When the full table reference is specified, the Dataset must exist and table must not exist. Accepted forms: * BigQuery path. For example: `bq://projectId` or `bq://projectId.bqDatasetId` or `bq://projectId.bqDatasetId.bqTableId`.
      },
      "gcsDestination": { # The Google Cloud Storage location where the output is to be written to. # Cloud Storage destination for evaluation output.
        "outputUriPrefix": "A String", # Required. Google Cloud Storage URI to output directory. If the uri doesn't end with '/', a '/' will be automatically appended. The directory is created if it doesn't exist.
      },
    },
    "promptTemplate": { # Prompt template used for inference. # The prompt template used for inference. The values for variables in the prompt template are defined in EvaluationItem.EvaluationPrompt.PromptTemplateData.values.
      "gcsUri": "A String", # Prompt template stored in Cloud Storage. Format: "gs://my-bucket/file-name.txt".
      "promptTemplate": "A String", # Inline prompt template. Template variables should be in the format "{var_name}". Example: "Translate the following from {source_lang} to {target_lang}: {text}"
    },
    "rubricConfigs": [ # Optional. The rubric configs for the evaluation run. They are used to generate rubrics which can be used by rubric-based metrics. Multiple rubric configs can be specified for rubric generation but only one rubric config can be used for a rubric-based metric. If more than one rubric config is provided, the evaluation metric must specify a rubric group key. Note that if a generation spec is specified on both a rubric config and an evaluation metric, the rubrics generated for the metric will be used for evaluation.
      { # Configuration for a rubric group to be generated/saved for evaluation.
        "predefinedRubricGenerationSpec": { # Specification for a pre-defined metric. # Dynamically generate rubrics using a predefined spec.
          "metricSpecName": "A String", # Required. The name of a pre-defined metric, such as "instruction_following_v1" or "text_quality_v1".
          "parameters": { # Optional. The parameters needed to run the pre-defined metric.
            "a_key": "", # Properties of the object.
          },
        },
        "rubricGenerationSpec": { # Specification for how rubrics should be generated. # Dynamically generate rubrics using this specification.
          "modelConfig": { # The autorater config used for the evaluation run. # Optional. Configuration for the model used in rubric generation. Configs including sampling count and base model can be specified here. Flipping is not supported for rubric generation.
            "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
            "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
              "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
              "candidateCount": 42, # Optional. Number of candidates to generate.
              "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
              "frequencyPenalty": 3.14, # Optional. Frequency penalties.
              "logprobs": 42, # Optional. Logit probabilities.
              "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
              "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
              "modelConfig": { # Config for model selection. # Optional. Config for model selection.
                "featureSelectionPreference": "A String", # Required. Feature selection preference.
              },
              "presencePenalty": 3.14, # Optional. Positive penalties.
              "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
              "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
              "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
              "responseModalities": [ # Optional. The modalities of the response.
                "A String",
              ],
              "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
                "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
                "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
                  # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                ],
                "default": "", # Optional. Default value of the data.
                "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
                  "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                },
                "description": "A String", # Optional. The description of the data.
                "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
                  "A String",
                ],
                "example": "", # Optional. Example of the object. Will only populated when the object is the root.
                "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
                "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
                "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
                "maxLength": "A String", # Optional. Maximum length of the Type.STRING
                "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
                "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
                "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
                "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
                "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
                "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
                "nullable": True or False, # Optional. Indicates if the value may be null.
                "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
                "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
                  "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                },
                "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
                  "A String",
                ],
                "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
                "required": [ # Optional. Required properties of Type.OBJECT.
                  "A String",
                ],
                "title": "A String", # Optional. The title of the Schema.
                "type": "A String", # Optional. The type of the data.
              },
              "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
                "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
                  "modelRoutingPreference": "A String", # The model routing preference.
                },
                "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
                  "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
                },
              },
              "seed": 42, # Optional. Seed.
              "speechConfig": { # The speech generation config. # Optional. The speech generation config.
                "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
                "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
                  "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
                    "voiceName": "A String", # The name of the preset voice to use.
                  },
                },
              },
              "stopSequences": [ # Optional. Stop sequences.
                "A String",
              ],
              "temperature": 3.14, # Optional. Controls the randomness of predictions.
              "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
                "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
                "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
              },
              "topK": 3.14, # Optional. If specified, top-k sampling will be used.
              "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
            },
            "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
          },
          "promptTemplate": "A String", # Optional. Template for the prompt used to generate rubrics. The details should be updated based on the most-recent recipe requirements.
          "rubricContentType": "A String", # Optional. The type of rubric content to be generated.
          "rubricTypeOntology": [ # Optional. An optional, pre-defined list of allowed types for generated rubrics. If this field is provided, it implies `include_rubric_type` should be true, and the generated rubric types should be chosen from this ontology.
            "A String",
          ],
        },
        "rubricGroupKey": "A String", # Required. The key used to save the generated rubrics. If a generation spec is provided, this key will be used for the name of the generated rubric group. Otherwise, this key will be used to look up the existing rubric group on the evaluation item. Note that if a rubric group key is specified on both a rubric config and an evaluation metric, the key from the metric will be used to select the rubrics for evaluation.
      },
    ],
  },
  "evaluationResults": { # The results of the evaluation run. # Output only. The results of the evaluation run. Only populated when the evaluation run's state is SUCCEEDED.
    "evaluationSet": "A String", # The evaluation set where item level results are stored.
    "summaryMetrics": { # The summary metrics for the evaluation run. # Optional. The summary metrics for the evaluation run.
      "failedItems": 42, # Optional. The number of items that failed to be evaluated.
      "metrics": { # Optional. Map of metric name to metric value.
        "a_key": "",
      },
      "totalItems": 42, # Optional. The total number of items that were evaluated.
    },
  },
  "evaluationSetSnapshot": "A String", # Output only. The specific evaluation set of the evaluation run. For runs with an evaluation set input, this will be that same set. For runs with BigQuery input, it's the sampled BigQuery dataset.
  "inferenceConfigs": { # Optional. The candidate to inference config map for the evaluation run. The candidate can be up to 128 characters long and can consist of any UTF-8 characters.
    "a_key": { # An inference config used for model inference during the evaluation run.
      "generationConfig": { # Generation config. # Optional. Generation config.
        "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
        "candidateCount": 42, # Optional. Number of candidates to generate.
        "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
        "frequencyPenalty": 3.14, # Optional. Frequency penalties.
        "logprobs": 42, # Optional. Logit probabilities.
        "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
        "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
        "modelConfig": { # Config for model selection. # Optional. Config for model selection.
          "featureSelectionPreference": "A String", # Required. Feature selection preference.
        },
        "presencePenalty": 3.14, # Optional. Positive penalties.
        "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
        "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
        "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
        "responseModalities": [ # Optional. The modalities of the response.
          "A String",
        ],
        "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
          "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
          "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
            # Object with schema name: GoogleCloudAiplatformV1beta1Schema
          ],
          "default": "", # Optional. Default value of the data.
          "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
            "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
          },
          "description": "A String", # Optional. The description of the data.
          "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
            "A String",
          ],
          "example": "", # Optional. Example of the object. Will only populated when the object is the root.
          "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
          "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
          "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
          "maxLength": "A String", # Optional. Maximum length of the Type.STRING
          "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
          "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
          "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
          "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
          "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
          "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
          "nullable": True or False, # Optional. Indicates if the value may be null.
          "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
          "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
            "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
          },
          "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
            "A String",
          ],
          "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
          "required": [ # Optional. Required properties of Type.OBJECT.
            "A String",
          ],
          "title": "A String", # Optional. The title of the Schema.
          "type": "A String", # Optional. The type of the data.
        },
        "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
          "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
            "modelRoutingPreference": "A String", # The model routing preference.
          },
          "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
            "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
          },
        },
        "seed": 42, # Optional. Seed.
        "speechConfig": { # The speech generation config. # Optional. The speech generation config.
          "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
          "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
            "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
              "voiceName": "A String", # The name of the preset voice to use.
            },
          },
        },
        "stopSequences": [ # Optional. Stop sequences.
          "A String",
        ],
        "temperature": 3.14, # Optional. Controls the randomness of predictions.
        "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
          "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
          "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
        },
        "topK": 3.14, # Optional. If specified, top-k sampling will be used.
        "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
      },
      "model": "A String", # Required. The fully qualified name of the publisher model or endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
    },
  },
  "labels": { # Optional. Labels for the evaluation run.
    "a_key": "A String",
  },
  "metadata": "", # Optional. Metadata about the evaluation run, can be used by the caller to store additional tracking information about the evaluation run.
  "name": "A String", # Identifier. The resource name of the EvaluationRun. This is a unique identifier. Format: `projects/{project}/locations/{location}/evaluationRuns/{evaluation_run}`
  "state": "A String", # Output only. The state of the evaluation run.
}

delete(name, x__xgafv=None)

Deletes an Evaluation Run.

Args:
  name: string, Required. The name of the EvaluationRun resource to be deleted. Format: `projects/{project}/locations/{location}/evaluationRuns/{evaluation_run}` (required)
  x__xgafv: string, V1 error format.
    Allowed values
      1 - v1 error format
      2 - v2 error format

Returns:
  An object of the form:

    { # This resource represents a long-running operation that is the result of a network API call.
  "done": True or False, # If the value is `false`, it means the operation is still in progress. If `true`, the operation is completed, and either `error` or `response` is available.
  "error": { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors). # The error result of the operation in case of failure or cancellation.
    "code": 42, # The status code, which should be an enum value of google.rpc.Code.
    "details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
      {
        "a_key": "", # Properties of the object. Contains field @type with type URL.
      },
    ],
    "message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
  },
  "metadata": { # Service-specific metadata associated with the operation. It typically contains progress information and common metadata such as create time. Some services might not provide such metadata. Any method that returns a long-running operation should document the metadata type, if any.
    "a_key": "", # Properties of the object. Contains field @type with type URL.
  },
  "name": "A String", # The server-assigned name, which is only unique within the same service that originally returns it. If you use the default HTTP mapping, the `name` should be a resource name ending with `operations/{unique_id}`.
  "response": { # The normal, successful response of the operation. If the original method returns no data on success, such as `Delete`, the response is `google.protobuf.Empty`. If the original method is standard `Get`/`Create`/`Update`, the response should be the resource. For other methods, the response should have the type `XxxResponse`, where `Xxx` is the original method name. For example, if the original method name is `TakeSnapshot()`, the inferred response type is `TakeSnapshotResponse`.
    "a_key": "", # Properties of the object. Contains field @type with type URL.
  },
}

get(name, x__xgafv=None)

Gets an Evaluation Run.

Args:
  name: string, Required. The name of the EvaluationRun resource. Format: `projects/{project}/locations/{location}/evaluationRuns/{evaluation_run}` (required)
  x__xgafv: string, V1 error format.
    Allowed values
      1 - v1 error format
      2 - v2 error format

Returns:
  An object of the form:

    { # EvaluationRun is a resource that represents a single evaluation run, which includes a set of prompts, model responses, evaluation configuration and the resulting metrics.
  "completionTime": "A String", # Output only. Time when the evaluation run was completed.
  "createTime": "A String", # Output only. Time when the evaluation run was created.
  "dataSource": { # The data source for the evaluation run. # Required. The data source for the evaluation run.
    "bigqueryRequestSet": { # The request set for the evaluation run. # Evaluation data in bigquery.
      "candidateResponseColumns": { # Optional. Map of candidate name to candidate response column name. The column will be in evaluation_item.CandidateResponse format.
        "a_key": "A String",
      },
      "promptColumn": "A String", # Optional. The name of the column that contains the requests to evaluate. This will be in evaluation_item.EvalPrompt format.
      "rubricsColumn": "A String", # Optional. The name of the column that contains the rubrics. This will be in evaluation_rubric.RubricGroup format (cl/762595858).
      "samplingConfig": { # The sampling config. # Optional. The sampling config for the bigquery resource.
        "samplingCount": 42, # Optional. The total number of logged data to import. If available data is less than the sampling count, all data will be imported. Default is 100.
        "samplingDuration": "A String", # Optional. How long to wait before sampling data from the BigQuery table. If not specified, defaults to 0.
        "samplingMethod": "A String", # Optional. The sampling method to use.
      },
      "uri": "A String", # Required. The URI of a BigQuery table. e.g. bq://projectId.bqDatasetId.bqTableId
    },
    "evaluationSet": "A String", # The EvaluationSet resource name. Format: `projects/{project}/locations/{location}/evaluationSets/{evaluation_set}`
  },
  "displayName": "A String", # Required. The display name of the Evaluation Run.
  "error": { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors). # Output only. Only populated when the evaluation run's state is FAILED or CANCELLED.
    "code": 42, # The status code, which should be an enum value of google.rpc.Code.
    "details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
      {
        "a_key": "", # Properties of the object. Contains field @type with type URL.
      },
    ],
    "message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
  },
  "evaluationConfig": { # The Evalution configuration used for the evaluation run. # Required. The configuration used for the evaluation.
    "autoraterConfig": { # The autorater config used for the evaluation run. # Optional. The autorater config for the evaluation run.
      "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
      "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
        "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
        "candidateCount": 42, # Optional. Number of candidates to generate.
        "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
        "frequencyPenalty": 3.14, # Optional. Frequency penalties.
        "logprobs": 42, # Optional. Logit probabilities.
        "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
        "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
        "modelConfig": { # Config for model selection. # Optional. Config for model selection.
          "featureSelectionPreference": "A String", # Required. Feature selection preference.
        },
        "presencePenalty": 3.14, # Optional. Positive penalties.
        "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
        "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
        "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
        "responseModalities": [ # Optional. The modalities of the response.
          "A String",
        ],
        "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
          "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
          "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
            # Object with schema name: GoogleCloudAiplatformV1beta1Schema
          ],
          "default": "", # Optional. Default value of the data.
          "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
            "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
          },
          "description": "A String", # Optional. The description of the data.
          "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
            "A String",
          ],
          "example": "", # Optional. Example of the object. Will only populated when the object is the root.
          "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
          "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
          "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
          "maxLength": "A String", # Optional. Maximum length of the Type.STRING
          "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
          "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
          "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
          "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
          "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
          "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
          "nullable": True or False, # Optional. Indicates if the value may be null.
          "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
          "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
            "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
          },
          "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
            "A String",
          ],
          "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
          "required": [ # Optional. Required properties of Type.OBJECT.
            "A String",
          ],
          "title": "A String", # Optional. The title of the Schema.
          "type": "A String", # Optional. The type of the data.
        },
        "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
          "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
            "modelRoutingPreference": "A String", # The model routing preference.
          },
          "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
            "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
          },
        },
        "seed": 42, # Optional. Seed.
        "speechConfig": { # The speech generation config. # Optional. The speech generation config.
          "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
          "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
            "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
              "voiceName": "A String", # The name of the preset voice to use.
            },
          },
        },
        "stopSequences": [ # Optional. Stop sequences.
          "A String",
        ],
        "temperature": 3.14, # Optional. Controls the randomness of predictions.
        "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
          "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
          "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
        },
        "topK": 3.14, # Optional. If specified, top-k sampling will be used.
        "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
      },
      "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
    },
    "metrics": [ # Required. The metrics to be calculated in the evaluation run.
      { # The metric used for evaluation runs.
        "llmBasedMetricSpec": { # Specification for an LLM based metric. # Spec for an LLM based metric.
          "additionalConfig": { # Optional. Optional additional configuration for the metric.
            "a_key": "", # Properties of the object.
          },
          "judgeAutoraterConfig": { # The autorater config used for the evaluation run. # Optional. Optional configuration for the judge LLM (Autorater).
            "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
            "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
              "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
              "candidateCount": 42, # Optional. Number of candidates to generate.
              "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
              "frequencyPenalty": 3.14, # Optional. Frequency penalties.
              "logprobs": 42, # Optional. Logit probabilities.
              "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
              "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
              "modelConfig": { # Config for model selection. # Optional. Config for model selection.
                "featureSelectionPreference": "A String", # Required. Feature selection preference.
              },
              "presencePenalty": 3.14, # Optional. Positive penalties.
              "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
              "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
              "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
              "responseModalities": [ # Optional. The modalities of the response.
                "A String",
              ],
              "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
                "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
                "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
                  # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                ],
                "default": "", # Optional. Default value of the data.
                "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
                  "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                },
                "description": "A String", # Optional. The description of the data.
                "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
                  "A String",
                ],
                "example": "", # Optional. Example of the object. Will only populated when the object is the root.
                "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
                "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
                "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
                "maxLength": "A String", # Optional. Maximum length of the Type.STRING
                "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
                "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
                "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
                "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
                "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
                "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
                "nullable": True or False, # Optional. Indicates if the value may be null.
                "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
                "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
                  "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                },
                "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
                  "A String",
                ],
                "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
                "required": [ # Optional. Required properties of Type.OBJECT.
                  "A String",
                ],
                "title": "A String", # Optional. The title of the Schema.
                "type": "A String", # Optional. The type of the data.
              },
              "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
                "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
                  "modelRoutingPreference": "A String", # The model routing preference.
                },
                "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
                  "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
                },
              },
              "seed": 42, # Optional. Seed.
              "speechConfig": { # The speech generation config. # Optional. The speech generation config.
                "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
                "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
                  "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
                    "voiceName": "A String", # The name of the preset voice to use.
                  },
                },
              },
              "stopSequences": [ # Optional. Stop sequences.
                "A String",
              ],
              "temperature": 3.14, # Optional. Controls the randomness of predictions.
              "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
                "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
                "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
              },
              "topK": 3.14, # Optional. If specified, top-k sampling will be used.
              "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
            },
            "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
          },
          "metricPromptTemplate": "A String", # Required. Template for the prompt sent to the judge model.
          "predefinedRubricGenerationSpec": { # Specification for a pre-defined metric. # Dynamically generate rubrics using a predefined spec.
            "metricSpecName": "A String", # Required. The name of a pre-defined metric, such as "instruction_following_v1" or "text_quality_v1".
            "parameters": { # Optional. The parameters needed to run the pre-defined metric.
              "a_key": "", # Properties of the object.
            },
          },
          "rubricGenerationSpec": { # Specification for how rubrics should be generated. # Dynamically generate rubrics using this specification.
            "modelConfig": { # The autorater config used for the evaluation run. # Optional. Configuration for the model used in rubric generation. Configs including sampling count and base model can be specified here. Flipping is not supported for rubric generation.
              "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
              "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
                "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
                "candidateCount": 42, # Optional. Number of candidates to generate.
                "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
                "frequencyPenalty": 3.14, # Optional. Frequency penalties.
                "logprobs": 42, # Optional. Logit probabilities.
                "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
                "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
                "modelConfig": { # Config for model selection. # Optional. Config for model selection.
                  "featureSelectionPreference": "A String", # Required. Feature selection preference.
                },
                "presencePenalty": 3.14, # Optional. Positive penalties.
                "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
                "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
                "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
                "responseModalities": [ # Optional. The modalities of the response.
                  "A String",
                ],
                "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
                  "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
                  "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
                    # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                  ],
                  "default": "", # Optional. Default value of the data.
                  "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
                    "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                  },
                  "description": "A String", # Optional. The description of the data.
                  "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
                    "A String",
                  ],
                  "example": "", # Optional. Example of the object. Will only populated when the object is the root.
                  "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
                  "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
                  "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
                  "maxLength": "A String", # Optional. Maximum length of the Type.STRING
                  "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
                  "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
                  "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
                  "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
                  "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
                  "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
                  "nullable": True or False, # Optional. Indicates if the value may be null.
                  "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
                  "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
                    "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                  },
                  "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
                    "A String",
                  ],
                  "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
                  "required": [ # Optional. Required properties of Type.OBJECT.
                    "A String",
                  ],
                  "title": "A String", # Optional. The title of the Schema.
                  "type": "A String", # Optional. The type of the data.
                },
                "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
                  "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
                    "modelRoutingPreference": "A String", # The model routing preference.
                  },
                  "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
                    "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
                  },
                },
                "seed": 42, # Optional. Seed.
                "speechConfig": { # The speech generation config. # Optional. The speech generation config.
                  "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
                  "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
                    "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
                      "voiceName": "A String", # The name of the preset voice to use.
                    },
                  },
                },
                "stopSequences": [ # Optional. Stop sequences.
                  "A String",
                ],
                "temperature": 3.14, # Optional. Controls the randomness of predictions.
                "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
                  "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
                  "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
                },
                "topK": 3.14, # Optional. If specified, top-k sampling will be used.
                "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
              },
              "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
            },
            "promptTemplate": "A String", # Optional. Template for the prompt used to generate rubrics. The details should be updated based on the most-recent recipe requirements.
            "rubricContentType": "A String", # Optional. The type of rubric content to be generated.
            "rubricTypeOntology": [ # Optional. An optional, pre-defined list of allowed types for generated rubrics. If this field is provided, it implies `include_rubric_type` should be true, and the generated rubric types should be chosen from this ontology.
              "A String",
            ],
          },
          "rubricGroupKey": "A String", # Use a pre-defined group of rubrics associated with the input. Refers to a key in the rubric_groups map of EvaluationInstance.
          "systemInstruction": "A String", # Optional. System instructions for the judge model.
        },
        "metric": "A String", # Required. The name of the metric.
        "predefinedMetricSpec": { # Specification for a pre-defined metric. # Spec for a pre-defined metric.
          "metricSpecName": "A String", # Required. The name of a pre-defined metric, such as "instruction_following_v1" or "text_quality_v1".
          "parameters": { # Optional. The parameters needed to run the pre-defined metric.
            "a_key": "", # Properties of the object.
          },
        },
        "rubricBasedMetricSpec": { # Specification for a metric that is based on rubrics. # Spec for rubric based metric.
          "inlineRubrics": { # Defines a list of rubrics, used when providing rubrics inline. # Use rubrics provided directly in the spec.
            "rubrics": [ # The list of rubrics.
              { # Message representing a single testable criterion for evaluation. One input prompt could have multiple rubrics.
                "content": { # Content of the rubric, defining the testable criteria. # Required. The actual testable criteria for the rubric.
                  "property": { # Defines criteria based on a specific property. # Evaluation criteria based on a specific property.
                    "description": "A String", # Description of the property being evaluated. Example: "The model's response is grammatically correct."
                  },
                },
                "importance": "A String", # Optional. The relative importance of this rubric.
                "rubricId": "A String", # Unique identifier for the rubric. This ID is used to refer to this rubric, e.g., in RubricVerdict.
                "type": "A String", # Optional. A type designator for the rubric, which can inform how it's evaluated or interpreted by systems or users. It's recommended to use consistent, well-defined, upper snake_case strings. Examples: "SUMMARIZATION_QUALITY", "SAFETY_HARMFUL_CONTENT", "INSTRUCTION_ADHERENCE".
              },
            ],
          },
          "judgeAutoraterConfig": { # The autorater config used for the evaluation run. # Optional. Optional configuration for the judge LLM (Autorater). The definition of AutoraterConfig needs to be provided.
            "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
            "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
              "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
              "candidateCount": 42, # Optional. Number of candidates to generate.
              "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
              "frequencyPenalty": 3.14, # Optional. Frequency penalties.
              "logprobs": 42, # Optional. Logit probabilities.
              "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
              "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
              "modelConfig": { # Config for model selection. # Optional. Config for model selection.
                "featureSelectionPreference": "A String", # Required. Feature selection preference.
              },
              "presencePenalty": 3.14, # Optional. Positive penalties.
              "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
              "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
              "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
              "responseModalities": [ # Optional. The modalities of the response.
                "A String",
              ],
              "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
                "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
                "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
                  # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                ],
                "default": "", # Optional. Default value of the data.
                "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
                  "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                },
                "description": "A String", # Optional. The description of the data.
                "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
                  "A String",
                ],
                "example": "", # Optional. Example of the object. Will only populated when the object is the root.
                "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
                "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
                "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
                "maxLength": "A String", # Optional. Maximum length of the Type.STRING
                "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
                "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
                "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
                "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
                "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
                "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
                "nullable": True or False, # Optional. Indicates if the value may be null.
                "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
                "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
                  "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                },
                "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
                  "A String",
                ],
                "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
                "required": [ # Optional. Required properties of Type.OBJECT.
                  "A String",
                ],
                "title": "A String", # Optional. The title of the Schema.
                "type": "A String", # Optional. The type of the data.
              },
              "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
                "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
                  "modelRoutingPreference": "A String", # The model routing preference.
                },
                "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
                  "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
                },
              },
              "seed": 42, # Optional. Seed.
              "speechConfig": { # The speech generation config. # Optional. The speech generation config.
                "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
                "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
                  "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
                    "voiceName": "A String", # The name of the preset voice to use.
                  },
                },
              },
              "stopSequences": [ # Optional. Stop sequences.
                "A String",
              ],
              "temperature": 3.14, # Optional. Controls the randomness of predictions.
              "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
                "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
                "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
              },
              "topK": 3.14, # Optional. If specified, top-k sampling will be used.
              "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
            },
            "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
          },
          "metricPromptTemplate": "A String", # Optional. Template for the prompt used by the judge model to evaluate against rubrics.
          "rubricGenerationSpec": { # Specification for how rubrics should be generated. # Dynamically generate rubrics for evaluation using this specification.
            "modelConfig": { # The autorater config used for the evaluation run. # Optional. Configuration for the model used in rubric generation. Configs including sampling count and base model can be specified here. Flipping is not supported for rubric generation.
              "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
              "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
                "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
                "candidateCount": 42, # Optional. Number of candidates to generate.
                "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
                "frequencyPenalty": 3.14, # Optional. Frequency penalties.
                "logprobs": 42, # Optional. Logit probabilities.
                "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
                "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
                "modelConfig": { # Config for model selection. # Optional. Config for model selection.
                  "featureSelectionPreference": "A String", # Required. Feature selection preference.
                },
                "presencePenalty": 3.14, # Optional. Positive penalties.
                "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
                "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
                "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
                "responseModalities": [ # Optional. The modalities of the response.
                  "A String",
                ],
                "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
                  "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
                  "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
                    # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                  ],
                  "default": "", # Optional. Default value of the data.
                  "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
                    "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                  },
                  "description": "A String", # Optional. The description of the data.
                  "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
                    "A String",
                  ],
                  "example": "", # Optional. Example of the object. Will only populated when the object is the root.
                  "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
                  "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
                  "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
                  "maxLength": "A String", # Optional. Maximum length of the Type.STRING
                  "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
                  "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
                  "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
                  "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
                  "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
                  "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
                  "nullable": True or False, # Optional. Indicates if the value may be null.
                  "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
                  "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
                    "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                  },
                  "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
                    "A String",
                  ],
                  "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
                  "required": [ # Optional. Required properties of Type.OBJECT.
                    "A String",
                  ],
                  "title": "A String", # Optional. The title of the Schema.
                  "type": "A String", # Optional. The type of the data.
                },
                "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
                  "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
                    "modelRoutingPreference": "A String", # The model routing preference.
                  },
                  "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
                    "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
                  },
                },
                "seed": 42, # Optional. Seed.
                "speechConfig": { # The speech generation config. # Optional. The speech generation config.
                  "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
                  "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
                    "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
                      "voiceName": "A String", # The name of the preset voice to use.
                    },
                  },
                },
                "stopSequences": [ # Optional. Stop sequences.
                  "A String",
                ],
                "temperature": 3.14, # Optional. Controls the randomness of predictions.
                "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
                  "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
                  "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
                },
                "topK": 3.14, # Optional. If specified, top-k sampling will be used.
                "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
              },
              "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
            },
            "promptTemplate": "A String", # Optional. Template for the prompt used to generate rubrics. The details should be updated based on the most-recent recipe requirements.
            "rubricContentType": "A String", # Optional. The type of rubric content to be generated.
            "rubricTypeOntology": [ # Optional. An optional, pre-defined list of allowed types for generated rubrics. If this field is provided, it implies `include_rubric_type` should be true, and the generated rubric types should be chosen from this ontology.
              "A String",
            ],
          },
          "rubricGroupKey": "A String", # Use a pre-defined group of rubrics associated with the input content. This refers to a key in the `rubric_groups` map of `RubricEnhancedContents`.
        },
      },
    ],
    "outputConfig": { # The output config for the evaluation run. # Optional. The output config for the evaluation run.
      "bigqueryDestination": { # The BigQuery location for the output content. # BigQuery destination for evaluation output.
        "outputUri": "A String", # Required. BigQuery URI to a project or table, up to 2000 characters long. When only the project is specified, the Dataset and Table is created. When the full table reference is specified, the Dataset must exist and table must not exist. Accepted forms: * BigQuery path. For example: `bq://projectId` or `bq://projectId.bqDatasetId` or `bq://projectId.bqDatasetId.bqTableId`.
      },
      "gcsDestination": { # The Google Cloud Storage location where the output is to be written to. # Cloud Storage destination for evaluation output.
        "outputUriPrefix": "A String", # Required. Google Cloud Storage URI to output directory. If the uri doesn't end with '/', a '/' will be automatically appended. The directory is created if it doesn't exist.
      },
    },
    "promptTemplate": { # Prompt template used for inference. # The prompt template used for inference. The values for variables in the prompt template are defined in EvaluationItem.EvaluationPrompt.PromptTemplateData.values.
      "gcsUri": "A String", # Prompt template stored in Cloud Storage. Format: "gs://my-bucket/file-name.txt".
      "promptTemplate": "A String", # Inline prompt template. Template variables should be in the format "{var_name}". Example: "Translate the following from {source_lang} to {target_lang}: {text}"
    },
    "rubricConfigs": [ # Optional. The rubric configs for the evaluation run. They are used to generate rubrics which can be used by rubric-based metrics. Multiple rubric configs can be specified for rubric generation but only one rubric config can be used for a rubric-based metric. If more than one rubric config is provided, the evaluation metric must specify a rubric group key. Note that if a generation spec is specified on both a rubric config and an evaluation metric, the rubrics generated for the metric will be used for evaluation.
      { # Configuration for a rubric group to be generated/saved for evaluation.
        "predefinedRubricGenerationSpec": { # Specification for a pre-defined metric. # Dynamically generate rubrics using a predefined spec.
          "metricSpecName": "A String", # Required. The name of a pre-defined metric, such as "instruction_following_v1" or "text_quality_v1".
          "parameters": { # Optional. The parameters needed to run the pre-defined metric.
            "a_key": "", # Properties of the object.
          },
        },
        "rubricGenerationSpec": { # Specification for how rubrics should be generated. # Dynamically generate rubrics using this specification.
          "modelConfig": { # The autorater config used for the evaluation run. # Optional. Configuration for the model used in rubric generation. Configs including sampling count and base model can be specified here. Flipping is not supported for rubric generation.
            "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
            "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
              "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
              "candidateCount": 42, # Optional. Number of candidates to generate.
              "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
              "frequencyPenalty": 3.14, # Optional. Frequency penalties.
              "logprobs": 42, # Optional. Logit probabilities.
              "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
              "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
              "modelConfig": { # Config for model selection. # Optional. Config for model selection.
                "featureSelectionPreference": "A String", # Required. Feature selection preference.
              },
              "presencePenalty": 3.14, # Optional. Positive penalties.
              "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
              "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
              "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
              "responseModalities": [ # Optional. The modalities of the response.
                "A String",
              ],
              "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
                "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
                "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
                  # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                ],
                "default": "", # Optional. Default value of the data.
                "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
                  "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                },
                "description": "A String", # Optional. The description of the data.
                "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
                  "A String",
                ],
                "example": "", # Optional. Example of the object. Will only populated when the object is the root.
                "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
                "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
                "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
                "maxLength": "A String", # Optional. Maximum length of the Type.STRING
                "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
                "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
                "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
                "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
                "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
                "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
                "nullable": True or False, # Optional. Indicates if the value may be null.
                "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
                "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
                  "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                },
                "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
                  "A String",
                ],
                "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
                "required": [ # Optional. Required properties of Type.OBJECT.
                  "A String",
                ],
                "title": "A String", # Optional. The title of the Schema.
                "type": "A String", # Optional. The type of the data.
              },
              "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
                "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
                  "modelRoutingPreference": "A String", # The model routing preference.
                },
                "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
                  "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
                },
              },
              "seed": 42, # Optional. Seed.
              "speechConfig": { # The speech generation config. # Optional. The speech generation config.
                "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
                "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
                  "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
                    "voiceName": "A String", # The name of the preset voice to use.
                  },
                },
              },
              "stopSequences": [ # Optional. Stop sequences.
                "A String",
              ],
              "temperature": 3.14, # Optional. Controls the randomness of predictions.
              "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
                "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
                "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
              },
              "topK": 3.14, # Optional. If specified, top-k sampling will be used.
              "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
            },
            "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
          },
          "promptTemplate": "A String", # Optional. Template for the prompt used to generate rubrics. The details should be updated based on the most-recent recipe requirements.
          "rubricContentType": "A String", # Optional. The type of rubric content to be generated.
          "rubricTypeOntology": [ # Optional. An optional, pre-defined list of allowed types for generated rubrics. If this field is provided, it implies `include_rubric_type` should be true, and the generated rubric types should be chosen from this ontology.
            "A String",
          ],
        },
        "rubricGroupKey": "A String", # Required. The key used to save the generated rubrics. If a generation spec is provided, this key will be used for the name of the generated rubric group. Otherwise, this key will be used to look up the existing rubric group on the evaluation item. Note that if a rubric group key is specified on both a rubric config and an evaluation metric, the key from the metric will be used to select the rubrics for evaluation.
      },
    ],
  },
  "evaluationResults": { # The results of the evaluation run. # Output only. The results of the evaluation run. Only populated when the evaluation run's state is SUCCEEDED.
    "evaluationSet": "A String", # The evaluation set where item level results are stored.
    "summaryMetrics": { # The summary metrics for the evaluation run. # Optional. The summary metrics for the evaluation run.
      "failedItems": 42, # Optional. The number of items that failed to be evaluated.
      "metrics": { # Optional. Map of metric name to metric value.
        "a_key": "",
      },
      "totalItems": 42, # Optional. The total number of items that were evaluated.
    },
  },
  "evaluationSetSnapshot": "A String", # Output only. The specific evaluation set of the evaluation run. For runs with an evaluation set input, this will be that same set. For runs with BigQuery input, it's the sampled BigQuery dataset.
  "inferenceConfigs": { # Optional. The candidate to inference config map for the evaluation run. The candidate can be up to 128 characters long and can consist of any UTF-8 characters.
    "a_key": { # An inference config used for model inference during the evaluation run.
      "generationConfig": { # Generation config. # Optional. Generation config.
        "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
        "candidateCount": 42, # Optional. Number of candidates to generate.
        "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
        "frequencyPenalty": 3.14, # Optional. Frequency penalties.
        "logprobs": 42, # Optional. Logit probabilities.
        "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
        "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
        "modelConfig": { # Config for model selection. # Optional. Config for model selection.
          "featureSelectionPreference": "A String", # Required. Feature selection preference.
        },
        "presencePenalty": 3.14, # Optional. Positive penalties.
        "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
        "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
        "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
        "responseModalities": [ # Optional. The modalities of the response.
          "A String",
        ],
        "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
          "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
          "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
            # Object with schema name: GoogleCloudAiplatformV1beta1Schema
          ],
          "default": "", # Optional. Default value of the data.
          "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
            "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
          },
          "description": "A String", # Optional. The description of the data.
          "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
            "A String",
          ],
          "example": "", # Optional. Example of the object. Will only populated when the object is the root.
          "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
          "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
          "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
          "maxLength": "A String", # Optional. Maximum length of the Type.STRING
          "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
          "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
          "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
          "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
          "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
          "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
          "nullable": True or False, # Optional. Indicates if the value may be null.
          "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
          "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
            "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
          },
          "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
            "A String",
          ],
          "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
          "required": [ # Optional. Required properties of Type.OBJECT.
            "A String",
          ],
          "title": "A String", # Optional. The title of the Schema.
          "type": "A String", # Optional. The type of the data.
        },
        "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
          "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
            "modelRoutingPreference": "A String", # The model routing preference.
          },
          "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
            "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
          },
        },
        "seed": 42, # Optional. Seed.
        "speechConfig": { # The speech generation config. # Optional. The speech generation config.
          "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
          "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
            "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
              "voiceName": "A String", # The name of the preset voice to use.
            },
          },
        },
        "stopSequences": [ # Optional. Stop sequences.
          "A String",
        ],
        "temperature": 3.14, # Optional. Controls the randomness of predictions.
        "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
          "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
          "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
        },
        "topK": 3.14, # Optional. If specified, top-k sampling will be used.
        "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
      },
      "model": "A String", # Required. The fully qualified name of the publisher model or endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
    },
  },
  "labels": { # Optional. Labels for the evaluation run.
    "a_key": "A String",
  },
  "metadata": "", # Optional. Metadata about the evaluation run, can be used by the caller to store additional tracking information about the evaluation run.
  "name": "A String", # Identifier. The resource name of the EvaluationRun. This is a unique identifier. Format: `projects/{project}/locations/{location}/evaluationRuns/{evaluation_run}`
  "state": "A String", # Output only. The state of the evaluation run.
}

list(parent, filter=None, orderBy=None, pageSize=None, pageToken=None, x__xgafv=None)

Lists Evaluation Runs.

Args:
  parent: string, Required. The resource name of the Location from which to list the Evaluation Runs. Format: `projects/{project}/locations/{location}` (required)
  filter: string, Optional. Filter expression that matches a subset of the EvaluationRuns to show. For field names both snake_case and camelCase are supported. For more information about filter syntax, see [AIP-160](https://google.aip.dev/160).
  orderBy: string, Optional. A comma-separated list of fields to order by, sorted in ascending order by default. Use `desc` after a field name for descending.
  pageSize: integer, Optional. The maximum number of Evaluation Runs to return.
  pageToken: string, Optional. A page token, received from a previous `ListEvaluationRuns` call. Provide this to retrieve the subsequent page.
  x__xgafv: string, V1 error format.
    Allowed values
      1 - v1 error format
      2 - v2 error format

Returns:
  An object of the form:

    { # Response message for EvaluationManagementService.ListEvaluationRuns.
  "evaluationRuns": [ # List of EvaluationRuns in the requested page.
    { # EvaluationRun is a resource that represents a single evaluation run, which includes a set of prompts, model responses, evaluation configuration and the resulting metrics.
      "completionTime": "A String", # Output only. Time when the evaluation run was completed.
      "createTime": "A String", # Output only. Time when the evaluation run was created.
      "dataSource": { # The data source for the evaluation run. # Required. The data source for the evaluation run.
        "bigqueryRequestSet": { # The request set for the evaluation run. # Evaluation data in bigquery.
          "candidateResponseColumns": { # Optional. Map of candidate name to candidate response column name. The column will be in evaluation_item.CandidateResponse format.
            "a_key": "A String",
          },
          "promptColumn": "A String", # Optional. The name of the column that contains the requests to evaluate. This will be in evaluation_item.EvalPrompt format.
          "rubricsColumn": "A String", # Optional. The name of the column that contains the rubrics. This will be in evaluation_rubric.RubricGroup format (cl/762595858).
          "samplingConfig": { # The sampling config. # Optional. The sampling config for the bigquery resource.
            "samplingCount": 42, # Optional. The total number of logged data to import. If available data is less than the sampling count, all data will be imported. Default is 100.
            "samplingDuration": "A String", # Optional. How long to wait before sampling data from the BigQuery table. If not specified, defaults to 0.
            "samplingMethod": "A String", # Optional. The sampling method to use.
          },
          "uri": "A String", # Required. The URI of a BigQuery table. e.g. bq://projectId.bqDatasetId.bqTableId
        },
        "evaluationSet": "A String", # The EvaluationSet resource name. Format: `projects/{project}/locations/{location}/evaluationSets/{evaluation_set}`
      },
      "displayName": "A String", # Required. The display name of the Evaluation Run.
      "error": { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors). # Output only. Only populated when the evaluation run's state is FAILED or CANCELLED.
        "code": 42, # The status code, which should be an enum value of google.rpc.Code.
        "details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
          {
            "a_key": "", # Properties of the object. Contains field @type with type URL.
          },
        ],
        "message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
      },
      "evaluationConfig": { # The Evalution configuration used for the evaluation run. # Required. The configuration used for the evaluation.
        "autoraterConfig": { # The autorater config used for the evaluation run. # Optional. The autorater config for the evaluation run.
          "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
          "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
            "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
            "candidateCount": 42, # Optional. Number of candidates to generate.
            "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
            "frequencyPenalty": 3.14, # Optional. Frequency penalties.
            "logprobs": 42, # Optional. Logit probabilities.
            "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
            "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
            "modelConfig": { # Config for model selection. # Optional. Config for model selection.
              "featureSelectionPreference": "A String", # Required. Feature selection preference.
            },
            "presencePenalty": 3.14, # Optional. Positive penalties.
            "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
            "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
            "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
            "responseModalities": [ # Optional. The modalities of the response.
              "A String",
            ],
            "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
              "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
              "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
                # Object with schema name: GoogleCloudAiplatformV1beta1Schema
              ],
              "default": "", # Optional. Default value of the data.
              "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
                "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
              },
              "description": "A String", # Optional. The description of the data.
              "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
                "A String",
              ],
              "example": "", # Optional. Example of the object. Will only populated when the object is the root.
              "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
              "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
              "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
              "maxLength": "A String", # Optional. Maximum length of the Type.STRING
              "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
              "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
              "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
              "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
              "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
              "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
              "nullable": True or False, # Optional. Indicates if the value may be null.
              "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
              "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
                "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
              },
              "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
                "A String",
              ],
              "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
              "required": [ # Optional. Required properties of Type.OBJECT.
                "A String",
              ],
              "title": "A String", # Optional. The title of the Schema.
              "type": "A String", # Optional. The type of the data.
            },
            "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
              "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
                "modelRoutingPreference": "A String", # The model routing preference.
              },
              "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
                "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
              },
            },
            "seed": 42, # Optional. Seed.
            "speechConfig": { # The speech generation config. # Optional. The speech generation config.
              "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
              "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
                "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
                  "voiceName": "A String", # The name of the preset voice to use.
                },
              },
            },
            "stopSequences": [ # Optional. Stop sequences.
              "A String",
            ],
            "temperature": 3.14, # Optional. Controls the randomness of predictions.
            "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
              "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
              "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
            },
            "topK": 3.14, # Optional. If specified, top-k sampling will be used.
            "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
          },
          "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
        },
        "metrics": [ # Required. The metrics to be calculated in the evaluation run.
          { # The metric used for evaluation runs.
            "llmBasedMetricSpec": { # Specification for an LLM based metric. # Spec for an LLM based metric.
              "additionalConfig": { # Optional. Optional additional configuration for the metric.
                "a_key": "", # Properties of the object.
              },
              "judgeAutoraterConfig": { # The autorater config used for the evaluation run. # Optional. Optional configuration for the judge LLM (Autorater).
                "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
                "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
                  "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
                  "candidateCount": 42, # Optional. Number of candidates to generate.
                  "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
                  "frequencyPenalty": 3.14, # Optional. Frequency penalties.
                  "logprobs": 42, # Optional. Logit probabilities.
                  "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
                  "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
                  "modelConfig": { # Config for model selection. # Optional. Config for model selection.
                    "featureSelectionPreference": "A String", # Required. Feature selection preference.
                  },
                  "presencePenalty": 3.14, # Optional. Positive penalties.
                  "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
                  "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
                  "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
                  "responseModalities": [ # Optional. The modalities of the response.
                    "A String",
                  ],
                  "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
                    "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
                    "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
                      # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                    ],
                    "default": "", # Optional. Default value of the data.
                    "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
                      "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                    },
                    "description": "A String", # Optional. The description of the data.
                    "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
                      "A String",
                    ],
                    "example": "", # Optional. Example of the object. Will only populated when the object is the root.
                    "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
                    "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
                    "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
                    "maxLength": "A String", # Optional. Maximum length of the Type.STRING
                    "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
                    "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
                    "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
                    "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
                    "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
                    "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
                    "nullable": True or False, # Optional. Indicates if the value may be null.
                    "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
                    "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
                      "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                    },
                    "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
                      "A String",
                    ],
                    "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
                    "required": [ # Optional. Required properties of Type.OBJECT.
                      "A String",
                    ],
                    "title": "A String", # Optional. The title of the Schema.
                    "type": "A String", # Optional. The type of the data.
                  },
                  "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
                    "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
                      "modelRoutingPreference": "A String", # The model routing preference.
                    },
                    "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
                      "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
                    },
                  },
                  "seed": 42, # Optional. Seed.
                  "speechConfig": { # The speech generation config. # Optional. The speech generation config.
                    "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
                    "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
                      "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
                        "voiceName": "A String", # The name of the preset voice to use.
                      },
                    },
                  },
                  "stopSequences": [ # Optional. Stop sequences.
                    "A String",
                  ],
                  "temperature": 3.14, # Optional. Controls the randomness of predictions.
                  "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
                    "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
                    "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
                  },
                  "topK": 3.14, # Optional. If specified, top-k sampling will be used.
                  "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
                },
                "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
              },
              "metricPromptTemplate": "A String", # Required. Template for the prompt sent to the judge model.
              "predefinedRubricGenerationSpec": { # Specification for a pre-defined metric. # Dynamically generate rubrics using a predefined spec.
                "metricSpecName": "A String", # Required. The name of a pre-defined metric, such as "instruction_following_v1" or "text_quality_v1".
                "parameters": { # Optional. The parameters needed to run the pre-defined metric.
                  "a_key": "", # Properties of the object.
                },
              },
              "rubricGenerationSpec": { # Specification for how rubrics should be generated. # Dynamically generate rubrics using this specification.
                "modelConfig": { # The autorater config used for the evaluation run. # Optional. Configuration for the model used in rubric generation. Configs including sampling count and base model can be specified here. Flipping is not supported for rubric generation.
                  "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
                  "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
                    "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
                    "candidateCount": 42, # Optional. Number of candidates to generate.
                    "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
                    "frequencyPenalty": 3.14, # Optional. Frequency penalties.
                    "logprobs": 42, # Optional. Logit probabilities.
                    "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
                    "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
                    "modelConfig": { # Config for model selection. # Optional. Config for model selection.
                      "featureSelectionPreference": "A String", # Required. Feature selection preference.
                    },
                    "presencePenalty": 3.14, # Optional. Positive penalties.
                    "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
                    "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
                    "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
                    "responseModalities": [ # Optional. The modalities of the response.
                      "A String",
                    ],
                    "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
                      "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
                      "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
                        # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                      ],
                      "default": "", # Optional. Default value of the data.
                      "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
                        "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                      },
                      "description": "A String", # Optional. The description of the data.
                      "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
                        "A String",
                      ],
                      "example": "", # Optional. Example of the object. Will only populated when the object is the root.
                      "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
                      "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
                      "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
                      "maxLength": "A String", # Optional. Maximum length of the Type.STRING
                      "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
                      "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
                      "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
                      "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
                      "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
                      "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
                      "nullable": True or False, # Optional. Indicates if the value may be null.
                      "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
                      "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
                        "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                      },
                      "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
                        "A String",
                      ],
                      "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
                      "required": [ # Optional. Required properties of Type.OBJECT.
                        "A String",
                      ],
                      "title": "A String", # Optional. The title of the Schema.
                      "type": "A String", # Optional. The type of the data.
                    },
                    "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
                      "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
                        "modelRoutingPreference": "A String", # The model routing preference.
                      },
                      "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
                        "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
                      },
                    },
                    "seed": 42, # Optional. Seed.
                    "speechConfig": { # The speech generation config. # Optional. The speech generation config.
                      "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
                      "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
                        "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
                          "voiceName": "A String", # The name of the preset voice to use.
                        },
                      },
                    },
                    "stopSequences": [ # Optional. Stop sequences.
                      "A String",
                    ],
                    "temperature": 3.14, # Optional. Controls the randomness of predictions.
                    "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
                      "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
                      "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
                    },
                    "topK": 3.14, # Optional. If specified, top-k sampling will be used.
                    "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
                  },
                  "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
                },
                "promptTemplate": "A String", # Optional. Template for the prompt used to generate rubrics. The details should be updated based on the most-recent recipe requirements.
                "rubricContentType": "A String", # Optional. The type of rubric content to be generated.
                "rubricTypeOntology": [ # Optional. An optional, pre-defined list of allowed types for generated rubrics. If this field is provided, it implies `include_rubric_type` should be true, and the generated rubric types should be chosen from this ontology.
                  "A String",
                ],
              },
              "rubricGroupKey": "A String", # Use a pre-defined group of rubrics associated with the input. Refers to a key in the rubric_groups map of EvaluationInstance.
              "systemInstruction": "A String", # Optional. System instructions for the judge model.
            },
            "metric": "A String", # Required. The name of the metric.
            "predefinedMetricSpec": { # Specification for a pre-defined metric. # Spec for a pre-defined metric.
              "metricSpecName": "A String", # Required. The name of a pre-defined metric, such as "instruction_following_v1" or "text_quality_v1".
              "parameters": { # Optional. The parameters needed to run the pre-defined metric.
                "a_key": "", # Properties of the object.
              },
            },
            "rubricBasedMetricSpec": { # Specification for a metric that is based on rubrics. # Spec for rubric based metric.
              "inlineRubrics": { # Defines a list of rubrics, used when providing rubrics inline. # Use rubrics provided directly in the spec.
                "rubrics": [ # The list of rubrics.
                  { # Message representing a single testable criterion for evaluation. One input prompt could have multiple rubrics.
                    "content": { # Content of the rubric, defining the testable criteria. # Required. The actual testable criteria for the rubric.
                      "property": { # Defines criteria based on a specific property. # Evaluation criteria based on a specific property.
                        "description": "A String", # Description of the property being evaluated. Example: "The model's response is grammatically correct."
                      },
                    },
                    "importance": "A String", # Optional. The relative importance of this rubric.
                    "rubricId": "A String", # Unique identifier for the rubric. This ID is used to refer to this rubric, e.g., in RubricVerdict.
                    "type": "A String", # Optional. A type designator for the rubric, which can inform how it's evaluated or interpreted by systems or users. It's recommended to use consistent, well-defined, upper snake_case strings. Examples: "SUMMARIZATION_QUALITY", "SAFETY_HARMFUL_CONTENT", "INSTRUCTION_ADHERENCE".
                  },
                ],
              },
              "judgeAutoraterConfig": { # The autorater config used for the evaluation run. # Optional. Optional configuration for the judge LLM (Autorater). The definition of AutoraterConfig needs to be provided.
                "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
                "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
                  "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
                  "candidateCount": 42, # Optional. Number of candidates to generate.
                  "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
                  "frequencyPenalty": 3.14, # Optional. Frequency penalties.
                  "logprobs": 42, # Optional. Logit probabilities.
                  "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
                  "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
                  "modelConfig": { # Config for model selection. # Optional. Config for model selection.
                    "featureSelectionPreference": "A String", # Required. Feature selection preference.
                  },
                  "presencePenalty": 3.14, # Optional. Positive penalties.
                  "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
                  "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
                  "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
                  "responseModalities": [ # Optional. The modalities of the response.
                    "A String",
                  ],
                  "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
                    "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
                    "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
                      # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                    ],
                    "default": "", # Optional. Default value of the data.
                    "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
                      "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                    },
                    "description": "A String", # Optional. The description of the data.
                    "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
                      "A String",
                    ],
                    "example": "", # Optional. Example of the object. Will only populated when the object is the root.
                    "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
                    "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
                    "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
                    "maxLength": "A String", # Optional. Maximum length of the Type.STRING
                    "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
                    "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
                    "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
                    "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
                    "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
                    "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
                    "nullable": True or False, # Optional. Indicates if the value may be null.
                    "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
                    "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
                      "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                    },
                    "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
                      "A String",
                    ],
                    "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
                    "required": [ # Optional. Required properties of Type.OBJECT.
                      "A String",
                    ],
                    "title": "A String", # Optional. The title of the Schema.
                    "type": "A String", # Optional. The type of the data.
                  },
                  "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
                    "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
                      "modelRoutingPreference": "A String", # The model routing preference.
                    },
                    "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
                      "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
                    },
                  },
                  "seed": 42, # Optional. Seed.
                  "speechConfig": { # The speech generation config. # Optional. The speech generation config.
                    "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
                    "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
                      "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
                        "voiceName": "A String", # The name of the preset voice to use.
                      },
                    },
                  },
                  "stopSequences": [ # Optional. Stop sequences.
                    "A String",
                  ],
                  "temperature": 3.14, # Optional. Controls the randomness of predictions.
                  "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
                    "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
                    "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
                  },
                  "topK": 3.14, # Optional. If specified, top-k sampling will be used.
                  "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
                },
                "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
              },
              "metricPromptTemplate": "A String", # Optional. Template for the prompt used by the judge model to evaluate against rubrics.
              "rubricGenerationSpec": { # Specification for how rubrics should be generated. # Dynamically generate rubrics for evaluation using this specification.
                "modelConfig": { # The autorater config used for the evaluation run. # Optional. Configuration for the model used in rubric generation. Configs including sampling count and base model can be specified here. Flipping is not supported for rubric generation.
                  "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
                  "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
                    "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
                    "candidateCount": 42, # Optional. Number of candidates to generate.
                    "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
                    "frequencyPenalty": 3.14, # Optional. Frequency penalties.
                    "logprobs": 42, # Optional. Logit probabilities.
                    "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
                    "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
                    "modelConfig": { # Config for model selection. # Optional. Config for model selection.
                      "featureSelectionPreference": "A String", # Required. Feature selection preference.
                    },
                    "presencePenalty": 3.14, # Optional. Positive penalties.
                    "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
                    "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
                    "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
                    "responseModalities": [ # Optional. The modalities of the response.
                      "A String",
                    ],
                    "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
                      "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
                      "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
                        # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                      ],
                      "default": "", # Optional. Default value of the data.
                      "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
                        "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                      },
                      "description": "A String", # Optional. The description of the data.
                      "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
                        "A String",
                      ],
                      "example": "", # Optional. Example of the object. Will only populated when the object is the root.
                      "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
                      "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
                      "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
                      "maxLength": "A String", # Optional. Maximum length of the Type.STRING
                      "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
                      "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
                      "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
                      "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
                      "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
                      "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
                      "nullable": True or False, # Optional. Indicates if the value may be null.
                      "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
                      "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
                        "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                      },
                      "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
                        "A String",
                      ],
                      "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
                      "required": [ # Optional. Required properties of Type.OBJECT.
                        "A String",
                      ],
                      "title": "A String", # Optional. The title of the Schema.
                      "type": "A String", # Optional. The type of the data.
                    },
                    "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
                      "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
                        "modelRoutingPreference": "A String", # The model routing preference.
                      },
                      "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
                        "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
                      },
                    },
                    "seed": 42, # Optional. Seed.
                    "speechConfig": { # The speech generation config. # Optional. The speech generation config.
                      "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
                      "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
                        "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
                          "voiceName": "A String", # The name of the preset voice to use.
                        },
                      },
                    },
                    "stopSequences": [ # Optional. Stop sequences.
                      "A String",
                    ],
                    "temperature": 3.14, # Optional. Controls the randomness of predictions.
                    "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
                      "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
                      "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
                    },
                    "topK": 3.14, # Optional. If specified, top-k sampling will be used.
                    "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
                  },
                  "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
                },
                "promptTemplate": "A String", # Optional. Template for the prompt used to generate rubrics. The details should be updated based on the most-recent recipe requirements.
                "rubricContentType": "A String", # Optional. The type of rubric content to be generated.
                "rubricTypeOntology": [ # Optional. An optional, pre-defined list of allowed types for generated rubrics. If this field is provided, it implies `include_rubric_type` should be true, and the generated rubric types should be chosen from this ontology.
                  "A String",
                ],
              },
              "rubricGroupKey": "A String", # Use a pre-defined group of rubrics associated with the input content. This refers to a key in the `rubric_groups` map of `RubricEnhancedContents`.
            },
          },
        ],
        "outputConfig": { # The output config for the evaluation run. # Optional. The output config for the evaluation run.
          "bigqueryDestination": { # The BigQuery location for the output content. # BigQuery destination for evaluation output.
            "outputUri": "A String", # Required. BigQuery URI to a project or table, up to 2000 characters long. When only the project is specified, the Dataset and Table is created. When the full table reference is specified, the Dataset must exist and table must not exist. Accepted forms: * BigQuery path. For example: `bq://projectId` or `bq://projectId.bqDatasetId` or `bq://projectId.bqDatasetId.bqTableId`.
          },
          "gcsDestination": { # The Google Cloud Storage location where the output is to be written to. # Cloud Storage destination for evaluation output.
            "outputUriPrefix": "A String", # Required. Google Cloud Storage URI to output directory. If the uri doesn't end with '/', a '/' will be automatically appended. The directory is created if it doesn't exist.
          },
        },
        "promptTemplate": { # Prompt template used for inference. # The prompt template used for inference. The values for variables in the prompt template are defined in EvaluationItem.EvaluationPrompt.PromptTemplateData.values.
          "gcsUri": "A String", # Prompt template stored in Cloud Storage. Format: "gs://my-bucket/file-name.txt".
          "promptTemplate": "A String", # Inline prompt template. Template variables should be in the format "{var_name}". Example: "Translate the following from {source_lang} to {target_lang}: {text}"
        },
        "rubricConfigs": [ # Optional. The rubric configs for the evaluation run. They are used to generate rubrics which can be used by rubric-based metrics. Multiple rubric configs can be specified for rubric generation but only one rubric config can be used for a rubric-based metric. If more than one rubric config is provided, the evaluation metric must specify a rubric group key. Note that if a generation spec is specified on both a rubric config and an evaluation metric, the rubrics generated for the metric will be used for evaluation.
          { # Configuration for a rubric group to be generated/saved for evaluation.
            "predefinedRubricGenerationSpec": { # Specification for a pre-defined metric. # Dynamically generate rubrics using a predefined spec.
              "metricSpecName": "A String", # Required. The name of a pre-defined metric, such as "instruction_following_v1" or "text_quality_v1".
              "parameters": { # Optional. The parameters needed to run the pre-defined metric.
                "a_key": "", # Properties of the object.
              },
            },
            "rubricGenerationSpec": { # Specification for how rubrics should be generated. # Dynamically generate rubrics using this specification.
              "modelConfig": { # The autorater config used for the evaluation run. # Optional. Configuration for the model used in rubric generation. Configs including sampling count and base model can be specified here. Flipping is not supported for rubric generation.
                "autoraterModel": "A String", # Optional. The fully qualified name of the publisher model or tuned autorater endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Tuned model endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
                "generationConfig": { # Generation config. # Optional. Configuration options for model generation and outputs.
                  "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
                  "candidateCount": 42, # Optional. Number of candidates to generate.
                  "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
                  "frequencyPenalty": 3.14, # Optional. Frequency penalties.
                  "logprobs": 42, # Optional. Logit probabilities.
                  "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
                  "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
                  "modelConfig": { # Config for model selection. # Optional. Config for model selection.
                    "featureSelectionPreference": "A String", # Required. Feature selection preference.
                  },
                  "presencePenalty": 3.14, # Optional. Positive penalties.
                  "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
                  "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
                  "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
                  "responseModalities": [ # Optional. The modalities of the response.
                    "A String",
                  ],
                  "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
                    "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
                    "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
                      # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                    ],
                    "default": "", # Optional. Default value of the data.
                    "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
                      "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                    },
                    "description": "A String", # Optional. The description of the data.
                    "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
                      "A String",
                    ],
                    "example": "", # Optional. Example of the object. Will only populated when the object is the root.
                    "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
                    "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
                    "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
                    "maxLength": "A String", # Optional. Maximum length of the Type.STRING
                    "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
                    "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
                    "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
                    "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
                    "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
                    "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
                    "nullable": True or False, # Optional. Indicates if the value may be null.
                    "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
                    "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
                      "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
                    },
                    "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
                      "A String",
                    ],
                    "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
                    "required": [ # Optional. Required properties of Type.OBJECT.
                      "A String",
                    ],
                    "title": "A String", # Optional. The title of the Schema.
                    "type": "A String", # Optional. The type of the data.
                  },
                  "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
                    "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
                      "modelRoutingPreference": "A String", # The model routing preference.
                    },
                    "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
                      "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
                    },
                  },
                  "seed": 42, # Optional. Seed.
                  "speechConfig": { # The speech generation config. # Optional. The speech generation config.
                    "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
                    "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
                      "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
                        "voiceName": "A String", # The name of the preset voice to use.
                      },
                    },
                  },
                  "stopSequences": [ # Optional. Stop sequences.
                    "A String",
                  ],
                  "temperature": 3.14, # Optional. Controls the randomness of predictions.
                  "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
                    "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
                    "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
                  },
                  "topK": 3.14, # Optional. If specified, top-k sampling will be used.
                  "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
                },
                "sampleCount": 42, # Optional. Number of samples for each instance in the dataset. If not specified, the default is 4. Minimum value is 1, maximum value is 32.
              },
              "promptTemplate": "A String", # Optional. Template for the prompt used to generate rubrics. The details should be updated based on the most-recent recipe requirements.
              "rubricContentType": "A String", # Optional. The type of rubric content to be generated.
              "rubricTypeOntology": [ # Optional. An optional, pre-defined list of allowed types for generated rubrics. If this field is provided, it implies `include_rubric_type` should be true, and the generated rubric types should be chosen from this ontology.
                "A String",
              ],
            },
            "rubricGroupKey": "A String", # Required. The key used to save the generated rubrics. If a generation spec is provided, this key will be used for the name of the generated rubric group. Otherwise, this key will be used to look up the existing rubric group on the evaluation item. Note that if a rubric group key is specified on both a rubric config and an evaluation metric, the key from the metric will be used to select the rubrics for evaluation.
          },
        ],
      },
      "evaluationResults": { # The results of the evaluation run. # Output only. The results of the evaluation run. Only populated when the evaluation run's state is SUCCEEDED.
        "evaluationSet": "A String", # The evaluation set where item level results are stored.
        "summaryMetrics": { # The summary metrics for the evaluation run. # Optional. The summary metrics for the evaluation run.
          "failedItems": 42, # Optional. The number of items that failed to be evaluated.
          "metrics": { # Optional. Map of metric name to metric value.
            "a_key": "",
          },
          "totalItems": 42, # Optional. The total number of items that were evaluated.
        },
      },
      "evaluationSetSnapshot": "A String", # Output only. The specific evaluation set of the evaluation run. For runs with an evaluation set input, this will be that same set. For runs with BigQuery input, it's the sampled BigQuery dataset.
      "inferenceConfigs": { # Optional. The candidate to inference config map for the evaluation run. The candidate can be up to 128 characters long and can consist of any UTF-8 characters.
        "a_key": { # An inference config used for model inference during the evaluation run.
          "generationConfig": { # Generation config. # Optional. Generation config.
            "audioTimestamp": True or False, # Optional. If enabled, audio timestamp will be included in the request to the model.
            "candidateCount": 42, # Optional. Number of candidates to generate.
            "enableAffectiveDialog": True or False, # Optional. If enabled, the model will detect emotions and adapt its responses accordingly.
            "frequencyPenalty": 3.14, # Optional. Frequency penalties.
            "logprobs": 42, # Optional. Logit probabilities.
            "maxOutputTokens": 42, # Optional. The maximum number of output tokens to generate per message.
            "mediaResolution": "A String", # Optional. If specified, the media resolution specified will be used.
            "modelConfig": { # Config for model selection. # Optional. Config for model selection.
              "featureSelectionPreference": "A String", # Required. Feature selection preference.
            },
            "presencePenalty": 3.14, # Optional. Positive penalties.
            "responseJsonSchema": "", # Optional. Output schema of the generated response. This is an alternative to `response_schema` that accepts [JSON Schema](https://json-schema.org/). If set, `response_schema` must be omitted, but `response_mime_type` is required. While the full JSON Schema may be sent, not all features are supported. Specifically, only the following properties are supported: - `$id` - `$defs` - `$ref` - `$anchor` - `type` - `format` - `title` - `description` - `enum` (for strings and numbers) - `items` - `prefixItems` - `minItems` - `maxItems` - `minimum` - `maximum` - `anyOf` - `oneOf` (interpreted the same as `anyOf`) - `properties` - `additionalProperties` - `required` The non-standard `propertyOrdering` property may also be set. Cyclic references are unrolled to a limited degree and, as such, may only be used within non-required properties. (Nullable properties are not sufficient.) If `$ref` is set on a sub-schema, no other properties, except for than those starting as a `$`, may be set.
            "responseLogprobs": True or False, # Optional. If true, export the logprobs results in response.
            "responseMimeType": "A String", # Optional. Output response mimetype of the generated candidate text. Supported mimetype: - `text/plain`: (default) Text output. - `application/json`: JSON response in the candidates. The model needs to be prompted to output the appropriate response type, otherwise the behavior is undefined. This is a preview feature.
            "responseModalities": [ # Optional. The modalities of the response.
              "A String",
            ],
            "responseSchema": { # Schema is used to define the format of input/output data. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema-object). More fields may be added in the future as needed. # Optional. The `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema). If set, a compatible response_mime_type must also be set. Compatible mimetypes: `application/json`: Schema for JSON response.
              "additionalProperties": "", # Optional. Can either be a boolean or an object; controls the presence of additional properties.
              "anyOf": [ # Optional. The value should be validated against any (one or more) of the subschemas in the list.
                # Object with schema name: GoogleCloudAiplatformV1beta1Schema
              ],
              "default": "", # Optional. Default value of the data.
              "defs": { # Optional. A map of definitions for use by `ref` Only allowed at the root of the schema.
                "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
              },
              "description": "A String", # Optional. The description of the data.
              "enum": [ # Optional. Possible values of the element of primitive type with enum format. Examples: 1. We can define direction as : {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]} 2. We can define apartment number as : {type:INTEGER, format:enum, enum:["101", "201", "301"]}
                "A String",
              ],
              "example": "", # Optional. Example of the object. Will only populated when the object is the root.
              "format": "A String", # Optional. The format of the data. Supported formats: for NUMBER type: "float", "double" for INTEGER type: "int32", "int64" for STRING type: "email", "byte", etc
              "items": # Object with schema name: GoogleCloudAiplatformV1beta1Schema # Optional. SCHEMA FIELDS FOR TYPE ARRAY Schema of the elements of Type.ARRAY.
              "maxItems": "A String", # Optional. Maximum number of the elements for Type.ARRAY.
              "maxLength": "A String", # Optional. Maximum length of the Type.STRING
              "maxProperties": "A String", # Optional. Maximum number of the properties for Type.OBJECT.
              "maximum": 3.14, # Optional. Maximum value of the Type.INTEGER and Type.NUMBER
              "minItems": "A String", # Optional. Minimum number of the elements for Type.ARRAY.
              "minLength": "A String", # Optional. SCHEMA FIELDS FOR TYPE STRING Minimum length of the Type.STRING
              "minProperties": "A String", # Optional. Minimum number of the properties for Type.OBJECT.
              "minimum": 3.14, # Optional. SCHEMA FIELDS FOR TYPE INTEGER and NUMBER Minimum value of the Type.INTEGER and Type.NUMBER
              "nullable": True or False, # Optional. Indicates if the value may be null.
              "pattern": "A String", # Optional. Pattern of the Type.STRING to restrict a string to a regular expression.
              "properties": { # Optional. SCHEMA FIELDS FOR TYPE OBJECT Properties of Type.OBJECT.
                "a_key": # Object with schema name: GoogleCloudAiplatformV1beta1Schema
              },
              "propertyOrdering": [ # Optional. The order of the properties. Not a standard field in open api spec. Only used to support the order of the properties.
                "A String",
              ],
              "ref": "A String", # Optional. Allows indirect references between schema nodes. The value should be a valid reference to a child of the root `defs`. For example, the following schema defines a reference to a schema node named "Pet": type: object properties: pet: ref: #/defs/Pet defs: Pet: type: object properties: name: type: string The value of the "pet" property is a reference to the schema node named "Pet". See details in https://json-schema.org/understanding-json-schema/structuring
              "required": [ # Optional. Required properties of Type.OBJECT.
                "A String",
              ],
              "title": "A String", # Optional. The title of the Schema.
              "type": "A String", # Optional. The type of the data.
            },
            "routingConfig": { # The configuration for routing the request to a specific model. # Optional. Routing configuration.
              "autoMode": { # When automated routing is specified, the routing will be determined by the pretrained routing model and customer provided model routing preference. # Automated routing.
                "modelRoutingPreference": "A String", # The model routing preference.
              },
              "manualMode": { # When manual routing is set, the specified model will be used directly. # Manual routing.
                "modelName": "A String", # The model name to use. Only the public LLM models are accepted. See [Supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#supported-models).
              },
            },
            "seed": 42, # Optional. Seed.
            "speechConfig": { # The speech generation config. # Optional. The speech generation config.
              "languageCode": "A String", # Optional. Language code (ISO 639. e.g. en-US) for the speech synthesization.
              "voiceConfig": { # The configuration for the voice to use. # The configuration for the speaker to use.
                "prebuiltVoiceConfig": { # The configuration for the prebuilt speaker to use. # The configuration for the prebuilt voice to use.
                  "voiceName": "A String", # The name of the preset voice to use.
                },
              },
            },
            "stopSequences": [ # Optional. Stop sequences.
              "A String",
            ],
            "temperature": 3.14, # Optional. Controls the randomness of predictions.
            "thinkingConfig": { # Config for thinking features. # Optional. Config for thinking features. An error will be returned if this field is set for models that don't support thinking.
              "includeThoughts": True or False, # Optional. Indicates whether to include thoughts in the response. If true, thoughts are returned only when available.
              "thinkingBudget": 42, # Optional. Indicates the thinking budget in tokens.
            },
            "topK": 3.14, # Optional. If specified, top-k sampling will be used.
            "topP": 3.14, # Optional. If specified, nucleus sampling will be used.
          },
          "model": "A String", # Required. The fully qualified name of the publisher model or endpoint to use. Publisher model format: `projects/{project}/locations/{location}/publishers/*/models/*` Endpoint format: `projects/{project}/locations/{location}/endpoints/{endpoint}`
        },
      },
      "labels": { # Optional. Labels for the evaluation run.
        "a_key": "A String",
      },
      "metadata": "", # Optional. Metadata about the evaluation run, can be used by the caller to store additional tracking information about the evaluation run.
      "name": "A String", # Identifier. The resource name of the EvaluationRun. This is a unique identifier. Format: `projects/{project}/locations/{location}/evaluationRuns/{evaluation_run}`
      "state": "A String", # Output only. The state of the evaluation run.
    },
  ],
  "nextPageToken": "A String", # A token to retrieve the next page of results.
}

list_next()

Retrieves the next page of results.

        Args:
          previous_request: The request for the previous page. (required)
          previous_response: The response from the request for the previous page. (required)

        Returns:
          A request object that you can call 'execute()' on to request the next
          page. Returns None if there are no more items in the collection.