Close httplib2 connections.
Deletes an evaluation run.
Gets details of the specified evaluation run.
list(parent, filter=None, orderBy=None, pageSize=None, pageToken=None, x__xgafv=None)
Lists all evaluation runs in the given app.
Retrieves the next page of results.
close()
Close httplib2 connections.
delete(name, x__xgafv=None)
Deletes an evaluation run.
Args:
name: string, Required. The resource name of the evaluation run to delete. (required)
x__xgafv: string, V1 error format.
Allowed values
1 - v1 error format
2 - v2 error format
Returns:
An object of the form:
{ # This resource represents a long-running operation that is the result of a network API call.
"done": True or False, # If the value is `false`, it means the operation is still in progress. If `true`, the operation is completed, and either `error` or `response` is available.
"error": { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors). # The error result of the operation in case of failure or cancellation.
"code": 42, # The status code, which should be an enum value of google.rpc.Code.
"details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
{
"a_key": "", # Properties of the object. Contains field @type with type URL.
},
],
"message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
},
"metadata": { # Service-specific metadata associated with the operation. It typically contains progress information and common metadata such as create time. Some services might not provide such metadata. Any method that returns a long-running operation should document the metadata type, if any.
"a_key": "", # Properties of the object. Contains field @type with type URL.
},
"name": "A String", # The server-assigned name, which is only unique within the same service that originally returns it. If you use the default HTTP mapping, the `name` should be a resource name ending with `operations/{unique_id}`.
"response": { # The normal, successful response of the operation. If the original method returns no data on success, such as `Delete`, the response is `google.protobuf.Empty`. If the original method is standard `Get`/`Create`/`Update`, the response should be the resource. For other methods, the response should have the type `XxxResponse`, where `Xxx` is the original method name. For example, if the original method name is `TakeSnapshot()`, the inferred response type is `TakeSnapshotResponse`.
"a_key": "", # Properties of the object. Contains field @type with type URL.
},
}
get(name, x__xgafv=None)
Gets details of the specified evaluation run.
Args:
name: string, Required. The resource name of the evaluation run to retrieve. (required)
x__xgafv: string, V1 error format.
Allowed values
1 - v1 error format
2 - v2 error format
Returns:
An object of the form:
{ # An evaluation run represents an all the evaluation results from an evaluation execution.
"appVersion": "A String", # Output only. The app version to evaluate. Format: `projects/{project}/locations/{location}/apps/{app}/versions/{version}`
"appVersionDisplayName": "A String", # Output only. The display name of the `app_version` that the evaluation ran against.
"changelog": "A String", # Output only. The changelog of the app version that the evaluation ran against. This is populated if user runs evaluation on latest/draft.
"changelogCreateTime": "A String", # Output only. The create time of the changelog of the app version that the evaluation ran against. This is populated if user runs evaluation on latest/draft.
"config": { # EvaluationConfig configures settings for running the evaluation. # Output only. The configuration used in the run.
"evaluationChannel": "A String", # Optional. The channel to evaluate.
"inputAudioConfig": { # InputAudioConfig configures how the CES agent should interpret the incoming audio data. # Optional. Configuration for processing the input audio.
"audioEncoding": "A String", # Required. The encoding of the input audio data.
"noiseSuppressionLevel": "A String", # Optional. Whether to enable noise suppression on the input audio. Available values are "low", "moderate", "high", "very_high".
"sampleRateHertz": 42, # Required. The sample rate (in Hertz) of the input audio data.
},
"outputAudioConfig": { # OutputAudioConfig configures how the CES agent should synthesize outgoing audio responses. # Optional. Configuration for generating the output audio.
"audioEncoding": "A String", # Required. The encoding of the output audio data.
"sampleRateHertz": 42, # Required. The sample rate (in Hertz) of the output audio data.
},
"toolCallBehaviour": "A String", # Optional. Specifies whether the evaluation should use real tool calls or fake tools.
},
"createTime": "A String", # Output only. Timestamp when the evaluation run was created.
"displayName": "A String", # Optional. User-defined display name of the evaluation run. default: " run - ".
"error": { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors). # Output only. Deprecated: Use error_info instead. Errors encountered during execution.
"code": 42, # The status code, which should be an enum value of google.rpc.Code.
"details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
{
"a_key": "", # Properties of the object. Contains field @type with type URL.
},
],
"message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
},
"errorInfo": { # Information about an error encountered during an evaluation execution. # Output only. Error information for the evaluation run.
"errorMessage": "A String", # Output only. The error message.
"errorType": "A String", # Output only. The type of error.
"sessionId": "A String", # Output only. The session ID for the conversation that caused the error.
},
"evaluationDataset": "A String", # Output only. The evaluation dataset that this run is associated with. This field is mutually exclusive with `evaluations`. Format: `projects/{project}/locations/{location}/apps/{app}/evaluationDatasets/{evaluationDataset}`
"evaluationResults": [ # Output only. The evaluation results that are part of this run. Format: `projects/{project}/locations/{location}/apps/{app}/evaluations/{evaluation}/results/{result}`
"A String",
],
"evaluationRunSummaries": { # Output only. Map of evaluation name to EvaluationRunSummary.
"a_key": { # Contains the summary of passed and failed result counts for a specific evaluation in an evaluation run.
"errorCount": 42, # Output only. Number of error results for the associated Evaluation in this run.
"failedCount": 42, # Output only. Number of failed results for the associated Evaluation in this run.
"passedCount": 42, # Output only. Number of passed results for the associated Evaluation in this run.
},
},
"evaluationType": "A String", # Output only. The type of the evaluations in this run.
"evaluations": [ # Output only. The evaluations that are part of this run. The list may contain evaluations of either type. This field is mutually exclusive with `evaluation_dataset`. Format: `projects/{project}/locations/{location}/apps/{app}/evaluations/{evaluation}`
"A String",
],
"goldenRunMethod": "A String", # Output only. The method used to run the evaluation.
"initiatedBy": "A String", # Output only. The user who initiated the evaluation run.
"latencyReport": { # Latency report for the evaluation run. # Output only. Latency report for the evaluation run.
"callbackLatencies": [ # Output only. Unordered list. Latency metrics for each callback.
{ # Latency metrics for a single callback.
"latencyMetrics": { # Latency metrics for a component. # Output only. The latency metrics for the callback.
"callCount": 42, # Output only. The number of times the resource was called.
"p50Latency": "A String", # Output only. The 50th percentile latency.
"p90Latency": "A String", # Output only. The 90th percentile latency.
"p99Latency": "A String", # Output only. The 99th percentile latency.
},
"stage": "A String", # Output only. The stage of the callback.
},
],
"guardrailLatencies": [ # Output only. Unordered list. Latency metrics for each guardrail.
{ # Latency metrics for a single guardrail.
"guardrail": "A String", # Output only. The name of the guardrail. Format: `projects/{project}/locations/{location}/apps/{app}/guardrails/{guardrail}`.
"guardrailDisplayName": "A String", # Output only. The display name of the guardrail.
"latencyMetrics": { # Latency metrics for a component. # Output only. The latency metrics for the guardrail.
"callCount": 42, # Output only. The number of times the resource was called.
"p50Latency": "A String", # Output only. The 50th percentile latency.
"p90Latency": "A String", # Output only. The 90th percentile latency.
"p99Latency": "A String", # Output only. The 99th percentile latency.
},
},
],
"llmCallLatencies": [ # Output only. Unordered list. Latency metrics for each LLM call.
{ # Latency metrics for a single LLM call.
"latencyMetrics": { # Latency metrics for a component. # Output only. The latency metrics for the LLM call.
"callCount": 42, # Output only. The number of times the resource was called.
"p50Latency": "A String", # Output only. The 50th percentile latency.
"p90Latency": "A String", # Output only. The 90th percentile latency.
"p99Latency": "A String", # Output only. The 99th percentile latency.
},
"model": "A String", # Output only. The name of the model.
},
],
"sessionCount": 42, # Output only. The total number of sessions considered in the latency report.
"toolLatencies": [ # Output only. Unordered list. Latency metrics for each tool.
{ # Latency metrics for a single tool.
"latencyMetrics": { # Latency metrics for a component. # Output only. The latency metrics for the tool.
"callCount": 42, # Output only. The number of times the resource was called.
"p50Latency": "A String", # Output only. The 50th percentile latency.
"p90Latency": "A String", # Output only. The 90th percentile latency.
"p99Latency": "A String", # Output only. The 99th percentile latency.
},
"tool": "A String", # Output only. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}`.
"toolDisplayName": "A String", # Output only. The display name of the tool.
"toolsetTool": { # A tool that is created from a toolset. # Output only. The toolset tool identifier.
"toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for.
"toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}`
},
},
],
},
"name": "A String", # Identifier. The unique identifier of the evaluation run. Format: `projects/{project}/locations/{location}/apps/{app}/evaluationRuns/{evaluationRun}`
"optimizationConfig": { # Configuration for running the optimization step after the evaluation run. # Optional. Configuration for running the optimization step after the evaluation run. If not set, the optimization step will not be run.
"assistantSession": "A String", # Output only. The assistant session to use for the optimization based on this evaluation run. Format: `projects/{project}/locations/{location}/apps/{app}/assistantSessions/{assistantSession}`
"errorMessage": "A String", # Output only. The error message if the optimization run failed.
"generateLossReport": True or False, # Optional. Whether to generate a loss report.
"lossReport": { # Output only. The generated loss report.
"a_key": "", # Properties of the object.
},
"reportSummary": "A String", # Output only. The summary of the loss report.
"shouldSuggestFix": True or False, # Output only. Whether to suggest a fix for the losses.
"status": "A String", # Output only. The status of the optimization run.
},
"personaRunConfigs": [ # Output only. The configuration to use for the run per persona.
{ # Configuration for running an evaluation for a specific persona.
"persona": "A String", # Optional. The persona to use for the evaluation. Format: `projects/{project}/locations/{location}/apps/{app}/evaluationPersonas/{evaluationPersona}`
"taskCount": 42, # Optional. The number of tasks to run for the persona.
},
],
"progress": { # The progress of the evaluation run. # Output only. The progress of the evaluation run.
"completedCount": 42, # Output only. Number of evaluation results that finished successfully. (EvaluationResult.execution_state is COMPLETED).
"errorCount": 42, # Output only. Number of evaluation results that failed to execute. (EvaluationResult.execution_state is ERROR).
"failedCount": 42, # Output only. Number of completed evaluation results with an outcome of FAIL. (EvaluationResult.execution_state is COMPLETED and EvaluationResult.evaluation_status is FAIL).
"passedCount": 42, # Output only. Number of completed evaluation results with an outcome of PASS. (EvaluationResult.execution_state is COMPLETED and EvaluationResult.evaluation_status is PASS).
"totalCount": 42, # Output only. Total number of evaluation results in this run.
},
"runCount": 42, # Output only. The number of times the evaluations inside the run were run.
"scheduledEvaluationRun": "A String", # Output only. The scheduled evaluation run resource name that created this evaluation run. This field is only set if the evaluation run was created by a scheduled evaluation run. Format: `projects/{project}/locations/{location}/apps/{app}/scheduledEvaluationRuns/{scheduled_evaluation_run}`
"state": "A String", # Output only. The state of the evaluation run.
}
list(parent, filter=None, orderBy=None, pageSize=None, pageToken=None, x__xgafv=None)
Lists all evaluation runs in the given app.
Args:
parent: string, Required. The resource name of the app to list evaluation runs from. (required)
filter: string, Optional. Filter to be applied when listing the evaluation runs. See https://google.aip.dev/160 for more details.
orderBy: string, Optional. Field to sort by. Only "name" and "create_time", and "update_time" are supported. Time fields are ordered in descending order, and the name field is ordered in ascending order. If not included, "update_time" will be the default. See https://google.aip.dev/132#ordering for more details.
pageSize: integer, Optional. Requested page size. Server may return fewer items than requested. If unspecified, server will pick an appropriate default.
pageToken: string, Optional. The next_page_token value returned from a previous list EvaluationService.ListEvaluationRuns call.
x__xgafv: string, V1 error format.
Allowed values
1 - v1 error format
2 - v2 error format
Returns:
An object of the form:
{ # Response message for EvaluationService.ListEvaluationRuns.
"evaluationRuns": [ # The list of evaluation runs.
{ # An evaluation run represents an all the evaluation results from an evaluation execution.
"appVersion": "A String", # Output only. The app version to evaluate. Format: `projects/{project}/locations/{location}/apps/{app}/versions/{version}`
"appVersionDisplayName": "A String", # Output only. The display name of the `app_version` that the evaluation ran against.
"changelog": "A String", # Output only. The changelog of the app version that the evaluation ran against. This is populated if user runs evaluation on latest/draft.
"changelogCreateTime": "A String", # Output only. The create time of the changelog of the app version that the evaluation ran against. This is populated if user runs evaluation on latest/draft.
"config": { # EvaluationConfig configures settings for running the evaluation. # Output only. The configuration used in the run.
"evaluationChannel": "A String", # Optional. The channel to evaluate.
"inputAudioConfig": { # InputAudioConfig configures how the CES agent should interpret the incoming audio data. # Optional. Configuration for processing the input audio.
"audioEncoding": "A String", # Required. The encoding of the input audio data.
"noiseSuppressionLevel": "A String", # Optional. Whether to enable noise suppression on the input audio. Available values are "low", "moderate", "high", "very_high".
"sampleRateHertz": 42, # Required. The sample rate (in Hertz) of the input audio data.
},
"outputAudioConfig": { # OutputAudioConfig configures how the CES agent should synthesize outgoing audio responses. # Optional. Configuration for generating the output audio.
"audioEncoding": "A String", # Required. The encoding of the output audio data.
"sampleRateHertz": 42, # Required. The sample rate (in Hertz) of the output audio data.
},
"toolCallBehaviour": "A String", # Optional. Specifies whether the evaluation should use real tool calls or fake tools.
},
"createTime": "A String", # Output only. Timestamp when the evaluation run was created.
"displayName": "A String", # Optional. User-defined display name of the evaluation run. default: " run - ".
"error": { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors). # Output only. Deprecated: Use error_info instead. Errors encountered during execution.
"code": 42, # The status code, which should be an enum value of google.rpc.Code.
"details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
{
"a_key": "", # Properties of the object. Contains field @type with type URL.
},
],
"message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
},
"errorInfo": { # Information about an error encountered during an evaluation execution. # Output only. Error information for the evaluation run.
"errorMessage": "A String", # Output only. The error message.
"errorType": "A String", # Output only. The type of error.
"sessionId": "A String", # Output only. The session ID for the conversation that caused the error.
},
"evaluationDataset": "A String", # Output only. The evaluation dataset that this run is associated with. This field is mutually exclusive with `evaluations`. Format: `projects/{project}/locations/{location}/apps/{app}/evaluationDatasets/{evaluationDataset}`
"evaluationResults": [ # Output only. The evaluation results that are part of this run. Format: `projects/{project}/locations/{location}/apps/{app}/evaluations/{evaluation}/results/{result}`
"A String",
],
"evaluationRunSummaries": { # Output only. Map of evaluation name to EvaluationRunSummary.
"a_key": { # Contains the summary of passed and failed result counts for a specific evaluation in an evaluation run.
"errorCount": 42, # Output only. Number of error results for the associated Evaluation in this run.
"failedCount": 42, # Output only. Number of failed results for the associated Evaluation in this run.
"passedCount": 42, # Output only. Number of passed results for the associated Evaluation in this run.
},
},
"evaluationType": "A String", # Output only. The type of the evaluations in this run.
"evaluations": [ # Output only. The evaluations that are part of this run. The list may contain evaluations of either type. This field is mutually exclusive with `evaluation_dataset`. Format: `projects/{project}/locations/{location}/apps/{app}/evaluations/{evaluation}`
"A String",
],
"goldenRunMethod": "A String", # Output only. The method used to run the evaluation.
"initiatedBy": "A String", # Output only. The user who initiated the evaluation run.
"latencyReport": { # Latency report for the evaluation run. # Output only. Latency report for the evaluation run.
"callbackLatencies": [ # Output only. Unordered list. Latency metrics for each callback.
{ # Latency metrics for a single callback.
"latencyMetrics": { # Latency metrics for a component. # Output only. The latency metrics for the callback.
"callCount": 42, # Output only. The number of times the resource was called.
"p50Latency": "A String", # Output only. The 50th percentile latency.
"p90Latency": "A String", # Output only. The 90th percentile latency.
"p99Latency": "A String", # Output only. The 99th percentile latency.
},
"stage": "A String", # Output only. The stage of the callback.
},
],
"guardrailLatencies": [ # Output only. Unordered list. Latency metrics for each guardrail.
{ # Latency metrics for a single guardrail.
"guardrail": "A String", # Output only. The name of the guardrail. Format: `projects/{project}/locations/{location}/apps/{app}/guardrails/{guardrail}`.
"guardrailDisplayName": "A String", # Output only. The display name of the guardrail.
"latencyMetrics": { # Latency metrics for a component. # Output only. The latency metrics for the guardrail.
"callCount": 42, # Output only. The number of times the resource was called.
"p50Latency": "A String", # Output only. The 50th percentile latency.
"p90Latency": "A String", # Output only. The 90th percentile latency.
"p99Latency": "A String", # Output only. The 99th percentile latency.
},
},
],
"llmCallLatencies": [ # Output only. Unordered list. Latency metrics for each LLM call.
{ # Latency metrics for a single LLM call.
"latencyMetrics": { # Latency metrics for a component. # Output only. The latency metrics for the LLM call.
"callCount": 42, # Output only. The number of times the resource was called.
"p50Latency": "A String", # Output only. The 50th percentile latency.
"p90Latency": "A String", # Output only. The 90th percentile latency.
"p99Latency": "A String", # Output only. The 99th percentile latency.
},
"model": "A String", # Output only. The name of the model.
},
],
"sessionCount": 42, # Output only. The total number of sessions considered in the latency report.
"toolLatencies": [ # Output only. Unordered list. Latency metrics for each tool.
{ # Latency metrics for a single tool.
"latencyMetrics": { # Latency metrics for a component. # Output only. The latency metrics for the tool.
"callCount": 42, # Output only. The number of times the resource was called.
"p50Latency": "A String", # Output only. The 50th percentile latency.
"p90Latency": "A String", # Output only. The 90th percentile latency.
"p99Latency": "A String", # Output only. The 99th percentile latency.
},
"tool": "A String", # Output only. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}`.
"toolDisplayName": "A String", # Output only. The display name of the tool.
"toolsetTool": { # A tool that is created from a toolset. # Output only. The toolset tool identifier.
"toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for.
"toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}`
},
},
],
},
"name": "A String", # Identifier. The unique identifier of the evaluation run. Format: `projects/{project}/locations/{location}/apps/{app}/evaluationRuns/{evaluationRun}`
"optimizationConfig": { # Configuration for running the optimization step after the evaluation run. # Optional. Configuration for running the optimization step after the evaluation run. If not set, the optimization step will not be run.
"assistantSession": "A String", # Output only. The assistant session to use for the optimization based on this evaluation run. Format: `projects/{project}/locations/{location}/apps/{app}/assistantSessions/{assistantSession}`
"errorMessage": "A String", # Output only. The error message if the optimization run failed.
"generateLossReport": True or False, # Optional. Whether to generate a loss report.
"lossReport": { # Output only. The generated loss report.
"a_key": "", # Properties of the object.
},
"reportSummary": "A String", # Output only. The summary of the loss report.
"shouldSuggestFix": True or False, # Output only. Whether to suggest a fix for the losses.
"status": "A String", # Output only. The status of the optimization run.
},
"personaRunConfigs": [ # Output only. The configuration to use for the run per persona.
{ # Configuration for running an evaluation for a specific persona.
"persona": "A String", # Optional. The persona to use for the evaluation. Format: `projects/{project}/locations/{location}/apps/{app}/evaluationPersonas/{evaluationPersona}`
"taskCount": 42, # Optional. The number of tasks to run for the persona.
},
],
"progress": { # The progress of the evaluation run. # Output only. The progress of the evaluation run.
"completedCount": 42, # Output only. Number of evaluation results that finished successfully. (EvaluationResult.execution_state is COMPLETED).
"errorCount": 42, # Output only. Number of evaluation results that failed to execute. (EvaluationResult.execution_state is ERROR).
"failedCount": 42, # Output only. Number of completed evaluation results with an outcome of FAIL. (EvaluationResult.execution_state is COMPLETED and EvaluationResult.evaluation_status is FAIL).
"passedCount": 42, # Output only. Number of completed evaluation results with an outcome of PASS. (EvaluationResult.execution_state is COMPLETED and EvaluationResult.evaluation_status is PASS).
"totalCount": 42, # Output only. Total number of evaluation results in this run.
},
"runCount": 42, # Output only. The number of times the evaluations inside the run were run.
"scheduledEvaluationRun": "A String", # Output only. The scheduled evaluation run resource name that created this evaluation run. This field is only set if the evaluation run was created by a scheduled evaluation run. Format: `projects/{project}/locations/{location}/apps/{app}/scheduledEvaluationRuns/{scheduled_evaluation_run}`
"state": "A String", # Output only. The state of the evaluation run.
},
],
"nextPageToken": "A String", # A token that can be sent as ListEvaluationRunsRequest.page_token to retrieve the next page. Absence of this field indicates there are no subsequent pages.
}
list_next()
Retrieves the next page of results.
Args:
previous_request: The request for the previous page. (required)
previous_response: The response from the request for the previous page. (required)
Returns:
A request object that you can call 'execute()' on to request the next
page. Returns None if there are no more items in the collection.