Close httplib2 connections.
create(parent, body=None, x__xgafv=None)
Creates evaluation of a generator.
Deletes an evaluation of generator.
Gets an evaluation of generator.
list(parent, pageSize=None, pageToken=None, x__xgafv=None)
Lists evaluations of generator.
Retrieves the next page of results.
close()
Close httplib2 connections.
create(parent, body=None, x__xgafv=None)
Creates evaluation of a generator.
Args:
parent: string, Required. The generator resource name. Format: `projects//locations//generators/` (required)
body: object, The request body.
The object takes the form of:
{ # Represents evaluation result of a generator.
"completeTime": "A String", # Output only. Completion time of this generator evaluation.
"createTime": "A String", # Output only. Creation time of this generator evaluation.
"displayName": "A String", # Optional. The display name of the generator evaluation. At most 64 bytes long.
"evaluationStatus": { # A common evalaution pipeline status. # Output only. The result status of the evaluation pipeline. Provides the status information including if the evaluation is still in progress, completed or failed with certain error and user actionable message.
"done": True or False, # Output only. If the value is `false`, it means the evaluation is still in progress. If `true`, the operation is completed, and either `error` or `response` is available.
"pipelineStatus": { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors). # Output only. The error result of the evaluation in case of failure in evaluation pipeline.
"code": 42, # The status code, which should be an enum value of google.rpc.Code.
"details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
{
"a_key": "", # Properties of the object. Contains field @type with type URL.
},
],
"message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
},
},
"generatorEvaluationConfig": { # Generator evaluation input config. # Required. The configuration of the evaluation task.
"inputDataConfig": { # Input data config details # Required. The config/source of input data.
"agentAssistInputDataConfig": { # The distinctive configs for Agent Assist conversations as the conversation source. # The distinctive configs for Agent Assist conversations as the conversation source.
"endTime": "A String", # Required. The end of the time range for conversations to be evaluated. Only conversations ended at or before this timestamp will be sampled.
"startTime": "A String", # Required. The start of the time range for conversations to be evaluated. Only conversations created at or after this timestamp will be sampled.
},
"datasetInputDataConfig": { # The distinctive configs for dataset as the conversation source. # The distinctive configs for dataset as the conversation source.
"dataset": "A String", # Required. The identifier of the dataset to be evaluated. Format: `projects//locations//datasets/`.
},
"endTime": "A String", # Optional. The end timestamp to fetch conversation data.
"inputDataSourceType": "A String", # Required. The source type of input data.
"isSummaryGenerationAllowed": True or False, # Optional. Whether the summary generation is allowed when the pre-existing qualified summaries are insufficient to cover the sample size.
"sampleSize": 42, # Optional. Desired number of conversation-summary pairs to be evaluated.
"startTime": "A String", # Optional. The start timestamp to fetch conversation data.
"summaryGenerationOption": "A String", # Optional. Option to control whether summaries are generated during evaluation.
},
"outputGcsBucketPath": "A String", # Required. The output Cloud Storage bucket path to store eval files, e.g. per_summary_accuracy_score report. This path is provided by customer and files stored in it are visible to customer, no internal data should be stored in this path.
"summarizationConfig": { # Evaluation configs for summarization generator. # Evaluation configs for summarization generator.
"accuracyEvaluationVersion": "A String", # Optional. Version for summarization accuracy. This will determine the prompt and model used at backend.
"completenessEvaluationVersion": "A String", # Optional. Version for summarization completeness. This will determine the prompt and model used at backend.
"enableAccuracyEvaluation": True or False, # Optional. Enable accuracy evaluation.
"enableCompletenessEvaluation": True or False, # Optional. Enable completeness evaluation.
"evaluatorVersion": "A String", # Output only. Version for summarization evaluation.
},
},
"initialGenerator": { # LLM generator. # Required. The initial generator that was used when creating this evaluation. This is a copy of the generator read from storage when creating the evaluation.
"createTime": "A String", # Output only. Creation time of this generator.
"description": "A String", # Optional. Human readable description of the generator.
"freeFormContext": { # Free form generator context that customer can configure. # Input of free from generator to LLM.
"text": "A String", # Optional. Free form text input to LLM.
},
"inferenceParameter": { # The parameters of inference. # Optional. Inference parameters for this generator.
"maxOutputTokens": 42, # Optional. Maximum number of the output tokens for the generator.
"temperature": 3.14, # Optional. Controls the randomness of LLM predictions. Low temperature = less random. High temperature = more random. If unset (or 0), uses a default value of 0.
"topK": 42, # Optional. Top-k changes how the model selects tokens for output. A top-k of 1 means the selected token is the most probable among all tokens in the model's vocabulary (also called greedy decoding), while a top-k of 3 means that the next token is selected from among the 3 most probable tokens (using temperature). For each token selection step, the top K tokens with the highest probabilities are sampled. Then tokens are further filtered based on topP with the final token selected using temperature sampling. Specify a lower value for less random responses and a higher value for more random responses. Acceptable value is [1, 40], default to 40.
"topP": 3.14, # Optional. Top-p changes how the model selects tokens for output. Tokens are selected from most K (see topK parameter) probable to least until the sum of their probabilities equals the top-p value. For example, if tokens A, B, and C have a probability of 0.3, 0.2, and 0.1 and the top-p value is 0.5, then the model will select either A or B as the next token (using temperature) and doesn't consider C. The default top-p value is 0.95. Specify a lower value for less random responses and a higher value for more random responses. Acceptable value is [0.0, 1.0], default to 0.95.
},
"name": "A String", # Output only. Identifier. The resource name of the generator. Format: `projects//locations//generators/`
"publishedModel": "A String", # Optional. The published Large Language Model name. * To use the latest model version, specify the model name without version number. Example: `text-bison` * To use a stable model version, specify the version number as well. Example: `text-bison@002`.
"summarizationContext": { # Summarization context that customer can configure. # Input of Summarization feature.
"fewShotExamples": [ # Optional. List of few shot examples.
{ # Providing examples in the generator (i.e. building a few-shot generator) helps convey the desired format of the LLM response.
"conversationContext": { # Context of the conversation, including transcripts. # Optional. Conversation transcripts.
"messageEntries": [ # Optional. List of message transcripts in the conversation.
{ # Represents a message entry of a conversation.
"createTime": "A String", # Optional. Create time of the message entry.
"languageCode": "A String", # Optional. The language of the text. See [Language Support](https://cloud.google.com/dialogflow/docs/reference/language) for a list of the currently supported language codes.
"role": "A String", # Optional. Participant role of the message.
"text": "A String", # Optional. Transcript content of the message.
},
],
},
"extraInfo": { # Optional. Key is the placeholder field name in input, value is the value of the placeholder. E.g. instruction contains "@price", and ingested data has <"price", "10">
"a_key": "A String",
},
"output": { # Suggestion generated using a Generator. # Required. Example output of the model.
"freeFormSuggestion": { # Suggestion generated using free form generator. # Optional. Free form suggestion.
"response": "A String", # Required. Free form suggestion.
},
"summarySuggestion": { # Suggested summary of the conversation. # Optional. Suggested summary.
"summarySections": [ # Required. All the parts of generated summary.
{ # A component of the generated summary.
"section": "A String", # Required. Name of the section.
"summary": "A String", # Required. Summary text for the section.
},
],
},
"toolCallInfo": [ # Optional. List of request and response for tool calls executed.
{ # Request and response for a tool call.
"toolCall": { # Represents a call of a specific tool's action with the specified inputs. # Required. Request for a tool call.
"action": "A String", # Optional. The name of the tool's action associated with this call.
"createTime": "A String", # Output only. Create time of the tool call.
"inputParameters": { # Optional. The action's input parameters.
"a_key": "", # Properties of the object.
},
"tool": "A String", # Optional. The tool associated with this call. Format: `projects//locations//tools/`.
},
"toolCallResult": { # The result of calling a tool's action. # Required. Response for a tool call.
"action": "A String", # Optional. The name of the tool's action associated with this call.
"content": "A String", # Only populated if the response content is utf-8 encoded.
"createTime": "A String", # Output only. Create time of the tool call result.
"error": { # An error produced by the tool call. # The tool call's error.
"message": "A String", # Optional. The error message of the function.
},
"rawContent": "A String", # Only populated if the response content is not utf-8 encoded. (by definition byte fields are base64 encoded).
"tool": "A String", # Optional. The tool associated with this call. Format: `projects//locations//tools/`.
},
},
],
},
"summarizationSectionList": { # List of summarization sections. # Summarization sections.
"summarizationSections": [ # Optional. Summarization sections.
{ # Represents the section of summarization.
"definition": "A String", # Optional. Definition of the section, for example, "what the customer needs help with or has question about."
"key": "A String", # Optional. Name of the section, for example, "situation".
"type": "A String", # Optional. Type of the summarization section.
},
],
},
},
],
"outputLanguageCode": "A String", # Optional. The target language of the generated summary. The language code for conversation will be used if this field is empty. Supported 2.0 and later versions.
"summarizationSections": [ # Optional. List of sections. Note it contains both predefined section sand customer defined sections.
{ # Represents the section of summarization.
"definition": "A String", # Optional. Definition of the section, for example, "what the customer needs help with or has question about."
"key": "A String", # Optional. Name of the section, for example, "situation".
"type": "A String", # Optional. Type of the summarization section.
},
],
"version": "A String", # Optional. Version of the feature. If not set, default to latest version. Current candidates are ["1.0"].
},
"tools": [ # Optional. Resource names of the tools that the generator can choose from. Format: `projects//locations//tools/`.
"A String",
],
"triggerEvent": "A String", # Optional. The trigger event of the generator. It defines when the generator is triggered in a conversation.
"updateTime": "A String", # Output only. Update time of this generator.
},
"name": "A String", # Output only. Identifier. The resource name of the evaluation. Format: `projects//locations//generators// evaluations/`
"summarizationMetrics": { # Evaluation metrics for summarization generator. # Output only. Only available when the summarization generator is provided.
"conversationDetails": [ # Output only. List of conversation details.
{ # Aggregated evaluation result on conversation level. This contains evaluation results of all the metrics and sections.
"messageEntries": [ # Output only. Conversation transcript that used for summarization evaluation as a reference.
{ # Represents a message entry of a conversation.
"createTime": "A String", # Optional. Create time of the message entry.
"languageCode": "A String", # Optional. The language of the text. See [Language Support](https://cloud.google.com/dialogflow/docs/reference/language) for a list of the currently supported language codes.
"role": "A String", # Optional. Participant role of the message.
"text": "A String", # Optional. Transcript content of the message.
},
],
"metricDetails": [ # Output only. List of metric details.
{ # Aggregated result on metric level. This contains the evaluation results of all the sections.
"metric": "A String", # Output only. Metrics name. e.g. accuracy, adherence, completeness.
"score": 3.14, # Output only. Aggregated(average) score on this metric across all sections.
"sectionDetails": [ # Output only. List of section details.
{ # Section level result.
"evaluationResults": [ # Output only. List of evaluation result. The list only contains one kind of the evaluation result.
{ # Evaluation result that contains one of accuracy, adherence or completeness evaluation result.
"accuracyDecomposition": { # Decomposition details for accuracy. # Only available for accuracy metric.
"accuracyReasoning": "A String", # Output only. The accuracy reasoning of the breakdown point.
"isAccurate": True or False, # Output only. Whether the breakdown point is accurate or not.
"point": "A String", # Output only. The breakdown point of the summary.
},
"adherenceRubric": { # Rubric result of the adherence evaluation. A rubric is ued to determine if the summary adheres to all aspects of the given instructions. # Only available for adherence metric.
"isAddressed": True or False, # Output only. A boolean that indicates whether the rubric question is addressed or not.
"question": "A String", # Output only. The question generated from instruction that used to evaluate summary.
"reasoning": "A String", # Output only. The reasoning of the rubric question is addressed or not.
},
"completenessRubric": { # Rubric details of the completeness evaluation result. # Only available for completeness metric.
"isAddressed": True or False, # Output only. A boolean that indicates whether the rubric question is addressed or not.
"question": "A String", # Output only. The question generated from instruction that used to evaluate summary.
},
},
],
"score": 3.14, # Output only. Aggregated(average) score on this section across all evaluation results. Either decompositions or rubrics.
"section": "A String", # Output only. The name of the summary instruction.
"sectionSummary": "A String", # Output only. Summary for this section
},
],
},
],
"sectionTokens": [ # Output only. Conversation level token count per section. This is an aggregated(sum) result of input token of summary acorss all metrics for a single conversation.
{ # A pair of section name and input token count of the input summary section.
"section": "A String", # Output only. The name of the summary instruction.
"tokenCount": "A String", # Output only. Token count.
},
],
"summarySections": [ # Output only. Summary sections that used for summarization evaluation as a reference.
{ # A component of the generated summary.
"section": "A String", # Required. Name of the section.
"summary": "A String", # Required. Summary text for the section.
},
],
},
],
"overallMetrics": [ # Output only. A list of aggregated(average) scores per metric section.
{ # Overall performance per metric. This is the aggregated score for each metric across all conversations that are selected for summarization evaluation.
"metric": "A String", # Output only. Metric name. e.g. accuracy, adherence, completeness.
},
],
"overallSectionTokens": [ # Output only. Overall token per section. This is an aggregated(sum) result of input token of summary acorss all conversations that are selected for summarization evaluation.
{ # A pair of section name and input token count of the input summary section.
"section": "A String", # Output only. The name of the summary instruction.
"tokenCount": "A String", # Output only. Token count.
},
],
"summarizationEvaluationMergedResultsUri": "A String", # Output only. User bucket uri for merged evaluation score and aggregation score csv.
"summarizationEvaluationResults": [ # Output only. A list of evaluation results per conversation(&summary), metric and section.
{ # Evaluation result per conversation(&summary), metric and section.
"decompositions": [ # Output only. List of decompostion details
{ # Decomposition details
"accuracyDecomposition": { # Decomposition details for accuracy. # only available for accuracy metric.
"accuracyReasoning": "A String", # Output only. The accuracy reasoning of the breakdown point.
"isAccurate": True or False, # Output only. Whether the breakdown point is accurate or not.
"point": "A String", # Output only. The breakdown point of the summary.
},
"adherenceDecomposition": { # Decomposition details for adherence. # only available for adherence metric.
"adherenceReasoning": "A String", # Output only. The adherence reasoning of the breakdown point.
"isAdherent": True or False, # Output only. Whether the breakdown point is adherent or not.
"point": "A String", # Output only. The breakdown point of the given instructions.
},
},
],
"evaluationResults": [ # Output only. List of evaluation results.
{ # Evaluation result that contains one of accuracy, adherence or completeness evaluation result.
"accuracyDecomposition": { # Decomposition details for accuracy. # Only available for accuracy metric.
"accuracyReasoning": "A String", # Output only. The accuracy reasoning of the breakdown point.
"isAccurate": True or False, # Output only. Whether the breakdown point is accurate or not.
"point": "A String", # Output only. The breakdown point of the summary.
},
"adherenceRubric": { # Rubric result of the adherence evaluation. A rubric is ued to determine if the summary adheres to all aspects of the given instructions. # Only available for adherence metric.
"isAddressed": True or False, # Output only. A boolean that indicates whether the rubric question is addressed or not.
"question": "A String", # Output only. The question generated from instruction that used to evaluate summary.
"reasoning": "A String", # Output only. The reasoning of the rubric question is addressed or not.
},
"completenessRubric": { # Rubric details of the completeness evaluation result. # Only available for completeness metric.
"isAddressed": True or False, # Output only. A boolean that indicates whether the rubric question is addressed or not.
"question": "A String", # Output only. The question generated from instruction that used to evaluate summary.
},
},
],
"metric": "A String", # Output only. metric name, e.g. accuracy, completeness, adherence, etc.
"score": 3.14, # Output only. score calculated from decompositions
"section": "A String", # Output only. section/task name, e.g. action, situation, etc
"sectionSummary": "A String", # Output only. Summary of this section
"sessionId": "A String", # Output only. conversation session id
},
],
},
}
x__xgafv: string, V1 error format.
Allowed values
1 - v1 error format
2 - v2 error format
Returns:
An object of the form:
{ # This resource represents a long-running operation that is the result of a network API call.
"done": True or False, # If the value is `false`, it means the operation is still in progress. If `true`, the operation is completed, and either `error` or `response` is available.
"error": { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors). # The error result of the operation in case of failure or cancellation.
"code": 42, # The status code, which should be an enum value of google.rpc.Code.
"details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
{
"a_key": "", # Properties of the object. Contains field @type with type URL.
},
],
"message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
},
"metadata": { # Service-specific metadata associated with the operation. It typically contains progress information and common metadata such as create time. Some services might not provide such metadata. Any method that returns a long-running operation should document the metadata type, if any.
"a_key": "", # Properties of the object. Contains field @type with type URL.
},
"name": "A String", # The server-assigned name, which is only unique within the same service that originally returns it. If you use the default HTTP mapping, the `name` should be a resource name ending with `operations/{unique_id}`.
"response": { # The normal, successful response of the operation. If the original method returns no data on success, such as `Delete`, the response is `google.protobuf.Empty`. If the original method is standard `Get`/`Create`/`Update`, the response should be the resource. For other methods, the response should have the type `XxxResponse`, where `Xxx` is the original method name. For example, if the original method name is `TakeSnapshot()`, the inferred response type is `TakeSnapshotResponse`.
"a_key": "", # Properties of the object. Contains field @type with type URL.
},
}
delete(name, x__xgafv=None)
Deletes an evaluation of generator.
Args:
name: string, Required. The generator evaluation resource name. Format: `projects//locations//generators// evaluations/` (required)
x__xgafv: string, V1 error format.
Allowed values
1 - v1 error format
2 - v2 error format
Returns:
An object of the form:
{ # A generic empty message that you can re-use to avoid defining duplicated empty messages in your APIs. A typical example is to use it as the request or the response type of an API method. For instance: service Foo { rpc Bar(google.protobuf.Empty) returns (google.protobuf.Empty); }
}
get(name, x__xgafv=None)
Gets an evaluation of generator.
Args:
name: string, Required. The generator evaluation resource name. Format: `projects//locations//generators//evaluations/` (required)
x__xgafv: string, V1 error format.
Allowed values
1 - v1 error format
2 - v2 error format
Returns:
An object of the form:
{ # Represents evaluation result of a generator.
"completeTime": "A String", # Output only. Completion time of this generator evaluation.
"createTime": "A String", # Output only. Creation time of this generator evaluation.
"displayName": "A String", # Optional. The display name of the generator evaluation. At most 64 bytes long.
"evaluationStatus": { # A common evalaution pipeline status. # Output only. The result status of the evaluation pipeline. Provides the status information including if the evaluation is still in progress, completed or failed with certain error and user actionable message.
"done": True or False, # Output only. If the value is `false`, it means the evaluation is still in progress. If `true`, the operation is completed, and either `error` or `response` is available.
"pipelineStatus": { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors). # Output only. The error result of the evaluation in case of failure in evaluation pipeline.
"code": 42, # The status code, which should be an enum value of google.rpc.Code.
"details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
{
"a_key": "", # Properties of the object. Contains field @type with type URL.
},
],
"message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
},
},
"generatorEvaluationConfig": { # Generator evaluation input config. # Required. The configuration of the evaluation task.
"inputDataConfig": { # Input data config details # Required. The config/source of input data.
"agentAssistInputDataConfig": { # The distinctive configs for Agent Assist conversations as the conversation source. # The distinctive configs for Agent Assist conversations as the conversation source.
"endTime": "A String", # Required. The end of the time range for conversations to be evaluated. Only conversations ended at or before this timestamp will be sampled.
"startTime": "A String", # Required. The start of the time range for conversations to be evaluated. Only conversations created at or after this timestamp will be sampled.
},
"datasetInputDataConfig": { # The distinctive configs for dataset as the conversation source. # The distinctive configs for dataset as the conversation source.
"dataset": "A String", # Required. The identifier of the dataset to be evaluated. Format: `projects//locations//datasets/`.
},
"endTime": "A String", # Optional. The end timestamp to fetch conversation data.
"inputDataSourceType": "A String", # Required. The source type of input data.
"isSummaryGenerationAllowed": True or False, # Optional. Whether the summary generation is allowed when the pre-existing qualified summaries are insufficient to cover the sample size.
"sampleSize": 42, # Optional. Desired number of conversation-summary pairs to be evaluated.
"startTime": "A String", # Optional. The start timestamp to fetch conversation data.
"summaryGenerationOption": "A String", # Optional. Option to control whether summaries are generated during evaluation.
},
"outputGcsBucketPath": "A String", # Required. The output Cloud Storage bucket path to store eval files, e.g. per_summary_accuracy_score report. This path is provided by customer and files stored in it are visible to customer, no internal data should be stored in this path.
"summarizationConfig": { # Evaluation configs for summarization generator. # Evaluation configs for summarization generator.
"accuracyEvaluationVersion": "A String", # Optional. Version for summarization accuracy. This will determine the prompt and model used at backend.
"completenessEvaluationVersion": "A String", # Optional. Version for summarization completeness. This will determine the prompt and model used at backend.
"enableAccuracyEvaluation": True or False, # Optional. Enable accuracy evaluation.
"enableCompletenessEvaluation": True or False, # Optional. Enable completeness evaluation.
"evaluatorVersion": "A String", # Output only. Version for summarization evaluation.
},
},
"initialGenerator": { # LLM generator. # Required. The initial generator that was used when creating this evaluation. This is a copy of the generator read from storage when creating the evaluation.
"createTime": "A String", # Output only. Creation time of this generator.
"description": "A String", # Optional. Human readable description of the generator.
"freeFormContext": { # Free form generator context that customer can configure. # Input of free from generator to LLM.
"text": "A String", # Optional. Free form text input to LLM.
},
"inferenceParameter": { # The parameters of inference. # Optional. Inference parameters for this generator.
"maxOutputTokens": 42, # Optional. Maximum number of the output tokens for the generator.
"temperature": 3.14, # Optional. Controls the randomness of LLM predictions. Low temperature = less random. High temperature = more random. If unset (or 0), uses a default value of 0.
"topK": 42, # Optional. Top-k changes how the model selects tokens for output. A top-k of 1 means the selected token is the most probable among all tokens in the model's vocabulary (also called greedy decoding), while a top-k of 3 means that the next token is selected from among the 3 most probable tokens (using temperature). For each token selection step, the top K tokens with the highest probabilities are sampled. Then tokens are further filtered based on topP with the final token selected using temperature sampling. Specify a lower value for less random responses and a higher value for more random responses. Acceptable value is [1, 40], default to 40.
"topP": 3.14, # Optional. Top-p changes how the model selects tokens for output. Tokens are selected from most K (see topK parameter) probable to least until the sum of their probabilities equals the top-p value. For example, if tokens A, B, and C have a probability of 0.3, 0.2, and 0.1 and the top-p value is 0.5, then the model will select either A or B as the next token (using temperature) and doesn't consider C. The default top-p value is 0.95. Specify a lower value for less random responses and a higher value for more random responses. Acceptable value is [0.0, 1.0], default to 0.95.
},
"name": "A String", # Output only. Identifier. The resource name of the generator. Format: `projects//locations//generators/`
"publishedModel": "A String", # Optional. The published Large Language Model name. * To use the latest model version, specify the model name without version number. Example: `text-bison` * To use a stable model version, specify the version number as well. Example: `text-bison@002`.
"summarizationContext": { # Summarization context that customer can configure. # Input of Summarization feature.
"fewShotExamples": [ # Optional. List of few shot examples.
{ # Providing examples in the generator (i.e. building a few-shot generator) helps convey the desired format of the LLM response.
"conversationContext": { # Context of the conversation, including transcripts. # Optional. Conversation transcripts.
"messageEntries": [ # Optional. List of message transcripts in the conversation.
{ # Represents a message entry of a conversation.
"createTime": "A String", # Optional. Create time of the message entry.
"languageCode": "A String", # Optional. The language of the text. See [Language Support](https://cloud.google.com/dialogflow/docs/reference/language) for a list of the currently supported language codes.
"role": "A String", # Optional. Participant role of the message.
"text": "A String", # Optional. Transcript content of the message.
},
],
},
"extraInfo": { # Optional. Key is the placeholder field name in input, value is the value of the placeholder. E.g. instruction contains "@price", and ingested data has <"price", "10">
"a_key": "A String",
},
"output": { # Suggestion generated using a Generator. # Required. Example output of the model.
"freeFormSuggestion": { # Suggestion generated using free form generator. # Optional. Free form suggestion.
"response": "A String", # Required. Free form suggestion.
},
"summarySuggestion": { # Suggested summary of the conversation. # Optional. Suggested summary.
"summarySections": [ # Required. All the parts of generated summary.
{ # A component of the generated summary.
"section": "A String", # Required. Name of the section.
"summary": "A String", # Required. Summary text for the section.
},
],
},
"toolCallInfo": [ # Optional. List of request and response for tool calls executed.
{ # Request and response for a tool call.
"toolCall": { # Represents a call of a specific tool's action with the specified inputs. # Required. Request for a tool call.
"action": "A String", # Optional. The name of the tool's action associated with this call.
"createTime": "A String", # Output only. Create time of the tool call.
"inputParameters": { # Optional. The action's input parameters.
"a_key": "", # Properties of the object.
},
"tool": "A String", # Optional. The tool associated with this call. Format: `projects//locations//tools/`.
},
"toolCallResult": { # The result of calling a tool's action. # Required. Response for a tool call.
"action": "A String", # Optional. The name of the tool's action associated with this call.
"content": "A String", # Only populated if the response content is utf-8 encoded.
"createTime": "A String", # Output only. Create time of the tool call result.
"error": { # An error produced by the tool call. # The tool call's error.
"message": "A String", # Optional. The error message of the function.
},
"rawContent": "A String", # Only populated if the response content is not utf-8 encoded. (by definition byte fields are base64 encoded).
"tool": "A String", # Optional. The tool associated with this call. Format: `projects//locations//tools/`.
},
},
],
},
"summarizationSectionList": { # List of summarization sections. # Summarization sections.
"summarizationSections": [ # Optional. Summarization sections.
{ # Represents the section of summarization.
"definition": "A String", # Optional. Definition of the section, for example, "what the customer needs help with or has question about."
"key": "A String", # Optional. Name of the section, for example, "situation".
"type": "A String", # Optional. Type of the summarization section.
},
],
},
},
],
"outputLanguageCode": "A String", # Optional. The target language of the generated summary. The language code for conversation will be used if this field is empty. Supported 2.0 and later versions.
"summarizationSections": [ # Optional. List of sections. Note it contains both predefined section sand customer defined sections.
{ # Represents the section of summarization.
"definition": "A String", # Optional. Definition of the section, for example, "what the customer needs help with or has question about."
"key": "A String", # Optional. Name of the section, for example, "situation".
"type": "A String", # Optional. Type of the summarization section.
},
],
"version": "A String", # Optional. Version of the feature. If not set, default to latest version. Current candidates are ["1.0"].
},
"tools": [ # Optional. Resource names of the tools that the generator can choose from. Format: `projects//locations//tools/`.
"A String",
],
"triggerEvent": "A String", # Optional. The trigger event of the generator. It defines when the generator is triggered in a conversation.
"updateTime": "A String", # Output only. Update time of this generator.
},
"name": "A String", # Output only. Identifier. The resource name of the evaluation. Format: `projects//locations//generators// evaluations/`
"summarizationMetrics": { # Evaluation metrics for summarization generator. # Output only. Only available when the summarization generator is provided.
"conversationDetails": [ # Output only. List of conversation details.
{ # Aggregated evaluation result on conversation level. This contains evaluation results of all the metrics and sections.
"messageEntries": [ # Output only. Conversation transcript that used for summarization evaluation as a reference.
{ # Represents a message entry of a conversation.
"createTime": "A String", # Optional. Create time of the message entry.
"languageCode": "A String", # Optional. The language of the text. See [Language Support](https://cloud.google.com/dialogflow/docs/reference/language) for a list of the currently supported language codes.
"role": "A String", # Optional. Participant role of the message.
"text": "A String", # Optional. Transcript content of the message.
},
],
"metricDetails": [ # Output only. List of metric details.
{ # Aggregated result on metric level. This contains the evaluation results of all the sections.
"metric": "A String", # Output only. Metrics name. e.g. accuracy, adherence, completeness.
"score": 3.14, # Output only. Aggregated(average) score on this metric across all sections.
"sectionDetails": [ # Output only. List of section details.
{ # Section level result.
"evaluationResults": [ # Output only. List of evaluation result. The list only contains one kind of the evaluation result.
{ # Evaluation result that contains one of accuracy, adherence or completeness evaluation result.
"accuracyDecomposition": { # Decomposition details for accuracy. # Only available for accuracy metric.
"accuracyReasoning": "A String", # Output only. The accuracy reasoning of the breakdown point.
"isAccurate": True or False, # Output only. Whether the breakdown point is accurate or not.
"point": "A String", # Output only. The breakdown point of the summary.
},
"adherenceRubric": { # Rubric result of the adherence evaluation. A rubric is ued to determine if the summary adheres to all aspects of the given instructions. # Only available for adherence metric.
"isAddressed": True or False, # Output only. A boolean that indicates whether the rubric question is addressed or not.
"question": "A String", # Output only. The question generated from instruction that used to evaluate summary.
"reasoning": "A String", # Output only. The reasoning of the rubric question is addressed or not.
},
"completenessRubric": { # Rubric details of the completeness evaluation result. # Only available for completeness metric.
"isAddressed": True or False, # Output only. A boolean that indicates whether the rubric question is addressed or not.
"question": "A String", # Output only. The question generated from instruction that used to evaluate summary.
},
},
],
"score": 3.14, # Output only. Aggregated(average) score on this section across all evaluation results. Either decompositions or rubrics.
"section": "A String", # Output only. The name of the summary instruction.
"sectionSummary": "A String", # Output only. Summary for this section
},
],
},
],
"sectionTokens": [ # Output only. Conversation level token count per section. This is an aggregated(sum) result of input token of summary acorss all metrics for a single conversation.
{ # A pair of section name and input token count of the input summary section.
"section": "A String", # Output only. The name of the summary instruction.
"tokenCount": "A String", # Output only. Token count.
},
],
"summarySections": [ # Output only. Summary sections that used for summarization evaluation as a reference.
{ # A component of the generated summary.
"section": "A String", # Required. Name of the section.
"summary": "A String", # Required. Summary text for the section.
},
],
},
],
"overallMetrics": [ # Output only. A list of aggregated(average) scores per metric section.
{ # Overall performance per metric. This is the aggregated score for each metric across all conversations that are selected for summarization evaluation.
"metric": "A String", # Output only. Metric name. e.g. accuracy, adherence, completeness.
},
],
"overallSectionTokens": [ # Output only. Overall token per section. This is an aggregated(sum) result of input token of summary acorss all conversations that are selected for summarization evaluation.
{ # A pair of section name and input token count of the input summary section.
"section": "A String", # Output only. The name of the summary instruction.
"tokenCount": "A String", # Output only. Token count.
},
],
"summarizationEvaluationMergedResultsUri": "A String", # Output only. User bucket uri for merged evaluation score and aggregation score csv.
"summarizationEvaluationResults": [ # Output only. A list of evaluation results per conversation(&summary), metric and section.
{ # Evaluation result per conversation(&summary), metric and section.
"decompositions": [ # Output only. List of decompostion details
{ # Decomposition details
"accuracyDecomposition": { # Decomposition details for accuracy. # only available for accuracy metric.
"accuracyReasoning": "A String", # Output only. The accuracy reasoning of the breakdown point.
"isAccurate": True or False, # Output only. Whether the breakdown point is accurate or not.
"point": "A String", # Output only. The breakdown point of the summary.
},
"adherenceDecomposition": { # Decomposition details for adherence. # only available for adherence metric.
"adherenceReasoning": "A String", # Output only. The adherence reasoning of the breakdown point.
"isAdherent": True or False, # Output only. Whether the breakdown point is adherent or not.
"point": "A String", # Output only. The breakdown point of the given instructions.
},
},
],
"evaluationResults": [ # Output only. List of evaluation results.
{ # Evaluation result that contains one of accuracy, adherence or completeness evaluation result.
"accuracyDecomposition": { # Decomposition details for accuracy. # Only available for accuracy metric.
"accuracyReasoning": "A String", # Output only. The accuracy reasoning of the breakdown point.
"isAccurate": True or False, # Output only. Whether the breakdown point is accurate or not.
"point": "A String", # Output only. The breakdown point of the summary.
},
"adherenceRubric": { # Rubric result of the adherence evaluation. A rubric is ued to determine if the summary adheres to all aspects of the given instructions. # Only available for adherence metric.
"isAddressed": True or False, # Output only. A boolean that indicates whether the rubric question is addressed or not.
"question": "A String", # Output only. The question generated from instruction that used to evaluate summary.
"reasoning": "A String", # Output only. The reasoning of the rubric question is addressed or not.
},
"completenessRubric": { # Rubric details of the completeness evaluation result. # Only available for completeness metric.
"isAddressed": True or False, # Output only. A boolean that indicates whether the rubric question is addressed or not.
"question": "A String", # Output only. The question generated from instruction that used to evaluate summary.
},
},
],
"metric": "A String", # Output only. metric name, e.g. accuracy, completeness, adherence, etc.
"score": 3.14, # Output only. score calculated from decompositions
"section": "A String", # Output only. section/task name, e.g. action, situation, etc
"sectionSummary": "A String", # Output only. Summary of this section
"sessionId": "A String", # Output only. conversation session id
},
],
},
}
list(parent, pageSize=None, pageToken=None, x__xgafv=None)
Lists evaluations of generator.
Args:
parent: string, Required. The generator resource name. Format: `projects//locations//generators/` Wildcard value `-` is supported on generator_id to list evaluations across all generators under same project. (required)
pageSize: integer, Optional. Maximum number of evaluations to return in a single page. By default 100 and at most 1000.
pageToken: string, Optional. The next_page_token value returned from a previous list request.
x__xgafv: string, V1 error format.
Allowed values
1 - v1 error format
2 - v2 error format
Returns:
An object of the form:
{ # Response of ListGeneratorEvaluations.
"generatorEvaluations": [ # The list of evaluations to return.
{ # Represents evaluation result of a generator.
"completeTime": "A String", # Output only. Completion time of this generator evaluation.
"createTime": "A String", # Output only. Creation time of this generator evaluation.
"displayName": "A String", # Optional. The display name of the generator evaluation. At most 64 bytes long.
"evaluationStatus": { # A common evalaution pipeline status. # Output only. The result status of the evaluation pipeline. Provides the status information including if the evaluation is still in progress, completed or failed with certain error and user actionable message.
"done": True or False, # Output only. If the value is `false`, it means the evaluation is still in progress. If `true`, the operation is completed, and either `error` or `response` is available.
"pipelineStatus": { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors). # Output only. The error result of the evaluation in case of failure in evaluation pipeline.
"code": 42, # The status code, which should be an enum value of google.rpc.Code.
"details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use.
{
"a_key": "", # Properties of the object. Contains field @type with type URL.
},
],
"message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
},
},
"generatorEvaluationConfig": { # Generator evaluation input config. # Required. The configuration of the evaluation task.
"inputDataConfig": { # Input data config details # Required. The config/source of input data.
"agentAssistInputDataConfig": { # The distinctive configs for Agent Assist conversations as the conversation source. # The distinctive configs for Agent Assist conversations as the conversation source.
"endTime": "A String", # Required. The end of the time range for conversations to be evaluated. Only conversations ended at or before this timestamp will be sampled.
"startTime": "A String", # Required. The start of the time range for conversations to be evaluated. Only conversations created at or after this timestamp will be sampled.
},
"datasetInputDataConfig": { # The distinctive configs for dataset as the conversation source. # The distinctive configs for dataset as the conversation source.
"dataset": "A String", # Required. The identifier of the dataset to be evaluated. Format: `projects//locations//datasets/`.
},
"endTime": "A String", # Optional. The end timestamp to fetch conversation data.
"inputDataSourceType": "A String", # Required. The source type of input data.
"isSummaryGenerationAllowed": True or False, # Optional. Whether the summary generation is allowed when the pre-existing qualified summaries are insufficient to cover the sample size.
"sampleSize": 42, # Optional. Desired number of conversation-summary pairs to be evaluated.
"startTime": "A String", # Optional. The start timestamp to fetch conversation data.
"summaryGenerationOption": "A String", # Optional. Option to control whether summaries are generated during evaluation.
},
"outputGcsBucketPath": "A String", # Required. The output Cloud Storage bucket path to store eval files, e.g. per_summary_accuracy_score report. This path is provided by customer and files stored in it are visible to customer, no internal data should be stored in this path.
"summarizationConfig": { # Evaluation configs for summarization generator. # Evaluation configs for summarization generator.
"accuracyEvaluationVersion": "A String", # Optional. Version for summarization accuracy. This will determine the prompt and model used at backend.
"completenessEvaluationVersion": "A String", # Optional. Version for summarization completeness. This will determine the prompt and model used at backend.
"enableAccuracyEvaluation": True or False, # Optional. Enable accuracy evaluation.
"enableCompletenessEvaluation": True or False, # Optional. Enable completeness evaluation.
"evaluatorVersion": "A String", # Output only. Version for summarization evaluation.
},
},
"initialGenerator": { # LLM generator. # Required. The initial generator that was used when creating this evaluation. This is a copy of the generator read from storage when creating the evaluation.
"createTime": "A String", # Output only. Creation time of this generator.
"description": "A String", # Optional. Human readable description of the generator.
"freeFormContext": { # Free form generator context that customer can configure. # Input of free from generator to LLM.
"text": "A String", # Optional. Free form text input to LLM.
},
"inferenceParameter": { # The parameters of inference. # Optional. Inference parameters for this generator.
"maxOutputTokens": 42, # Optional. Maximum number of the output tokens for the generator.
"temperature": 3.14, # Optional. Controls the randomness of LLM predictions. Low temperature = less random. High temperature = more random. If unset (or 0), uses a default value of 0.
"topK": 42, # Optional. Top-k changes how the model selects tokens for output. A top-k of 1 means the selected token is the most probable among all tokens in the model's vocabulary (also called greedy decoding), while a top-k of 3 means that the next token is selected from among the 3 most probable tokens (using temperature). For each token selection step, the top K tokens with the highest probabilities are sampled. Then tokens are further filtered based on topP with the final token selected using temperature sampling. Specify a lower value for less random responses and a higher value for more random responses. Acceptable value is [1, 40], default to 40.
"topP": 3.14, # Optional. Top-p changes how the model selects tokens for output. Tokens are selected from most K (see topK parameter) probable to least until the sum of their probabilities equals the top-p value. For example, if tokens A, B, and C have a probability of 0.3, 0.2, and 0.1 and the top-p value is 0.5, then the model will select either A or B as the next token (using temperature) and doesn't consider C. The default top-p value is 0.95. Specify a lower value for less random responses and a higher value for more random responses. Acceptable value is [0.0, 1.0], default to 0.95.
},
"name": "A String", # Output only. Identifier. The resource name of the generator. Format: `projects//locations//generators/`
"publishedModel": "A String", # Optional. The published Large Language Model name. * To use the latest model version, specify the model name without version number. Example: `text-bison` * To use a stable model version, specify the version number as well. Example: `text-bison@002`.
"summarizationContext": { # Summarization context that customer can configure. # Input of Summarization feature.
"fewShotExamples": [ # Optional. List of few shot examples.
{ # Providing examples in the generator (i.e. building a few-shot generator) helps convey the desired format of the LLM response.
"conversationContext": { # Context of the conversation, including transcripts. # Optional. Conversation transcripts.
"messageEntries": [ # Optional. List of message transcripts in the conversation.
{ # Represents a message entry of a conversation.
"createTime": "A String", # Optional. Create time of the message entry.
"languageCode": "A String", # Optional. The language of the text. See [Language Support](https://cloud.google.com/dialogflow/docs/reference/language) for a list of the currently supported language codes.
"role": "A String", # Optional. Participant role of the message.
"text": "A String", # Optional. Transcript content of the message.
},
],
},
"extraInfo": { # Optional. Key is the placeholder field name in input, value is the value of the placeholder. E.g. instruction contains "@price", and ingested data has <"price", "10">
"a_key": "A String",
},
"output": { # Suggestion generated using a Generator. # Required. Example output of the model.
"freeFormSuggestion": { # Suggestion generated using free form generator. # Optional. Free form suggestion.
"response": "A String", # Required. Free form suggestion.
},
"summarySuggestion": { # Suggested summary of the conversation. # Optional. Suggested summary.
"summarySections": [ # Required. All the parts of generated summary.
{ # A component of the generated summary.
"section": "A String", # Required. Name of the section.
"summary": "A String", # Required. Summary text for the section.
},
],
},
"toolCallInfo": [ # Optional. List of request and response for tool calls executed.
{ # Request and response for a tool call.
"toolCall": { # Represents a call of a specific tool's action with the specified inputs. # Required. Request for a tool call.
"action": "A String", # Optional. The name of the tool's action associated with this call.
"createTime": "A String", # Output only. Create time of the tool call.
"inputParameters": { # Optional. The action's input parameters.
"a_key": "", # Properties of the object.
},
"tool": "A String", # Optional. The tool associated with this call. Format: `projects//locations//tools/`.
},
"toolCallResult": { # The result of calling a tool's action. # Required. Response for a tool call.
"action": "A String", # Optional. The name of the tool's action associated with this call.
"content": "A String", # Only populated if the response content is utf-8 encoded.
"createTime": "A String", # Output only. Create time of the tool call result.
"error": { # An error produced by the tool call. # The tool call's error.
"message": "A String", # Optional. The error message of the function.
},
"rawContent": "A String", # Only populated if the response content is not utf-8 encoded. (by definition byte fields are base64 encoded).
"tool": "A String", # Optional. The tool associated with this call. Format: `projects//locations//tools/`.
},
},
],
},
"summarizationSectionList": { # List of summarization sections. # Summarization sections.
"summarizationSections": [ # Optional. Summarization sections.
{ # Represents the section of summarization.
"definition": "A String", # Optional. Definition of the section, for example, "what the customer needs help with or has question about."
"key": "A String", # Optional. Name of the section, for example, "situation".
"type": "A String", # Optional. Type of the summarization section.
},
],
},
},
],
"outputLanguageCode": "A String", # Optional. The target language of the generated summary. The language code for conversation will be used if this field is empty. Supported 2.0 and later versions.
"summarizationSections": [ # Optional. List of sections. Note it contains both predefined section sand customer defined sections.
{ # Represents the section of summarization.
"definition": "A String", # Optional. Definition of the section, for example, "what the customer needs help with or has question about."
"key": "A String", # Optional. Name of the section, for example, "situation".
"type": "A String", # Optional. Type of the summarization section.
},
],
"version": "A String", # Optional. Version of the feature. If not set, default to latest version. Current candidates are ["1.0"].
},
"tools": [ # Optional. Resource names of the tools that the generator can choose from. Format: `projects//locations//tools/`.
"A String",
],
"triggerEvent": "A String", # Optional. The trigger event of the generator. It defines when the generator is triggered in a conversation.
"updateTime": "A String", # Output only. Update time of this generator.
},
"name": "A String", # Output only. Identifier. The resource name of the evaluation. Format: `projects//locations//generators// evaluations/`
"summarizationMetrics": { # Evaluation metrics for summarization generator. # Output only. Only available when the summarization generator is provided.
"conversationDetails": [ # Output only. List of conversation details.
{ # Aggregated evaluation result on conversation level. This contains evaluation results of all the metrics and sections.
"messageEntries": [ # Output only. Conversation transcript that used for summarization evaluation as a reference.
{ # Represents a message entry of a conversation.
"createTime": "A String", # Optional. Create time of the message entry.
"languageCode": "A String", # Optional. The language of the text. See [Language Support](https://cloud.google.com/dialogflow/docs/reference/language) for a list of the currently supported language codes.
"role": "A String", # Optional. Participant role of the message.
"text": "A String", # Optional. Transcript content of the message.
},
],
"metricDetails": [ # Output only. List of metric details.
{ # Aggregated result on metric level. This contains the evaluation results of all the sections.
"metric": "A String", # Output only. Metrics name. e.g. accuracy, adherence, completeness.
"score": 3.14, # Output only. Aggregated(average) score on this metric across all sections.
"sectionDetails": [ # Output only. List of section details.
{ # Section level result.
"evaluationResults": [ # Output only. List of evaluation result. The list only contains one kind of the evaluation result.
{ # Evaluation result that contains one of accuracy, adherence or completeness evaluation result.
"accuracyDecomposition": { # Decomposition details for accuracy. # Only available for accuracy metric.
"accuracyReasoning": "A String", # Output only. The accuracy reasoning of the breakdown point.
"isAccurate": True or False, # Output only. Whether the breakdown point is accurate or not.
"point": "A String", # Output only. The breakdown point of the summary.
},
"adherenceRubric": { # Rubric result of the adherence evaluation. A rubric is ued to determine if the summary adheres to all aspects of the given instructions. # Only available for adherence metric.
"isAddressed": True or False, # Output only. A boolean that indicates whether the rubric question is addressed or not.
"question": "A String", # Output only. The question generated from instruction that used to evaluate summary.
"reasoning": "A String", # Output only. The reasoning of the rubric question is addressed or not.
},
"completenessRubric": { # Rubric details of the completeness evaluation result. # Only available for completeness metric.
"isAddressed": True or False, # Output only. A boolean that indicates whether the rubric question is addressed or not.
"question": "A String", # Output only. The question generated from instruction that used to evaluate summary.
},
},
],
"score": 3.14, # Output only. Aggregated(average) score on this section across all evaluation results. Either decompositions or rubrics.
"section": "A String", # Output only. The name of the summary instruction.
"sectionSummary": "A String", # Output only. Summary for this section
},
],
},
],
"sectionTokens": [ # Output only. Conversation level token count per section. This is an aggregated(sum) result of input token of summary acorss all metrics for a single conversation.
{ # A pair of section name and input token count of the input summary section.
"section": "A String", # Output only. The name of the summary instruction.
"tokenCount": "A String", # Output only. Token count.
},
],
"summarySections": [ # Output only. Summary sections that used for summarization evaluation as a reference.
{ # A component of the generated summary.
"section": "A String", # Required. Name of the section.
"summary": "A String", # Required. Summary text for the section.
},
],
},
],
"overallMetrics": [ # Output only. A list of aggregated(average) scores per metric section.
{ # Overall performance per metric. This is the aggregated score for each metric across all conversations that are selected for summarization evaluation.
"metric": "A String", # Output only. Metric name. e.g. accuracy, adherence, completeness.
},
],
"overallSectionTokens": [ # Output only. Overall token per section. This is an aggregated(sum) result of input token of summary acorss all conversations that are selected for summarization evaluation.
{ # A pair of section name and input token count of the input summary section.
"section": "A String", # Output only. The name of the summary instruction.
"tokenCount": "A String", # Output only. Token count.
},
],
"summarizationEvaluationMergedResultsUri": "A String", # Output only. User bucket uri for merged evaluation score and aggregation score csv.
"summarizationEvaluationResults": [ # Output only. A list of evaluation results per conversation(&summary), metric and section.
{ # Evaluation result per conversation(&summary), metric and section.
"decompositions": [ # Output only. List of decompostion details
{ # Decomposition details
"accuracyDecomposition": { # Decomposition details for accuracy. # only available for accuracy metric.
"accuracyReasoning": "A String", # Output only. The accuracy reasoning of the breakdown point.
"isAccurate": True or False, # Output only. Whether the breakdown point is accurate or not.
"point": "A String", # Output only. The breakdown point of the summary.
},
"adherenceDecomposition": { # Decomposition details for adherence. # only available for adherence metric.
"adherenceReasoning": "A String", # Output only. The adherence reasoning of the breakdown point.
"isAdherent": True or False, # Output only. Whether the breakdown point is adherent or not.
"point": "A String", # Output only. The breakdown point of the given instructions.
},
},
],
"evaluationResults": [ # Output only. List of evaluation results.
{ # Evaluation result that contains one of accuracy, adherence or completeness evaluation result.
"accuracyDecomposition": { # Decomposition details for accuracy. # Only available for accuracy metric.
"accuracyReasoning": "A String", # Output only. The accuracy reasoning of the breakdown point.
"isAccurate": True or False, # Output only. Whether the breakdown point is accurate or not.
"point": "A String", # Output only. The breakdown point of the summary.
},
"adherenceRubric": { # Rubric result of the adherence evaluation. A rubric is ued to determine if the summary adheres to all aspects of the given instructions. # Only available for adherence metric.
"isAddressed": True or False, # Output only. A boolean that indicates whether the rubric question is addressed or not.
"question": "A String", # Output only. The question generated from instruction that used to evaluate summary.
"reasoning": "A String", # Output only. The reasoning of the rubric question is addressed or not.
},
"completenessRubric": { # Rubric details of the completeness evaluation result. # Only available for completeness metric.
"isAddressed": True or False, # Output only. A boolean that indicates whether the rubric question is addressed or not.
"question": "A String", # Output only. The question generated from instruction that used to evaluate summary.
},
},
],
"metric": "A String", # Output only. metric name, e.g. accuracy, completeness, adherence, etc.
"score": 3.14, # Output only. score calculated from decompositions
"section": "A String", # Output only. section/task name, e.g. action, situation, etc
"sectionSummary": "A String", # Output only. Summary of this section
"sessionId": "A String", # Output only. conversation session id
},
],
},
},
],
"nextPageToken": "A String", # Token to retrieve the next page of results, or empty if there are no more results in the list.
}
list_next()
Retrieves the next page of results.
Args:
previous_request: The request for the previous page. (required)
previous_response: The response from the request for the previous page. (required)
Returns:
A request object that you can call 'execute()' on to request the next
page. Returns None if there are no more items in the collection.