Dialogflow API . projects . conversationModels . evaluations

close()

Close httplib2 connections.

get(name, x__xgafv=None)

Gets an evaluation of conversation model.

Args:
  name: string, Required. The conversation model evaluation resource name. Format: `projects//conversationModels//evaluations/` (required)
  x__xgafv: string, V1 error format.
    Allowed values
      1 - v1 error format
      2 - v2 error format

Returns:
  An object of the form:

    { # Represents evaluation result of a conversation model.
  "createTime": "A String", # Output only. Creation time of this model.
  "displayName": "A String", # Optional. The display name of the model evaluation. At most 64 bytes long.
  "evaluationConfig": { # The configuration for model evaluation. # Optional. The configuration of the evaluation task.
    "datasets": [ # Required. Datasets used for evaluation.
      { # InputDataset used to create model or do evaluation. NextID:5
        "dataset": "A String", # Required. ConversationDataset resource name. Format: `projects//locations//conversationDatasets/`
      },
    ],
    "smartComposeConfig": { # Smart compose specific configuration for evaluation job. # Configuration for smart compose model evalution.
      "allowlistDocument": "A String", # The allowlist document resource name. Format: `projects//knowledgeBases//documents/`. Only used for smart compose model.
      "maxResultCount": 42, # Required. The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.
    },
    "smartReplyConfig": { # Smart reply specific configuration for evaluation job. # Configuration for smart reply model evalution.
      "allowlistDocument": "A String", # The allowlist document resource name. Format: `projects//knowledgeBases//documents/`. Only used for smart reply model.
      "maxResultCount": 42, # Required. The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.
    },
  },
  "name": "A String", # The resource name of the evaluation. Format: `projects//conversationModels//evaluations/`
  "rawHumanEvalTemplateCsv": "A String", # Output only. Human eval template in csv format. It tooks real-world conversations provided through input dataset, generates example suggestions for customer to verify quality of the model. For Smart Reply, the generated csv file contains columns of Context, (Suggestions,Q1,Q2)*3, Actual reply. Context contains at most 10 latest messages in the conversation prior to the current suggestion. Q1: "Would you send it as the next message of agent?" Evaluated based on whether the suggest is appropriate to be sent by agent in current context. Q2: "Does the suggestion move the conversation closer to resolution?" Evaluated based on whether the suggestion provide solutions, or answers customer's question or collect information from customer to resolve the customer's issue. Actual reply column contains the actual agent reply sent in the context.
  "smartReplyMetrics": { # The evaluation metrics for smart reply model. # Output only. Only available when model is for smart reply.
    "allowlistCoverage": 3.14, # Percentage of target participant messages in the evaluation dataset for which similar messages have appeared at least once in the allowlist. Should be [0, 1].
    "conversationCount": "A String", # Total number of conversations used to generate this metric.
    "topNMetrics": [ # Metrics of top n smart replies, sorted by TopNMetric.n.
      { # Evaluation metrics when retrieving `n` smart replies with the model.
        "n": 42, # Number of retrieved smart replies. For example, when `n` is 3, this evaluation contains metrics for when Dialogflow retrieves 3 smart replies with the model.
        "recall": 3.14, # Defined as `number of queries whose top n smart replies have at least one similar (token match similarity above the defined threshold) reply as the real reply` divided by `number of queries with at least one smart reply`. Value ranges from 0.0 to 1.0 inclusive.
      },
    ],
  },
}

list(parent, pageSize=None, pageToken=None, x__xgafv=None)

Lists evaluations of a conversation model.

Args:
  parent: string, Required. The conversation model resource name. Format: `projects//conversationModels/` (required)
  pageSize: integer, Optional. Maximum number of evaluations to return in a single page. By default 100 and at most 1000.
  pageToken: string, Optional. The next_page_token value returned from a previous list request.
  x__xgafv: string, V1 error format.
    Allowed values
      1 - v1 error format
      2 - v2 error format

Returns:
  An object of the form:

    { # The response message for ConversationModels.ListConversationModelEvaluations
  "conversationModelEvaluations": [ # The list of evaluations to return.
    { # Represents evaluation result of a conversation model.
      "createTime": "A String", # Output only. Creation time of this model.
      "displayName": "A String", # Optional. The display name of the model evaluation. At most 64 bytes long.
      "evaluationConfig": { # The configuration for model evaluation. # Optional. The configuration of the evaluation task.
        "datasets": [ # Required. Datasets used for evaluation.
          { # InputDataset used to create model or do evaluation. NextID:5
            "dataset": "A String", # Required. ConversationDataset resource name. Format: `projects//locations//conversationDatasets/`
          },
        ],
        "smartComposeConfig": { # Smart compose specific configuration for evaluation job. # Configuration for smart compose model evalution.
          "allowlistDocument": "A String", # The allowlist document resource name. Format: `projects//knowledgeBases//documents/`. Only used for smart compose model.
          "maxResultCount": 42, # Required. The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.
        },
        "smartReplyConfig": { # Smart reply specific configuration for evaluation job. # Configuration for smart reply model evalution.
          "allowlistDocument": "A String", # The allowlist document resource name. Format: `projects//knowledgeBases//documents/`. Only used for smart reply model.
          "maxResultCount": 42, # Required. The model to be evaluated can return multiple results with confidence score on each query. These results will be sorted by the descending order of the scores and we only keep the first max_result_count results as the final results to evaluate.
        },
      },
      "name": "A String", # The resource name of the evaluation. Format: `projects//conversationModels//evaluations/`
      "rawHumanEvalTemplateCsv": "A String", # Output only. Human eval template in csv format. It tooks real-world conversations provided through input dataset, generates example suggestions for customer to verify quality of the model. For Smart Reply, the generated csv file contains columns of Context, (Suggestions,Q1,Q2)*3, Actual reply. Context contains at most 10 latest messages in the conversation prior to the current suggestion. Q1: "Would you send it as the next message of agent?" Evaluated based on whether the suggest is appropriate to be sent by agent in current context. Q2: "Does the suggestion move the conversation closer to resolution?" Evaluated based on whether the suggestion provide solutions, or answers customer's question or collect information from customer to resolve the customer's issue. Actual reply column contains the actual agent reply sent in the context.
      "smartReplyMetrics": { # The evaluation metrics for smart reply model. # Output only. Only available when model is for smart reply.
        "allowlistCoverage": 3.14, # Percentage of target participant messages in the evaluation dataset for which similar messages have appeared at least once in the allowlist. Should be [0, 1].
        "conversationCount": "A String", # Total number of conversations used to generate this metric.
        "topNMetrics": [ # Metrics of top n smart replies, sorted by TopNMetric.n.
          { # Evaluation metrics when retrieving `n` smart replies with the model.
            "n": 42, # Number of retrieved smart replies. For example, when `n` is 3, this evaluation contains metrics for when Dialogflow retrieves 3 smart replies with the model.
            "recall": 3.14, # Defined as `number of queries whose top n smart replies have at least one similar (token match similarity above the defined threshold) reply as the real reply` divided by `number of queries with at least one smart reply`. Value ranges from 0.0 to 1.0 inclusive.
          },
        ],
      },
    },
  ],
  "nextPageToken": "A String", # Token to retrieve the next page of results, or empty if there are no more results in the list.
}

list_next()

Retrieves the next page of results.

        Args:
          previous_request: The request for the previous page. (required)
          previous_response: The response from the request for the previous page. (required)

        Returns:
          A request object that you can call 'execute()' on to request the next
          page. Returns None if there are no more items in the collection.