batchImport(parent, body=None, x__xgafv=None)

Imports a list of externally generated EvaluatedAnnotations.

Args:
parent: string, Required. The name of the parent ModelEvaluationSlice resource. Format: `projects/{project}/locations/{location}/models/{model}/evaluations/{evaluation}/slices/{slice}` (required)
body: object, The request body.
The object takes the form of:

{ # Request message for ModelService.BatchImportEvaluatedAnnotations
"evaluatedAnnotations": [ # Required. Evaluated annotations resource to be imported.
{ # True positive, false positive, or false negative. EvaluatedAnnotation is only available under ModelEvaluationSlice with slice of `annotationSpec` dimension.
"dataItemPayload": "", # Output only. The data item payload that the Model predicted this EvaluatedAnnotation on.
"errorAnalysisAnnotations": [ # Annotations of model error analysis results.
{ # Model error analysis for each annotation.
"attributedItems": [ # Attributed items for a given annotation, typically representing neighbors from the training sets constrained by the query type.
{ # Attributed items for a given annotation, typically representing neighbors from the training sets constrained by the query type.
"annotationResourceName": "A String", # The unique ID for each annotation. Used by FE to allocate the annotation in DB.
"distance": 3.14, # The distance of this item to the annotation.
},
],
"outlierScore": 3.14, # The outlier score of this annotated item. Usually defined as the min of all distances from attributed items.
"outlierThreshold": 3.14, # The threshold used to determine if this annotation is an outlier or not.
"queryType": "A String", # The query type used for finding the attributed items.
},
],
"evaluatedDataItemViewId": "A String", # Output only. ID of the EvaluatedDataItemView under the same ancestor ModelEvaluation. The EvaluatedDataItemView consists of all ground truths and predictions on data_item_payload.
"explanations": [ # Explanations of predictions. Each element of the explanations indicates the explanation for one explanation Method. The attributions list in the EvaluatedAnnotationExplanation.explanation object corresponds to the predictions list. For example, the second element in the attributions list explains the second element in the predictions list.
{ # Explanation result of the prediction produced by the Model.
"explanation": { # Explanation of a prediction (provided in PredictResponse.predictions) produced by the Model on a given instance. # Explanation attribution response details.
"attributions": [ # Output only. Feature attributions grouped by predicted outputs. For Models that predict only one output, such as regression Models that predict only one score, there is only one attibution that explains the predicted output. For Models that predict multiple outputs, such as multiclass Models that predict multiple classes, each element explains one specific item. Attribution.output_index can be used to identify which output this attribution is explaining. By default, we provide Shapley values for the predicted class. However, you can configure the explanation request to generate Shapley values for any other classes too. For example, if a model predicts a probability of `0.4` for approving a loan application, the model's decision is to reject the application since `p(reject) = 0.6 > p(approve) = 0.4`, and the default Shapley values would be computed for rejection decision and not approval, even though the latter might be the positive class. If users set ExplanationParameters.top_k, the attributions are sorted by instance_output_value in descending order. If ExplanationParameters.output_indices is specified, the attributions are stored by Attribution.output_index in the same order as they appear in the output_indices.
{ # Attribution that explains a particular prediction output.
"approximationError": 3.14, # Output only. Error of feature_attributions caused by approximation used in the explanation method. Lower value means more precise attributions. * For Sampled Shapley attribution, increasing path_count might reduce the error. * For Integrated Gradients attribution, increasing step_count might reduce the error. * For XRAI attribution, increasing step_count might reduce the error. See [this introduction](/vertex-ai/docs/explainable-ai/overview) for more information.
"baselineOutputValue": 3.14, # Output only. Model predicted output if the input instance is constructed from the baselines of all the features defined in ExplanationMetadata.inputs. The field name of the output is determined by the key in ExplanationMetadata.outputs. If the Model's predicted output has multiple dimensions (rank > 1), this is the value in the output located by output_index. If there are multiple baselines, their output values are averaged.
"featureAttributions": "", # Output only. Attributions of each explained feature. Features are extracted from the prediction instances according to explanation metadata for inputs. The value is a struct, whose keys are the name of the feature. The values are how much the feature in the instance contributed to the predicted result. The format of the value is determined by the feature's input format: * If the feature is a scalar value, the attribution value is a floating number. * If the feature is an array of scalar values, the attribution value is an array. * If the feature is a struct, the attribution value is a struct. The keys in the attribution value struct are the same as the keys in the feature struct. The formats of the values in the attribution struct are determined by the formats of the values in the feature struct. The ExplanationMetadata.feature_attributions_schema_uri field, pointed to by the ExplanationSpec field of the Endpoint.deployed_models object, points to the schema file that describes the features and their attribution values (if it is populated).
"instanceOutputValue": 3.14, # Output only. Model predicted output on the corresponding explanation instance. The field name of the output is determined by the key in ExplanationMetadata.outputs. If the Model predicted output has multiple dimensions, this is the value in the output located by output_index.
"outputDisplayName": "A String", # Output only. The display name of the output identified by output_index. For example, the predicted class name by a multi-classification Model. This field is only populated iff the Model predicts display names as a separate field along with the explained output. The predicted display name must has the same shape of the explained output, and can be located using output_index.
"outputIndex": [ # Output only. The index that locates the explained prediction output. If the prediction output is a scalar value, output_index is not populated. If the prediction output has multiple dimensions, the length of the output_index list is the same as the number of dimensions of the output. The i-th element in output_index is the element index of the i-th dimension of the output vector. Indices start from 0.
42,
],
"outputName": "A String", # Output only. Name of the explain output. Specified as the key in ExplanationMetadata.outputs.
},
],
"neighbors": [ # Output only. List of the nearest neighbors for example-based explanations. For models deployed with the examples explanations feature enabled, the attributions field is empty and instead the neighbors field is populated.
{ # Neighbors for example-based explanations.
"neighborDistance": 3.14, # Output only. The neighbor distance.
"neighborId": "A String", # Output only. The neighbor id.
},
],
},
"explanationType": "A String", # Explanation type. For AutoML Image Classification models, possible values are: * `image-integrated-gradients` * `image-xrai`
},
],
"groundTruths": [ # Output only. The ground truth Annotations, i.e. the Annotations that exist in the test data the Model is evaluated on. For true positive, there is one and only one ground truth annotation, which matches the only prediction in predictions. For false positive, there are zero or more ground truth annotations that are similar to the only prediction in predictions, but not enough for a match. For false negative, there is one and only one ground truth annotation, which doesn't match any predictions created by the model. The schema of the ground truth is stored in ModelEvaluation.annotation_schema_uri
"",
],
"predictions": [ # Output only. The model predicted annotations. For true positive, there is one and only one prediction, which matches the only one ground truth annotation in ground_truths. For false positive, there is one and only one prediction, which doesn't match any ground truth annotation of the corresponding data_item_view_id. For false negative, there are zero or more predictions which are similar to the only ground truth annotation in ground_truths but not enough for a match. The schema of the prediction is stored in ModelEvaluation.annotation_schema_uri
"",
],
"type": "A String", # Output only. Type of the EvaluatedAnnotation.
},
],
}

x__xgafv: string, V1 error format.
Allowed values
1 - v1 error format
2 - v2 error format

Returns:
An object of the form:

{ # Response message for ModelService.BatchImportEvaluatedAnnotations
"importedEvaluatedAnnotationsCount": 42, # Output only. Number of EvaluatedAnnotations imported.
}

close()

Close httplib2 connections.

get(name, x__xgafv=None)

Gets a ModelEvaluationSlice.

Args:
name: string, Required. The name of the ModelEvaluationSlice resource. Format: `projects/{project}/locations/{location}/models/{model}/evaluations/{evaluation}/slices/{slice}` (required)
x__xgafv: string, V1 error format.
Allowed values
1 - v1 error format
2 - v2 error format

Returns:
An object of the form:

{ # A collection of metrics calculated by comparing Model's predictions on a slice of the test data against ground truth annotations.
"createTime": "A String", # Output only. Timestamp when this ModelEvaluationSlice was created.
"metrics": "", # Output only. Sliced evaluation metrics of the Model. The schema of the metrics is stored in metrics_schema_uri
"metricsSchemaUri": "A String", # Output only. Points to a YAML file stored on Google Cloud Storage describing the metrics of this ModelEvaluationSlice. The schema is defined as an OpenAPI 3.0.2 [Schema Object](https://github.com/OAI/OpenAPI-Specification/blob/main/versions/3.0.2.md#schemaObject).
"modelExplanation": { # Aggregated explanation metrics for a Model over a set of instances. # Output only. Aggregated explanation metrics for the Model's prediction output over the data this ModelEvaluation uses. This field is populated only if the Model is evaluated with explanations, and only for tabular Models.
"meanAttributions": [ # Output only. Aggregated attributions explaining the Model's prediction outputs over the set of instances. The attributions are grouped by outputs. For Models that predict only one output, such as regression Models that predict only one score, there is only one attibution that explains the predicted output. For Models that predict multiple outputs, such as multiclass Models that predict multiple classes, each element explains one specific item. Attribution.output_index can be used to identify which output this attribution is explaining. The baselineOutputValue, instanceOutputValue and featureAttributions fields are averaged over the test data. NOTE: Currently AutoML tabular classification Models produce only one attribution, which averages attributions over all the classes it predicts. Attribution.approximation_error is not populated.
{ # Attribution that explains a particular prediction output.
"approximationError": 3.14, # Output only. Error of feature_attributions caused by approximation used in the explanation method. Lower value means more precise attributions. * For Sampled Shapley attribution, increasing path_count might reduce the error. * For Integrated Gradients attribution, increasing step_count might reduce the error. * For XRAI attribution, increasing step_count might reduce the error. See [this introduction](/vertex-ai/docs/explainable-ai/overview) for more information.
"baselineOutputValue": 3.14, # Output only. Model predicted output if the input instance is constructed from the baselines of all the features defined in ExplanationMetadata.inputs. The field name of the output is determined by the key in ExplanationMetadata.outputs. If the Model's predicted output has multiple dimensions (rank > 1), this is the value in the output located by output_index. If there are multiple baselines, their output values are averaged.
"featureAttributions": "", # Output only. Attributions of each explained feature. Features are extracted from the prediction instances according to explanation metadata for inputs. The value is a struct, whose keys are the name of the feature. The values are how much the feature in the instance contributed to the predicted result. The format of the value is determined by the feature's input format: * If the feature is a scalar value, the attribution value is a floating number. * If the feature is an array of scalar values, the attribution value is an array. * If the feature is a struct, the attribution value is a struct. The keys in the attribution value struct are the same as the keys in the feature struct. The formats of the values in the attribution struct are determined by the formats of the values in the feature struct. The ExplanationMetadata.feature_attributions_schema_uri field, pointed to by the ExplanationSpec field of the Endpoint.deployed_models object, points to the schema file that describes the features and their attribution values (if it is populated).
"instanceOutputValue": 3.14, # Output only. Model predicted output on the corresponding explanation instance. The field name of the output is determined by the key in ExplanationMetadata.outputs. If the Model predicted output has multiple dimensions, this is the value in the output located by output_index.
"outputDisplayName": "A String", # Output only. The display name of the output identified by output_index. For example, the predicted class name by a multi-classification Model. This field is only populated iff the Model predicts display names as a separate field along with the explained output. The predicted display name must has the same shape of the explained output, and can be located using output_index.
"outputIndex": [ # Output only. The index that locates the explained prediction output. If the prediction output is a scalar value, output_index is not populated. If the prediction output has multiple dimensions, the length of the output_index list is the same as the number of dimensions of the output. The i-th element in output_index is the element index of the i-th dimension of the output vector. Indices start from 0.
42,
],
"outputName": "A String", # Output only. Name of the explain output. Specified as the key in ExplanationMetadata.outputs.
},
],
},
"name": "A String", # Output only. The resource name of the ModelEvaluationSlice.
"slice": { # Definition of a slice. # Output only. The slice of the test data that is used to evaluate the Model.
"dimension": "A String", # Output only. The dimension of the slice. Well-known dimensions are: * `annotationSpec`: This slice is on the test data that has either ground truth or prediction with AnnotationSpec.display_name equals to value. * `slice`: This slice is a user customized slice defined by its SliceSpec.
"sliceSpec": { # Specification for how the data should be sliced. # Output only. Specification for how the data was sliced.
"configs": { # Mapping configuration for this SliceSpec. The key is the name of the feature. By default, the key will be prefixed by "instance" as a dictionary prefix for Vertex Batch Predictions output format.
"a_key": { # Specification message containing the config for this SliceSpec. When `kind` is selected as `value` and/or `range`, only a single slice will be computed. When `all_values` is present, a separate slice will be computed for each possible label/value for the corresponding key in `config`. Examples, with feature zip_code with values 12345, 23334, 88888 and feature country with values "US", "Canada", "Mexico" in the dataset: Example 1: { "zip_code": { "value": { "float_value": 12345.0 } } } A single slice for any data with zip_code 12345 in the dataset. Example 2: { "zip_code": { "range": { "low": 12345, "high": 20000 } } } A single slice containing data where the zip_codes between 12345 and 20000 For this example, data with the zip_code of 12345 will be in this slice. Example 3: { "zip_code": { "range": { "low": 10000, "high": 20000 } }, "country": { "value": { "string_value": "US" } } } A single slice containing data where the zip_codes between 10000 and 20000 has the country "US". For this example, data with the zip_code of 12345 and country "US" will be in this slice. Example 4: { "country": {"all_values": { "value": true } } } Three slices are computed, one for each unique country in the dataset. Example 5: { "country": { "all_values": { "value": true } }, "zip_code": { "value": { "float_value": 12345.0 } } } Three slices are computed, one for each unique country in the dataset where the zip_code is also 12345. For this example, data with zip_code 12345 and country "US" will be in one slice, zip_code 12345 and country "Canada" in another slice, and zip_code 12345 and country "Mexico" in another slice, totaling 3 slices.
"allValues": True or False, # If all_values is set to true, then all possible labels of the keyed feature will have another slice computed. Example: `{"all_values":{"value":true}}`
"range": { # A range of values for slice(s). `low` is inclusive, `high` is exclusive. # A range of values for a numerical feature. Example: `{"range":{"low":10000.0,"high":50000.0}}` will capture 12345 and 23334 in the slice.
"high": 3.14, # Exclusive high value for the range.
"low": 3.14, # Inclusive low value for the range.
},
"value": { # Single value that supports strings and floats. # A unique specific value for a given feature. Example: `{ "value": { "string_value": "12345" } }`
"floatValue": 3.14, # Float type.
"stringValue": "A String", # String type.
},
},
},
},
"value": "A String", # Output only. The value of the dimension in this slice.
},
}

list(parent, filter=None, pageSize=None, pageToken=None, readMask=None, x__xgafv=None)

Lists ModelEvaluationSlices in a ModelEvaluation.

Args:
parent: string, Required. The resource name of the ModelEvaluation to list the ModelEvaluationSlices from. Format: `projects/{project}/locations/{location}/models/{model}/evaluations/{evaluation}` (required)
filter: string, The standard list filter. * `slice.dimension` - for =.
pageSize: integer, The standard list page size.
pageToken: string, The standard list page token. Typically obtained via ListModelEvaluationSlicesResponse.next_page_token of the previous ModelService.ListModelEvaluationSlices call.
readMask: string, Mask specifying which fields to read.
x__xgafv: string, V1 error format.
Allowed values
1 - v1 error format
2 - v2 error format

Returns:
An object of the form:

{ # Response message for ModelService.ListModelEvaluationSlices.
"modelEvaluationSlices": [ # List of ModelEvaluations in the requested page.
{ # A collection of metrics calculated by comparing Model's predictions on a slice of the test data against ground truth annotations.
"createTime": "A String", # Output only. Timestamp when this ModelEvaluationSlice was created.
"metrics": "", # Output only. Sliced evaluation metrics of the Model. The schema of the metrics is stored in metrics_schema_uri
"metricsSchemaUri": "A String", # Output only. Points to a YAML file stored on Google Cloud Storage describing the metrics of this ModelEvaluationSlice. The schema is defined as an OpenAPI 3.0.2 [Schema Object](https://github.com/OAI/OpenAPI-Specification/blob/main/versions/3.0.2.md#schemaObject).
"modelExplanation": { # Aggregated explanation metrics for a Model over a set of instances. # Output only. Aggregated explanation metrics for the Model's prediction output over the data this ModelEvaluation uses. This field is populated only if the Model is evaluated with explanations, and only for tabular Models.
"meanAttributions": [ # Output only. Aggregated attributions explaining the Model's prediction outputs over the set of instances. The attributions are grouped by outputs. For Models that predict only one output, such as regression Models that predict only one score, there is only one attibution that explains the predicted output. For Models that predict multiple outputs, such as multiclass Models that predict multiple classes, each element explains one specific item. Attribution.output_index can be used to identify which output this attribution is explaining. The baselineOutputValue, instanceOutputValue and featureAttributions fields are averaged over the test data. NOTE: Currently AutoML tabular classification Models produce only one attribution, which averages attributions over all the classes it predicts. Attribution.approximation_error is not populated.
{ # Attribution that explains a particular prediction output.
"approximationError": 3.14, # Output only. Error of feature_attributions caused by approximation used in the explanation method. Lower value means more precise attributions. * For Sampled Shapley attribution, increasing path_count might reduce the error. * For Integrated Gradients attribution, increasing step_count might reduce the error. * For XRAI attribution, increasing step_count might reduce the error. See [this introduction](/vertex-ai/docs/explainable-ai/overview) for more information.
"baselineOutputValue": 3.14, # Output only. Model predicted output if the input instance is constructed from the baselines of all the features defined in ExplanationMetadata.inputs. The field name of the output is determined by the key in ExplanationMetadata.outputs. If the Model's predicted output has multiple dimensions (rank > 1), this is the value in the output located by output_index. If there are multiple baselines, their output values are averaged.
"featureAttributions": "", # Output only. Attributions of each explained feature. Features are extracted from the prediction instances according to explanation metadata for inputs. The value is a struct, whose keys are the name of the feature. The values are how much the feature in the instance contributed to the predicted result. The format of the value is determined by the feature's input format: * If the feature is a scalar value, the attribution value is a floating number. * If the feature is an array of scalar values, the attribution value is an array. * If the feature is a struct, the attribution value is a struct. The keys in the attribution value struct are the same as the keys in the feature struct. The formats of the values in the attribution struct are determined by the formats of the values in the feature struct. The ExplanationMetadata.feature_attributions_schema_uri field, pointed to by the ExplanationSpec field of the Endpoint.deployed_models object, points to the schema file that describes the features and their attribution values (if it is populated).
"instanceOutputValue": 3.14, # Output only. Model predicted output on the corresponding explanation instance. The field name of the output is determined by the key in ExplanationMetadata.outputs. If the Model predicted output has multiple dimensions, this is the value in the output located by output_index.
"outputDisplayName": "A String", # Output only. The display name of the output identified by output_index. For example, the predicted class name by a multi-classification Model. This field is only populated iff the Model predicts display names as a separate field along with the explained output. The predicted display name must has the same shape of the explained output, and can be located using output_index.
"outputIndex": [ # Output only. The index that locates the explained prediction output. If the prediction output is a scalar value, output_index is not populated. If the prediction output has multiple dimensions, the length of the output_index list is the same as the number of dimensions of the output. The i-th element in output_index is the element index of the i-th dimension of the output vector. Indices start from 0.
42,
],
"outputName": "A String", # Output only. Name of the explain output. Specified as the key in ExplanationMetadata.outputs.
},
],
},
"name": "A String", # Output only. The resource name of the ModelEvaluationSlice.
"slice": { # Definition of a slice. # Output only. The slice of the test data that is used to evaluate the Model.
"dimension": "A String", # Output only. The dimension of the slice. Well-known dimensions are: * `annotationSpec`: This slice is on the test data that has either ground truth or prediction with AnnotationSpec.display_name equals to value. * `slice`: This slice is a user customized slice defined by its SliceSpec.
"sliceSpec": { # Specification for how the data should be sliced. # Output only. Specification for how the data was sliced.
"configs": { # Mapping configuration for this SliceSpec. The key is the name of the feature. By default, the key will be prefixed by "instance" as a dictionary prefix for Vertex Batch Predictions output format.
"a_key": { # Specification message containing the config for this SliceSpec. When `kind` is selected as `value` and/or `range`, only a single slice will be computed. When `all_values` is present, a separate slice will be computed for each possible label/value for the corresponding key in `config`. Examples, with feature zip_code with values 12345, 23334, 88888 and feature country with values "US", "Canada", "Mexico" in the dataset: Example 1: { "zip_code": { "value": { "float_value": 12345.0 } } } A single slice for any data with zip_code 12345 in the dataset. Example 2: { "zip_code": { "range": { "low": 12345, "high": 20000 } } } A single slice containing data where the zip_codes between 12345 and 20000 For this example, data with the zip_code of 12345 will be in this slice. Example 3: { "zip_code": { "range": { "low": 10000, "high": 20000 } }, "country": { "value": { "string_value": "US" } } } A single slice containing data where the zip_codes between 10000 and 20000 has the country "US". For this example, data with the zip_code of 12345 and country "US" will be in this slice. Example 4: { "country": {"all_values": { "value": true } } } Three slices are computed, one for each unique country in the dataset. Example 5: { "country": { "all_values": { "value": true } }, "zip_code": { "value": { "float_value": 12345.0 } } } Three slices are computed, one for each unique country in the dataset where the zip_code is also 12345. For this example, data with zip_code 12345 and country "US" will be in one slice, zip_code 12345 and country "Canada" in another slice, and zip_code 12345 and country "Mexico" in another slice, totaling 3 slices.
"allValues": True or False, # If all_values is set to true, then all possible labels of the keyed feature will have another slice computed. Example: `{"all_values":{"value":true}}`
"range": { # A range of values for slice(s). `low` is inclusive, `high` is exclusive. # A range of values for a numerical feature. Example: `{"range":{"low":10000.0,"high":50000.0}}` will capture 12345 and 23334 in the slice.
"high": 3.14, # Exclusive high value for the range.
"low": 3.14, # Inclusive low value for the range.
},
"value": { # Single value that supports strings and floats. # A unique specific value for a given feature. Example: `{ "value": { "string_value": "12345" } }`
"floatValue": 3.14, # Float type.
"stringValue": "A String", # String type.
},
},
},
},
"value": "A String", # Output only. The value of the dimension in this slice.
},
},
],
"nextPageToken": "A String", # A token to retrieve next page of results. Pass to ListModelEvaluationSlicesRequest.page_token to obtain that page.
}

Vertex AI API . projects . locations . models . evaluations . slices

Instance Methods

Method Details