Retrieve information about LineageProcesses associated with specific links. LineageProcesses are transformation pipelines that result in data flowing from **source** to **target** assets. Links between assets represent this operation. If you have specific link names, you can use this method to verify which LineageProcesses contribute to creating those links. See the SearchLinks method for more information on how to retrieve link name. You can retrieve the LineageProcess information in every project where you have the `datalineage.events.get` permission. The project provided in the URL is used for Billing and Quota.

Creates new lineage events together with their parents: process and run. Updates the process and run if they already exist. Mapped from Open Lineage specification: https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json.

Retrieves a streaming response of lineage links connected to the requested assets by performing a breadth-first search in the given direction. Links represent the data flow between **source** (upstream) and **target** (downstream) assets in transformation pipelines. Links are stored in the same project as the Lineage Events that create them. This method retrieves links from all valid locations provided in the request. This method supports Column-Level Lineage (CLL) along with wildcard support to retrieve all CLL for an Entity FQN. Following permissions are required to retrieve links: * `datalineage.events.get` permission for the project where the link is stored for entity-level lineage. * `datalineage.events.getFields` permission for the project where the link is stored for column-level lineage. This method also returns processes that created the links if explicitly requested by setting [max_process_per_link](google.cloud.datacatalog.lineage.v1.SearchLineageStreamingRequest.limits.max_process_per_link) is non-zero and full process details are requested via `links.processes.process` in the [FieldMask](https://developers.google.com/workspace/docs/api/how-tos/field-masks#read_with_a_field_mask). Permission required to retrieve processes: * `datalineage.processes.get` permission for the project where the process is stored.

Retrieve a list of links connected to a specific asset. Links represent the data flow between **source** (upstream) and **target** (downstream) assets in transformation pipelines. Links are stored in the same project as the Lineage Events that create them. You can retrieve links in every project where you have the `datalineage.events.get` permission. The project provided in the URL is used for Billing and Quota.

Method Details

batchSearchLinkProcesses(parent, body=None, x__xgafv=None)

Args:
parent: string, Required. The project and location where you want to search. (required)
body: object, The request body.
The object takes the form of:

{ # Request message for BatchSearchLinkProcesses.
"links": [ # Required. An array of links to check for their associated LineageProcesses. The maximum number of items in this array is 100. If the request contains more than 100 links, it returns the `INVALID_ARGUMENT` error. Format: `projects/{project}/locations/{location}/links/{link}`.
"A String",
],
"pageSize": 42, # Optional. The maximum number of processes to return in a single page of the response. A page may contain fewer results than this value.
"pageToken": "A String", # The page token received from a previous `BatchSearchLinkProcesses` call. Use it to get the next page. When requesting subsequent pages of a response, remember that all parameters must match the values you provided in the original request.
}

x__xgafv: string, V1 error format.
Allowed values
1 - v1 error format
2 - v2 error format

Returns:
An object of the form:

{ # Response message for BatchSearchLinkProcesses.
"nextPageToken": "A String", # The token to specify as `page_token` in the subsequent call to get the next page. Omitted if there are no more pages in the response.
"processLinks": [ # An array of processes associated with the specified links.
{ # Links associated with a specific process.
"links": [ # An array containing link details objects of the links provided in the original request. A single process can result in creating multiple links. If any of the links you provide in the request are created by the same process, they all are included in this array.
{ # Link details.
"endTime": "A String", # The end of the last event establishing this link-process tuple.
"link": "A String", # The name of the link in the format of `projects/{project}/locations/{location}/links/{link}`.
"startTime": "A String", # The start of the first event establishing this link-process tuple.
},
],
"process": "A String", # The process name in the format of `projects/{project}/locations/{location}/processes/{process}`.
},
],
}

batchSearchLinkProcesses_next()

Retrieves the next page of results.

        Args:
          previous_request: The request for the previous page. (required)
          previous_response: The response from the request for the previous page. (required)

        Returns:
          A request object that you can call 'execute()' on to request the next
          page. Returns None if there are no more items in the collection.

close()

Close httplib2 connections.

processOpenLineageRunEvent(parent, body=None, requestId=None, x__xgafv=None)

Creates new lineage events together with their parents: process and run. Updates the process and run if they already exist. Mapped from Open Lineage specification: https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json.

Args:
  parent: string, Required. The name of the project and its location that should own the process, run, and lineage event. (required)
  body: object, The request body.
  requestId: string, Optional. A unique identifier for this request. Restricted to 36 ASCII characters. A random UUID is recommended. This request is idempotent only if a `request_id` is provided.
  x__xgafv: string, V1 error format.
    Allowed values
      1 - v1 error format
      2 - v2 error format

Returns:
  An object of the form:

    { # Response message for ProcessOpenLineageRunEvent.
  "lineageEvents": [ # Created lineage event names. Format: `projects/{project}/locations/{location}/processes/{process}/runs/{run}/lineageEvents/{lineage_event}`.
    "A String",
  ],
  "process": "A String", # Created process name. Format: `projects/{project}/locations/{location}/processes/{process}`.
  "run": "A String", # Created run name. Format: `projects/{project}/locations/{location}/processes/{process}/runs/{run}`.
}

searchLineageStreaming(parent, body=None, x__xgafv=None)

Args:
parent: string, Required. The project and location to initiate the search from. (required)
body: object, The request body.
The object takes the form of:

{ # Request message for SearchLineageStreaming.
"direction": "A String", # Required. Direction of the search.
"filters": { # Filters for the search. # Optional. Filters for the search.
"dependencyTypes": [ # Optional. Types of dependencies between entities to retrieve. If unspecified, all dependency types are returned.
"A String",
],
"entitySet": "A String", # Optional. Entity set restriction. If unspecified, the method returns all entities.
"timeRange": { # Represents a time interval, encoded as a Timestamp start (inclusive) and a Timestamp end (exclusive). The start must be less than or equal to the end. When the start equals the end, the interval is empty (matches no time). When both start and end are unspecified, the interval matches any time. # Optional. Time interval to search for lineage. If unspecified, all lineage is returned. Currently, at most one of `start_time` and `end_time` can be set.
"endTime": "A String", # Optional. Exclusive end of the interval. If specified, a Timestamp matching this interval will have to be before the end.
"startTime": "A String", # Optional. Inclusive start of the interval. If specified, a Timestamp matching this interval will have to be the same or after the start.
},
},
"limits": { # Limits for the search results. # Optional. Limits for the search.
"maxDepth": 42, # Optional. The maximum depth of the search. The default value is 5 and maximum value is 100.
"maxProcessPerLink": 42, # Optional. The maximum number of processes to return per link. The default value is 0 and the maximum value is 100. If this value is non-zero, the response will contain process names for the links. To retrieve full process details in the response, include `links.processes.process` in the [FieldMask](https://developers.google.com/workspace/docs/api/how-tos/field-masks#read_with_a_field_mask).
"maxResults": 42, # Optional. The maximum number of links to return in the response. The default value is 1_000 and the maximum value is 10_000.
},
"locations": [ # Required. The locations to search in.
"A String",
],
"rootCriteria": { # Criteria for the root of the search. # Required. Criteria for the root of the search.
"entities": { # Multiple entity reference for SearchLinksRequest. # Optional. The entities to initiate the search from. Entities can be specified by FQN only, or by FQN and field. To search by FQN and all available fields for that FQN, use the wildcard `*` as the field value.
"entities": [ # Optional. The list of entities to search for links. The maximum number of entities is 20.
{ # The soft reference to everything you can attach a lineage event to.
"field": [ # Optional. Field path within the entity. Each nesting level should be a separate value in the repeated field. The order matters. Must be empty for asset level lineage For example to address "salary.net" subfield where "salary" is a column and "net" is a proto field two values in the `field` should be reported, the first is "salary" and the second is "net". Each field length is limited to 500 characters. Maximum supported nesting level is 20.
"A String",
],
"fullyQualifiedName": "A String", # Required. [Fully Qualified Name (FQN)](https://cloud.google.com/dataplex/docs/fully-qualified-names) of the entity.
},
],
},
},
}

x__xgafv: string, V1 error format.
Allowed values
1 - v1 error format
2 - v2 error format

Returns:
An object of the form:

{ # Response message for SearchLineageStreaming.
"links": [ # Output only. The lineage links that match the search criteria. Can be empty if no links match.
{ # Lineage link between two entities.
"dependencyInfo": [ # Describes how the target entity is dependent on the source entity.
{ # Dependency info describes how one entity is dependent on another.
"dependencyType": "A String", # The type of dependency.
},
],
"depth": 42, # Depth of the current link in the graph starting from 1.
"location": "A String", # The location where the LineageEvent that created the link is stored.
"processes": [ # Processes metadata associated with the link.
{ # Process metadata for the link.
"process": { # A process is the definition of a data transformation operation. # Process that created the link.
"attributes": { # Optional. The attributes of the process. Should only be used for the purpose of non-semantic management (classifying, describing or labeling the process). Up to 100 attributes are allowed.
"a_key": "",
},
"displayName": "A String", # Optional. A human-readable name you can set to display in a user interface. Must be not longer than 200 characters and only contain UTF-8 letters or numbers, spaces or characters like `_-:&.`
"name": "A String", # Immutable. The resource name of the lineage process. Format: `projects/{project}/locations/{location}/processes/{process}`. Can be specified or auto-assigned. {process} must be not longer than 200 characters and only contain characters in a set: `a-zA-Z0-9_-:.`
"origin": { # Origin of a process. # Optional. The origin of this process and its runs and lineage events.
"name": "A String", # If the source_type isn't CUSTOM, the value of this field should be a Google Cloud resource name of the system, which reports lineage. The project and location parts of the resource name must match the project and location of the lineage resource being created. Examples: - `{source_type: COMPOSER, name: "projects/foo/locations/us/environments/bar"}` - `{source_type: BIGQUERY, name: "projects/foo/locations/eu"}` - `{source_type: CUSTOM, name: "myCustomIntegration"}`
"sourceType": "A String", # Type of the source. Use of a source_type other than `CUSTOM` for process creation or updating is highly discouraged. It might be restricted in the future without notice. There will be increase in cost if you use any of the source types other than `CUSTOM`.
},
},
},
],
"source": { # The soft reference to everything you can attach a lineage event to. # The entity that is the **source** of this link.
"field": [ # Optional. Field path within the entity. Each nesting level should be a separate value in the repeated field. The order matters. Must be empty for asset level lineage For example to address "salary.net" subfield where "salary" is a column and "net" is a proto field two values in the `field` should be reported, the first is "salary" and the second is "net". Each field length is limited to 500 characters. Maximum supported nesting level is 20.
"A String",
],
"fullyQualifiedName": "A String", # Required. [Fully Qualified Name (FQN)](https://cloud.google.com/dataplex/docs/fully-qualified-names) of the entity.
},
"target": { # The soft reference to everything you can attach a lineage event to. # The entity that is the **target** of this link.
"field": [ # Optional. Field path within the entity. Each nesting level should be a separate value in the repeated field. The order matters. Must be empty for asset level lineage For example to address "salary.net" subfield where "salary" is a column and "net" is a proto field two values in the `field` should be reported, the first is "salary" and the second is "net". Each field length is limited to 500 characters. Maximum supported nesting level is 20.
"A String",
],
"fullyQualifiedName": "A String", # Required. [Fully Qualified Name (FQN)](https://cloud.google.com/dataplex/docs/fully-qualified-names) of the entity.
},
},
],
"unreachable": [ # Unordered list. Unreachable resources. If non-empty, the result set might be incomplete. Currently, only locations are supported. Format: `projects/[PROJECT_NUMBER]/locations/[LOCATION]` Example: projects/123456789/locations/us-east1
"A String",
],
}

searchLinks(parent, body=None, x__xgafv=None)

Args:
parent: string, Required. The project and location you want search in. (required)
body: object, The request body.
The object takes the form of:

{ # Request message for SearchLinks.
"pageSize": 42, # Optional. The maximum number of links to return in a single page of the response. A page may contain fewer links than this value. If unspecified, at most 10 links are returned. Maximum value is 100; values greater than 100 are reduced to 100.
"pageToken": "A String", # Optional. The page token received from a previous `SearchLinksRequest` call. Use it to get the next page. When requesting subsequent pages of a response, remember that all parameters must match the values you provided in the original request.
"source": { # The soft reference to everything you can attach a lineage event to. # Optional. Send asset information in the **source** field to retrieve all links that lead from the specified asset to downstream assets.
"field": [ # Optional. Field path within the entity. Each nesting level should be a separate value in the repeated field. The order matters. Must be empty for asset level lineage For example to address "salary.net" subfield where "salary" is a column and "net" is a proto field two values in the `field` should be reported, the first is "salary" and the second is "net". Each field length is limited to 500 characters. Maximum supported nesting level is 20.
"A String",
],
"fullyQualifiedName": "A String", # Required. [Fully Qualified Name (FQN)](https://cloud.google.com/dataplex/docs/fully-qualified-names) of the entity.
},
"sources": { # Multiple entity reference for SearchLinksRequest. # Optional. Send a list of asset information in the **sources** field to retrieve all links that lead from the specified assets to downstream assets. This field is similar to the `source` source field but allows providing multiple entities. All entities within the `MultipleEntityReference` must have the same `fully_qualified_name`.
"entities": [ # Optional. The list of entities to search for links. The maximum number of entities is 20.
{ # The soft reference to everything you can attach a lineage event to.
"field": [ # Optional. Field path within the entity. Each nesting level should be a separate value in the repeated field. The order matters. Must be empty for asset level lineage For example to address "salary.net" subfield where "salary" is a column and "net" is a proto field two values in the `field` should be reported, the first is "salary" and the second is "net". Each field length is limited to 500 characters. Maximum supported nesting level is 20.
"A String",
],
"fullyQualifiedName": "A String", # Required. [Fully Qualified Name (FQN)](https://cloud.google.com/dataplex/docs/fully-qualified-names) of the entity.
},
],
},
"target": { # The soft reference to everything you can attach a lineage event to. # Optional. Send asset information in the **target** field to retrieve all links that lead from upstream assets to the specified asset.
"field": [ # Optional. Field path within the entity. Each nesting level should be a separate value in the repeated field. The order matters. Must be empty for asset level lineage For example to address "salary.net" subfield where "salary" is a column and "net" is a proto field two values in the `field` should be reported, the first is "salary" and the second is "net". Each field length is limited to 500 characters. Maximum supported nesting level is 20.
"A String",
],
"fullyQualifiedName": "A String", # Required. [Fully Qualified Name (FQN)](https://cloud.google.com/dataplex/docs/fully-qualified-names) of the entity.
},
"targets": { # Multiple entity reference for SearchLinksRequest. # Optional. Send a list of asset information in the **targets** field to retrieve all links that lead from upstream assets to the specified assets. This field is similar to the `target` target field but allows providing multiple entities. All entities within the `MultipleEntityReference` must have the same `fully_qualified_name`.
"entities": [ # Optional. The list of entities to search for links. The maximum number of entities is 20.
{ # The soft reference to everything you can attach a lineage event to.
"field": [ # Optional. Field path within the entity. Each nesting level should be a separate value in the repeated field. The order matters. Must be empty for asset level lineage For example to address "salary.net" subfield where "salary" is a column and "net" is a proto field two values in the `field` should be reported, the first is "salary" and the second is "net". Each field length is limited to 500 characters. Maximum supported nesting level is 20.
"A String",
],
"fullyQualifiedName": "A String", # Required. [Fully Qualified Name (FQN)](https://cloud.google.com/dataplex/docs/fully-qualified-names) of the entity.
},
],
},
}

x__xgafv: string, V1 error format.
Allowed values
1 - v1 error format
2 - v2 error format

Returns:
An object of the form:

{ # Response message for SearchLinks.
"links": [ # The list of links for a given asset. Can be empty if the asset has no relations of requested type (source or target).
{ # Links represent the data flow between **source** (upstream) and **target** (downstream) assets in transformation pipelines. Links are created when LineageEvents record data transformation between related assets.
"dependencyInfo": [ # Optional. The dependency info of the link (applies only to column level links).
{ # Dependency info describes how one entity depends on another.
"dependencyType": "A String", # The type of dependency.
},
],
"endTime": "A String", # The end of the last event establishing this link.
"name": "A String", # Output only. Immutable. The name of the link. Format: `projects/{project}/locations/{location}/links/{link}`.
"source": { # The soft reference to everything you can attach a lineage event to. # The pointer to the entity that is the **source** of this link.
"field": [ # Optional. Field path within the entity. Each nesting level should be a separate value in the repeated field. The order matters. Must be empty for asset level lineage For example to address "salary.net" subfield where "salary" is a column and "net" is a proto field two values in the `field` should be reported, the first is "salary" and the second is "net". Each field length is limited to 500 characters. Maximum supported nesting level is 20.
"A String",
],
"fullyQualifiedName": "A String", # Required. [Fully Qualified Name (FQN)](https://cloud.google.com/dataplex/docs/fully-qualified-names) of the entity.
},
"startTime": "A String", # The start of the first event establishing this link.
"target": { # The soft reference to everything you can attach a lineage event to. # The pointer to the entity that is the **target** of this link.
"field": [ # Optional. Field path within the entity. Each nesting level should be a separate value in the repeated field. The order matters. Must be empty for asset level lineage For example to address "salary.net" subfield where "salary" is a column and "net" is a proto field two values in the `field` should be reported, the first is "salary" and the second is "net". Each field length is limited to 500 characters. Maximum supported nesting level is 20.
"A String",
],
"fullyQualifiedName": "A String", # Required. [Fully Qualified Name (FQN)](https://cloud.google.com/dataplex/docs/fully-qualified-names) of the entity.
},
},
],
"nextPageToken": "A String", # The token to specify as `page_token` in the subsequent call to get the next page. Omitted if there are no more pages in the response.
}

Data Lineage API . projects . locations

Instance Methods

Method Details