Types for BigQuery Storage API Client#

class google.cloud.bigquery_storage_v1beta1.types.ArrowRecordBatch#

Arrow RecordBatch.

serialized_record_batch#

IPC serialized Arrow RecordBatch.

row_count#

The count of rows in the returning block.

row_count

Field google.cloud.bigquery.storage.v1beta1.ArrowRecordBatch.row_count

serialized_record_batch

Field google.cloud.bigquery.storage.v1beta1.ArrowRecordBatch.serialized_record_batch

class google.cloud.bigquery_storage_v1beta1.types.ArrowSchema#

Arrow schema.

serialized_schema#

IPC serialized Arrow schema.

serialized_schema

Field google.cloud.bigquery.storage.v1beta1.ArrowSchema.serialized_schema

class google.cloud.bigquery_storage_v1beta1.types.AvroRows#

Avro rows.

serialized_binary_rows#

Binary serialized rows in a block.

row_count#

The count of rows in the returning block.

row_count

Field google.cloud.bigquery.storage.v1beta1.AvroRows.row_count

serialized_binary_rows

Field google.cloud.bigquery.storage.v1beta1.AvroRows.serialized_binary_rows

class google.cloud.bigquery_storage_v1beta1.types.AvroSchema#

Avro schema.

schema#

Json serialized schema, as described at https://avro.apache.org/docs/1.8.1/spec.html

schema

Field google.cloud.bigquery.storage.v1beta1.AvroSchema.schema

class google.cloud.bigquery_storage_v1beta1.types.BatchCreateReadSessionStreamsRequest#

Information needed to request additional streams for an established read session.

session#

Required. Must be a non-expired session obtained from a call to CreateReadSession. Only the name field needs to be set.

requested_streams#

Required. Number of new streams requested. Must be positive. Number of added streams may be less than this, see CreateReadSessionRequest for more information.

requested_streams

Field google.cloud.bigquery.storage.v1beta1.BatchCreateReadSessionStreamsRequest.requested_streams

session

Field google.cloud.bigquery.storage.v1beta1.BatchCreateReadSessionStreamsRequest.session

class google.cloud.bigquery_storage_v1beta1.types.BatchCreateReadSessionStreamsResponse#

The response from BatchCreateReadSessionStreams returns the stream identifiers for the newly created streams.

streams#

Newly added streams.

streams

Field google.cloud.bigquery.storage.v1beta1.BatchCreateReadSessionStreamsResponse.streams

class google.cloud.bigquery_storage_v1beta1.types.CreateReadSessionRequest#

Creates a new read session, which may include additional options such as requested parallelism, projection filters and constraints.

table_reference#

Required. Reference to the table to read.

parent#

Required. String of the form projects/{project_id} indicating the project this ReadSession is associated with. This is the project that will be billed for usage.

table_modifiers#

Optional. Any modifiers to the Table (e.g. snapshot timestamp).

requested_streams#

Optional. Initial number of streams. If unset or 0, we will provide a value of streams so as to produce reasonable throughput. Must be non-negative. The number of streams may be lower than the requested number, depending on the amount parallelism that is reasonable for the table and the maximum amount of parallelism allowed by the system. Streams must be read starting from offset 0.

read_options#

Optional. Read options for this session (e.g. column selection, filters).

format#

Data output format. Currently default to Avro.

sharding_strategy#

The strategy to use for distributing data among multiple streams. Currently defaults to liquid sharding.

format

Field google.cloud.bigquery.storage.v1beta1.CreateReadSessionRequest.format

parent

Field google.cloud.bigquery.storage.v1beta1.CreateReadSessionRequest.parent

read_options

Field google.cloud.bigquery.storage.v1beta1.CreateReadSessionRequest.read_options

requested_streams

Field google.cloud.bigquery.storage.v1beta1.CreateReadSessionRequest.requested_streams

sharding_strategy

Field google.cloud.bigquery.storage.v1beta1.CreateReadSessionRequest.sharding_strategy

table_modifiers

Field google.cloud.bigquery.storage.v1beta1.CreateReadSessionRequest.table_modifiers

table_reference

Field google.cloud.bigquery.storage.v1beta1.CreateReadSessionRequest.table_reference

class google.cloud.bigquery_storage_v1beta1.types.Empty#
class google.cloud.bigquery_storage_v1beta1.types.FinalizeStreamRequest#

Request information for invoking FinalizeStream.

stream#

Stream to finalize.

stream

Field google.cloud.bigquery.storage.v1beta1.FinalizeStreamRequest.stream

class google.cloud.bigquery_storage_v1beta1.types.ReadRowsRequest#

Requesting row data via ReadRows must provide Stream position information.

read_position#

Required. Identifier of the position in the stream to start reading from. The offset requested must be less than the last row read from ReadRows. Requesting a larger offset is undefined.

read_position

Field google.cloud.bigquery.storage.v1beta1.ReadRowsRequest.read_position

class google.cloud.bigquery_storage_v1beta1.types.ReadRowsResponse#

Response from calling ReadRows may include row data, progress and throttling information.

rows#

Row data is returned in format specified during session creation.

avro_rows#

Serialized row data in AVRO format.

arrow_record_batch#

Serialized row data in Arrow RecordBatch format.

row_count#

Number of serialized rows in the rows block. This value is recorded here, in addition to the row_count values in the output-specific messages in rows, so that code which needs to record progress through the stream can do so in an output format-independent way.

status#

Estimated stream statistics.

throttle_status#

Throttling status. If unset, the latest response still describes the current throttling status.

arrow_record_batch

Field google.cloud.bigquery.storage.v1beta1.ReadRowsResponse.arrow_record_batch

avro_rows

Field google.cloud.bigquery.storage.v1beta1.ReadRowsResponse.avro_rows

row_count

Field google.cloud.bigquery.storage.v1beta1.ReadRowsResponse.row_count

status

Field google.cloud.bigquery.storage.v1beta1.ReadRowsResponse.status

throttle_status

Field google.cloud.bigquery.storage.v1beta1.ReadRowsResponse.throttle_status

class google.cloud.bigquery_storage_v1beta1.types.ReadSession#

Information returned from a CreateReadSession request.

name#

Unique identifier for the session, in the form projects/{pro ject_id}/locations/{location}/sessions/{session_id}.

expire_time#

Time at which the session becomes invalid. After this time, subsequent requests to read this Session will return errors.

schema#

The schema for the read. If read_options.selected_fields is set, the schema may be different from the table schema as it will only contain the selected fields.

avro_schema#

Avro schema.

arrow_schema#

Arrow schema.

streams#

Streams associated with this session.

table_reference#

Table that this ReadSession is reading from.

table_modifiers#

Any modifiers which are applied when reading from the specified table.

sharding_strategy#

The strategy to use for distributing data among the streams.

arrow_schema

Field google.cloud.bigquery.storage.v1beta1.ReadSession.arrow_schema

avro_schema

Field google.cloud.bigquery.storage.v1beta1.ReadSession.avro_schema

expire_time

Field google.cloud.bigquery.storage.v1beta1.ReadSession.expire_time

name

Field google.cloud.bigquery.storage.v1beta1.ReadSession.name

sharding_strategy

Field google.cloud.bigquery.storage.v1beta1.ReadSession.sharding_strategy

streams

Field google.cloud.bigquery.storage.v1beta1.ReadSession.streams

table_modifiers

Field google.cloud.bigquery.storage.v1beta1.ReadSession.table_modifiers

table_reference

Field google.cloud.bigquery.storage.v1beta1.ReadSession.table_reference

class google.cloud.bigquery_storage_v1beta1.types.SplitReadStreamRequest#

Request information for SplitReadStream.

original_stream#

Stream to split.

fraction#

A value in the range (0.0, 1.0) that specifies the fractional point at which the original stream should be split. The actual split point is evaluated on pre-filtered rows, so if a filter is provided, then there is no guarantee that the division of the rows between the new child streams will be proportional to this fractional value. Additionally, because the server-side unit for assigning data is collections of rows, this fraction will always map to to a data storage boundary on the server side.

fraction

Field google.cloud.bigquery.storage.v1beta1.SplitReadStreamRequest.fraction

original_stream

Field google.cloud.bigquery.storage.v1beta1.SplitReadStreamRequest.original_stream

class google.cloud.bigquery_storage_v1beta1.types.SplitReadStreamResponse#

Response from SplitReadStream.

primary_stream#

Primary stream, which contains the beginning portion of |original_stream|. An empty value indicates that the original stream can no longer be split.

remainder_stream#

Remainder stream, which contains the tail of |original_stream|. An empty value indicates that the original stream can no longer be split.

primary_stream

Field google.cloud.bigquery.storage.v1beta1.SplitReadStreamResponse.primary_stream

remainder_stream

Field google.cloud.bigquery.storage.v1beta1.SplitReadStreamResponse.remainder_stream

class google.cloud.bigquery_storage_v1beta1.types.Stream#

Information about a single data stream within a read session.

name#

Name of the stream, in the form projects/{project_id}/locati ons/{location}/streams/{stream_id}.

name

Field google.cloud.bigquery.storage.v1beta1.Stream.name

class google.cloud.bigquery_storage_v1beta1.types.StreamPosition#

Expresses a point within a given stream using an offset position.

stream#

Identifier for a given Stream.

offset#

Position in the stream.

offset

Field google.cloud.bigquery.storage.v1beta1.StreamPosition.offset

stream

Field google.cloud.bigquery.storage.v1beta1.StreamPosition.stream

class google.cloud.bigquery_storage_v1beta1.types.StreamStatus#

Progress information for a given Stream.

estimated_row_count#

Number of estimated rows in the current stream. May change over time as different readers in the stream progress at rates which are relatively fast or slow.

fraction_consumed#

A value in the range [0.0, 1.0] that represents the fraction of rows assigned to this stream that have been processed by the server. In the presence of read filters, the server may process more rows than it returns, so this value reflects progress through the pre-filtering rows. This value is only populated for sessions created through the BALANCED sharding strategy.

is_splittable#

Whether this stream can be split. For sessions that use the LIQUID sharding strategy, this value is always false. For BALANCED sessions, this value is false when enough data have been read such that no more splits are possible at that point or beyond. For small tables or streams that are the result of a chain of splits, this value may never be true.

estimated_row_count

Field google.cloud.bigquery.storage.v1beta1.StreamStatus.estimated_row_count

fraction_consumed

Field google.cloud.bigquery.storage.v1beta1.StreamStatus.fraction_consumed

is_splittable

Field google.cloud.bigquery.storage.v1beta1.StreamStatus.is_splittable

class google.cloud.bigquery_storage_v1beta1.types.TableModifiers#

All fields in this message optional.

snapshot_time#

The snapshot time of the table. If not set, interpreted as now.

snapshot_time

Field google.cloud.bigquery.storage.v1beta1.TableModifiers.snapshot_time

class google.cloud.bigquery_storage_v1beta1.types.TableReadOptions#

Options dictating how we read a table.

selected_fields#

Optional. Names of the fields in the table that should be read. If empty, all fields will be read. If the specified field is a nested field, all the sub-fields in the field will be selected. The output field order is unrelated to the order of fields in selected_fields.

row_restriction#

Optional. SQL text filtering statement, similar to a WHERE clause in a query. Currently, only a single predicate that is a comparison between a column and a constant value is supported. Aggregates are not supported. Examples: “int_field > 5” “date_field = CAST(‘2014-9-27’ as DATE)” “nullable_field is not NULL” “st_equals(geo_field, st_geofromtext(“POINT(2, 2)”))” “numeric_field BETWEEN 1.0 AND 5.0”

row_restriction

Field google.cloud.bigquery.storage.v1beta1.TableReadOptions.row_restriction

selected_fields

Field google.cloud.bigquery.storage.v1beta1.TableReadOptions.selected_fields

class google.cloud.bigquery_storage_v1beta1.types.TableReference#

Table reference that includes just the 3 strings needed to identify a table.

project_id#

The assigned project ID of the project.

dataset_id#

The ID of the dataset in the above project.

table_id#

The ID of the table in the above dataset.

dataset_id

Field google.cloud.bigquery.storage.v1beta1.TableReference.dataset_id

project_id

Field google.cloud.bigquery.storage.v1beta1.TableReference.project_id

table_id

Field google.cloud.bigquery.storage.v1beta1.TableReference.table_id

class google.cloud.bigquery_storage_v1beta1.types.ThrottleStatus#

Information on if the current connection is being throttled.

throttle_percent#

How much this connection is being throttled. 0 is no throttling, 100 is completely throttled.

throttle_percent

Field google.cloud.bigquery.storage.v1beta1.ThrottleStatus.throttle_percent

class google.cloud.bigquery_storage_v1beta1.types.Timestamp#
nanos#

Field google.protobuf.Timestamp.nanos

seconds#

Field google.protobuf.Timestamp.seconds