google.resumable_media.requests.upload

Support for resumable uploads.

Also supported here are simple (media) uploads and multipart uploads that contain both metadata and a small file as payload.

class google.resumable_media.requests.upload.MultipartUpload(upload_url, headers=None)

Bases: google.resumable_media.requests._helpers.RequestsMixin, google.resumable_media._upload.MultipartUpload

Upload a resource with metadata to a Google API.

A multipart upload sends both metadata and the resource in a single (multipart) request.

Parameters:
  • upload_url (str) – The URL where the content will be uploaded.
  • headers (Optional [ Mapping [ str, str ] ]) – Extra headers that should be sent with the request, e.g. headers for encrypted data.
upload_url

The URL where the content will be uploaded.

Type:str
property finished

Flag indicating if the upload has completed.

Type:bool
transmit(transport, data, metadata, content_type)

Transmit the resource to be uploaded.

Parameters:
  • transport (Session) – A requests object which can make authenticated requests.
  • data (bytes) – The resource content to be uploaded.
  • metadata (Mapping [ str, str ]) – The resource metadata, such as an ACL list.
  • content_type (str) – The content type of the resource, e.g. a JPEG image has content type image/jpeg.
Returns:

The HTTP response returned by transport.

Return type:

Response

class google.resumable_media.requests.upload.ResumableUpload(upload_url, chunk_size, headers=None)

Bases: google.resumable_media.requests._helpers.RequestsMixin, google.resumable_media._upload.ResumableUpload

Initiate and fulfill a resumable upload to a Google API.

A resumable upload sends an initial request with the resource metadata and then gets assigned an upload ID / upload URL to send bytes to. Using the upload URL, the upload is then done in chunks (determined by the user) until all bytes have been uploaded.

When constructing a resumable upload, only the resumable upload URL and the chunk size are required:

>>> from google.resumable_media.requests import ResumableUpload
>>>
>>> url_template = (
...     u'https://www.googleapis.com/upload/storage/v1/b/{bucket}/o?'
...     u'uploadType=resumable')
>>> upload_url = url_template.format(bucket=bucket)
>>>
>>> chunk_size = 3 * 1024 * 1024  # 3MB
>>> upload = ResumableUpload(upload_url, chunk_size)

When initiating an upload (via initiate()), the caller is expected to pass the resource being uploaded as a file-like stream. If the size of the resource is explicitly known, it can be passed in directly:

>>> import os
>>>
>>> upload.total_bytes is None
True
>>>
>>> stream = open(filename, u'rb')
>>> total_bytes = os.path.getsize(filename)
>>> metadata = {u'name': filename}
>>> response = upload.initiate(
...     transport, stream, metadata, u'text/plain',
...     total_bytes=total_bytes)
>>> response
<Response [200]>
>>>
>>> upload.total_bytes == total_bytes
True

If the stream is in a “final” state (i.e. it won’t have any more bytes written to it), the total number of bytes can be determined implicitly from the stream itself:

>>> stream = io.BytesIO(data)
>>> response = upload.initiate(
...     transport, stream, metadata, content_type)
>>>
>>> upload.total_bytes == len(data)
True

If the size of the resource is unknown when the upload is initiated, the stream_final argument can be used. This might occur if the resource is being dynamically created on the client (e.g. application logs). To use this argument:

>>> response = upload.initiate(
...     transport, stream, metadata, content_type,
...     stream_final=False)
>>>
>>> upload.total_bytes is None
True
Parameters:
  • upload_url (str) – The URL where the resumable upload will be initiated.
  • chunk_size (int) – The size of each chunk used to upload the resource.
  • headers (Optional [ Mapping [ str, str ] ]) – Extra headers that should be sent with the initiate() request, e.g. headers for encrypted data. These will not be sent with transmit_next_chunk() or recover() requests.
upload_url

The URL where the content will be uploaded.

Type:str
Raises:ValueError – If chunk_size is not a multiple of UPLOAD_CHUNK_SIZE.
property bytes_uploaded

Number of bytes that have been uploaded.

Type:int
property chunk_size

The size of each chunk used to upload the resource.

Type:int
property finished

Flag indicating if the upload has completed.

Type:bool
initiate(transport, stream, metadata, content_type, total_bytes=None, stream_final=True)

Initiate a resumable upload.

By default, this method assumes your stream is in a “final” state ready to transmit. However, stream_final=False can be used to indicate that the size of the resource is not known. This can happen if bytes are being dynamically fed into stream, e.g. if the stream is attached to application logs.

If stream_final=False is used, chunk_size bytes will be read from the stream every time transmit_next_chunk() is called. If one of those reads produces strictly fewer bites than the chunk size, the upload will be concluded.

Parameters:
  • transport (Session) – A requests object which can make authenticated requests.
  • stream (IO[bytes]) – The stream (i.e. file-like object) that will be uploaded. The stream must be at the beginning (i.e. stream.tell() == 0).
  • metadata (Mapping [ str, str ]) – The resource metadata, such as an ACL list.
  • content_type (str) – The content type of the resource, e.g. a JPEG image has content type image/jpeg.
  • total_bytes (Optional [ int ]) – The total number of bytes to be uploaded. If specified, the upload size will not be determined from the stream (even if stream_final=True).
  • stream_final (Optional [ bool ]) – Indicates if the stream is “final” (i.e. no more bytes will be added to it). In this case we determine the upload size from the size of the stream. If total_bytes is passed, this argument will be ignored.
Returns:

The HTTP response returned by transport.

Return type:

Response

property invalid

Indicates if the upload is in an invalid state.

This will occur if a call to transmit_next_chunk() fails. To recover from such a failure, call recover().

Type:bool
recover(transport)

Recover from a failure.

This method should be used when a ResumableUpload is in an invalid state due to a request failure.

This will verify the progress with the server and make sure the current upload is in a valid state before transmit_next_chunk() can be used again.

Parameters:transport (Session) – A requests object which can make authenticated requests.
Returns:The HTTP response returned by transport.
Return type:Response
property resumable_url

The URL of the in-progress resumable upload.

Type:Optional [ str ]
property total_bytes

The total number of bytes to be uploaded.

If this upload is initiated (via initiate()) with stream_final=True, this value will be populated based on the size of the stream being uploaded. (By default stream_final=True.)

If this upload is initiated with stream_final=False, total_bytes will be None since it cannot be determined from the stream.

Type:Optional [ int ]
transmit_next_chunk(transport)

Transmit the next chunk of the resource to be uploaded.

If the current upload was initiated with stream_final=False, this method will dynamically determine if the upload has completed. The upload will be considered complete if the stream produces fewer than chunk_size bytes when a chunk is read from it.

In the case of failure, an exception is thrown that preserves the failed response:

>>> error = None
>>> try:
...     upload.transmit_next_chunk(transport)
... except resumable_media.InvalidResponse as caught_exc:
...     error = caught_exc
...
>>> error
InvalidResponse('Request failed with status code', 400,
                'Expected one of', <HTTPStatus.OK: 200>, 308)
>>> error.response
<Response [400]>
Parameters:transport (Session) – A requests object which can make authenticated requests.
Returns:The HTTP response returned by transport.
Return type:Response
Raises:InvalidResponse – If the status code is not 200 or 308.
class google.resumable_media.requests.upload.SimpleUpload(upload_url, headers=None)

Bases: google.resumable_media.requests._helpers.RequestsMixin, google.resumable_media._upload.SimpleUpload

Upload a resource to a Google API.

A simple media upload sends no metadata and completes the upload in a single request.

Parameters:
  • upload_url (str) – The URL where the content will be uploaded.
  • headers (Optional [ Mapping [ str, str ] ]) – Extra headers that should be sent with the request, e.g. headers for encrypted data.
upload_url

The URL where the content will be uploaded.

Type:str
property finished

Flag indicating if the upload has completed.

Type:bool
transmit(transport, data, content_type)

Transmit the resource to be uploaded.

Parameters:
  • transport (Session) – A requests object which can make authenticated requests.
  • data (bytes) – The resource content to be uploaded.
  • content_type (str) – The content type of the resource, e.g. a JPEG image has content type image/jpeg.
Returns:

The HTTP response returned by transport.

Return type:

Response