Skip to content

fix(storage): remove partial file on any download_to_filename error#16727

Open
RyanCunhaCosta wants to merge 1 commit intogoogleapis:mainfrom
RyanCunhaCosta:fix/storage-download-cleanup-on-error
Open

fix(storage): remove partial file on any download_to_filename error#16727
RyanCunhaCosta wants to merge 1 commit intogoogleapis:mainfrom
RyanCunhaCosta:fix/storage-download-cleanup-on-error

Conversation

@RyanCunhaCosta
Copy link
Copy Markdown

Problem

Blob.download_to_filename() opens the destination file for writing before the network request begins. If the download fails (network error, timeout, ConnectionError, etc.) the method raises an exception but leaves an empty or partially-written file on disk.

The existing code only cleaned up on DataCorruption and NotFound, but every other failure mode — including common requests.exceptions.ConnectionError / Timeout seen with GCS timeouts — still left a corrupt artifact.

Observed in the wild:

2026-04-18 07:02:07,368 | ERROR | Failed to download decomp file:
  Timeout of 120.0s exceeded, last exception:
  HTTPSConnectionPool(host='storage.googleapis.com', port=443):
  Max retries exceeded ...
  (Caused by NewConnectionError(...): [Errno 101] Network is unreachable)

After the above error the destination file exists on disk with 0 bytes, causing subsequent pipeline runs to treat the file as already downloaded and skip the retry.

Fix

In _handle_filename_and_download(), replace except (DataCorruption, NotFound) with except Exception so the file is removed before re-raising, regardless of the failure type. The inner try/except OSError is a safety guard for the edge case where the file was never successfully created.

# Before
try:
    with open(filename, "wb") as file_obj:
        self._prep_and_do_download(file_obj, ...)
except (DataCorruption, NotFound):
    os.remove(filename)
    raise

# After
with open(filename, "wb") as file_obj:
    try:
        self._prep_and_do_download(file_obj, ...)
    except Exception:
        try:
            os.remove(filename)
        except OSError:
            pass
        raise

Tests

Four new unit tests added to tests/unit/test_blob.py:

Test Validates
test_download_to_filename_cleans_up_on_connection_error ConnectionError → file removed
test_download_to_filename_cleans_up_on_timeout Timeout → file removed
test_download_to_filename_cleans_up_on_generic_exception Any RuntimeError → file removed
test_download_to_filename_keeps_file_on_success Success path unaffected

Existing tests (test_download_to_filename_corrupted, test_download_to_filename_notfound) pass unchanged.

Notes

  • No behaviour change on success path.
  • Backward-compatible: callers that previously checked os.path.exists() after a DataCorruption/NotFound error would already receive False; this extends that guarantee to all failure modes.
  • Docstring updated to document the cleanup guarantee.

Blob.download_to_filename() opens the destination file for writing
before the network request begins. If the download fails, the method
raises an exception but previously left an empty or partially-written
file on disk for most error types (only DataCorruption and NotFound
were cleaned up).

This extends the cleanup to catch all exceptions, ensuring the
partially-written file is removed before re-raising. This prevents
callers from seeing a corrupt/empty file after network errors,
timeouts, or any other unexpected failures.

The inner try/except OSError is a safety guard for the edge case
where the file was never successfully created.
@RyanCunhaCosta RyanCunhaCosta requested a review from a team as a code owner April 20, 2026 14:41
@google-cla
Copy link
Copy Markdown

google-cla Bot commented Apr 20, 2026

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces automatic cleanup for destination files when a download fails in download_to_filename, extending the previous error handling to cover all exceptions such as network errors and timeouts. It also adds comprehensive unit tests to verify cleanup behavior during various failure modes. Feedback was provided regarding the placement of the exception handler: nesting it within the with open(...) block prevents file deletion on Windows due to file locking. It is recommended to wrap the entire context manager in the try...except block and include logging for the caught exception to improve debuggability.

Comment thread packages/google-cloud-storage/google/cloud/storage/blob.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants