Skip to content

Stream template build-context upload from disk instead of buffering in memory#1435

Draft
mishushakov wants to merge 1 commit into
mainfrom
mishushakov/stream-template-build-upload
Draft

Stream template build-context upload from disk instead of buffering in memory#1435
mishushakov wants to merge 1 commit into
mainfrom
mishushakov/stream-template-build-upload

Conversation

@mishushakov

Copy link
Copy Markdown
Member

Summary

Template builds previously buffered the entire gzipped build-context tar archive in memory before uploading it. This PR spools the archive to a temporary file and streams it from disk during upload — in the JS SDK and both sync and async Python SDKs — so memory usage no longer scales with the size of the build context.

The upload keeps an explicit Content-Length header (taken from the spooled file's size), which S3 presigned PUT URLs require — they reject Transfer-Encoding: chunked with 501 NotImplemented (#1243).

Changes

  • JS (packages/js-sdk/src/template/): tarFileStream/tarFileStreamUpload are replaced by tarFileToTempFile, which writes the archive to a temp file and returns its path, size, and a cleanup callback. uploadFile streams the file via fs.createReadStream with duplex: 'half' and an explicit Content-Length, and removes the temp file afterwards.
  • Python (packages/python-sdk/e2b/template/utils.py, template_async/build_api.py, template_sync/build_api.py): tar_file_stream now writes to a tempfile.TemporaryFile instead of io.BytesIO and returns the file object positioned at the start; the upload streams from it and closes it (deleting the temp file) when done.
  • Tests updated for the new return shapes (JS tarFileToTempFile.test.ts, Python upload/tar tests).

Usage

No API changes — Template.build() / template builds behave the same, just without holding the build context in memory:

await Template.build(template, { alias: 'my-template' })
Template.build(template, alias="my-template")

Split out of #1433, which covers streaming for sandbox/volume file uploads and downloads.

🤖 Generated with Claude Code

…uffering in memory

Spool the build-context tar archive to a temporary file and stream it
from disk during upload (JS and Python, sync and async), keeping the
explicit Content-Length required by S3 presigned PUT URLs, which reject
Transfer-Encoding: chunked with 501 NotImplemented (#1243).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cla-bot cla-bot Bot added the cla-signed label Jun 12, 2026
@changeset-bot

changeset-bot Bot commented Jun 12, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 0a76482

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
Name Type
e2b Patch
@e2b/python-sdk Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@cursor

cursor Bot commented Jun 12, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Touches presigned S3 PUT behavior for template builds; a regression on Content-Length or chunked encoding would break uploads (#1243).

Overview
Template build context uploads no longer load the full gzipped tar into RAM. JS writes the archive with tarFileToTempFile, streams it via createReadStream with explicit Content-Length and duplex: 'half', then deletes the temp dir. Python tar_file_stream uses tempfile.TemporaryFile; sync upload passes the file to httpx, async upload uses aiter_io_chunks plus Content-Length. Public template build APIs are unchanged; tests were updated for the new shapes.

Reviewed by Cursor Bugbot for commit 0a76482. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

Copy link
Copy Markdown
Contributor

Package Artifacts

Built from 1642b31. Download artifacts from this workflow run.

JS SDK (e2b@2.29.2-mishushakov-stream-template-build-upload.0):

npm install ./e2b-2.29.2-mishushakov-stream-template-build-upload.0.tgz

CLI (@e2b/cli@2.11.2-mishushakov-stream-template-build-upload.0):

npm install ./e2b-cli-2.11.2-mishushakov-stream-template-build-upload.0.tgz

Python SDK (e2b==2.28.2+mishushakov-stream-template-build-upload):

pip install ./e2b-2.28.2+mishushakov.stream.template.build.upload-py3-none-any.whl

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant