Stream template build-context upload from disk instead of buffering in memory#1435
Stream template build-context upload from disk instead of buffering in memory#1435mishushakov wants to merge 1 commit into
Conversation
…uffering in memory Spool the build-context tar archive to a temporary file and stream it from disk during upload (JS and Python, sync and async), keeping the explicit Content-Length required by S3 presigned PUT URLs, which reject Transfer-Encoding: chunked with 501 NotImplemented (#1243). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 0a76482 The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
PR SummaryMedium Risk Overview Reviewed by Cursor Bugbot for commit 0a76482. Bugbot is set up for automated code reviews on this repo. Configure here. |
Package ArtifactsBuilt from 1642b31. Download artifacts from this workflow run. JS SDK ( npm install ./e2b-2.29.2-mishushakov-stream-template-build-upload.0.tgzCLI ( npm install ./e2b-cli-2.11.2-mishushakov-stream-template-build-upload.0.tgzPython SDK ( pip install ./e2b-2.28.2+mishushakov.stream.template.build.upload-py3-none-any.whl |
Summary
Template builds previously buffered the entire gzipped build-context tar archive in memory before uploading it. This PR spools the archive to a temporary file and streams it from disk during upload — in the JS SDK and both sync and async Python SDKs — so memory usage no longer scales with the size of the build context.
The upload keeps an explicit
Content-Lengthheader (taken from the spooled file's size), which S3 presigned PUT URLs require — they rejectTransfer-Encoding: chunkedwith501 NotImplemented(#1243).Changes
packages/js-sdk/src/template/):tarFileStream/tarFileStreamUploadare replaced bytarFileToTempFile, which writes the archive to a temp file and returns its path, size, and acleanupcallback.uploadFilestreams the file viafs.createReadStreamwithduplex: 'half'and an explicitContent-Length, and removes the temp file afterwards.packages/python-sdk/e2b/template/utils.py,template_async/build_api.py,template_sync/build_api.py):tar_file_streamnow writes to atempfile.TemporaryFileinstead ofio.BytesIOand returns the file object positioned at the start; the upload streams from it and closes it (deleting the temp file) when done.tarFileToTempFile.test.ts, Python upload/tar tests).Usage
No API changes —
Template.build()/ template builds behave the same, just without holding the build context in memory:Split out of #1433, which covers streaming for sandbox/volume file uploads and downloads.
🤖 Generated with Claude Code