Skip to content

feat(artifacts): list and download pipeline-run artifacts#42

Open
arseniy-pplx wants to merge 1 commit into
TangleML:masterfrom
arseniy-pplx:transfer/artifact-list-download
Open

feat(artifacts): list and download pipeline-run artifacts#42
arseniy-pplx wants to merge 1 commit into
TangleML:masterfrom
arseniy-pplx:transfer/artifact-list-download

Conversation

@arseniy-pplx

Copy link
Copy Markdown

Summary

  • extend tangle sdk artifacts get with --list, --out-dir, --only and --include-children
  • accept a run id or root execution id for listing and fetching artifact output
  • JSON-encode inline values as <name>.json; stream signed-URL content to a file with owner-only permissions on POSIX (0o600; Windows ACLs are not adjusted) under a sanitized artifact name
  • refuse symlinks, collisions and path traversal; check the initial signed-URL host and every redirect hop against loopback, private, link-local and CGNAT address literals by default
  • validate --query top-level and nested shapes (tasks/executions/components/artifact_ids) as clean CLI errors; null per-entry output filters mean all outputs (same as []), and an explicit --query '{}' is a valid empty query rather than a missing one
  • add the public TangleApiClient.request_raw seam so artifact fetching does not depend on private client methods

Context

The current CLI reports artifact metadata but cannot retrieve artifact contents. This completes the pipeline-run output workflow while keeping large files streaming and applying conservative filesystem, redirect and error-redaction safeguards. Self-hosted deployments with private-network object storage can opt out with TANGLE_ALLOW_INTERNAL_ARTIFACT_HOSTS=1; the host-literal guard is a conservative default, not a DNS-rebinding defense.

Testing

  • uv run pytest tests/test_artifacts.py tests/test_artifacts_cli.py tests/test_client.py
  • uvx ruff check on the touched source files
  • uv lock --check
  • git diff --check

@arseniy-pplx
arseniy-pplx marked this pull request as ready for review July 20, 2026 18:10
of passing the run id straight to ``get_execution_details``.
"""

details = self._require_client().get_run_details(run_id)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(AI-assisted)

Could root resolution stay shallow here?

get_run_details() calls get_execution_details(), which recursively _enrich_execution_tree() across every descendant. Consequently, even --list / --out-dir without --include-children makes one API call per descendant and can fail because a nested execution the command never displays cannot be fetched.

Could we add a shallow helper that maps run ID → root execution ID and calls executions_details once, fetching direct children only when include_children is set? A real-client call-count test would pin that boundary.

raise RuntimeError("Artifact downloads require a client with raw request support")
endpoint = _data_endpoint(artifact_id)
try:
response = request("GET", endpoint, stream=True)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(AI-assisted)

Can we avoid reading/logging the body for streamed requests before this ships?

This stream=True request still traverses _request_with_same_origin_redirects, whose TANGLE_VERBOSE branch evaluates response.text before returning. That buffers the entire artifact—defeating the large-file streaming guarantee—and writes artifact contents verbatim to diagnostics.

PR #39 has the centralized fix: log <streaming body omitted> when request_kwargs["stream"] is set. Since #42 is an independent branch, could this PR either include that guard or explicitly depend on/rebase onto #39?

Extend tangle sdk artifacts get with list, out-dir, only and include-children modes. Inline values are JSON-encoded; signed-URL content streams to owner-only (POSIX 0o600) files with path, collision, symlink, redirect and internal-address checks. Signed URLs are redacted from errors. Query top-level and nested shapes are validated as clean errors, null per-entry output filters select all outputs (same as an empty list), and an explicit empty --query object stays distinct from an absent one. A public TangleApiClient.request_raw seam keeps artifact fetching independent of private client methods.
@arseniy-pplx
arseniy-pplx force-pushed the transfer/artifact-list-download branch from ba76057 to 6203bb5 Compare July 24, 2026 10:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants