Decouple IngestionDocumentReader from IngestionPipeline constructor

[Copilot]

IngestionPipeline<T> required an IngestionDocumentReader at construction time, making it impossible to use with in-memory or programmatically created documents.

Changes

• IngestionPipeline<T> constructor — removed reader parameter; pipeline now only requires chunker and writer
• New overload — ProcessAsync(IAsyncEnumerable<IngestionDocument>, CancellationToken) processes documents directly without any file-system dependency
• File-system overloads — ProcessAsync(IngestionDocumentReader, DirectoryInfo, ...) and ProcessAsync(IngestionDocumentReader, IEnumerable<FileInfo>, ...) now take reader as a mandatory first argument
• DiagnosticsConstants — added ProcessDocuments / ProcessDocument activity names for the new overload's tracing
• Tests — updated existing tests for the new signatures; added CanProcessDocumentsWithoutReader demonstrating direct document ingestion
• Template + snapshots — updated DataIngestor.cs and all 5 integration-test snapshots
• READMEs / CHANGELOG — updated Microsoft.Extensions.DataIngestion, MarkItDown, and Markdig docs

Usage without a reader

using IngestionPipeline<string> pipeline = new(CreateChunker(), CreateWriter());

IngestionDocument document = new("my-doc-id");
document.Sections.Add(new IngestionDocumentSection());
document.Sections[0].Elements.Add(new IngestionDocumentParagraph("In-memory content."));

await foreach (IngestionResult result in pipeline.ProcessAsync(new[] { document }.ToAsyncEnumerable()))
{
    Console.WriteLine($"{result.DocumentId}: {result.Succeeded}");
}

Usage with a reader (file system)

IngestionDocumentReader reader = new MarkdownReader();
using IngestionPipeline<string> pipeline = new(CreateChunker(), CreateWriter());

await foreach (IngestionResult result in pipeline.ProcessAsync(reader, new DirectoryInfo("docs"), "*.md"))
{
    Console.WriteLine($"{result.DocumentId}: {result.Succeeded}");
}

Microsoft Reviewers: Open in CodeFlow

[adamsitnik]

@copilot fetch latest changes from https://github.com/dotnet/extensions/tree/data-ingestion-preview2 and solve the merge conflicts

[Copilot]

@copilot fetch latest changes from https://github.com/dotnet/extensions/tree/data-ingestion-preview2 and solve the merge conflicts

Done — fetched the latest from data-ingestion-preview2 and resolved all merge conflicts in commit b862662.

Key resolutions:

• CHANGELOG.md — accepted deletion from the base branch
• IngestionPipeline.cs — combined both changes: removed generic <T> parameter (from base branch's non-generic refactor) and kept reader out of the constructor / on ProcessAsync overloads (from this PR)
• IngestionPipelineTests.cs — resolved 4 conflict blocks plus fixed auto-merged PipelineWorksWithEmbeddingGenerator and CanProcessDocumentsWithoutReader tests to use the non-generic types
• Template + 5 snapshots — DataIngestor.cs now uses new IngestionPipeline(chunker, writer) with reader passed to ProcessAsync

[adamsitnik]

@copilot address my feedback

[adamsitnik]

IngestionPipeline is no longer generic, update all readme files

Suggested change

	using IngestionPipeline<string> pipeline = new(CreateChunker(), CreateWriter());
	using IngestionPipeline pipeline = new(CreateChunker(), CreateWriter());

[Copilot]

Fixed in the latest commit — updated all three README files (Markdig, MarkItDown, and DataIngestion) to use IngestionPipeline without the generic type parameter, and also updated the prose description in the DataIngestion README from IngestionPipeline<T> to IngestionPipeline.

[Copilot]

@copilot address my feedback

Done — all README files now use non-generic IngestionPipeline.

[dotnet-comment-bot]

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.Extensions.Diagnostics.Testing	Line	99	98.65 🔻
Microsoft.Extensions.Telemetry	Line	93	91.95 🔻
Microsoft.Extensions.AI	Line	89	88.54 🔻
Microsoft.Extensions.AI	Branch	89	88.57 🔻
Microsoft.Extensions.AI.OpenAI	Line	75	62.89 🔻
Microsoft.Extensions.AI.OpenAI	Branch	75	50.41 🔻
Microsoft.Extensions.DataIngestion.Abstractions	Branch	75	72.73 🔻
Microsoft.Extensions.DataIngestion.MarkItDown	Line	75	4.46 🔻
Microsoft.Extensions.DataIngestion.MarkItDown	Branch	75	0 🔻
Microsoft.Extensions.Diagnostics.ResourceMonitoring	Line	99	96.03 🔻
Microsoft.Extensions.Diagnostics.ResourceMonitoring	Branch	99	94.39 🔻
Microsoft.Extensions.Diagnostics.ResourceMonitoring.Kubernetes	Line	99	97.73 🔻
Microsoft.Extensions.ServiceDiscovery.Dns	Line	75	68.32 🔻
Microsoft.Extensions.ServiceDiscovery.Abstractions	Line	75	42.11 🔻
Microsoft.Extensions.ServiceDiscovery.Abstractions	Branch	75	42.86 🔻
Microsoft.Extensions.ServiceDiscovery	Line	75	68.11 🔻
Microsoft.Extensions.ServiceDiscovery	Branch	75	71.43 🔻
Microsoft.Extensions.ServiceDiscovery.Yarp	Line	75	73.85 🔻
Microsoft.Extensions.ServiceDiscovery.Yarp	Branch	75	70 🔻
Microsoft.Extensions.VectorData.Abstractions	Line	75	37.39 🔻
Microsoft.Extensions.VectorData.Abstractions	Branch	75	22.73 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Gen.BuildMetadata	97	100
Microsoft.Gen.MetadataExtractor	57	73
Microsoft.Gen.MetricsReports	67	69
Microsoft.Extensions.AI.Abstractions	82	85
Microsoft.Extensions.AI.Evaluation.NLP	0	78
Microsoft.Extensions.Caching.Hybrid	82	89
Microsoft.Extensions.DataIngestion	75	87
Microsoft.Extensions.DataIngestion.Markdig	75	90
Microsoft.Extensions.Http.Resilience	97	100

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1485806&view=codecoverage-tab