cdc: add kafka max-message-bytes related explain#23352
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
ti-chi-bot
Bot
added
missing-translation-status
📝 WalkthroughWalkthroughThe documentation updates describe Open Protocol event batching, Kafka message-size thresholds, oversized-message handling, and version-specific troubleshooting. They also remove superseded FAQ guidance and make minor formatting corrections. ChangesKafka message sizing
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested labels: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 91844770-f488-4ff5-b9a0-9a994f1c6a90
📒 Files selected for processing (4)
ticdc/ticdc-faq.mdticdc/ticdc-open-protocol.mdticdc/ticdc-sink-to-kafka.mdticdc/troubleshoot-ticdc.md
| ```ini | ||
| # The maximum message size accepted by the topic | ||
| max.message.bytes=<a value no smaller than the message to be sent> | ||
| # The maximum message size accepted by the broker; configure this when using the broker default limit | ||
| message.max.bytes=<a value no smaller than the message to be sent> | ||
| # The maximum message size that the broker can replicate | ||
| replica.fetch.max.bytes=<a value no smaller than message.max.bytes> | ||
| # The maximum message size that the consumer can read | ||
| fetch.message.max.bytes=<a value no smaller than the effective Kafka message size limit> |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Candidate files =="
git ls-files | rg '(^|/)ticdc/troubleshoot-ticdc\.md$|resources/doc-templates|scripts/(markdownlint|verify-links)' || true
echo
echo "== Target snippet =="
if [ -f ticdc/troubleshoot-ticdc.md ]; then
nl -ba ticdc/troubleshoot-ticdc.md | sed -n '100,150p'
fi
echo
echo "== Kafka config name occurrences =="
rg -n "fetch\.message\.max\.bytes|fetch\.max\.bytes|max\.partition\.fetch\.bytes|replica\.fetch\.max\.bytes|message\.max\.bytes|max\.message\.bytes" -S . || true
echo
echo "== Surrounding context in target =="
nl -ba ticdc/troubleshoot-ticdc.md | sed -n '110,135p'Repository: pingcap/docs
Length of output: 597
🌐 Web query:
Apache Kafka 4.3 configuration consumer fetch.max.bytes max.partition.fetch.bytes fetch.message.max.bytes replica.fetch.max.bytes
💡 Result:
In Apache Kafka 4.3, these configurations manage the amount of data transferred during fetch operations. Note that fetch.message.max.bytes is not a standard configuration key in modern Kafka versions; it is often confused with broker-level or topic-level message size limits [1][2]. 1. fetch.max.bytes This is a consumer-side configuration [1]. It defines the maximum amount of data the server should return for a fetch request [1]. While it serves as a soft limit, if the first record batch in the first non-empty partition is larger than this value, the batch is still returned to ensure consumer progress [1]. The default value is 52,428,800 bytes (50 MiB) [1]. 2. max.partition.fetch.bytes This is a consumer-side configuration [1]. It sets the maximum amount of data the server will return for each partition in a fetch request [1]. Like fetch.max.bytes, if the first record batch is larger than this limit, it will still be returned to ensure progress [1]. The default value is 1,048,576 bytes (1 MiB) [1]. 3. replica.fetch.max.bytes This is a broker-side configuration that governs the maximum number of bytes the broker will attempt to fetch from another broker during replication for a single partition [3]. Note that historical documentation shows this setting exists to manage inter-broker traffic, separate from consumer-facing fetch settings [3]. 4. Regarding fetch.message.max.bytes There is no standard configuration named fetch.message.max.bytes in Apache Kafka 4.3 [1][2][4]. You are likely referring to the broker-level message size limit, message.max.bytes, or the topic-level equivalent, max.message.bytes [1][2][4]. These settings define the maximum allowed size of a record batch and are crucial because if they are set larger than the consumer's fetch settings, the consumer may be unable to fetch batches, potentially causing processing issues [1][4]. Summary Table Configuration | Type | Purpose | Default (4.3) fetch.max.bytes | Consumer | Max data for a full fetch request [1] | 52,428,800 [1] max.partition.fetch.bytes | Consumer | Max data per partition in a fetch [1] | 1,048,576 [1] replica.fetch.max.bytes | Broker | Max data per partition during replication [3] | 1,048,576 (varies) message.max.bytes | Broker | Max batch size allowed by broker [1][4] | 1,048,588 [4]
Citations:
- 1: https://kafka.apache.org/43/configuration/consumer-configs/
- 2: https://kafka.apache.org/43/configuration/broker-configs/
- 3: https://kafka.apache.org/23/generated/kafka_config.html
- 4: https://kafka.apache.org/43/configuration/topic-configs/
Use the current Kafka consumer fetch settings.
fetch.message.max.bytes is not a Kafka config key for Kafka 4.3; replace it with the consumer configs fetch.max.bytes and max.partition.fetch.bytes. Keep replica.fetch.max.bytes sized for the effective Kafka message size limit, including topic settings that override broker defaults.
Committable replacement
| ```ini | |
| # The maximum message size accepted by the topic | |
| max.message.bytes=<a value no smaller than the message to be sent> | |
| # The maximum message size accepted by the broker; configure this when using the broker default limit | |
| message.max.bytes=<a value no smaller than the message to be sent> | |
| # The maximum message size that the broker can replicate | |
| replica.fetch.max.bytes=<a value no smaller than message.max.bytes> | |
| # The maximum message size that the consumer can read | |
| fetch.message.max.bytes=<a value no smaller than the effective Kafka message size limit> | |
| # The maximum message size accepted by the topic | |
| max.message.bytes=<a value no smaller than the message to be sent> | |
| # The maximum message size accepted by the broker; configure this when using the broker default limit | |
| message.max.bytes=<a value no smaller than the message to be sent> | |
| # The maximum message size that a broker replica fetches per partition | |
| replica.fetch.max.bytes=<a value no smaller than the effective Kafka message size limit> | |
| # The maximum amount of data returned in a consumer.fetch request | |
| fetch.max.bytes=<a value no smaller than the effective Kafka message size limit> | |
| # The maximum amount of data returned per partition to the consumer | |
| max.partition.fetch.bytes=<a value no smaller than the effective Kafka message size limit> |
Source: Path instructions
1 participant
First-time contributors' checklist
What is changed, added or deleted? (Required)
Which TiDB version(s) do your changes apply to? (Required)
Tips for choosing the affected version(s):
By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.
For details, see tips for choosing the affected versions.
What is the related PR or file link(s)?
AI agent involvement
Do your changes match any of the following descriptions?
Summary by CodeRabbit
max-batch-sizeandmax-message-bytes.ErrMessageTooLarge, including version-specific recovery instructions and alternative handling options.