Skip to content

[CUSTOMMEM] only track the newly added heap when brk changed#4055

Merged
ptitSeb merged 2 commits into
ptitSeb:mainfrom
Shirowwww:brk-catchup-preserve-prot
Jul 17, 2026
Merged

[CUSTOMMEM] only track the newly added heap when brk changed#4055
ptitSeb merged 2 commits into
ptitSeb:mainfrom
Shirowwww:brk-catchup-preserve-prot

Conversation

@Shirowwww

Copy link
Copy Markdown
Contributor

Proper fix for #4052, replacing the CheckExec workaround in #4053.

find31bitBlockNearHint / find47bitBlockNearHint re-tracked the whole heap from pbrk as RW every time the program break moved. setProtection only writes the tracking, not the kernel, so every page the guest had made executable inside the heap silently lost PROT_EXEC in the tracking while the kernel still mapped it rwx. CheckExec then saw prot=3 against a kernel 0x7 and raised 0xecec forever at that RIP.

Hit it with UE4SS on a Palworld dedicated server: it puts hook trampolines on heap pages, the heap grows, the trampolines lose their exec bit in the tracking, and the server dies in a signal storm about one boot in three.

Now only the range actually added since last time is tracked, so existing heap pages keep their protection.

Tested on ARM64 (Neoverse-N1), Palworld dedicated server + UE4SS, 8 boots: no storm. I kept the CheckExec resync from #4053 in the test binary purely as a canary that logs when tracking and kernel disagree — with the same binary before this fix it fired on 5 of 8 boots, after it never fired once.

#4053 can be closed if you take this one.

find31/47bitBlockNearHint re-tracked the whole heap from pbrk as RW every
time the program break moved. setProtection only writes the tracking, not
the kernel, so any page the guest had made executable inside the heap (JIT
or hook trampolines) silently lost PROT_EXEC while the kernel still mapped
it rwx, and CheckExec then raised 0xecec forever on it. Track only the
range that was actually added. Fixes ptitSeb#4052
Copilot AI review requested due to automatic review settings July 17, 2026 11:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Comment thread src/custommem.c
@ptitSeb

ptitSeb commented Jul 17, 2026

Copy link
Copy Markdown
Owner

Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants