SEGV (stack-user-after-return) with simple exec::task in a stdexec::sync_wait (Linux 32-bit)

[armpribb]

The following minimal code sample segfaults on Linux Intel 32-bit. The same cannot be reproduced on Intel 64-bit, aarch64, or armhf.

https://godbolt.org/z/1Yobafcz8

Was able to git-bisect it to the changes in the following PR: #1974.
Before it was working fine.

Sanitizers seems to indicate stack-user-after-return (stacktrace is intentionally incomplete, skips confidential parts). The exact backtrace might differ depending on context. Basically all parts of our code-base using stdexec::sync_wait and the exec::task type are affected.
The godbolt link above is just a minimal reproducing sample (sanitizer output might look different), unfortunately I was not able to do a 32-bit sanitizer build on godbolt (seems required static clang-rt libs are missing).

ASan/UBSan sample output (from one occurrence in our code-base):

=================================================================
==732791==ERROR: AddressSanitizer: stack-use-after-return on address 0xe8b1d0e5 at pc 0x67a74cd1 bp 0xffe5d918 sp 0xffe5d910
READ of size 1 at 0xe8b1d0e5 thread T0
    #0 0x67a74cd0 in stdexec::__opt::__optional<experimental::execution::__task::__default_awaiter_context<stdexec::__connect_await::__promise<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>, stdexec::__rcvr<stdexec::schedule_from_t, stdexec::__detail::__state<stdexec::__rcvr<stdexec::continues_on_t, stdexec::__trnsfr::__state<experimental::execution::__task::__any_scheduler, stdexec::(anonymous namespace)::__sexpr<stdexec::'lambda14'(){}>, stdexec::__as_awaitable::__async_receiver<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>::__promise, void>>, 0u>, stdexec::__>, 0u>>>>::reset() /build/external/stdexec/include/exec/../stdexec/__detail/__optional.hpp:193:13
    #1 0x67a747c3 in experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>::__task_awaiter<stdexec::__connect_await::__promise<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>, stdexec::__rcvr<stdexec::schedule_from_t, stdexec::__detail::__state<stdexec::__rcvr<stdexec::continues_on_t, stdexec::__trnsfr::__state<experimental::execution::__task::__any_scheduler, stdexec::(anonymous namespace)::__sexpr<stdexec::'lambda14'(){}>, stdexec::__as_awaitable::__async_receiver<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>::__promise, void>>, 0u>, stdexec::__>, 0u>>>::await_resume() /build/external/stdexec/include/exec/task.hpp:682:22
    #2 0x67a74519 in stdexec::__connect_await::__awaitable_wrapper<stdexec::__connect_await::__awaitable_state<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>, stdexec::__connect_await::__promise<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>, stdexec::__rcvr<stdexec::schedule_from_t, stdexec::__detail::__state<stdexec::__rcvr<stdexec::continues_on_t, stdexec::__trnsfr::__state<experimental::execution::__task::__any_scheduler, stdexec::(anonymous namespace)::__sexpr<stdexec::'lambda14'(){}>, stdexec::__as_awaitable::__async_receiver<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>::__promise, void>>, 0u>, stdexec::__>, 0u>>>::__state>::await_resume() /build/external/stdexec/include/exec/../stdexec/__detail/__connect_awaitable.hpp:141:28
    #3 0x67a7438a in stdexec::__connect_await::__awaitable_wrapper<stdexec::__connect_await::__awaitable_state<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>, stdexec::__connect_await::__promise<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>, stdexec::__rcvr<stdexec::schedule_from_t, stdexec::__detail::__state<stdexec::__rcvr<stdexec::continues_on_t, stdexec::__trnsfr::__state<experimental::execution::__task::__any_scheduler, stdexec::(anonymous namespace)::__sexpr<stdexec::'lambda14'(){}>, stdexec::__as_awaitable::__async_receiver<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>::__promise, void>>, 0u>, stdexec::__>, 0u>>>>::await_resume() /build/external/stdexec/include/exec/../stdexec/__detail/__connect_awaitable.hpp:141:28
    #4 0x67a73ee1 in stdexec::__connect_await::__opstate<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>, stdexec::__rcvr<stdexec::schedule_from_t, stdexec::__detail::__state<stdexec::__rcvr<stdexec::continues_on_t, stdexec::__trnsfr::__state<experimental::execution::__task::__any_scheduler, stdexec::(anonymous namespace)::__sexpr<stdexec::'lambda14'(){}>, stdexec::__as_awaitable::__async_receiver<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>::__promise, void>>, 0u>, stdexec::__>, 0u>>::__on_resume() /build/external/stdexec/include/exec/../stdexec/__detail/__connect_awaitable.hpp:467:24
    #5 0x67a73ce7 in stdexec::__connect_await::__promise<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>, stdexec::__rcvr<stdexec::schedule_from_t, stdexec::__detail::__state<stdexec::__rcvr<stdexec::continues_on_t, stdexec::__trnsfr::__state<experimental::execution::__task::__any_scheduler, stdexec::(anonymous namespace)::__sexpr<stdexec::'lambda14'(){}>, stdexec::__as_awaitable::__async_receiver<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>::__promise, void>>, 0u>, stdexec::__>, 0u>>::__resume(void*) /build/external/stdexec/include/exec/../stdexec/__detail/__connect_awaitable.hpp:103:15
    #6 0x67aba385 in stdexec::__as_awaitable::__async_receiver<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>::__promise, void>::__done() /build/external/stdexec/include/exec/../stdexec/__detail/__as_awaitable.hpp:294:9
    #7 0x67abbdad in void stdexec::__as_awaitable::__async_receiver<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>::__promise, void>::set_value<>() /build/external/stdexec/include/exec/../stdexec/__detail/__as_awaitable.hpp:249:9
    #8 0x67abbcac in void stdexec::__detail::__applier::operator()<stdexec::__as_awaitable::__async_receiver<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>::__promise, void>, stdexec::set_value_t>(stdexec::__as_awaitable::__async_receiver<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>::__promise, void>&, stdexec::set_value_t) /build/external/stdexec/include/exec/../stdexec/__detail/__storage.hpp:37:9
    #9 0x67abb8e2 in void stdexec::__detail::__visitor::operator()<stdexec::__as_awaitable::__async_receiver<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>::__promise, void>, stdexec::__tup::__tuple<stdexec::set_value_t>>(stdexec::__as_awaitable::__async_receiver<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>::__promise, void>&, stdexec::__tup::__tuple<stdexec::set_value_t>&&) /build/external/stdexec/include/exec/../stdexec/__detail/__storage.hpp:46:9
    #10 0x67a632a9 in void stdexec::__var::__visit_alt<0u, void, stdexec::__detail::__visitor, stdexec::__results_storage<stdexec::set_value_t (), stdexec::set_error_t (std::__exception_ptr::exception_ptr), stdexec::set_stopped_t ()>, stdexec::__as_awaitable::__async_receiver<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>::__promise, void>&>(stdexec::__detail::__visitor&&, stdexec::__results_storage<stdexec::set_value_t (), stdexec::set_error_t (std::__exception_ptr::exception_ptr), stdexec::set_stopped_t ()>&&, stdexec::__as_awaitable::__async_receiver<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>::__promise, void>&) /build/external/stdexec/include/exec/../stdexec/__detail/__variant.hpp:93:14
    #11 0x67acf081 in decltype(auto) stdexec::__var::__variant<(stdexec::__iota<0u, 1u, 2u>*)0, stdexec::__tup::__tuple<stdexec::set_value_t>, stdexec::__tup::__tuple<stdexec::set_error_t, std::__exception_ptr::exception_ptr>, stdexec::__tup::__tuple<stdexec::set_stopped_t>>::__visit<stdexec::__detail::__visitor, stdexec::__results_storage<stdexec::set_value_t (), stdexec::set_error_t (std::__exception_ptr::exception_ptr), stdexec::set_stopped_t ()>, stdexec::__as_awaitable::__async_receiver<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>::__promise, void>&>(stdexec::__detail::__visitor&&, stdexec::__results_storage<stdexec::set_value_t (), stdexec::set_error_t (std::__exception_ptr::exception_ptr), stdexec::set_stopped_t ()>&&, T1&&...) /build/external/stdexec/include/exec/../stdexec/__detail/__variant.hpp:373:16
    #12 0x67aceb67 in void stdexec::__results_storage<stdexec::set_value_t (), stdexec::set_error_t (std::__exception_ptr::exception_ptr), stdexec::set_stopped_t ()>::__static___complete<stdexec::__results_storage<stdexec::set_value_t (), stdexec::set_error_t (std::__exception_ptr::exception_ptr), stdexec::set_stopped_t ()>, stdexec::__as_awaitable::__async_receiver<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>::__promise, void>>(stdexec::__results_storage<stdexec::set_value_t (), stdexec::set_error_t (std::__exception_ptr::exception_ptr), stdexec::set_stopped_t ()>&&, stdexec::__as_awaitable::__async_receiver<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>::__promise, void>&) /build/external/stdexec/include/exec/../stdexec/__detail/__storage.hpp:75:7
    #13 0x67cd595d in stdexec::__trnsfr::__receiver2<stdexec::(anonymous namespace)::__sexpr<stdexec::'lambda23'(){}>, stdexec::__as_awaitable::__async_receiver<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>::__promise, void>>::set_value() /build/external/stdexec/include/exec/../stdexec/__detail/__continues_on.hpp:85:38
    #14 0x67ccdb58 in experimental::execution::_any::_ireceiver_memfn<stdexec::set_value_t ()>::_interface_<stdexec::__any::__reference_root<experimental::execution::_any::_ireceiver<stdexec::completion_signatures<stdexec::set_value_t (), stdexec::set_error_t (std::__exception_ptr::exception_ptr), stdexec::set_stopped_t ()>, experimental::execution::queries<>>::_isemi_receiver, experimental::execution::_any::_state<stdexec::__trnsfr::__receiver2<stdexec::(anonymous namespace)::__sexpr<stdexec::'lambda23'(){}>, stdexec::__as_awaitable::__async_receiver<experimental::execution::__task::basic_task<void, experimental::execution::__task::__default_task_context_impl<(experimental::execution::__task::__scheduler_affinity)1>>::__promise, void>>, stdexec::never_stop_token>, experimental::execution::_any::_ireceiver<stdexec::completion_signatures<stdexec::set_value_t (), stdexec::set_error_t (std::__exception_ptr::exception_ptr), stdexec::set_stopped_t ()>, experimental::execution::queries<>>::_isemi_receiver<experimental::execution::_any::_ireceiver_memfn<stdexec::set_stopped_t ()>::_interface_<experimental::execution::_any::_ireceiver_memfn<stdexec::set_error_t (std::__exception_ptr::exception_ptr)>::_interface_<experimental::execution::_any::_ireceiver_memfn<stdexec::set_value_t ()>::_interface_<stdexec::__any::__iroot>>>>>>::set_value() && /build/external/stdexec/include/exec/any_sender_of.hpp:235:53
    #15 0x67aaf228 in experimental::execution::_any::_ireceiver_memfn<stdexec::set_value_t ()>::_interface_<stdexec::__any::__reference_proxy_root<experimental::execution::_any::_ireceiver<stdexec::completion_signatures<stdexec::set_value_t (), stdexec::set_error_t (std::__exception_ptr::exception_ptr), stdexec::set_stopped_t ()>, experimental::execution::queries<>>::_isemi_receiver>>::set_value() && /build/external/stdexec/include/exec/any_sender_of.hpp:235:53
    #16 0x67b4e2c3 in void stdexec::__pointer_receiver<stdexec::__any::__any_ptr<experimental::execution::_any::_ireceiver<stdexec::completion_signatures<stdexec::set_value_t (), stdexec::set_error_t (std::__exception_ptr::exception_ptr), stdexec::set_stopped_t ()>, experimental::execution::queries<>>::_isemi_receiver>, stdexec::env<>>::set_value<>() /build/external/stdexec/include/exec/../stdexec/__detail/__receiver_ref.hpp:59:7
    #17 0x67bead18 in stdexec::__run_loop::__run_loop_base::__opstate_t<stdexec::__pointer_receiver<stdexec::__any::__any_ptr<experimental::execution::_any::_ireceiver<stdexec::completion_signatures<stdexec::set_value_t (), stdexec::set_error_t (std::__exception_ptr::exception_ptr), stdexec::set_stopped_t ()>, experimental::execution::queries<>>::_isemi_receiver>, stdexec::env<>>>::__execute_impl(stdexec::__run_loop::__run_loop_base::__task*) /build/external/stdexec/include/exec/../stdexec/__detail/__run_loop.hpp:126:13
    #18 0x67bf5594 in stdexec::__run_loop::__run_loop_base::__task::__execute() /build/external/stdexec/include/exec/../stdexec/__detail/__run_loop.hpp:104:11
    #19 0x67bf3071 in stdexec::__run_loop::__run_loop_base::__execute_all() /build/external/stdexec/include/exec/../stdexec/__detail/__run_loop.hpp:177:22
    #20 0x67b64c43 in stdexec::__run_loop::__run_loop_base::run() /build/external/stdexec/include/exec/../stdexec/__detail/__run_loop.hpp:58:11
    #21 0x67b5d521 in std::optional<stdexec::__sync_wait::__value_tuple_for<stdexec::(anonymous namespace)::__sexpr<stdexec::'lambda15'(){}>>::__t> stdexec::sync_wait_t::apply_sender<stdexec::(anonymous namespace)::__sexpr<stdexec::'lambda15'(){}>>(stdexec::(anonymous namespace)::__sexpr<stdexec::'lambda15'(){}>&&) const /build/external/stdexec/include/exec/../stdexec/__detail/__sync_wait.hpp:382:29
    #22 0x67b5c728 in auto stdexec::sync_wait_t::operator()<stdexec::(anonymous namespace)::__sexpr<stdexec::'lambda15'(){}>>(stdexec::(anonymous namespace)::__sexpr<stdexec::'lambda15'(){}>&&) const /build/external/stdexec/include/exec/../stdexec/__detail/__sync_wait.hpp:317:48

[armpribb]

Also reproduces with an even more minimal sample using stdexec::task:
https://godbolt.org/z/vqdKx3nTb

Again, x64 is working fine, only x86 crashes.

The issue might be related to #2098 where an eerily similar code sample has been posted (#2098 (comment)).

[ispeters]

That minimal example can also be seen here, with the generated assembly: https://godbolt.org/z/qaYs89hPn.

Lines 30-38 are:

.LBB0_4:
        mov     edx, dword ptr [ebp - 64]
        mov     eax, dword ptr [ebp - 68]
        mov     dword ptr [ebp - 76], eax
        lea     ecx, [edx + foo() (.resume)@GOTOFF]
        mov     dword ptr [eax], ecx
        mov     bl, 1
        lea     ecx, [edx + foo() (.cleanup)@GOTOFF]
        lea     edx, [edx + foo() (.destroy)@GOTOFF]

I'd guess the segfault comes from the foo coroutine having both a .cleanup and a .destroy; the hack in #1974 assumes the coroutine frame only requires two function pointers (.resume and .destroy, which we force to be a no-op). I'd guess the compiler is unhappy that we're not leaving enough space for all the stuff it wants to exist in the frame. We might be able to fix it by modifying __synthetic_coro_frame to take up more space on 32-bit platforms, but I don't have the necessary hardware to validate such a hypothesis, I don't think.

[armpribb]

Found out it's working fine with gcc 13 and clang trunk (23.x), see https://godbolt.org/z/8vTa7sPvd for comparison of the three.
So it's a clang bug... kinda.
Was able to bisect it to the following PR in llvm/llvm-project, which resolves the issue: llvm/llvm-project#198943.
It changed the calling conventions (fastcc -> ccc) for resume/destroy functions of C++ coroutines, which is an ABI breaking change for i686 and some others (only in this case it fixed instead of broke something).

[armpribb]

Summary:

• clang specific issue: uses (llvm internal) fastcc calling convention for C++ coroutines
• not an issue until optimization in Shrink the inline storage in connect_awaitable #1974, most likely because the C calling convention of the noop-destroy does not work well with fastcc used internally by llvm for resume/destroy functions of C++ coroutines on Linux i686!? maybe someone can explain the details better
• fixed in clang 23.x by [coro] Use C calling convention for C++20 coroutines llvm/llvm-project#198943 (ABI breaking change for i686 and some other platforms, not an issue for amd64)

See godbolt sample comparing clang 22 (SEGV), gcc 13 and clang trunk (23.x): https://godbolt.org/z/8vTa7sPvd

@ispeters, @ericniebler do you see any possibility to make #1974 work for linux i686 with clang 21, 22 (any clang before llvm/llvm-project#198943)?

Unfortunately we are stuck with clang 21 for the time being and currently still need to support linux i686.
We were able to work around the issue in our CI (basically picking the relevant parts of llvm/llvm-project#198943 to clang 21, which we have to use for the time being).