Connection limits

[rylev]

This PR is probably best reviewed commit by commit. Each factor's changes are largely independent and follow the same pattern.

This adds connection limits (both per factor and global) for the following factors (which originally had separate PRs which have been closed in favor of a single PR):

• Wasi (sockets): Cap Number of Open TCP and UDP Sockets #3537
• Redis: Limit the number of redis connections an app can have across instances #3545
• MySQL: Limit the number of pg/mysql connections an app can have across instances #3546
• Postgres: Limit the number of pg/mysql connections an app can have across instances #3546
• MQTT

Additionally the global connection limit is connected to the existing outbound http connection limiting introduced in #3285. The runtime config toml has also been updated to be consistent across all factors while retaining the now deprecated existing toml naming for backwards compatibility.

Open Issues/Questions

• I was not able to wrap the wasi P3 Socket implementation in a wrapper type that could hold on to a ConnectionSemaphore. I may need to talk with @dicej about introducing changes in wasmtime-wasi to allow this.
• The wasi socket connections error immediately when no quota is available where as all other implementations .await for quote blocking the guest from making progress. Should we always immediately error so that guests can decide what to do when no capacity is available? We treated sockets differently because waiting would risk deadlock if a component holds sockets open across async yield points.
• KV is not limited. Should we limit KV with the same mechanism, a different mechanism, or not at all?

Breaking changes

This does contain one breaking change (beyond factor APIs which I believe we do not have any stability guarantees at this point): the outbound_http.concurrent_connection_permits_acquired metric has been renamed to outbound_connection_permits_acquired with the label kind set to outbound_http. Maintaining backward compatibility here is ugly but doable if we believe it is important.

[dicej]

I was not able to wrap the wasi P3 Socket implementation in a wrapper type that could hold on to a ConnectionSemaphore. I may need to talk with @dicej about introducing changes in wasmtime-wasi to allow this.

What issues did you run into there? I'm happy to start working on any changes needed.

[rylev]

@dicej you can see code plus a print out of the compilation errors in this commit. Essentially there's no way to make the store.with_getter call to work since the generic T is just a plain generic that the implementor cannot set.

[dicej]

@dicej you can see code plus a print out of the compilation errors in this commit. Essentially there's no way to make the store.with_getter call to work since the generic T is just a plain generic that the implementor cannot set.

Thanks, I'll check out that commit and play around with it.

[rylev]

I was not able to wrap the wasi P3 Socket implementation in a wrapper type that could hold on to a ConnectionSemaphore. I may need to talk with @dicej about introducing changes in wasmtime-wasi to allow this.

Update: thanks to @dicej and @alexcrichton for helping figure this out. With the addition of bytecodealliance/wasmtime#13549, we will be able to support this. The code for that can be see in https://github.com/dicej/spin/tree/connection-limits. We'll want to wait until the linked changes land in a release of wasmtime to adopt this, but the path forward is now clear.

[lann]

Do we have an issue tracking p3 socket support?

[rylev]

Do we have an issue tracking p3 socket support?

I do not see any.