[no sq] Generic IPC mechanism between Lua envs

[sfan5]

the first two commits are totally unrelated but this is a package deal, sorry.

To do

This PR is ready.

TODO:

• CAS primitive
• Polling function that blocks

How to test

1. read and verify the unittests
2. the forbidden ping-pong:

-- Note: this will just not work if you have less than 2 async workers.
-- It also hangs the server at shutdown but that's expected.

local function fa()
	while true do
		core.ipc_poll("btoa", 60 * 1000)
		local n = core.ipc_get("btoa")
		core.ipc_set("btoa", nil)
		print("a:", n)
		core.ipc_poll("", 100) -- sleep
		core.ipc_set("atob", n + 1)
	end
end
local function fb()
	while true do
		core.ipc_poll("atob", 60 * 1000)
		local n = core.ipc_get("atob")
		core.ipc_set("atob", nil)
		print("b:", n)
		core.ipc_poll("", 200) -- sleep
		core.ipc_set("btoa", n + 1)
	end
end
core.handle_async(fa, function() end)
core.handle_async(fb, function() end)
core.ipc_set("atob", 0)

4. (optional) invent your own code

[grorp]

only looked at the documentation

[Kimapr]

I think channels, as done in love2d would be preferable to this.

[sfan5]

Channels are definitely more flexible, but there are two reasons that made me choose this approach:

• The main usecase (also mentioned in the OP issue) is sharing some more or less static data with other environments. This is possible with channels but requires more setup/work.
• Minetest does not allow arbitrary thread spawning, and you have to make do with the ones that exist. Except for the emerge thread none of these should be blocked for extended periods of time. This reduces the usefulness of channels that can be blocked on.

[BuckarooBanzay]

nice work 👍

do i get this right: this is basically a persistent and global key/value store and i have to make sure the items don't pile up and fill my memory (as a modder)?

[sfan5]

do i get this right: this is basically a persistent and global key/value store and i have to make sure the items don't pile up and fill my memory (as a modder)?

Correct.

[Kimapr]

Minetest does not allow arbitrary thread spawning, and you have to make do with the ones that exist. Except for the emerge thread none of these should be blocked for extended periods of time. This reduces the usefulness of channels that can be blocked on.

What about the async environment?

For my usecase I want a long-running task in its own thread. This is so I can run user-provided code in my (yet to be made, though i have implemented most of the VM for my fictional CPU architecture in Lua already) computer mod without blocking the main thread. The main thread and the code execution thread would communicate with each other by sending messages, where the main thread would send events to respond to and the VM in the execution thread would send actions it wants to perform in the world. Without an IPC mechanism i would have to serialize, send the entire VM state back and forth and reinitalize it on every event or action, which might be far too much overhead for this to be practical.

The main usecase (also mentioned in the OP issue) is sharing some more or less static data with other environments. This is possible with channels but requires more setup/work.

If the data is static, no IPC mechanism is needed. Something similar to the way its done in async env could be done for mapgen env, allowing modders to pass initial values. If more communication is needed, pass a channel to the env. Peekable channels could be used to store values.

[sfan5]

What about the async environment?

There is a fixed maximum amount of async workers, so if you block one of them you risk stalling all other jobs.

If the data is static, no IPC mechanism is needed. Something similar to the way its done in async env could be done for mapgen env, allowing modders to pass initial values.

There is in fact no good way to pass static data into the async env. You can pass data together with every job, but then you pay the transfer cost every time.
I considered implementing a mechanism for one-way static data copies from the main -> mapgen env, but a shared data table like it is now is bother simpler and more powerful.

Regardless in #14451 OP requested the ability to change this data at runtime. So it's not really static.

[Kimapr]

There is a fixed maximum amount of async workers, so if you block one of them you risk stalling all other jobs.

I think this limit should be removed. It's not even mentioned in the docs. Blocking only one of them also shouldn't stall anything as the work stealing algorithm will distribute the jobs among remaining workers.

[sfan5]

If the maximum of worker threads is 1, then blocking a single one will block everything.
A generalized thread API could definitely be added one day but I think "repurposing" the async jobs for this is wrong.

---

Due to lack of usecase I have decided to not implement a polling function. This PR is now ready.

[Kimapr]

Due to lack of usecase I have decided to not implement a polling function. This PR is now ready.

:/

[sfan5]

I changed my mind. People will find uses for it when it exists.

[Desour]

It might also be useful to have an async polling function with a callback. With this one could do the same as checking a value in each server-step, but with less latency:

core.ipc_set("mymod:please_write_it")
-- wait for value
local function wait_for_it()
  local val = core.ipc_get("mymod:val")
  if val == nil then
-   core.after(0, wait_for_it) -- not called until next step
+   core.poll_async("mymod:val", nil, wait_for_it) -- called as soon as possible. (nil given for comparison, similar to futex API)
  else
    print(val)
  end
end
wait_for_it()

(Mods can also use it for synchronous polling by yielding a coroutine.)

[sfan5]

It might also be useful to have an async polling function with a callback

This would not share any code with the blocking implementation and have no advantages over manually polling with core.after(0, ...).