Implementing multi-agent interactions

[Aceticia]

Hi, I'm new to the RL field and this package not only served as a great tool but also helped me learn to understand this field better. I'm still very new in this field so please pardon my ignorance...

I'm starting a multi-agent project, and the policy involves 2 AC-type networks for each agent. These agents have an AC-type network to interact with the environment, called net 1, and they have another network for interacting with each other, called net 2.

The problem for me was the interactions: The interaction between agents generates a reward on its own, and I'm not sure how I can approach it. A few ideas I have in mind:

1. Write a wrapper for the environment I use, and handle the interactions inside the wrapper. However, the environment only returns 1 reward. I can hard code something, but features such as reward normalization won't work on both of them.
2. Handle it inside the policy forward() or learn() calls. However, I will have to call collectors and construct replay buffers manually or sequentially evaluate this interaction between agents, which will very likely cause slow speed and I imagine could potentially break multi-processing.
3. Write an independent environment to handle agent interaction. But since observations from the actual gym environment are needed, I'm not sure how I can sync the replay buffer to the interaction env?

Any suggestion helps. Thank you so much.

[Trinkle23897]

The interaction between agents generates a reward on its own

How will you use this reward? It affects the design choice.

[Aceticia]

This reward is passed to net 2 of each agent and trains them with policy gradient given the observations.

My current solution actually works alright. I'm currently handling everything inside the learn() function by:

1. Iterating over all agents to create a new buffer from the interactions,
2. Pass the newly constructed buffer to update() the net 2 (which can be any built-in policy from Tianshou).

The speed turns out to be ok. I might turn to more sophisticated methods in the future when I need to scale up the number of agents. (Considering that in the future distributed training might be supported, this approach might also prevent me from easily using that feature)

The only downside to this approach, as I just found out, is that I can't update net 1 and net 2 simultaneously. If I do two separate update() calls as-is, the model would have changed when I'm updating the second sub-policy. I'll need to return the loss from these sub-policy' update(), and sum them, and then I can call backward and step. It would be nice to have an argument for update() that accumulates the gradient in its learn() call, this way an arbitrary number of sub-policies can be updated together by accumulating the gradient. Not a major change in functionality, but it won't require much change to the code.

[Trinkle23897]

cool, good to hear that!

[MischaPanch]

This is stale, and MARL likely won't be a focus of development in the immediate future. Closing this