code_acme

Control Ordinary Differential Equations using Deepmind acme framework.

• Implement simple environments with ode physics to investigate continuous action space control with deep reinforcement learning.
• Provide implementation examples of custom environments with dm-control framework specifics.
• Provide examples of using acme framework to train deep reinforcement learning agent for continuous action space control.

The implemented environments are meant to be lightwise to enable training with limited computational resources. However, thanks to acme framework, the experiments can be easily scaled up to allow for distributed learning.

Ideally the project would serve as a tutorial for students and researchers to interfacing their own custom environments with acme framework and exploiting deep reinforcement learning for continuous control purposes.

Installation

We recommend installation with docker container.

Click the drop-down below

docker container

• Build docker image.

make build 

• Run bash shell on docker image.

No gpus

make bash

With GPU

make bash-gpu

The docker image includes all the package dependencies for training including tensorflow, acme,dm_control. Running make bash will mount code_acme folder as a docker volume. This allows to develop your code within or outside the docker container in your favourite environment.

• Install code_acme

From the docker container bash, install code_acme package to make sure you are using the latest version of the package including eventually your local modifications.

pip install .

• Test installation.

pytest

virtualenv

Make sure your user has the writing rights. If not, run with `sudo`.

Generate and activate virtual env.

pip install virtualenv
virtualenv .code-acme
source .code-acme/bin/activate

Install code_acme packages.

pip install .[dev]

Test installation.

pytest

If you are developing with devcontainer and VScode, code_acme contains .json files to set-up the environment.

Available environments and tasks

Here you can find the developed environments.

• tank: Control water level in a tank with nozzle acting on inflow. 1D ode with, stable system, 1 continuous space action.
• moving_coil: Control position of a linear wire attracted by 2 wires fixed in space, acting on the static wire currents. 1D ode, unstable system, 2 competing continuous space actions.

Examples/tutorial

We provide a colab notebook with an introductive tutorial explaning how to implement a simple physical environment with dm-control framework, train an agent for continuous action space control with acme and evaluate the results with tensorboard.

The script examples/train.py provides examples on how to train mpo, d4pg, dmpo agents on the developed environments. From a docker container bash or virtualenv with code_acme installed run,

python examples/train.py --environment tank --agent mpo

Policy checkpoints are store in ~acme path on a dedicate folder for each process. To evaluate and visualize the results of the last updated policy a simple script examples/eval.py is given.

Future plans

Below a high priority wishing list:

• Examples of training agent with distributed learning using launchpad, #12
• Flexibility in storing checkpoints and snapshots during training specifying the storing path following the solution in mava, #13
• Utils for hyperparameters scan with wandb, #14
• Parameters handling with hydra, #16
• Implement more complex tasks and compare DRL performances against linear control solutions.

Contributing

If you have any question reach out at francescocarpanese [at] hotmail [dot] it or in the dicussion.