Deep Reinforcement Learning for Adaptive Traffic Light Signal Control

This repository aims to provide a framework that can be used to easily define an environment compatible with OpenAI Gym for an arbitrary road network defined in a SUMO simulation configuration.

How to train an agent [LINUX ONLY]

The pre-trained agents are not provided. However, you can train one by yourself within 4-8 hours, dependinding on your hardware. Just run the following script:

python tls/agents/agent_dqn.py \
  --net-file path/to/*.net.xml \
  --config-file path/to/*.sumocfg \
  --additional-file path/to/*.det.xml \
  --num-iters 1000 \
  --checkpoint-freq 100 \
  --mode train

To evaluate the trained agent, replace path to the trained agent in tls/agents/agent_dqn.py and all the files needed for the environment initialization in tls/rollout.py . Then run the following script:

python tls/agents/agent_dqn.py \
  --mode eval

The repository also provides several other scripts to train APEX DQN and PPO agents in the directory tls/agents/.

How it works

The simulation environment is based on an open-source microscopic traffic simulation package Simulation of Urban MObility (SUMO). The implementation of all RL algorithms is taken from RLlib: Scalable Reinforcement Learning.

On the diagram below the interactions between the components of the framework is shown. The Environment component is the key one because it abstracts the interaction with the simulation and provides an interface to initialize, step through and reset the simulation.

During initialization of the Environment component the following things happen:

1. The SUMO road network definition is pre-processed and for each traffic light extracted an internal representation of the corresponding intersection that can be used to observe the situation at the intersection by the reinforcement learning agent in a convenient way;
2. Inside Controller component, from the additional file, defining the detectors in the simulation, extracted information about installed in the road network induction-loop detectors;
3. Created Controller component that is responsible for the interaction with the environment and initialized with the extracted traffic light skeletons and information about the detectors; Additionally, the following things happen
   • The controller Trafficlight is created for each traffic light;
   • The state observer Observer is created for each traffic light.

After an environment has been created and the SUMO process with the appropriate configuration files has been started, the reinforcement learning agent can start to interact with the environment by calling the step through function repeatedly, passing in a joint action. The actions are applied to the system, then the simulation is progressed one step further, and the result of the simulation step is returned back to the agent.

Intersection observation

The intersection defined in the sumo network configuration is represented as a 1 or 0 valued matrix. An example of how an agent see the environment shown below.

Internal network representation

The internal representation of a road network definition is a JSON object and its schema is presented below. The keys of the JSON object store information about the relative position of lanes, which constitute the observed part of the road network. The intersection is split into several segments, one for each side of the world. Each segment contains a list of lanes that are adjacent to the intersection. Because in the SUMO network definition the lanes are separated by connections, each lane in the list is represented as another list, where the sequence of lanes actually corresponds to a single physical lane with direction and offset additionally specified. Sometimes segment can connect two intersections, where only one of them must be uncontrolled, then additionally the segment may contain internal representation of a nested intersection. The nested intersection has the same structure, however, with the specified offset from the main intersection and unspecified segment that creates the connection.

{
  "id": %JUNCTION_ID%,
  "offset": [%X_OFFSET%, %Y_OFFSET%],
  "segments": {
    "bottom": {
      "junction": {
        %NESTED_JUNCTION%
      },
      "lanes": [
        [
          [%START_POSITION%, %DIRECTION%, %LANE_ID%],
          ...
        ],
        ...
      ]
    },
    "right": {
      %ADJACENT_SEGMENT%
    },
    "top": {
      %ADJACENT_SEGMENT%
    },
    "left": {
      %ADJACENT_SEGMENT%
    }
  }
}