Simulation-Based Design of Humanoid Robots (Project 170)

[robertogl]

Contribute to the discussion by asking and/or answering questions, commenting, or sharing your ideas for solutions to project #170

[rhyde005]

This is a paper that gives some insights into how we control our limbs and which should be useful when working on this project:
https://www.researchgate.net/publication/7901827_Threading_neural_feedforward_into_a_mechanical_spring_How_biology_exploits_physics_in_limb_control

[Lizzyv00]

Hi,
We're just starting to make some progress on the model, but have a few questions.

How do we implement a balance algorithm alongside reinforcement learning without them interfering with each other? Should we assign a set of motors to balance, and a set to reinforcement learning? We need to do more research on this - would you be able to point us to some useful resources?

From our research we've encountered HER, which seems like it could be great to use in the model. However, the only examples of implementation we've seen are in python - are there any examples of its use in MATLAB, or would we be better off not focusing on it?

We've made great use of the onramp facilities for various topics, but sadly there isn't one for genetic algorithms. Are there any resources for this?

Thank you!
Lizzy

[rhyde005]

Hi Lizzy

Good you're making progress with the model, and thank you for your questions.

As regards balance algorithms mixing with reinforcement learning, yes, you can use the two together. One approach is to use successive feedback loop closing. If you first design your balance controller, you can then add the reinforcement as an outer loop, this providing demands to your balance controller. So you'd need to think about what the balance controller should be doing when the robot is commanded to move forwards, for example. When standing still, then the demand for robot centre of gravity needs to be within the contact area mapped out by the feet contact points on the ground, but when walking the demand might need to move outside of this area in the desired direction of travel. Another approach is to switch between controllers based on whether the robot is requested stationary or moving.

I'm not sure what HER is - a quick google search didn't show any AI or reinforcement learning related results. My advice would be to stay within the MathWorks toolset - we have many capabilities within the reinforcement learning toolbox, deep learning toolbox plus support for multiple optimization methods. More important to think about is how to set the problem up for AI/optimization/classical feedback design.

For genetic algorithms, our documentation help should be more than sufficient to figure out how to configure an optimization. The robot example you're starting with already has one training option that used genetic algorithms, so perhaps take a look at that for a good example. If you're thinking about using genetic algorithms, I'd suggest looking for a textbook that gives some insight into what types of problem genetic algorithms are good for and examples of their use.

Most important piece of advice: when looking at/assessing design methods, experiment with a simple problem first. Control of an inverted pendulum is a great place to start, for example.

Hope that helps

Rick

[Partwolf]

Hi,

I'm Paul Roberts, working on the Environment sub-team for the project

I have been trying to alter the dimensions of the simulation topography, from checking into the Example system, which uses a single plane, I had planned to make changes to the height variable, using some simple geometry equations to affect slope, but this has turned out to be a problem, as far as I can establish this is because of the blocks used in Multibody

I have started trying to look into multibody and have not been able to find an onramp or similar for multibody, are there any specific resources that you would recommend?

Thanks

Paul

[rhyde005]

We do not have an on-ramp for Simscape Multibody. However, you should be able to work though the links and examples on this page to get started: Get Started with Simscape Multibody

I suggest you start with modelling something simple like a pendulum rather than starting with the robot example.

Hope that helps

Rick

[martinasciola]

A few more resources on Simscape Multibody:

• tutorials on Youtube: Physical Modeling Tutorial
• video 47 min: Multibody Simulation with Simscape Multibody

[Lizzyv00]

Hi,

I'm attempting to create an inverse kinematics equation for the legs of the robot to use a state space control. I've got the mass and inertia measurements from the inertia blocks within the urdf file, but the centre of mass for each link is [0,0,0]. I understand that this is relative to the link, as the frames of reference are changing, but I need some direction for what to do next. Should I extract the centres of mass of each link relative to a certain point (e.g. the centre of the hips) and go from there, or is there a more elegant way to do this that I am missing?

I also can't seem to find where the lengths of each link are, and only the initial height of the robot. Do you know where these might be found?

Thank you,
Lizzy

[rhyde005]

Hi Lizzy
I think you'll find it easier to get the information from the Simscape model rather than the urdf file. Much of this you can get by opening the robot definition in Simulink and interrogating the individual block parameters (bodies and transformations). For the inverse kinematics solve, you might find this useful: https://www.mathworks.com/help/physmod/sm/ref/simscape.multibody.kinematicssolver.html. This creates a MATLAB object from the Simscape Multibody model that you can use to solve the inverse kinematics problem. Hope that helps
-Rick

[Lizzyv00]

Hi,
We've created a state-space equation of the model, but have found that there are so many trig terms of the inputs, that the system would be classified as non-linear state-space. Therefore, we were wondering if you had any recommendations for resources for non-linear state-space systems. We were wanting to explore this avenue before deciding between it and solely reinforcement learning.
Thank you,
Lizzy

[rhyde005]

Hi Lizzy

Some thoughts. When a foot is not in contact with the ground, things you might want to control are landing position and leg landing angle. It is known that, when animals run/walk, leg landing angle is important when trying to regulate the trajectory from one stride to the next. When a foot is in contact with the ground, then things you might want to control are planar (x direction, y direction) forces and force in the direction of the leg.

Inverse kinematics can be used to work out joint angles required to place the foot and leg angle, and this would be one way to get target demands for the individual joint actuators. This is what conventional manufacturing robots do. However, here the actuation is assumed reversible (you can back-drive the motor and sink energy as well as acting as an energy source). This reduces energy consumption, energy being recaptured when the foot its the ground during a walk or run. This makes inverse kinematics less useful, and for better accuracy, you would need to invert the dynamics i.e. include energy stored by the mass and inertia of the limbs. One approach to this is nonlinear dynamic inversion (NDI).

I suggest that, in the first instance, you use inverse kinematics to design a controller that tells you what the individual actuator positions should be to meet a demanded foot position and leg angle, and then repeat for foot forces when in contact with the ground. This gives you a baseline that you can then use with reinforcement learning, it working with demanded foot and leg quantities plus target limb stiffnesses. Once you've done that, then read up on NDI and first apply it to a pendulum and then double pendulum to get familiar with the method. You can then have a go at applying it to the robot leg.

State-space methods could, in theory, be applied to this problem if you gain scheduled your controller with limb angles, but the dimension of your controller quickly becomes large with a humanoid robot given the number of degrees of freedom.

Hope that helps

Rick

[Partwolf]

I'm struggling to find a great deal in the way of research papers relating to simulated environments in AI that don't immediately link back to video games and/or are not speculative and from the late 90's...any ideas on where to look?

[rhyde005]

I'm not immediately familiar with the AI literature - my background is in conventional control and system modelling. That said, I'm sure there is a lot written about AI in the context of robotics. One thing is to try varying your search criteria, for example using "reinforcement learning" in place of "AI" (which is much more general). I tried "reinforcement learning in robotics" and got lots of useful hits in Google. Try using Google Scholar in place of just Google to find the more useful references. You could try replacing "robotics" with terms like "Locomotion" or "walking". If it is the actual robot-environment interaction you're looking for, then video games are good place to look - my understanding is that the physics in many video games is reasonably accurate.

[Lizzyv00]

I'm struggling to find research papers on NDI that aren't solely for use in aerospace engineering. Even the control theory is in terms of parameters needed for flight. What I'm going with for now is to correspond roll, pitch, and yaw of a plane to that of the foot. I just wanted to check before ploughing further that you don't have any ideas of where to look at NDI in a solely control or robotics field?

[rhyde005]

This looks like a good resource:
https://ethz.ch/content/dam/ethz/special-interest/mavt/robotics-n-intelligent-systems/rsl-dam/documents/RobotDynamics2017/RD_HS2017script.pdf
I suggest discussing with your project supervisor too. NDI would normally be a a topic for a research Masters or PhD, and much depends how much time you have allocated to your project as there is quite a learning curve. You might want instead to consider focusing on inverse kinematics plus some conventional PID control, perhaps gain scheduling PID gains with leg posture if needed.

[rhyde005]

Here's another useful reference: Dynamics, Modelling, Simulation and Control of a Spacecraft Manipulator System

[Jack9Taylor]

Hi, I'm Jack Taylor from the Motion Subteam working on the project.
In order to allow continuous actions/observations with sparse binary rewards, I have used DDPG reinforcement learning with HER (Hindsight Experience Replay). Since HER is seemingly not supported by the reinforcement learning toolbox, I have had to code this as a MATLAB script using the actual algorithms. This has led to me realising a problem in that the part of the algorithm that interacts with our simulated environment needs to input an action to the environment at each timestep in a loop and then receive the resulting new state. Oppositely, a Simulink file started in a MATLAB script runs to completion before any following code is run and therefore I don't know how to interact with it during runtime to alter the action performed at different times during the simulation.
I presume that in the reinforcement learning toolbox functions, the two programs interact step-wise at a lower level. Is there a way for me to do this or is there some other existing function for this that I don't know about?

Thanks,
Jack Taylor

[rhyde005]

Hi Jack
Take a look at the Simulink /User-Defined Functions/ sublibrary. There you will find various ways to call MATLAB during a simulation. The Function Caller block is a good place to start. I don't think pausing the simulation will work.
Rick

[Jack9Taylor]

I'm currently using set_param to 'start' the simulation, then using set_param to 'update' after each time I change the actions, then using pause() to pause the matlab script for a timestep whilst the simulation keeps running, and then taking the new observations. Would this be more likely to work given that the simulation doesn't get paused?

[rhyde005]

I'm not clear what you're trying to do. Normally you would run the model from Simulink (either from the toolstrip, or invoke the simulation from MATLAB). Then during the simulation, at every controller discrete sample time, the controller gets updated. My suggest relates to this scenario and using the Function Caller block to call your control algorithm. Is this the set-up you have, or are you trying the run the Simulink model as part of a predictor e.g. simulate 5 seconds ahead to see what the state will be given some given control demand?

Perhaps if you could send me a diagram or short Word document summary then that would help me answer your question.

Kind regards

Rick

[Jack9Taylor]

After some thinking, I believe my problem is actually that I can't find a function that either: pauses a matlab script (but not simulink) until a condition is met (like waitfor but it accepts boolean inputs), or acts as a pause toggle for a matlab script but not simulink (like uiwait and uiresume but without pausing simulink). There are condition-based pause functions but the "condition" always seems to be non-programmatic like a user pushing a key.

[rhyde005]

Take a look at Simulink enabled subsystems - with these you can pause/run the content of the subsystem based on a logical condition. Another way would be to write your MATLAB code with an enable flag input, and to use this within the MATLAB to control when state is updated or not. Without seeing a summary of your your simulation set-up, it's hard to be more specific.

[Lizzyv00]

Hi,
I'm starting to work on implementing controllers based on inverse kinematics to an inverted pendulum. Whilst I'm doing this, I've found some great youtube videos by MATLAB, for example 'Modeling and Simulation of Walking Robots' by Sebastian Castro. The resources he's used are supposedly on the MATLAB file exchange, but I can't seem to find them. I don't know if the case is that they are not available for public use, but I was wondering if there was anything you could do to make them accessible for us? They contain information on contact points, creating a GUI, and using nested PID controllers - all within simscape multibody.
Thank you,
Lizzy

[rhyde005]

Hi Lizzy
I believe this is the example: https://www.mathworks.com/help/reinforcement-learning/ug/train-biped-robot-to-walk-using-reinforcement-learning-agents.html
It ships with the Reinforcement Learning Toolbox which is why you couldn't find it on File Exchange.
Rick

[Jack9Taylor]

Thank you for the previous help.
Paul and I are currently trying to find a way to run our Simulink environment just in terms of its calculations whilst suppressing any graphical simulation or real-time (1 second of simulation taking 1 real second to run) simulation. This is so that we can suppress these aspects during training (when we just need outputs for each timestep as quickly as possible) and allow them when calling the simulation for display purposes (when graphics and delays are needed). Is there an input to the sim() command that toggles this? Or perhaps a block in Simulink that toggles this based on a variable in the MatLab workspace?

Thanks,
Jack

[rhyde005]

You can simulate a model without opening the model canvas and scopes. You can use load_system('modelName') to preload the model into memory, and then call it with the sim command. However, I don't expect it will save you much simulation time - but you can give it a try.
Thanks
Rick

[Anirudha7]

Hi, I am a student at DYPCOE Pune and am interested in working on and collaborating on this project.

[rhyde005]

Welcome Anirudha. Please put any questions you have into this discussion board and let us know how you get on.
Many thanks
Rick

[patweatharva]

Hello, I am currently studying robotics and working on this project. I am using TD3 RL algo, which will take reference commands from a planner for doing things like walking, jumping, running, etc. I was thinking of using the ref commands demanded from the planner to shape the reward for that particular episode. Will you be able to comment on my approach?

[rhyde005]

I'm no expert on different types of reinforcement learning, so suggest discuss this with your academic supervisor. The one piece of advice I do have is to build what you can into the problem formulation so that the learning part has fewer inputs to work with and/or a smaller search space. An example might be to build in some inverse kinematics into the problem formulation to avoid just presenting raw joint angle commands to the controller. For example, leg landing angle is known to be an important gait control variable in animal locomotion; finding a formulation that exploits this knowledge may result in faster convergence or a better solution.
Rick