Cookies help us deliver our services. By using our services, you agree to our use of cookies. Learn more

crowdAI is shutting down - please read our blog post for more information

Learning How to Walk

Reinforcement learning environments with musculoskeletal models


What is exactly considered "walking"

Posted by ViktorF over 4 years ago

According to the competition rules the goal is to maximize pelvis_tx at end of 500th iteration while keeping pelvis_ty above 0.7 all the time.

It does not state what is considered walking, so jumping like a kangaroo is also considered as a valid solution. It is a lower degree of freedom model which is much easier to train. Since the human model does not seem to have hands it might actually be faster to jump forward than to walk…

What is considered as “walking” exactly?

Also: The human body model contains activation minimum 0.0 and maximum 1.0, but the grader accepts and happily executes activation values above 1.0.

Posted by Lukasz_  over 4 years ago |  Quote

Yes, for this challenge we define walking as moving as quickly as possible on the x axis. For the challenge of this kind it would be hard to define ‘walking’ well without using tracking data. The hope is that the winning algorithm can also easily adapt to other constraints if needed. Let’s try :) Thanks for your comment!

Posted by spMohanty  over 4 years ago |  Quote

Hi @ViktorF,

Regarding the point you made about activations outside the range [0,1] . They are actually clipped to this range internally before being applied. So whenever you apply an activation say N > 1.0 then it will be clipped to 1.0 and when N < 0.0 it will be clipped to 0.0

Cheers, Mohanty

Posted by ViktorF  over 4 years ago |  Quote

It looks like a kangaroo will win :)

Posted by ViktorF  over 4 years ago |  Quote

syllogismos, how does your question belong into this topic?

Please create a new topic and ask there to make it more visible.

Posted by ViktorF  over 4 years ago |  Quote

Also, we should create a classifier telling us whether a solution belongs to one of the following categories:

  • Kangaroo
  • Human
  • Rolling robot
  • Other



Posted by spMohanty  over 4 years ago |  Quote

@syllogismos: Weird. I tried it out myself and it seems to work as expected for me.

Can you tell me what OS are you on when you run this script ? Small variations in the state space can in principle be explained by difference in running environments (OS, opensim version, simbody version).

The grader currently runs on a Ubuntu 16.04 instance. Assuming you used the osim-rl conda package, I dont think there will be any variations in the opensim or simbody version. So the most likely culprit could be the variation in the OS.

Can you try to quantify the variation in some way ? Maybe plot the euclidean distance between the actual and expected over time ?

Cheers, Mohanty

Posted by syllogismos  over 4 years ago |  Quote

@VictorF sorry, I was clumsy and started discussing here.. spammed also quite a bit, because I was worried I was training on different version of GaitEnv. I apologise. Will delete my posts.

@Mohanty, I installed everything on ubuntu, and there I don’t see the difference. I only see it on my personal OSX computer. Will work on ubuntu from now or reinstall everything on my computer and see. Thanks for the clarification. So it’s just on my computer.. Sorry for the trouble..

Posted by syllogismos  over 4 years ago |  Quote

On topic, @victorF, yours is my favourite bot. Its actually the one walking like human, In my case somehow its not able to learn that there is symmetry and both right and left side of the body is similar. What you call kangaroo is also symmetric because both legs jump together. I want to use imitation learning from one of my trained agents and switching left and right side observations and make it work. I don’t know if I will be able to do it or if its feasible.. Or if I will make any significant progress by doing so. Maybe I will give it a shot after the competition.

Posted by ViktorF  over 4 years ago |  Quote

Cheat: Jump high at 45 degrees ;)

Detailed idea: * Duck down in a few iterations. * Jump forward at 45 degrees with very high force. (It does not seem to be limited by the current grader.) * It would cause the skeleton to fly on ballistic curve, like a cannon ball. * Win: It would be 10k+ score in 500 iterations.

Not walking, granted.

Can it be trained? Sure.

Why haven’t our networks figured it out already?

Posted by shawn  over 4 years ago |  Quote

@ViktorF: Interesting, are you saying there is no force limit?

Posted by ViktorF  over 4 years ago |  Quote

As far as I see there is no explicit clipping of action values in the Python code.

I tried to train such a model out of curiosity, but it does not jump high. So the OSIM library seems to limit the force somewhere in the simulation code.

Posted by tomvw  over 4 years ago |  Quote

@ViktorF: The activation values are indeed clipped between 0 and 1 by the OSIM code. Also their relation to forces produced by the muscles is physiologically modeled in the OSIM code and is quite nonlinear. If you are interested, the following pages provide good explanation: http://simtk-confluence.stanford.edu:8080/display/OpenSim/Muscle+Model+Theory+and+Publications.