Cookies help us deliver our services. By using our services, you agree to our use of cookies. Learn more

crowdAI is shutting down - please read our blog post for more information

NIPS 2017: Learning to Run

Reinforcement learning environments with musculoskeletal models


Second round evaluation

Posted by Anton_Pechenko over 4 years ago

As it is said in “In order to avoid overfitting to the training environment, the top 10 participants will be asked to resubmit their solutions in the second round of the challenge. Environments in the second round will have the same structure but they will be initialized with different seeds. The final ranking will be based on results from the second round.” there will be reevaluation with another seeds. So, how will it be? Do we need to reevaluate exactly the same NN with exactly the same weights? Or we allowed to submit another solution? And how will it be checked that participants used the same weights if it is the constraint?


Posted by spMohanty  over 4 years ago |  Quote

Hi @Anton,

You will get you choose the best performing model according to you. It is not necessary that your current best performing model performs the best there in the final round.

From your end, it will be the exact same process, and you just have to submit again like you would right now.

In the second round, the key points are, the models will be tested using different env initialisation seeds (different than what is being used right now on the grading server), and secondly, the total number of trials (which is 3 at the moment) will be increased to a larger number to ensure that the mean score is as close as possible to the mean performance of the model.

Does this answer your question ?

Cheers, Mohanty

Posted by Anton_Pechenko  over 4 years ago |  Quote

So, We will be able to test number of trained networks and choose the best fitted without limitations by number of submissions, right? Just like as it is now?

Posted by spMohanty  over 4 years ago |  Quote

Hi @Anton Pechenko,

Yeah, you would be allowed to test a number of trained networks in the second round, but there will still be a limit on the number of submissions you can make in a single day.

Cheers, Mohanty