Cookies help us deliver our services. By using our services, you agree to our use of cookies. Learn more

crowdAI is shutting down - please read our blog post for more information

Visual Doom AI Competition 2018 - Singleplayer Track (1)

ViZDoom Reinforcement Learning


What does evaluation log mean?

Posted by Trinkle almost 2 years ago

I have submitted a random policy bot, but I’m confused about the result in gitlab issue.

  1. “At the beginning, test evaluation will take place on really simple maps.” But in the issue, I saw 10 episode being graded, and they have different frag counts. I don’t know whether 10 maps are generated from the same config or not, and what does “frag counts” and “number of bots” mean?

  2. How can the bot get into the next level if these 10 maps are generated from same config? Will I get into the next level as soon as I pass one of these games or pass all of these 10 episodes?

Many thanks if you response the questions at your earliest convenience!

Posted by spMohanty  almost 2 years ago |  Quote

@Trinkle : For for 1), the evaluation happens across 10 different maps. The current instructions to participants are that, when one game ends, they have to continue trying to connect to the next game, which basically allows us to run an arbitrary number of games with a single container instantiation. The configs you are using are all appropriately overridden internally, so you should not worry about it. “Frag Count” has the classic Doom definition of a Frag, which can be found here : http://doom.wikia.com/wiki/Frag.

And for 2), as I mentioned, the instructions currently as you to connect to the next doom game as soon as one is over , so basically when a single game is over, the game sever closes the current game, and starts a new game with a new map, and expects you to join it. So, each of the episodes are different games (on different maps)

Hope this answers all your questions. Cheers,