Loading
Cookies help us deliver our services. By using our services, you agree to our use of cookies. Learn more

crowdAI is shutting down - please read our blog post for more information

Train Schedule Optimisation Challenge

Optimizing train schedules


Completed
209
Submissions
444
Participants
29677
Views

Solver time

Posted by Raluca about 1 year ago

Hello,

Is there any way to find out if the solutions submitted take 15 min to solve? It can be quite misleading if people post solutions which take a much longer time despite the low objective function.

Thank you!

Posted by jordiju  about 1 year ago |  Quote

Hi @Raluca there isn’t really, no. We cannot prevent participants to upload submissions which took longer than 15mins for some of the intances.

Of course, before awarding any prizes, we will verify the time limits were observed by actually running the solvers. So there really isn’t much point for people to submit solutions which would be disqualified later and it would also cause unnecessary work for everybody involved.

We appeal to people’s common sense and fair play to not upload submissions which took longer to produce than the allotted time limit.

If someone accidentally uploaded such a submission, please let us know and we will remove it.

Posted by jordiju  about 1 year ago |  Quote

If someone accidentally uploaded such a submission, please let us know and we will remove it.

That includes submissions that violate one of the other challenge rules, of course.

Posted by hur  about 1 year ago |  Quote

@jordiju unless everyone has access to a computer equivalent to yours, self-detection of time limit violation is not realistic. We tried to run the calculations on a VM with 32-cores but the results were unexpectedly way worse than my personal 8-cores laptop and we don’t have a more powerful (physical) machine. Thus I might upload results that take 1h to solve hoping that it will take 15mins on your computer but not necessarily intending to just take the lead in the leaderboard.

Posted by jordiju  about 1 year ago |  Quote

@hur Yes, we realize the situation is not perfect. Still, I strongly urge anyone to only submit what they have been able to run themselves (locally or on some VM/Pod) within 15min/instance under the given hardware-restrictions. If your results on the large VM were worse than on your local laptop, just use the best solution you have been able to obtain locally. It is unlikely we have better hardware than you, so there is no reason to believe that our VMs would be more powerful than the one you tried.

Generally, the strength of the algorithm is more important than raw computing power. So don’t feel too bad if you run your algorithm on 4 or 8 cores only. In many cases - as you also experienced - the performance is even better with less cores.

Posted by LeoB  about 1 year ago |  Quote

It might bei worth to run your final computation in a machine similar to the one of the organizers. I guess a AWS ec2 r4.8xlarge instance might hit the characteristic of their machine pretty closely (as long as they die nothing stupid ;). It’s the Machine I ended using for my final submission.

If you need more power, a c5.9xlarge instance with 4 cores dectivated might give you someone mir Speed and is still within competition rules.

Posted by jordiju  about 1 year ago |  Quote

@LeoB, @hur That is certainly an option, yes. We generally do not use AWS internally but our “own” cloud provider. It may be that our instances are slower than an AWS r4.8xlarge (we honestly don’t even know what kind of CPU we have under our machines). However, if you can show that it works with an AWS r4.8xlarge, we would certainly accept that.

And, again, if the results are actually worse than from your local notebook - just use the local solution. Don’t extrapolate, hoping we would miraculously be able to find hardware that brings computation time down to 15 minutes.

Thanks all!

Posted by hur  about 1 year ago |  Quote

@LeoB thanks, I will check AWS