Loading
Cookies help us deliver our services. By using our services, you agree to our use of cookies. Learn more

crowdAI is shutting down - please read our blog post for more information

Train Schedule Optimisation Challenge

Optimizing train schedules


Completed
209
Submissions
444
Participants
29670
Views

Questions about the Competition Rules

Posted by LeoB about 1 year ago

I have some questions about the competition rules.

You write that the results must be reproducible. I think this is very good to make this competion fair and I try everything to make this possible. However given the wall-clock time limit it might be realy hard to make the result reproducable if you give the solver a timelimit after which he needs to stop.

In the rules it’s writen that “the solutions must be found by the participants’ solver in no more than 15 minutes per problem instance”. By the “participants’ solver” you mean the hole pipline including pre and post processing?

I guess “more exotic hardware” does not include a server with just a lot more (conventional) cores? (Just to be sure ;-)

The rules do not state anything about “overfitting”. By that I mean tunning certain parameters or even writing special code for certain instances. Is this practice allowed according to the rules?

Posted by Palstek  about 1 year ago |  Quote

I see these same problems too. In principle you can just use a terribly overfitted ML-Algorithm that will output solutions that were used during training but would not be computable in 15min with the programmed algorithm.

Posted by jordiju  about 1 year ago |  Quote

Hi guys. Several answers in sequence of your remarks

However given the wall-clock time limit it might be realy hard to make the result reproducable if you give the solver a timelimit after which he needs to stop.

Sure, at the end of the day we will need to use common sense when we evaluate people’s code. Many solvers are actually surprisingly deterministic, so reproducing the results is rather easy. If someone’s solution is more random, that is ok. However, we will certainly check that we are able to reliably reproduce the solutions a participant submitted in the best submission. By ‘reliably’ I mean that if we run the code 5 times and get to (or very close to) the best submission of the particpant 4 times, then what would be considered reliable. If it only gets there 50’% of the time, then…not so much.

Posted by jordiju  about 1 year ago |  Quote

By the “participants’ solver” you mean the hole pipline including pre and post processing? Yes, the whole pipeline.

Posted by jordiju  about 1 year ago |  Quote

I guess “more exotic hardware” does not include a server with just a lot more (conventional) cores? (Just to be sure ;-) Depends what you mean by ‘a lot’ ;). Up to 32 cores is fine. Do you need more?

Posted by jordiju  about 1 year ago |  Quote

The rules do not state anything about “overfitting”. By that I mean tunning certain parameters or even writing special code for certain instances. Is this practice allowed according to the rules?

and

I see these same problems too. In principle you can just use a terribly overfitted ML-Algorithm that will output solutions that were used during training but would not be computable in 15min with the programmed algorithm.

We realize that. We conciously allow this practice, yes. The reason is that such “overfitting” might actually be a good strategy also in the realy word - for example in a fixed region of Switzerland. So we’re interested to find interesting ideas in this area.

Sure, if you just train which CPLEX-parameters are optimal for a particular instance, we probably don’t learn much. But we don’t forbid it.

Posted by jordiju  about 1 year ago |  Quote

..and sorry for messing up with the quoting…just now realized how it works…

Posted by jordiju  about 1 year ago |  Quote

Just to add, because this is quite a good question:

If someone tries to be funny and runs a solver for, say, a week, then saves the best solutions, then builds another “solver” that basically just reconstructs these solutions, we would - as you might imagine - not award this.

The rules say “your solutions must be produced from the problem instances by your code”. Taken literally, one might argue that a “solver” as described above does not violate this rule.

But, as mentioned, we use common sense when juding the code, and I think everybody would agree that such a solver should not be awarded.

Generally, we would expect that the solvers also work for similar instances that we have not published. This should be a good rule of thumb to decide how far you can go with tuning it for the published instances.

Posted by jordiju  about 1 year ago |  Quote

Follow-up: We have added the following clarification to the Challenge Rules. “In order to be eligible for prizes, The solver must generic enough that it is able to handle problem instances that are similar to the nine published problem instances. Custom tuning of the solver to the nine published instances is allowed, but must be within reasonable limits. For example, (hyper)parameter-tuning of an algorithm based on the nine instances is ok. Reading in solutions, or partial solutions, to the problem instances as part of the ‘algorithm’ is not ok.”