Loading
Cookies help us deliver our services. By using our services, you agree to our use of cookies. Learn more

crowdAI is shutting down - please read our blog post for more information

Train Schedule Optimisation Challenge

Optimizing train schedules


Completed
209
Submissions
444
Participants
29677
Views

Validator calculating penalty for problem instance 7 wrong?

Posted by LeoB about 1 year ago

What I get from the grader for my solution for problem instance 7 is:

There are 3 warnings and 0 errors

the solution has 3 warnings. It will be accepted as a feasible solution. However, it will incur 47.183334 penalty points in the grader.

Warnings: - Solution with VP-Label “07_V1.22_FWA” and problem_instance_hash “-1799980237” has a wrong Hash! Hash: 1539370788, expected: 1570521172 - Entry time 07:52:14 after einMax 07:52 for Train run sections with route_section_id “18225#302” and Section Marker “PF_Halt” in service_intention “18225” - Exit time 06:44:57 after exit_latest 06:44 for Train run sections with route_section_id “68333#30” and Section Marker “ZAU” in service_intention “68333”

Controlling the solution manually I do not see any weights for entry or exit delay other than one. I do not see any other penalties. So when I calculate the above delays I should get (14+57)/60 = 1.183. Why does the validator disagree with me on this one?

Posted by jordiju  about 1 year ago |  Quote

The total penalty is the sum of delay and route penalties. My first guess would be that the difference can be attributed to route penalties.

Unfortuntaley the grader does not print out the details and since routing penalties do not produce warnings, it’s a bit hard to follow what exactly the grader does. We will improve it so it prints out the components of the total penalty.

In the meantime: the validation_result object has an attribute ‘detail’ which contains the individual penalties making up the total. Could you check what they contain? If they don’t sum up to 47.183, we would indeed have a problem.

Example: notice the ‘minutes_of_dealy’ and ‘route_penalty’ fields. The ‘objective_value’ must be the sum of those two. ~~~~ print(validation_result)

{‘business_rules_violations’: [], ‘solution_hash’: 1538680897, ‘objective_value’: 0.0, ‘details’: {‘minutes_of_delay’: ‘0.0’, ‘route_penalty’: ‘0.0’}, ‘request_uuid’: None} ~~~~

Posted by LeoB  about 1 year ago |  Quote

Thank you for your fast answer!

So the route penalty is 46, so this sums nicely up. But if I check on my own validation script, I do not have a route with a penalty on it. Off course my script might also be buggy. But as I more or less just read in the penalties from the source file its hard to mess it up. It would be realy great to have some more output from the validator to check whats wrong.

Posted by ms123  about 1 year ago |  Quote

I have a similar issue with instance 3 (and probably others, I did not check all), e.g., when I check with my own validation script, I should not have any penalty, but with your validation script, I have a penalty of 0.7, but your script does not give any output, where this penalty is coming from, so I guess it is a route penalty? However, can route penalties be fractional?

Posted by LeoB  about 1 year ago |  Quote

Don’t have any code to check at the moment. But route penalties can be fractional. In problem instance 8 you have a lot of penalties of 0.5.

Posted by ms123  about 1 year ago |  Quote

@LeoB Thank you, I missed the fractionality of the route penalties

Posted by jordiju  about 1 year ago |  Quote

but your script does not give any output, where this penalty is coming from, so I guess it is a route penalty? However, can route penalties be fractional?

Yeah, sorry, I realise now that this would be helpful. Unfortunately it is too big a change in the validator to add it now. Maybe somebody could come up with a little script that checks a solution for which sections incur route penalty? We could add it to the utils…

Posted by LeoB  about 1 year ago |  Quote

@jordiju but would it be possible to debug with my solution if I send it to you by mail. I just can not find an error in my code and therefor still think the validator might calculate the route penalty wrong. BTW my script agrees with the penalty on all other solution (which I have) except for solution 7.

Posted by jordiju  about 1 year ago |  Quote

@jordiju but would it be possible to debug with my solution if I send it to you by mail. I just can not find an error in my code and therefor still think the validator might calculate the route penalty wrong. BTW my script agrees with the penalty on all other solution (which I have) except for solution 7.

Sure, go ahead and send to julian.jordi@sbb.ch

Will look at it tomorrow.

Best, julian

Posted by jordiju  about 1 year ago |  Quote

Hi all

We have added a simple script that basically joins the route_section information from the problem instance to the solution. This way you can easily see which route_sections resulted in how much routing penalty.

You find the script here: https://github.com/crowdAI/train-schedule-optimisation-challenge-starter-kit/blob/master/utils/route_penalty.py

It’s quick and dirty and not executable, you have to run it in a REPL or an IDE or copy-paste into a notebook. (If somebody wants to make it executable, pull requests welcome ;) ).

Posted by ms123  about 1 year ago |  Quote

I have a solution for instance 8, where the validator-script says

the solution has 2 warnings. It will be accepted as a feasible solution. However, it will incur0.15penalty points in the grader.

Warnings: - Solution with VP-Label “test” and problem_instance_hash “-1187628662” has a wrong Hash! Hash: 42, expected: -402269277 - Entry time 11:22:09 after einMax 11:22 for Train run sections with route_section_id “18239#302” and Section Marker “PF_Halt” in service_intention “18239”

However, the solution uses some route sections with a penalty of 1, e.g., 20539#50075, thus it seems the validator may not be calculating the penalty correctly?

Posted by jordiju  about 1 year ago |  Quote

However, the solution uses some route sections with a penalty of 1, e.g., 20539#50075, thus it seems the validator may not be calculating the penalty correctly?

Please forward this solution to julian.jordi@sbb.ch so we can debug. Summing up penalty should be simple, but can’t rule out that some bug sneaked in…

Posted by jordiju  about 1 year ago |  Quote

Hi all

The example provided by @ms123 pointed us in the right direction. Indeed there is another bug in the way the routing penalty is calculated. A fix will be deployed on Monday. This does not affect the validity of solutions. Any solution that is considered valid by the current validator will remain valid. The only difference will be a possible increase in the routing penalty.

Posted by Raphael  about 1 year ago |  Quote

The fixed version of the grader has been deployed.