Loading
Cookies help us deliver our services. By using our services, you agree to our use of cookies. Learn more

NIPS 2017: Learning to Run

Reinforcement learning environments with musculoskeletal models


Completed
2154
Submissions
618
Participants
84429
Views

Submission problems - server error

Posted by hagrid67 over 1 year ago

I seem to be getting these errors repeatedly, as do some others.

This is one of the errors: the HTML (rather than JSON) returned by the the server looks like this, which of course fails JSON parsing:

b’<!DOCTYPE HTML PUBLIC “-//W3C//DTD HTML 3.2 Final//EN”>\n

500 Internal Server Error\n<h1>Internal Server Error</h1>\n<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>\n’

This one happened before the first step.

I have also had errors happening several hundred steps into the submission process (I don’t have the response content for these). If you look at the leaderboard right now, the only submissions since Sep 4th are quite short episodes (reward<10), which might indicate there is more of a problem for the longer submissions.

The problem is compounded by the limit of 5 submissions per 24 hours. I am getting repeated failures, and these use up the limit. Can I suggest that failures either should not use up the limit, or the limit for failures should be higher?

Many thanks.

4

Posted by reason8.ai  over 1 year ago |  Quote

I have similar problem and cant submit for last 12 hours. Question for organizers: are you working on this issue, because topic was created 19 hours ago and there is no response yet? Thanks in advance.

2

Posted by Sean  over 1 year ago |  Quote

The grader is back online now, thanks for the comments.

2

Posted by hagrid67  over 1 year ago |  Quote

Hi thanks Sean - can I ask if it still works for you now? It’s failing for me at the moment (about 2 hours after your comment). (I’m in the UK / UTC+1 / BST timezone)

Might it be possible for you to run a test instance of the grader, or share the code (minus the random seed / special sauce) so I could test again my own instance? (Is the code out there already? Haven’t looked for it yet)

[2017-09-08 12:37:29,210] POST http://grader.crowdai.org:1729/v1/envs/ {“env_id”: “Run”, “token”: “XXXX_real_token_hidden___XXXXX”, “version”: “1.4.1”} JW JSON Decode error: b’<!DOCTYPE HTML PUBLIC “-//W3C//DTD HTML 3.2 Final//EN”>\n500 Internal Server Error\n<h1>Internal Server Error</h1>\n<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>\n’

2

Posted by hagrid67  over 1 year ago |  Quote

OK I just got past the initial HTTP POST and it’s uploading now (step 165). Let’s see if it completes.

1

Posted by Yujin  over 1 year ago |  Quote

I’m getting 502 Server Error: Bad Gateway for url: http://grader.crowdai.org:1729/v1/envs/, it was 500 Server Error couple of hours ago. Is the grader’s URL changed?

1

Posted by Sean  over 1 year ago |  Quote

OK we are looking into further issues on the grader. A message will be posted here when it is running again.

1

Posted by spMohanty  over 1 year ago |  Quote

Hi everyone,

Is anyone still having problems with submissions ? This should have had been solved by the changes on Friday evening. But if you still have any issues with the grader, do let us know here.

Cheers, Mohanty

2

Posted by reason8.ai  over 1 year ago |  Quote

Hi, I have problem with submission. When trying to submit I get this error: HTTPError: 500 Server Error: INTERNAL SERVER ERROR for url … I hope you fix this problem, if you need more details, please ask. Thanks in advance.

Posted by spMohanty  over 1 year ago |  Quote

Hi @mpavlov,

I restarted the server just to be sure, and tried making a submission. It works for me.

Occasionally you will run into an internal server error if multiple participants are making the submission at the same time (and hence, the server runs out of resources). The advice in that case will be to retry the submission again after 10-15 mins.

Also, this problem will not be seen in the second round of the challenge, as each of the top-10 participants will have an independent grader assigned to them.

Cheers, Mohanty

1

Posted by reason8.ai  over 1 year ago |  Quote

Hi @spMohanty, thanks! I will try to submit now. It also looks like that all unsuccessful attempts to submit are accounted in total sumbission per day. I dont know if it is easy to fix, but I think that submits when server error was encountered should not be accounted into total sumbissions. Thanks in advance.

1

Posted by reason8.ai  over 1 year ago |  Quote

Hello. I tried to submit one solution 3 times at different times. And always get error at the same step. It seem like that this particular step simulation take a lot of time, may be timeouts are too short?: HTTPError: 504 Server Error: Gateway Time-out for url:

Posted by spMohanty  over 1 year ago |  Quote

Hi @mpavlov ,

The submissions seem to be working fine. I just now made a submission. In your case, it could be the weird opensim bug which causes opensim to choke on certain actions. We are looking into it, but I believe if you hit the same case again, you could simply add a “little” but of noise at the said step and see if that solves the issue. I would be curious to know if that helps.

Cheers, Mohanty

Posted by reason8.ai  over 1 year ago |  Quote

Hi @spMohanty, thanks, little noise did not help, but pure random action helped, but it looked liked agent fell with this action. Is it possible to increase timeout? I see from local runs that some actions took a lot of time but I never get simulator errors.