Cookies help us deliver our services. By using our services, you agree to our use of cookies. Learn more

crowdAI is shutting down - please read our blog post for more information

IEEE Investment Ranking Challenge

Explore methodology to identify assets with extreme positive or negative returns.


Details for Round 2

Posted by benharlander about 2 years ago

Round 2 is open to all challenge participants. Round 1 focused on prototyping models that maximized statistical measures and Round 2 will enhance this with a deeper dive into your methodology and a new set of holdout data from 2017. To compete, all participants must submit the following items:

  • Final predictions for all time periods

  • A brief written solution using the IEEE template for conference proceedings. MS Word and LaTeX are both acceptable. At a minimum, the document should include an introduction, description of your methodology, results, and any other information needed to understand your solution and its merit. Tables, charts, and other visuals are highly encouraged.

  • All code and files needed to reproduce your results uploaded to Gitlab. (More details on this soon)

The top 6 solutions will be selected based on their statistical performance, calculated in the following manner:

Final score = (A+B+C+D)/4


  • A = Rank of spearman correlation on holdout data from 2002 – 2016

  • B = Rank of NDCG score on holdout data from 2002 – 2016

  • C = Rank of spearman correlation on holdout data from 2017

  • D = Rank of NDCG score on holdout data from 2017

All ranks will be determined using 3 significant digits. Performance on the data from 2017 will be used as a tiebreaker if needed. Participants will still have access to the crowdAI API to test their submissions on the original dataset, but no feedback on the holdout from 2017 will be provided until the challenge closes.

Posted by lance  about 2 years ago |  Quote

Thanks @benharlander

So according to the above there will be no penalty for overfitting the validation data, in order to boost A and B?

Posted by BearStrikesBack  about 2 years ago |  Quote


Can you answer a question about overfitting on 2002-2016 data? As far as I can see at the moment the winning strategy in not to create robust model, but to overfit on 2002-2016 data and make neutral submission for 2017 data. That’s apparently will maximize scores, but would have poor generalization ability.