Challenge Guidelines

As part of the benchmarking effort, we are hosting a competition and offering prizes for the best-performing submissions on the EvalAI leaderboard. Phase 1 of the challenge ended on 1/7/2022, won by AE Studio. Phase 2 of the challenge is ongoing, and will end on April 3rd, 2022, anywhere on Earth (i.e. 23:59 April 3rd UTC-12). Phase 2 is graciously funded by AE Studio.

Ranking and Prizes

The challenge evaluates co-smoothing performance on 5ms bins, on individual datasets and across all datasets. There are four primary datasets: MC_Maze, MC_RTT, Area2_Bump, DMFC_RSG, and three scaling datasets: MC_Maze-Large, MC_Maze-Medium, MC_Maze-Small. The prizes are allotted as follows:

• New: A prize of 2,000 USD will be given to the top model as selected by an external panel of judges according to criteria of model interpretability, usability, and accuracy. There will also be two runner-up prizes of 500 USD. (Co-smoothing performance, along with other metrics, will be considered under “accuracy.”) Eligibility for this prize will require a model writeup/open sourcing.
• A prize of 1,000 USD will be given for best rank on each of MC_Maze, MC_RTT, Area2_Bump, and DMFC_RSG.
• A prize of 1,000 USD will be given for best average rank on MC_Maze-Large, MC_Maze-Medium, and MC_Maze-Small. Prizes will be provided in the form of Visa gift cards.

Eligibility

To be considered for challenge prizes, teams must submit their methods to the EvalAI challenge before the deadline and commit to releasing code for reproducibility. This commitment means that, after the deadline, top contenders will be notified to share their code with the NLB team so that we can validate their submissions, and moreover, that this code must be open-sourced before we transfer funds. Note that NLB organizers are ineligible for collecting prize money.

We recommend participants to publicly share their code to help promote progress in the field.