The leaderboard evaluates all possible scenes (scenes with at least 4 true stars) and determines the number of subsets needed to find a valid subset. If the subset is not found the number of subsets tried + the number of existing subsets is used as score for this scene. The final score is then determined as the average score over all scenes.
The submission is a zip file containing a file called sequence_20_4.csv and the Python source code.
The naming scheme is sequence_N_M.csv, where N is the number of star candidates in the scene and M is the subset size, ie. the number of stars that need to be queried at once. These csv files contain no header and each row is one query for the database. The file has M columns and every cell contains a number between 0 and N-1 inclusive, which is the star candidate ID.
The python code will be evaluated manually whenever possible and submissions will be marked checked on the leaderboard and the participants' submission page. Random algorithms should fix the seed to always produce the same result, but still need to be able to achieve a similar score with a different seed and on other scenarios.
The winner of the competition will be evaluated by running the best submission in the leaderboard of every participant on different scenarios of N with M=4. These scores of these scenarios get summed up and the person with the lowest score wins the competition. In case multiple people have the same lowest score, the submission time decides the winner. Of course only submissions following the rules will be considered, participants with submissions not following these rules will be disqualified.