Submission Rules

How to make a submission

Pose estimated for images of the test and real_test sets must be submitted in a single .csv file. In the file each line should correspond to an image, containing the following information:

• the filename of the image (e.g. img000014.jpg, img000021.jpg, img000001real.jpg, etc.)
• 4 floats describing the orientation quaternion
• 3 floats standing for the relative position vector.

$$
\scriptsize \begin{array}{cccccccccl} \hline \mbox{line in file} & filename &[q_1 & q_2 & q_3 & q_4]& [r_1 & r_2 & r_3] & [m] \\ \hline [0]& string & float,& float,& float,& float,& float,& float,& float & \\ [1]& string & float, & float,& float,& float,& float,& float,& float & \\ ... \\ \hline \end{array} $$

We provide a simple example of writing valid submissions as part of the speed-utils repository can be cloned here: https://gitlab.com/EuropeanSpaceAgency/speed-utils

Submission processing

As soon as a file is submitted two steps will happen in our servers: a) preliminary validation and b) scoring step. During the validation step we try to filter any invalid submissions (e.g. invalid fields in the submission files, unknown filenames, etc.), and make sure that the submission contains pose estimates for all test and real_test images.

If a problem arises at this stage an error code and a description of the problem will appear to help the user debug the problem. Otherwise the preliminary validation is passed, and the submitted file will be stored in our servers and appear in the submission list.

Then scoring is run on the submission. If scoring is successful, the result will appear on the leaderboard, otherwise it will appear in the private submission list as an invalid submission. Score on the real_test set is listed on the leaderboard as Real Image Score, while score on the (synthetic) test set is displayed in the Best Score column.