Scoring

Your submissions will be ranked in two leaderboards. Predictions for images from the lightbox category will be evaluated in the lightbox leaderboard and images from the sunlamp category will be evaluated in the sunlamp leaderboard. The total score is indicated on each leaderboard in the Best Score column. There is no ranking for performance on synthetic images, as the task is to bridge the domain gap between synthetic and real images.

In additon to the score, a separate column will describe the orientation score and position score separately (see pose score calculation for details).

During the competition, submissions are evaluated on a subset of all test images. We do this to encourage the development of genuine pose estimation methods and diminish the impact of exploiting the aggregated feedback of the leaderboard, i.e. overfitting to increase one's rank. At the end of the submission period, all submission will be re-evaluated on the complete test set and each team will be ranked by this final score.

Pose score calculation

The pose score is calculated in two steps: first, we compute the orientation and positions score, then we combine these two.

The position error for image i is the difference between the estimated and the ground truth position vectors, normalized by the ground truth distance of the satellite.

\(\text{err}_{position}^{(i)} =\frac{\mid r_{gt}^{(i)} - r_{est}^{(i)} \mid_2}{\mid r_{gt}^{(i)} \mid_2}\)

For each 2.173mm per m groundtruth distance, we correct the position error to zero to account for our machine precision. Thus, the position score for image i is defined as 

\(\text{score}_{position}^{(i)}= \begin{cases} 0,\qquad~~~~~~~~~~\text{if}~ \text{err}_{position}^{(i)} < 0.002173\\ \text{err}_{position}^{(i)},\quad\text{else}\\ \end{cases}\)

We define the orientation error as the angle of the rotation, that aligns the estimated and ground truth orientations:

\(\text{err}_{orientation}^{(i)} = 2 \cdot \arccos(\mid \langle q_{est}^{(i)}, q_{gt}^{(i)}\rangle \mid)\)

We correct the orientation error within 0.169° to be zero to account for our machine precision. Thus, the orientation score for an image i is defined as:

\(\text{score}_{orientation}^{(i)}= \begin{cases} 0,\qquad~~~~~~~~~~~~~~\text{if}~ \text{err}_{orientation}^{(i)} < 0.169°\\ \text{err}_{orientation}^{(i)},\quad\text{else}\\ \end{cases}\)

The pose score for an image i is the sum of its orientation and position scores:

\(\text{score}_{pose}^{(i)} = \text{score}_{orientation}^{(i)} + \text{score}_{position}^{(i)}\)

Finally, the total score is the average of pose scores over all images N corresponding to the test set (sunlamp or lightbox):

\(\text{score} = \frac{1}{N} \sum\limits_{i=1}^{N} \text{score}_{pose}^{(i)}\)