Oct. 25, 2021, 9 a.m. UTC

March 31, 2022, midnight UTC

The competition is over.

Your submissions will be ranked in two leaderboards. Predictions for images from the *lightbox category* will be evaluated in the *lightbox leaderboard* and images from the *sunlamp category* will be evaluated in the *sunlamp leaderboard*. The** total score** is indicated on each leaderboard in the **Best Score **column. There is no ranking for performance on synthetic images, as the task is to bridge the domain gap between synthetic and real images.

In additon to the score, a separate column will describe the **orientation score** and** position score** separately (see pose score calculation for details).

During the competition, submissions are evaluated on a subset of all test images. We do this to encourage the development of genuine pose estimation methods and diminish the impact of exploiting the aggregated feedback of the leaderboard, i.e. overfitting to increase one's rank. At the end of the submission period, all submission will be re-evaluated on the complete test set and each team will be ranked by this final score.

The **pose score** is calculated in two steps: first, we compute the orientation and positions score, then we combine these two.

The **position error **for image **i **is the difference between the estimated and the ground truth position vectors, normalized by the ground truth distance of the satellite.

\(\text{err}_{position}^{(i)} =\frac{\mid r_{gt}^{(i)} - r_{est}^{(i)} \mid_2}{\mid r_{gt}^{(i)} \mid_2}\)

For each **2.173mm per m groundtruth** distance, we correct the position error to zero to account for our machine precision. Thus, the **position score** for image **i** is defined as

\(\text{score}_{position}^{(i)}= \begin{cases} 0,\qquad~~~~~~~~~~\text{if}~ \text{err}_{position}^{(i)} < 0.002173\\ \text{err}_{position}^{(i)},\quad\text{else}\\ \end{cases}\)

We define the** orientation error **as the angle of the rotation, that aligns the estimated and ground truth orientations:

\(\text{err}_{orientation}^{(i)} = 2 \cdot \arccos(\mid \langle q_{est}^{(i)}, q_{gt}^{(i)}\rangle \mid)\)

We correct the orientation error within **0.169°** to be zero to account for our machine precision. Thus, the **orientation score** for an image **i** is defined as:

\(\text{score}_{orientation}^{(i)}= \begin{cases} 0,\qquad~~~~~~~~~~~~~~\text{if}~ \text{err}_{orientation}^{(i)} < 0.169°\\ \text{err}_{orientation}^{(i)},\quad\text{else}\\ \end{cases}\)

The **pose score** for an image** i** is the sum of its orientation and position scores:

\(\text{score}_{pose}^{(i)} = \text{score}_{orientation}^{(i)} + \text{score}_{position}^{(i)}\)

Finally, the **total score** is the average of pose scores over all images **N** corresponding to the test set (*sunlamp *or *lightbox*):

\(\text{score} = \frac{1}{N} \sum\limits_{i=1}^{N} \text{score}_{pose}^{(i)}\)