Scoring

Formally, for a panorama \(i\) let \((x_i, y_i)\) be the groundtruth coordinates (viewpoint) with respect to the topdown map and \((\hat{x_i}, \hat{y_i})\) the predicted position. The localization error of the panorama is the Euclidean distance between those coordinates:

\(d_i = \sqrt{(\hat{x_i} - x_i)^2 + (\hat{y_i} - y_i)^2}\)

Depending on the localization error, a punishment term is calculated as a gated exponential, which is more sensitive to small deviations than a standard linear error. We tuned this exponential such that there is a punishment value of roughly 0.1 for an error of about 1m and about 0.68 for an error of about 10m. Any localization that is off by more than 80m is capped to a flat penalty of 1.0.

\(p_i = \begin{cases} 1.0 - \exp\left(-\log(100)\,\dfrac{d_i}{40}\right), & d_i < 80\,\mathrm{m}, \\[6pt] 1.0, & d_i \geq 80\,\mathrm{m}. \end{cases} \)

Let \(I\) be the set of all test panoramas and \(P \subset I\) the subset of test panoramas for the public leaderboard (about 50% of \(I\)). Your leaderboard score will be

\(\text{public score} = \frac{1}{\mid P \mid} \sum\limits_{i \in P} p_i\)

and your final score (revealed at the end of the competition) is

\(\text{final score} = \frac{1}{\mid I \mid} \sum\limits_{i \in I}p_i\)

As you would expect from a competition portal named Kelvins: nothing else but reaching the absolute zero can be the goal! Thus, the lower your score, the cooler you are (i.e.: the lowest score wins).