Scoring

The score of your submission is conceptually a normalized mean-square error of the ground-truth velocity and your estimate of it.

Formally, for a sequence \(i\) over a set of \(T(i)\) timestamps, let  \(v_{x,gt}^{(i)}, v_{y,gt}^{(i)}, v_{z,gt}^{(i)}\) be vectors of length \(|T(i)|\) containing the x,y,z component of the lander's velocity for all timestamps. Analogously, let \(v_{x,est}^{(i)}, v_{y,est}^{(i)}, v_{z,est}^{(i)}\) be the estimate of those velocity components as given by your submission. The error \(E(i)\) of sequence \(i\) is defined as

\(E(i) = \frac{1}{|T(i)|} \cdot \frac{ \sqrt{(v_{x,est}^{(i)} - v_{x,gt}^{(i)})^2 + (v_{y,est}^{(i)} - v_{y,gt}^{(i)})^2 + (v_{z,est}^{(i)} - v_{z,gt}^{(i)})^2 }} {z_{gt}^{(i)}}\)

where \(z_{gt}^{(i)}\) is the ground-truth altitude in sequence \(i\) at the corresponding timestamps.

Let \(I\) be the set of all test sequences and \(P \subset I\) the subset of test sequence for the public leaderboard (about 50% of \(I\)). Your leaderboard score is thus

\(\text{public score} = \frac{1}{|P|} \sum\limits_{i \in P} E(i)\)

and your final score (revealed at the end of the competition) is

\(\text{final score} = \frac{1}{|I|} \sum\limits_{i \in I} E(i)\).

As the score is based on the error, a low score is better than a high score. If you find that confusing, remember that Kelvins is about reaching absolute zero!