Discussion

Hi,

In the definition of the score, the error of a sequence is normalized by dividing by the ground-truth altitude z. However, the altitudes in the dataset are negative, which would lead to a negative score. So, to confirm, the score is actually computed by dividing by the absolute value |z| of the ground-truth altitude. Is that correct?

Looking at the scoring function in more detail, I noticed some other things I am confused about:

• You are describing the scoring conceptually as a "normalized mean-squared error". However, in your definition of the error, you are taking the square root, which does not agree with the definition of mean-squared error.
• How should we interpret squaring a vector? Here: \((v_{x,est}^{(i)} - v_{x,gt}^{(i)})^2\) we are squaring the difference between the vectors \(v_{x,est}^{(i)}\) and \(v_{x,gt}^{(i)}\), which is again a vector. Should we understand this square as the dot product of the vector with itself, or as element-wise squaring?
  • If it's the dot product, then the result would be just a number. Which means we are taking the sum of three numbers, and then the square root of that, which results in just a number. But then we want to divide by \(z_{gt}^{(i)}\), which is a vector. So that doesn't make sense.
  • If it's the element-wise square, I assume the square root would also be element-wise, and the division by \(z_{gt}^{(i)}\) as well. So then the final score would be a vector, and not a number, which also doesn't make sense.

* The error used is the Root-MSE (RMSE) now clarified in the text. Thanks for pointing this out.
* The coordinate \(z\) in the dataset is NOT the altitude, but a coordinate in some reference system with z axis pointing towards the Moon center. Normalization happens indeed with respect to its asolute value (\(|z|\))
* \(v^{(i)}_{x,est}\) as well as \(z^{(i)}_{gt}\) are scalars, not vectors. Please let us know if somewhere we wrongly hint they are vectors. The final score is indeed a scalar. The math in https://kelvins.esa.int/elope/scoring/ seems coherent with this.

Thank you so much for the quick response!

• Ok, thank you for the fix!
• Thank you for confirming that the absolute value is used! On the scoring page, \(z_{gt}^{(i)}\) is referred to as "the ground-truth altitude", which is why I was calling z the "altitude".
• On the scoring page, it says "let \(v_{x,gt}^{(i)}, v_{y,gt}^{(i)}, v_{z,gt}^{(i)}\) be vectors of length \(|T(i)|\)". And it makes sense to me that they should be vectors of length |T(i)|, since to calculate the error per sequence, we want to average over all the errors for each timestamp in that sequence.

Right, I see now the confusion, thanks for pointing me to the exact text, I agree its not crystal clear, we assumed a lot of elementwise operations 

The correct interpretation we use in scoring is:

\(E(i) = \frac 1{|T(i)|}\sum_{j=1}^{|T(i)|} \frac{\sqrt{(v_{j_{x, est}}^{(i)}-v_{j_{x, gt}}^{(i)} )^2+(\cdot)^2+(\cdot)^2}}{|z_{j_{gt}}|}\)

in code:
 

    err_total = np.sum(np.sqrt(((vx_est - vx_gt)**2) + ((vy_est - vy_gt)**2) + ((vz_est - vz_gt)**2)) / z_gt)

hope this clarify. Let us know.

... in the code above the division by \(|T(i)|\) was omitted, but it is part of the scoring in kelvins.

Ok, thank you for clarifying that!