Scoring

Submission format

You need to provide a UTF-8 encoded textfile for the submission which encodes the same parameters to be found in parameters.csv for the remaining 100 lightcurves. That is, the first row in your submission file should correspond to lcvold201.dat and lcvnew201.dat and continue incrementally.

Only \(\beta\) and \(a/c\) are used to evaluate your score (the numerical values provided for \(J_2\) in the second column are ignored). Values for \(\beta\) have to be larger than \(1.0\) and values for \(a/c\) between \(1.3\) and \(2.5\).

Evaluation metric

Your score is computed using a variant on the Mean Columnwise Root Mean Squared Error (MCRMSE). Let \(\beta_i, r_i\) denote the submitted values for the \(\beta\)-factor and the axial ratio \(a/c\) for lightcurve \(i\) and \(\hat{\beta_i}, \hat{r_i}\) the true values. Then

$$ MCRMSE = \frac{1}{2} \cdot \left(\sqrt{\frac{1}{N} \sum\limits_{i=1}^{N} (\beta_i - \hat{\beta_i})^2} + \sqrt{\frac{1}{N} \sum\limits_{i=1}^{N} (r_i - \hat{r_i})^2}\right) $$

with \(N\) being the number of lightcurves under evaluation.

A perfect submission will have a value of \(0\), true to the inofficial motto of Kelvins to be absolute cool.

Intracompetition Score vs Final Score

To avoid leaking information to participants who might try to probe our evaluation data by multiple submissions, the scoring during the competition will only be computed on a subset of 50 out of the 100 submitted lightcurves. After the competition has concluded, the submission file that achieved the highest score on those 50 lightcurves from each team is reevaluated on the full set of 100 lightcurves to determine the final score and overall ranking.