Can you predict the position and orientation of our spacecraft in realistic images while only being provided with labels from computer generated examples?
The autonomous estimation of the pose (i.e., position and orientation) of a noncooperative spacecraft given camera images is a vital tool for various satellite servicing missions of scientific, economic, and societal benefits, such as RemoveDEBRIS by Surrey Space Centre, Restore-L by NASA, Phoenix program by DARPA, and several missions planned by new space start-ups such as Astroscale and Infinite Orbits. Pose estimation also enables debris removal technologies required for ensuring mankind’s continued access to space, refurbishment of expensive space assets, and the development of space depots to facilitate travel towards distant destinations.
The Next challenge in pose estimation
In the previous Satellite Pose Estimation Competition (SPEC2019), the participants were tasked to estimate the pose of the Tango spacecraft based on the first Spacecraft PosE Estimation Dataset (SPEED). This dataset provided 15.000 images generated by computer graphics and 300 real images taken from a robotic testbed. Both types of imagery were split into a training and a test set, with the task to predict the hidden labels of the synthetic testset as the objective for ranking. Already at this time we were interested in the question if techniques developed for synthetic images would carry over to more realistic images? Although some images from the robotic testbed were already included and evaluated out of interest, their score was not part of the leaderboard ranking.
The new challenge SPEC2021 is explicitly designed to investigate the domain gap between those two types of images. Conducting this challenge will shed more light on the next big question for spaceborne computer vision algorithms: how can one validate on-ground the pose estimation algorithm on spaceborne image targets that are simply unavailable prior to the mission? After all, unlike on Earth, autonomous driving in space prohibits habitual road tests and on-site debugging.
In this competition, you will have 4 times the amount of synthetic images to work with, but they are no longer part of the test-set. Instead, your rank is solely based on your performance on 9,531 Hardware-In-the-Loop (HIL) test images of the half-scale mockup model of the Tango spacecraft captured from the Testbed for Rendezvous and Optical Navigation (TRON) facility at Stanford’s Space Rendezvous Laboratory (SLAB). The HIL images can be from either lightbox or sunlamp category created with different sources of illumination. The number of poses, the fidelity of the pose labels and the variance in lightning conditions have been considerably improved in comparison to the real images from the old SPEED dataset. Having a larger variation in environmental factors will put your pose estimation algorithms to the test: will you be able to deal with extreme reflections, under/overexposure, optical artifacts and noise levels that are similar to the conditions in space?
The Generation of SPEED+
At the center of SPEED+ are the hardware-in-the-loop images captured by the TRON facility at the Space Rendezvous Laboratory (SLAB). Specifically, these two domains are constructed to mimic realistic illumination conditions by use of lightboxes with diffuser plates for albedo simulation and a sunlamp to simulate direct high-intensity homogeneous illumination from the Sun. This setup allows for interesting visual features like stray lights, shadowing, reflections and extreme exposures on finest level of detail which would be in general difficult to obtain by conventional computer graphics.
The TRON facility features two 6 Degrees-of-freedom KUKA robot arms. One of them is fixed to the ground and holds a lightweight reduced-scale model of the Tango satellite as target object. The second arm is mountain on ceiling-rail which allows it to move freely accross the lab. In order to obtain precise pose labels, 12 Vicon Vero cameras are used to track infrared markers attached to the target and the camera in combination with the internal telemetry of the robotic arms. A total of 10 Earth albedo lightboxes are deployable to simulate diffuse light conditions (lightbox images) and a metal halide arc lamp was used to simulate direct sunlight (sunlamp). Further details on the collection of the images are documented in reference .
All images from the robotic testbed are provided without labels. An accompanying set of synthetic images created using high fidelty texture models of the Tango spacecraft is provided fully-labeled, covering the range of possible poses and orientations that can be found in the real images.
 Park, T. H., Märtens, M., Lecuyer, G., Izzo, D., D’Amico, S. SPEED+: Next Generation Dataset for Spacecraft Pose Estimation across Domain Gap. arXiv preprint arXiv:2110.03101 (2021). [pdf]
 Park, T. H., Bosse, J., D’Amico, S. Robotic Testbed for Rendezvous and Optical Navigation: Multi-Source Calibration and Machine Learning Use Cases, 2021 AAS/AIAA Astrodynamics Specialist Conference, Big Sky, Virtual, August 9-11 (2021). [pdf]