As fast as light is not fast enough

Space is not only utterly empty, but also big beyond imagination. So big that, even at the incredible speed of light, radio waves take 4 hours to reach the orbit of Neptune. When you add the time to travel back it is not uncommon that the ground stations for the New Horizons mission were transmitting one full night, spent the day in other duties and only by the next sunset they started to receive the answer from the spacecraft (past the orbit of Pluto the two-way light travel time went above 12 hours).

The long response time has two direct implications for space research: on the one hand, any emergency response has to by built into the spacecraft, because a response from Earth could easily take hours in the best case; even for nominal operations, if a response from ground is necessary, the activities are going to be very, very boring, because most of the time will be spent waiting for the radio waves to travel between the spacecraft and the Earth or back.

Photo: PxHere

Considering these limitations, we have developed over time two end-case operational paradigms. In the interactive mode, instructions are sent to the spacecraft one by one or in small batches and the ground team waits for a confirmation that everything has happened as expected before proceeding with the next batch. If something unexpected happens, the activities do not progress until the problem is assessed, just in case it might have a knock-on effect on subsequent activities. However, even if the operations go well, this is a very slow way of operating. For regular science we normally operate in the non-interactive mode, where one or more days' worth of commands are sent to the spacecraft in one single batch, with their desired execution times and the ground team just gets the recordings after the fact. Not having to wait for a confirmation before the execution of the next command saves a lot of time, but it has the disadvantage that, if you make a mistake in the sequence of commands that gets uploaded the instrument can get stuck at some point you can lose a lot of science because you do not even know that something has gone wrong.

This is what happened to us yesterday: trying to cram as many activities as possible in the time slots that we were allocated we ended up shooting ourselves on the foot and ordering a competing activity before the previous one was finished. How is it possible that we make mistakes when flying the instrument? The answer is fairly simply: we have been short of hands in producing the software and the operation tools and have been forced to cut corners at some points. Last month I mentioned one of the solutions that I have contributed is a program that checks if the timing of the commands is enough to avoid collisions, and that has saved us more than once already in one year of mission. However, this tool relies on having the information of how long each command takes, so if the information is not detailed enough, the results are not reliable: if one of the scientists insists on taking a set of 58 images instead of 51 as originally planed, the system does not break, but it takes longer to run, so if you have another command coming after it you should ensure that there is enough margin to allow for the extra duration (we didn't) so that the instrument does not crash (it did).

An alternative to the fully non-interactive mode is the execution in visibility: the planning takes place just as in the non-interactive mode but the activities are scheduled to happen while the spacecraft is communicating with ground, so that the team can instantly (i.e. with several minutes of delay) see if something has gone wrong and respond quickly (i.e. within hours instead of within days) to any anomaly that might happen. The main problem for this mode of operation is that there are many space mission and just a bunch of ground stations, so the use time of the stations is disputed fiercely and has to be justified. Besides, during the ground contacts the spacecraft is normally busy transmitting the recording of the previous days, and this playback normally "fills up" the channel, so it is hard to add the near-real-time information on top of it.

In the end, we have agreed that we need to have a more thorough testing of the sequences on ground before they get sent to the spacecraft. Luckily, we have a functional replica of the instrument, so we can measure how long each individual command takes, so that they do not end up stepping on each others' toes. It will take a couple of months to put the system in place, but we cannot keep making operations mistakes. I hope we manage. Enjoy the evening.