Landing Rockets Using AI

It’s impossible to have a successful mission if your rocket doesn’t land correctly. When it comes to space exploration, there are going to be many times in the future that we will have to land a manned-spacecraft on planetary terrain. Since we can’t have one of those smooth landing pads in a planet we have never been to, we have to use the landscape that already exists, and it’s not an option to land on uneven terrain. Significant rocks can damage the rocket boosters, and sloped landscapes can tip over the rocket after touchdown. One possible approach is to have self-adjusting legs under the rocket. However I argue that there is a better alternative to self adjusting legs to go about this problem. Below I will discuss an AI algorithm that I developed which may eliminate this problem all together.

And this isn’t some sort of futuristic precautionary thought, something similar actually happened recently. On March 4th, 2021, SpaceX launched their 10th prototype of Starship, which ended up exploding a few minutes after landing.

What ended up happening to SN10, was when it landed, it crushed some of its engines underneath, causing the rocket to blow up 5 minutes after a seemingly “perfect” touchdown. Now, this didn’t happen because the rocket landed on a boulder, it was because the rocket came down too fast, like said, crushing the boosters underneath. Even though landing on a rock, and landing on a smooth surface is nowhere near the same thing, they lead to similar outcomes since they both severely damage the engines of the rocket.

To give you a good sense of how fragile boosters are, here is what a raptor engine looks like (the ones SpaceX is planning to use for the starship):

Don’t get deceived by the metallic look, it is nothing but a thin metal tube when being sandwiched in between a 11 million pound beast and Martian soil. These engines are so complex, that even one metal tube being ruptured or bent could lead to a potential explosion like it did on March 4th. What’s so dangerous is that the oxidizer is flowing through tubes only a few centimeters away from the fuel. And if you look back at your 9th grade chemistry class, you remember that fuel + oxidizer = boom. The whole point of the booster is to make the two react at the correct spot, the reaction chamber, or else the reaction can carry on backwards to the whole tank making all of your fuel react with all your oxidizer, which nothing manmade can contain without being destroyed.

For this reason we have to protect these engines from anything that is potentially dangerous, including rocks that can pierce right through all the complicated parts of the engine.

The solution to this is not as simple as “lets just include fins at the bottom so that the surface of mars never touches the engines in the first place”, since fins don’t eliminate the danger of tip-over.

Tip-over happens when the center of mass of an object (red circle in this case) reflected perpendicularly to the surface of landing (orange dotted line + orange point) is not directly over the base of the object (yellow line). This causes the object to tip, which we can all agree is bad news. After landing on it’s side, one of two things can happen. One, the rocket flops to its side, leaving everybody stranded on Mars until a rescue mission arrives. The second possibility is much worse, the fuel and oxidizer tanks cannot withhold the pressure of the fuel/oxidizer inside of them + the strike from the side so they both blow up, turning the rocket into dust.

Hopefully this is obvious by now, but sadly rockets + rocks aren’t the most dynamic duo…

The Current Approach — Self-Adjusting Legs

The current solution that the space industry is thinking of is to have a bunch of “legs” on the rocket that self-adjust according to the terrain underneath them, to keep the main body of the rocket upright (lets call this solution SAL, self-adjusting-legs). The thing is, there are much needed adjustments to make to SAL to be able to actually use it without risking a space mission and human lives. In what position will the rocket be in while the legs are still extending? When will the legs know to stop extending? What if there is a rock in such a spot that the legs can’t adapt to it? What if a rock is taller than the maximum expansion of the legs? What would end up happening if the rock was already slightly dislodged, slipping away from underneath the leg that’s holding on to it? What if the extreme weight that the spacecraft exerting on to that rock cracks it, giving way under one of the legs causing the spacecraft to tip over? It isn’t impossible to make SAL happen, but there are still many questions that we have to answer and adapt to accordingly. We need to get this technological artifact as prefect as possible before we put it under the very machine that will be carrying the first interplanetary explorers, the pioneers of humanity which will take us one big leap closer to populating the entire solar system as a species.

Now here’s my proposal, we completely avoid rocks all together, and land the rocket on one of those safe Martian plains that have a straight surface and no rocks at all to worry about. We can scope out perfect landing sites using AI-incorporated computer vision, and advanced rocket-steering technology which already exists.

This approach that we can take with the AI is proactive, since we avoid dangerous spots completely. However, with SAL, we don’t completely solve the problem, we just take measures to prevent a catastrophe, which is reactive. It is comparable to a vaccine vs. social distancing. Social distancing is a reaction, a secondary step, while the vaccine is a step taken even before a virus has a chance to present itself.

My Proposed Approach — Leveraging AI

Extraterrestrial Landscape Analysis & Trajectory Adjustment — ELATA for short.

When I first started this project, my main vision was to have an AI algorithm that identified dangerous landing surfaces, and recommended adjustments to the trajectory of the rocket in order to secure a safe landing. This leads to 4 main components that should appear in the final product.

• Identify & Mark Dangerous landing areas
• Identify & Mark Safe landing areas
• Mark the current trajectory of the vehicle, where it is going to land
• Mark the best landing spot in the whole map overall

After I pressed “run” for the final version of the code, it was nowhere near disappointing!

Let me explain what all of these colorful marks mean on the image, so that the output actually makes sense, and you can judge the performance for yourself!

• The red dots on the image point out landing spots that have a rough surface, including valleys bumps, rocks etc. Basically, it isn’t a great idea to land the rocket near the red dots.
• The green boxes are spots on the image that look like it’s safe to land on, and are a certain distance away from the red dots.
• The yellow plus sign, or the “cross-heir” in the middle of the image points to where the rocket is approaching currently. The cross-heir is always in the middle, since the cameras that capture this view are on the rocket, and the angle of the camera (and it’s center) is the same as where the rocket is pointing at.
• The yellow box which is near the center of the image is the most optimal landing spot, which has to qualify for 2 criteria; it has to be on a green box, and the closest green box to the middle of the image. The reason we want it to be as close as possible is to have the rocket not maneuver much when it’s readjusting its course from the yellow cross-heir to the “highlighted” or “golden square.”

Having ELATA run real-time, we can constantly narrow down on better locations for the spacecraft and eventually land at a near-perfect spot.

How Do We Incorporate This Software On A Rocket?

The first thing we want to do is get a good look of the ground. This can be done with cameras on the bottom of the rocket (blue), near the boosters. We want to communicate the information to a processing unit (red arrows). After the processing unit runs the algorithm, we relay the information over to the hydraulic pistons (orange) that change the angle of the engines, and cold gas thrusters (green) that adjust the position of the rocket. As the pistons and thrusters get to work, we can steer the rocket towards the golden square (the most desired location).

What Does The Code Look Like?

In this section I will talk about how the code works, and I will explain what each part does in easily digestible pieces. The general flow of the whole program is:

Importing → Processing Dataset → Defining Useful Functions → Creating The Training Data → Training the Neural Network → Displaying Outputs

Importing

First things first, we need to import all of the packages that we will need further down the road. Here’s a quick rundown of what each package does:

• Keras is the machine learning package that I used for this algorithm. It shortens the process of coding the neural network, so you have more time on experimenting with different architectures, which can dramatically lower the amount of time that you spend designing the neural network. It took me around 15 different neural networks to get to the final version of the AI.
• matplotlib.pyplot is a great package that helps graph numerical values. This way I could graph any sequential number series, say, the accuracy of the neural network over time during training. This can get crucial to understand whether the neural network is actually learning, memorizing or not improving at all.
• matplotlib.image is really similar to pyplot, except it is designed for turning numpy arrays into images that you can see plotted on a coordinate grid in the console. This is great for visualizing mid-steps inside of the code, for example displaying an image of where the AI put the red dots on to make sure that everything is flowing smoothly.
• PIL is a package that enables us to turn PNG’s into numpy arrays which are essential for neural networks, especially when working with Keras.
• Numpy helps us do operations with arrays. Keras uses numpy arrays as input and training, so we have to use numpy arrays when feeding into the neural network, and taking outputs from it.
• Random is pretty self-explanatory. We get to pick random numbers using this library.
• Pandas is a package that helps us read in CSV files and turn them into arrays so that they can easily be used in the code.
• Math is also pretty self explanatory. It lets us do more complex mathematical operations such as square root.

Reading the CSV

Next, we want to read in our dataset, so we can use the processed information to train the NN that will help out with ELATA. However in order to understand the code that processes the dataset, you need to have an understanding on what the dataset looks like.

Essentially, the dataset is a bunch of bounding boxes on many different images that contain rocks inside of them. To be able to define the location of a bounding box, you need 5 variables. The first variable would be which image of the dataset the bounding box is in. This is important since a rock might be in a specific spot on one image, but there would be barren landscape on the corresponding location in another image of the 9700+ images in the dataset. The second variable would represent the x-coordinate of the top left corner of the bounding box. The third variable is the y-coordinate of the top left corner of the bounding box. With this information only, we can determine where the upper left corner of the bounding box is in the dataset. Now to figure out the actual size of the bounding box, we can take a look at the fourth variable, the length of the box, a line extending from the upper left corner of the bounding box parallel to the x-axis, eastwards. To figure out the width of the box, we look at the fifth variable, the width which is a line extending parallel to the y-axis southwards from the upper left corner of the box.

So we split the table up into its rows, or in other words into 5 different arrays, each carrying all the numbers in a row. For example, BBFrame is an array of all the numbers in the first row by order. We have this for all of the rows, each array having their own row allocated to them.

Functions

Since this algorithm processes images, we have a lot of patterns in the code of things that we do, for example, taking a snippet from an image. Since it would extend the code uselessly to re-write all of those pieces of code, I just made a few functions to shorten the algorithm and make the source code more understandable.

Red Box Function: places a red box in a specified spot in an image:

This function draws a red box with a border width of 5 pixels, given the upper left corner’s x-coordinate, y-coordinate, width of the box and the length of the box, and which image to actually draw the box on.

Green Box Function: similar to the red box function, places a green box in any specified spot in any image

The green box function draws a green box with a border width of 2 pixels, given the box’s upper left corner’s x and y coordinate, along with the width and height of the box that we want to draw. Similar to the input of the red box function, we also need the image that we want the box on to be specified.

Yellow Box Function: Same thing as the last 2, except this time it’s a yellow box.

It’s really the exact same thing as the green box and the red box. We give the function the upper left corner’s x and y coordinates, the width and height of the box and the image that we want the box on. The width of the border of the yellow box is specified to be 5.

Cross-heir Function: places a cross-heir in the specified spot of given image

Given the x-coordinate, y-coordinate, width and height this function draws a cross-heir (basically a plus sign) on the image we tell it to. The width of the cross-heir’s lines is specified to be 4 pixels.

Snip Function: Snips out the specified portion of any image given to it

Here, we input the function the x-coordinate, y-coordinate, width, length and the image that we want to snip a specific piece of. It is basically the opposite of the box functions. It snips that part of the image and saves it as another image. This way we can look at specific portions of any image that we specify.

Path of Image Function: Gives the Image’s Path Name

Every file in your computer has a path name, which sort of looks like some text with a bunch of forward slashes. An example of a file path is “/users/timucin1erbas/desktop/images/2020/December/summer.jpg”. There is a specific format of how I can convert a number into a file path in the dataset, and this is what I set this function to do. For example, if I wanted the first image in the dataset, then I could call this function, give it the number “1”, and the function would return the name of the path of the file. This is crucial to do since you can’t use an image at all if you don’t know its location (or path name) in the first place.

Is Rock Function: Returns whether a given snippet of an image contains a rock in it.

It would be a completely valid question to ask: “Wouldn’t we need AI to determine if there is a rock in an image?” Yes, we definitely would, but conveniently the dataset has a “heat-map” image to every real image in the dataset.

If a pixel in the heat-map is completely black (RGB (0, 0, 0)), this would mean that the specific corresponding pixel on the real image would be nothing but just ground. If the image has a light green color to it (RGB (0, 255, 0)) this means that there is a moderate-sized rock on that corresponding pixel. If the pixel is red (RGB (255, 0, 0)), then the corresponding pixel in the real image is a pixel that represents space. A blue pixel (RGB (0, 0, 255)) would indicate a big rock in the real image.

What we do in the algorithm is given coordinates for a bounding box, we take a look at what’s inside. If the bounding box reflected to the heat-map has a lot of green/blue pixels, then that means there is a rock in the coordinates of the bounding box that we are analyzing. However, if the bounding box is mostly black/red, this would mean there isn’t a rock in the position of that bounding box.

Creating Positive Train Set

Now that we have all of the functions settled, we want to create the training data that the Neural Network will learn on. The main goal for the neural network is that when given an image, it can give an accurate output of whether the image contains a rock, or doesn’t contain one. So when you think of it, we need a bunch of negative and positive training data which in this case will be images with rocks (positive data) and images with no rocks (negative data).

In the chunk of code above, we create out positive training data. Remember all of those bounding boxes? Well, as you probably know by now all of the bounding boxes are around significant rocks. You know what that means? If we take a snippet of the images inside of the bounding boxes given in the dataset, then we will end up with pictures of rocks that we can put in an array as our positive data.

Creating Negative Train Set

This is similar to creating the positive train set, but not quite the same. We cycle through 5000 images in the dataset and pick 2 random numbers; one representing the x-coordinate, and the other representing the y-coordinate when we pick our random bounding box. Now, remember the isRock function? We put the random bounding box though that function, and if the random bounding box doesn’t contain a rock inside of it, we append it to the negative training set.

After some runtime, we end up with a bunch of images like this in our negative training set which is exactly what we want. As you can see there are no rocks, just some soil on the ground which sets us up for some great training data.

Merging Test Sets

Now that we have a bunch of “positive” images, and a bunch of “negative” images, we want to merge all of our training data into one big array so that the Neural Network can learn on the dataset automatically.

One really important thing when preparing test sets is to make sure that the shuffling of the positive vs. negative images is completely random. For example, if the first 1000 images in your dataset were positive, and the next 1000 were negative, then the neural network would suck because it would first get the impression that 1 is always the correct answer since the first 1000 images are positive. Then you suddenly start pumping in negative images, and now it thinks that 0 is always the way to go. To avoid this trap, we want the order of positive vs. negative images to be random.

To do this, we take advantage of the random package that we imported all the way at the beginning. The random number generator allows only one of the isolated sets to append to the combined set. If the random number generator outputs a 0, then the negative set gets to put one negative image into the combined set. If the number generator gives a 1, then the positive set is allowed to put one positive image into the combined set, making the final dataset perfectly shuffled.

Neural Network Training

Now that we are officially completely prepared the training data, it’s time to create the coolest part of this project, the AI itself.

Even though this step seems plain and simple, it isn’t. I tried at least 15 different neural network architectures until I found the one that worked (which was the one above, with a whopping 99.184% accuracy).

You might realize that I constantly use layers like “Conv2D” or “MaxPooling2D”. The idea behind that is with Conv2D we can detect edges in the input image. With the MaxPool we filter the very important (high activation areas of the image.). After we get a bunch of values from the edge detection and max pooling, we flatten the neural network’s layer into a dense layer, so that the computer has time to “think” about what the numbers mean from the convolutional part of the neural network, and synthesize a result of whether the image is a rock, or just the ground.

Placing Red Dots — Identifying Uneven Terrain

Now it’s time to use the AI that we just trained. When we look back at what our vision was with the final processed image, the first step to take was obviously placing red dots on surfaces that are bumpy with rocks. This is exactly what we do here.

Looks big — but it’s really the same thing over and over again.

We pass a filter of a certain size (rectangular filter) and shift it through the image that is given. In every instance of the filter, we input the snippet (derived from the inside of the filter) into the neural network. If the neural network returns a 1, that means that in that spot there is uneven terrain, unsafe for a rocket. As a result we put a red dot in the center of that filter’s rectangle, if there is a rock present.

Speaking in terms of the example above, we originally start out with the red box, and put it into the neural network. Depending on what the neural network outputs, we either place a dot on that part of the image, or we simply move on. We shift the red box into the position of the dashed orange boxes and repeat the whole process all over again. This way we eventually get red dots on the image that represent an unsafe landing site.

Now — in the sake of processing rocks with different dimensions, we pass a filter (red box) with different dimensions itself. For example, say that you have a very narrow but tall rock. The filter might never embrace the whole rock within its borders which could lead us to ignoring it… with a small chance. However, to further minimize that risk, we run another filter over the image that is tall and wide rather than a square. In the algorithm we precisely use 4 different filter dimensions: 50 pixels by 50 pixels, 100 pixels by 100 pixels, 100 pixels by 50 pixels and 50 pixels by 100 pixels. As a result of all of this searching, we arrive at an image that is quite accurate when it comes to expressing bad landing cites.

Green Box Placing:

Now that the red dots are present, it’s time to place the green boxes. At this point we are completely done with AI, and the rest of the algorithm is straightforward math.

In this part of the algorithm, we do something similar to the red-dot placement. Think of it this way: we have a point that starts on the upper left corner. For every red dot on the image, we find the distance (Euclidian distance/length of the straight line connecting the 2 points). If the distance of the point that we are “on” is far away from every red dot in the image, then we put a green box around it. If not, we don’t put a box.

The line between “far away” and “close by” is determined by the radius of the rocket. If we select a point to land on, the rocket will cover all things within the radius of the base of the rocket from that point. So if there is a rock within the radius of the rocket when compared to the landing point, then that rock will most likely cause a danger.

For example, the orange square isn’t on a rock, but it does have 3 rocks in such a distance that if the rocket was to land on exactly that point, then the rocket would also land on the rocks because of it’s radius. However the green circle has no rocks at all within the radius that we are speaking of, qualifying it as a safe landing spot. If the spot that we are analyzing at the moment is safe, then we place a square at that point in such a way that when a bunch of squares happen to be directly next to each other, they form a grid-looking structure.

To find Euclidian distance, we take advantage of the Pythagorean theorem since you can’t technically go diagonally on a pixel image. As an end result of this step, we end up with green squares on our image that represent safe landing spots.

Cross-Heir and Yellow Square

Only 2 more things to do, and the algorithm is complete. At this step of the algorithm we want to place a cross-heir in the middle of the image, and a yellow box overlapping the best green box to land on.

First things first, we place a cross-heir in the middle of the image, since that is where the rocket is heading with its current trajectory. It might be confusing at first why the cross-heir’s position is always static, but when you think of it, the camera is mounted at the rocket’s bottom. Therefore, wherever the rocket pointing at, so will the camera, and so will the cross-heir. We add the cross-heir using the “Add Cross-Heir function” that we defined earlier.

Next, we find the closest green square to the middle of the image using the same Idea of Euclidian distance, and we highlight it with yellow by drawing a yellow square over the green square.

Why? Since the square that we chose would be the closest safe landing spot to the rocket’s current trajectory. We don’t want any other square since it would be unnecessary to do all the maneuvers and waste fuel, when you can get to a place with the same qualities, just closer by. Our job is to get the crew of the spaceship safely on the planet, not to provide a fun roller-coaster to their destination.

Outputs

When looking at the inputs there is something interesting to realize… The fact that the neural network can not only classify rocks, but can also classify bumpy ridges on the floor because of its structure. Although this was an unintended consequence it does happen to improve the performance of AI a whole lot.

Closing Thoughts & Goodbyes

Rocket boosters are fragile compared to their modern expectations for landing a rocket upright. To protect them from any damage, we can take proactive action and avoid uneven terrain altogether so that this essential component of a rocket doesn’t fail its crew.

Thanks so much for checking out my article, this thing really took so much time and effort to make (however it was a whole lot of fun!), so it would really mean a whole lot to me if you could drop some claps 👏 !

I have all of the code posted and ready to use in my GitHub.

If this got recommend to you in any way, it is a great idea to subscribe to my monthly newsletter where I send regular monthly updates, you will definitely enjoy them :)

Twitter | LinkedIn | YouTube