In February 2023, a Chinese reconnaissance balloon was discovered and shot down over the United States.
Did you know that satellite data and cutting-edge AI technology were used behind the scenes?
Planet Labs PBC (hereafter referred to as Planet) and Synthetaic, which have been tracking the reconnaissance balloon, and were able to hear more about them.

Planet currently operates hundreds of satellites which observe the Earth and provide satellite data services in a wide variety of fields, including agriculture, infrastructure management, and climate and environmental monitoring. Synthetaic was founded in 2019 as a company that provides AI solutions as a result of current CEO Corey Jaskolski’s aid activities for the endangered Sumatran rhinoceros.

Planet, which continues to be at the forefront of Earth observation satellite data utilization, and Synthetaic, which developed RAIC (Rapid Automatic Image Categorization), an AI platform, are both key to this balloon tracking. I will introduce it based on a case study.
[Persons interviewed]

▼Jim Thomasonさん／VP Product, Imagery and New Missions, at Planet
Jim is the VP of Product, Imagery and New Missions, at Planet, where he oversees a world-class team building innovative imagery and data solutions that make global change visible, accessible, and actionable. He joined Planet in 2017 and has worked closely with hundreds of customers in agriculture, government, finance, insurance, and beyond – bringing their needs to the center of Planet’s mission. Prior to Planet, Jim has built his career managing high functioning technical teams to deliver market-making data products with companies like Cisco and Schneider.

▼Corey Jaskolsk i/ Founder & CEO Synthetaic

Corey Jaskolski, founder and CEO of Synthetaic, is a National Geographic Fellow and MIT graduate. He’s dedicated his career to developing and deploying technologies for global exploration, conservation, and security. His work spans all seven continents, from diving among Antarctic icebergs to exploring the Titanic’s depths 12,500 feet below. He’s led projects like high-res Everest LIDAR mapping and digitizing Maya sacrifice victims and Ice Age bear skeletons in flooded caves. In 2020, he received the Rolex National Geographic Explorer of the Year Award for his pioneering contributions to conservation, science, and technology.

(1) What is China's reconnaissance balloon tracking project?

In February 2023, a Chinese reconnaissance balloon was spotted over the United States and was featured in the news in Japan. After being shot down by a fighter jet off the coast of South Carolina, amidst various speculations as to its origin and trajectory, Planet provided more than 2,400 satellite images from all locations to Synethiac which enabled them to find the balloon. It is said that they thought that they could not do it as it would be difficult for people to find balloons by looking at about 400 satellite images one by one every day.
So an AI-based tracking project with Synthetaic,, began. It is said that Planet and Synthetaic have worked together before this project, such as supporting data research for tracking greenhouse gas emissions.

So how does RAIC differ from traditional AI techniques? Conventional AI algorithms require a lot of training data in the machine learning process, so some companies use billions of labeled data to train AI to detect specific objects.

However, in this case, it was necessary to find balloons that no one could confirm on satellite images, so it was not possible to prepare training data. This is where RAIC (Rapid Automated Image Categorization) comes into play. A feature of RAIC is that it does not require labeled data or pre-training for data detection, and instantly detects and classifies arbitrary objects contained in image data with AI and human support. In fact, from a number of images taken over South Carolina before the balloon was shot down, the RAIC reportedly captured the balloon within two minutes. How RAIC detected balloons with human support will be explained in detail in the next chapter.

After the balloon was detected by RAIC, the balloon was tracked by calculating backwards over the Americas using balloon sighting information and wind forecasts. After being unable to track the balloon over Alaska, RAIC read about 60 terabytes of Earth observation data for seven days, and was able to track the balloon and locate it to its launch point on an island in China’s South China Sea.In total, RAIC detected the balloon’s two-week journey from China to South Carolina 12 times.

Why was RAIC able to find the balloon in the absence of training data?
From chapter 2 onwards, we will introduce the contents of the interview about AI and human support, which is the true value of RAIC.

(2) How did RAIC track the balloon from satellite images?

RAIC, an AI pipeline developed by Synthetaic, takes in unlabeled image data, presents objects that are similar to the object the user wants to find, and improves its accuracy interactively to find the object that the user really wants. You can easily generate an AI model to find it.

In this project, they used RAIC to find a single balloon from a wide range of satellite data, using hand-drawn illustrations as input.

Sorabatake:Please tell us in detail how RAIC tracked the balloon. I read in an article that the first clue to finding a Chinese reconnaissance balloon was an illustration of red, blue, and green circles. Was this idea from Planet or Synthetaic?

Synthetaic: It was our idea. From working with Planet in the past, I knew how the sensors on Planet’s satellites would capture moving objects. Our project begins by loading RAIC with satellite data of South Carolina, where a reconnaissance balloon was shot down, and a hand-drawn picture of the balloon.

Sorabatake memo: Moving objects appear in three colors on satellite data! ?
(add explanation)

First, I started drawing, predicting what the balloon would look like when Planet satellites shot it from above at different angles, with red, blue, and green bands. The result is a red, blue, and green snowman shape as shown below.

(If you have an image of a circle that is actually separated, I would like to use it.)

In the satellite images of the balloons actually detected by RAIC, the red, blue, and green circles were not connected, but separated from each other. But RAIC was able to detect the red, blue and green circles in my drawing and find the balloon. Once the balloon was found on the satellite data, RAIC was able to find a series of images of the actual balloon in Planet’s data and trace it back to the launch site of the balloon in China.

Sorabatake:For RAIC’s AI engine, it’s important for detection to be able to read clue information at the beginning.

Synthetaic: That’s right. Anything can be a clue. For example, if you are looking for a military facility, if you know what kind of place it is and what it looks like, you can prepare pictures, 3D models, and generated AI data, and use these as clues to explore the whole world with RAIC. You can search.

Sorabatake:In other words, could RAIC have used other clues besides the hand-drawn picture of the balloon to find the balloon?

Planet: We’ve seen a lot of satellite images of airplanes in the past, and the planes appear to shift red, blue, and green in the same way that a balloon draws red, blue, and green circles.So I could have loaded RAIC with planes, found the balloons in the data, and adjusted it to identify only the balloons from then on.

Sorabatake:Did you add any other clues when tracking the balloon?

Synthetaic: We found that satellite data can change the appearance of circles. I noticed that sometimes the white balloon can actually be seen as two circles, one yellow and one green. We were looking for red, blue and green balloons, but would have missed the white balloon if we were looking for another balloon.

That’s why RAIC calls it AI-human collaboration. From the big data, select the one that most closely resembles the clue and repeat until you find exactly the object you are looking for.

Even if you don’t have the perfect clue from the start, you can manipulate and tune the RAIC until you find it. Even in this case, the balloon was not necessarily a beautiful round balloon, and depending on how the sun hit it, only one side was bright, and sometimes it wasn’t. If I was just thinking about exactly how the AI works and how RAIC works, I would have missed the balloons in various ways many times. RAIC allows humans to interact with AI, so it can guide what it is learning as it is.

Traditional AI technology labels the data and trains the AI to be ready for use, but RAIC does not require months of analysis.

Sorabatake:How many people in the company can operate RAIC?

Synthetaic: We have 47 employees and almost everyone can use RAIC. It takes 15 minutes to teach you how to use it and you can even find a Chinese balloon. It’s that easy to use.

Sorabatake:What kind of spatial resolution and temporal resolution (observation frequency) of satellite images will be required to track a reconnaissance balloon using RAIC’s capabilities? Are there other factors as well?

Synthetaic: Temporal resolution was the most important part of balloon tracking. Because we needed satellite data for every day of the whole planet. If we had used high-resolution data, the imaged range would have been smaller. We were able to track the balloon in this project thanks to the time resolution of Planet’s satellite data .

Planet: From the time we designed the first satellite, we have been particular about the balance between spatial resolution and temporal resolution, and have aimed to secure an imaging range and acquire satellite images of the entire Earth every day. While it is common to image only a specific region, our medium-resolution Earth observation Dove satellites image all of Earth’s landmass every day.. We believe that this is a temporal resolution that enhances the possibilities of satellite data utilization.

By balancing spatial resolution, imaging range, and temporal resolution, we have built a system that can capture images of any place on Earth every day. Especially for temporal resolution, we aim to image the whole world at least once a day.

Now, by using high resolution SkySat, it is possible to switch between medium resolution for daily data and high resolution for detailed data. Satellite imagery with higher spatial resolution can be used to complement the data. Specifically, while monitoring a wide area every day with satellites, if you want to know more information, you can use a high-resolution SkySat to zoom in on a specific location. We believe that the proper use of medium- and high-resolution satellites is necessary to obtain more detailed information and make decisions.

SkySat allows you to photograph a specific location up to 12 times a day, but with our next-generation high-definition satellite Pelican, we believe it is important to improve that observation frequency, allowing us to capture a specific location up to 30 times, and be imaged multiple times.

Synthetaic: Many countries have their own satellites, but as far as we know, no one collects all of the Earth’s data every day like Planet does.

I think another important thing is to make all the archived images public. Because we publish all images, we can make the most of the performance of RAIC.

Some companies with high-resolution satellites do not release satellite images to the public. Such companies have contracts with the Ministry of Defense and others, and satellite images will not be disclosed. We intentionally publish a wealth of data sets so that satellite data can be widely used in society.

(3) Value of AI technology in satellite data utilization

Sorabatake:For Planet, which has a huge amount of satellite data, it must have been epoch-making to be able to search for the required object using AI. Please tell us about the value of AI technology in utilizing satellite data.

Planet: The development of AI technology like RAIC has been needed by us for at least five years. When we started using satellites to collect the entire planet, we felt that satellite data was too much for humans to analyze. It would take hundreds of people for humans to examine satellite imagery. We thought that it was necessary to build a system that can quickly process data from a huge amount of satellite data and make decisions.

We are looking forward to partnering with Synthetaic because our customers have been looking for tools that allow them to visualize changes in satellite imagery and understand what is happening on Earth. In addition, by using past satellite images, we can go back in time and investigate what we need, so I think RAIC is a very effective tool.

Synthetaic: We also want to use AI to make quick decisions. In order for AI to make decisions, it needs the algorithms we have and data. Our algorithm alone cannot do the job.

Sorabatake:When applying satellite data to machine learning, the more data the better, of course, but it also requires a large amount of computers. What do you think about this point?

Synthetaic: I think the important thing is the amount of data. Once you’ve acquired a day’s worth of satellite data for the entire globe and successfully searched it with RAIC, you’ll be able to search anything after that. RAIC is not just an AI model that finds balloons. Once you create a dataset to search with RAIC, you can use RAIC to explore it when you want to find something else.

Conventional AI and machine learning required a large amount of training data to find something, and re-learning was necessary each time the object to be examined changed. RAIC has a mechanism that allows the created dataset to be applied when detecting other objects. Therefore, even if the amount of data to be read increases, you can use the created dataset to find another object without new learning, using the principle of chatGPT, which is currently a hot topic.

Currently, satellite data is loaded and detected when it is necessary for each project like this time, but in the future, if all the captured satellite data is loaded from the end every day, the search accuracy and speed will improve. I think it will go up further.

Sorabatake:It is true that the larger the amount of data, the more accurate information can be obtained. As the amount of data increases, the storage capacity required to store it also increases. Are there any storage cost challenges?

Planet: As you say, the amount of satellite image data is increasing day by day, but I feel that cloud costs are becoming more efficient. Thanks to the development of cloud services, we are able to appropriately use everything from data processing to storage.

Synthetaic: We believe that using RAIC to make more effective use of satellite data will create new value. In other words, we believe that as the amount of data increases, more customers will use the data, and we do not see storage costs as an issue.

Sorabatake:What aspects of satellite images, such as image quality, are important for using AI to examine satellite images?

Planet: We do a lot of processing in-house to prepare data that is easy for AI to analyze. The colors of all images are adjusted so that AI can easily compare yesterday’s satellite image, today’s image, and tomorrow’s image.

We do a lot of processing to ensure that the building is consistently in the same location throughout the archive. This is standard processing for making data usable by both humans and AI. This question is also interesting to us. In particular, we wanted to confirm how important band registration* is for AI.

Synthetaic: From an AI perspective, RAIC doesn’t really care what an image looks like. For example, if the satellite data were black and white instead of red, blue, green, the blue RAIC would still work. If there is a slight difference in the image, it can be searched with RAIC, and there is no need for humans to manipulate the image.

Sorabatake:Does that mean that RAIC can import multiple data without data fusion?

Sorabatake Memo: Data Fusion
Correction of missing and uneven data so that multiple satellite images can be handled in one analysis.

Synthetaic: That’s right. Although it is not satellite data, as a related experiment, we are currently conducting an experiment to search for one object from three different types of images obtained by a camera installed on the ground, a drone, and a camera mounted on a car. No image adjustment is required, RAIC adjusts for your specific data set.

The Chinese reconnaissance balloon tracking used optical imagery, but just as SAR images are difficult for humans to read, RAIC will need retraining to find detailed information. But other than that, it rarely needs retraining.

The same RAIC can be used for other satellite imagery such as MAXAR as well as SkySat using the same algorithm.

Sorabatake:There are companies in Japan that operate satellites and sell data, such as Axelspace Corporation and QPS Research Institute. Is there a possibility of collaborating with a Japanese satellite company that has fewer satellites than Planet in the future?

Planet: I think we can. Although not RAIC, we are already cooperating with a Japanese SAR data company to provide judgment and analysis services from new perspectives that combine satellite data.

Sorabatake:What about hyperspectral images? I believe that Planet has a development plan for a satellite equipped with a hyperspectral sensor, “Tanager.” Are you also considering using RAIC?

Synthetaic: We are very excited about Planet’s hyperspectral satellites. RAIC does not only operate at three wavelengths: red, blue and green. Theoretically 200 wavelengths can be handled by RAIC and I am very excited to discover something new in satellite data that was not possible before.

Sorabatake:Did China’s balloon tracking use only three wavelengths, red, blue, and green?

Synthetaic: 3 wavelengths are used to detect balloons. Since Planet’s satellite constellation Dove, which captures the whole world, observes at 8 wavelengths, we are investigating whether it is possible to acquire better images using 8 wavelengths.

(4) Utilization of satellite data and AI technology in Japan

Sorabatake:What kind of possibilities do you see in the deployment of satellite data and AI technology in Japan?

Planet: I think it is important to expand into the security field. Since the security sector has a large budget, it is not possible to solve the problem with satellite data alone, and as a service that should be provided jointly with AI technology, there is a need in Japan for a method of searching for necessary information from a huge satellite data archive.

We believe that by combining the functions that enable users to search for necessary information and anything that may be of interest to them with the vast amount of satellite data, we believe that it can be applied to fields other than security.

For example, there are many trading companies in Japan, and if AI technology can be used to estimate the amount of wood stockpiled by finding mountains of timber around the world from satellite data, it will be extremely powerful information for trading companies.A wood traceability service that combines satellite data and AI is also very easy to implement with RAIC.

Sorabatake:What are the challenges in Japan in introducing satellite data and AI technology? Please give us your opinion on the future development of Planet and Synthetaic in Japan.

Planet: At this stage, we do not feel there are any issues. In the Japanese market, I think that there is a positive attitude toward new technologies, such as processing satellite data with AI. If the government-promoted DX penetrates all levels, including local governments and companies, the movement will further accelerate.

Synthetaic: In many countries, it can take years to introduce new technology, but compared to those countries, Japan seems to have a better environment for new technology.

Planet: Certainly, it takes time to make a decision in Japan, but once a decision is made to introduce it, the movement after that is quick, isn’t it? The COVID-19 pandemic has brought about major changes in society and our lives, but I also believe that it has also served as an opportunity to introduce new technologies. I look forward to future developments in Japan.

(5) Summary

Behind the shooting down of a reconnaissance balloon in China that I saw in the news, the technological strengths of Planet and Synthetaic enabled balloon tracking on a global scale. I was able to know.

When I heard about AI and machine learning, I had the impression that it takes time to learn data before it can actually be used, but RAIC is an AI service that overturns such conventional wisdom. In the utilization of satellite data, it takes a lot of manpower to interpret satellite data, and especially in the case of SAR satellites, it is difficult to train people who can decipher it. By using RAIC to quickly identify global changes that have been difficult to track with the human eye, I strongly felt the potential for efficient analysis of vast amounts of satellite data.

As mentioned in the interview, we believe that these technologies have the potential to create value in various fields such as disaster response and environmental conservation. In addition, three wavelengths of red, blue, and green were used for the balloon detection this time.
The success of China’s reconnaissance balloon tracking project opens up new possibilities for the future, and I was excited to see the value of satellite data increase as AI technology evolves.

I would like to pay attention to the future activities of Planet and Synthetaic.