On August 4, during a talk session, we held titled; “An Introduction of Services That Use Tellus and Their Future”, we heard about businesses that use satellite data as a part of “Tellus SPACE xData Fes. -Online Weeks 2020-“.

In this article, we will be discussing the potential of a technology developed by Sharp for super resolution imagery, used along with a joint research and development project for automatic parking-lot space detection using satellite data. This project is a collaboration between parking space sharing service provider “akippa,” AI consultant “Ridge-i” and “Sakura Internet,” the company that runs the satellite data platform “Tellus”.

This event discusses the following.
-How people used to look at maps to figure out where to find potential parking lots locations
- Using satellite data to cut time and costs in order to find such potential locations
- One of the challenges faced is the cost of high-resolution data.
- Potentially cutting costs by using super resolution imagery to convert mid-resolution into high-resolution

The project we discussed during this talk session was announced by each of the three collaborating companies. Be sure to check out their press releases and videos archiving this project.

(1) Speaker Introduction

Let’s introduce the participants in today’s talk session.
– Tanaka Daiki
Head of Marketing Office & Active Business Operation Group, akippa Inc.,
– Issei Sugiyama
Executive Officer, Business Structuring Manager, Ridge-i Inc.,
– Ito Norio
Chief Researcher, Research and Development HQ, No.3 Communication and Video Technology Research Institute, Sharp Corporation
Yamazaki Hidehito of Sakura Internet Inc.’s cross data department in the business development division will work as the facilitator for this discussion.

Yamazaki: Hello everyone, let’s have a great talk session. We’ll start off with a quick introduction of each of our guests’ businesses that use Tellus.
In order to paint a picture of what kind of services use Tellus, let start off with a description of how each of their businesses use satellite data, and discuss what led them to start their projects, the challenges they face, and how they plan on improving their services.

So let’s start with introducing each of the speakers. Would you mind starting us off, Mr. Tanaka from akippa Inc.?

Tanaka: Hello everyone. My name is Tanaka and I am in charge of operations and business strategy at akippa, a parking space sharing service company. Thanks for having me today.
I’d like to give a quick introduction of what we do at akippa, the service at the center of this project.

akippa is a part of the sharing economy. We are like Airbnb, except we rent out parking spaces instead of empty rooms.
People who own empty plots of land that can be used for parking register on akippa for users to park their cars in, like a make-shift parking lot.

Our service is pretty simple. It’s available for anyone with open land to register their property as a parking space.
It doesn’t cost anything to sign up and there is no subscription fee. It only starts to cost money when a user rents an owner’s space.

An example of how our business model works is, if a user rents a parking space on akippa for 1000 yen, 500 yen goes to akippa, and 500 yen goes to the owner.

Tanaka: As of July 2020, we currently have 37,000 spaces registered on akippa.
Many of these are parking lots rented monthly that have available space as well as private driveways of homes that aren’t being used.

Our desire to find spots like these with more efficiency was what brought us on board with this project. That pretty much sums up what we do at akippa.

Yamazaki: Thank you very much. Next, let’s hear about Ridge-i from Mr. Sugiyama.

Sugiyama: Hello, I’m Sugiyama. I am an executive officer at Ridge-i Inc.. Thanks for having me as well.
Ridge-i is a 4-year-old AI company with 40 employees. Our main mission is to use cutting-edge technology to overcome challenges faced by businesses to create real change in society. using technology to solve issues for our clients and society allows us to re-envision society and create new businesses.

One of the fields we work on is called AI consulting/solution development, where we use both machine learning, a new technology called deep learning to create solutions for our clients’ managerial and operational challenges.
Another is called joint ventures/product development, where we work with a client to create a new service or with our own products to provide a service.

Due to the covid-19 pandemic, we have recently been working on different products such as a solution to analyze how crowded certain spots are, or ways, using video to monitor irregularity in the burns.

Sugiyama: What separates us from other AI start-ups is that in the short span of four years since our establishment—well actually five after this month—we have already passed the PoC  (proof of concept) stage and are currently implementing many different projects.

An example of this is an AI we developed to categorize different types of garbage within an incinerator at a garbage disposal plant. This AI product has been in operating without malfunction for over a year in Funabashi City.
Another example is a project we developed with NHK where we created a technology that adds color to black-and-white images. Older images tend to be black-and-white, but by using our technology NHK has been able to increase their image quality for broadcasting.

Sugiyama: We started to take an interest in satellite images about 2–3 years ago, and have a few examples of projects in which we used them.

On the left of the diagram below, you can see a satellite image we used to detect spots in the ocean where there have been oil leaks.
In the middle, there is a satellite image that shows the aftermath of a landslide, which was part of a project we worked on with JAXA to figure out quickly and accurately where and how large landslides were happening.
The right shows a project run by our own research and development unit in which we use aerial photography as satellite images to detect cars.

These are some of the other satellite projects we are currently working on, similar to the one we will be introducing today. Thanks for having me today.

Yamazaki: Thank you very much. Let’s move on to Mr. Ito from Sharp Inc.

Ito: Hello, I am Ito and I work at the research and development division of Sharp Inc. I look forward to today’s discussion. I would like to explain why we at Sharp, a manufacturer of consumer electronics and LG display, that uses super resolution, were called in to be a part of this project.

Sharp has been working hard these past 10 years to bring 8K resolution into the world. We have a department called the “8K Lab” where we develop 8K system solutions to meet our customer’s needs beyond just using 8K technology for TV.

Ito: We work with lots of different technologies at the 8K Lab, but for this project, we use one called “super resolution imagery” to turn low-resolution images into high-resolution images.
We would like to optimize super resolution imagery for processing satellite images so that it may be used on Tellus.

There are a few different methods Sharp uses for performing super resolution technologies, all of which are are needed for LG displays which display images in multiple resolutions.
An 8K TV, for example, displays television that is currently only shown in HD by expanding it to 8K. It does the same with 4K images, all in real-time using an engine built into the TV.
Video needs to process 60 images a second, and it is our job to optimize the algorithm used to make this possible.

For this project, we are working with satellites, which is a new technology that excels beyond what our current super resolution imagery is capable of handling. We would like to use Sharp’s machine learning and super resolution technology to help with this project.

Ito: The image above shows what happens when we use super resolution to process satellite images.
To the left, you can see an image that has been expanded to 6 m resolution. Compared to that, on the right, you can see how much clearer an image that has been processed using super resolution is.
By creating a pseudo-image at about 3 m from the satellite’s 6 m resolution image, we can use cheaper, low resolution data for remote sensing services that require high-quality data. We hope that by participating today we will be able to contribute to making such a task possible. I’m really looking forward to continuing this.

Yamazaki: Thank you.

(2) Introduction of the Joint Project of Sakura Internet, a kippa, and Ridge-i

Yamazaki: Alright, so I would like to give a quick introduction of a project using satellite data that akippa, Ridge-i, and Sakura Internet have collaborated on.

Let’s start this section off by talking about remote sensing. This started off as a technology used in the scientific community, which was even a technical challenge for JAXA and the World Space Agency.
On the other hand, we are on the business side of things, which means we are more focused on what kind of services we can provide, using space data and machine learning, that try and help solve challenges companies face today.

akippa was a client of ours, and we spoke to them about how we could make their sales team more efficient whilst decreasing costs by helping them find new potential clients more efficiently. We started a project to take this on last year.

Yamazaki: On Tellus, we have data from the ASNAR01 satellite available. This will get a little technical, but this satellite can gather optical data at a resolution of 50 cm, just like a digital camera.
In other words, each pixel contains about 50 cm of data, making it possible to get extremely high-resolution data.
We are using this satellite data, using Ridge-i’s machine learning technology, to look at areas in Fukuoka and Sapporo for potential parking spot locations for akippa’s sales personnel to target.

After developing the program for 3 months and before undergoing a trial, we were able to make a program in its early stages, I was able to find potential parking spots with a 75% accuracy.

Yamazaki: A picture on the slide above shows purple spaces that could potentially be used as parking spots.
Since this is machine learning, we used teaching data, so we repeated the process of adding satellite data, and then checked whether it was correct, to make the program smarter.

Mr. Tanaka from akippa will get into what we mean by a 75% accuracy later, but this is where we stand after 3 months.

We have a video that describes the service I just talked about, which I would like everyone to see.

(3) Evaluating Project Success

Yamazaki:  I would like to hear your opinions on the results so far, as well as your future expectations. Let’s start with Mr. Tanaka.

Tanaka: Before I share my opinion, I’d like to talk a little about one of the challenges we faced.

The way akippa approaches sales is, firstly, they decide on a landmark as a target.
Right now, we are seeing less of this due to covid-19, but where there is an event, during it, there is a lot more demand for parking spaces around the venue.

For example, we want spaces in the area around the stadium used for the national high-school baseball tournament, or the surrounding area of Tokyo Dome when a popular artist is performing a concert.
First, we find locations like this, and then draw circles around them in increments of 10, 20, and 30 minutes by foot journeys to the venue. We then set out, by foot, in search of available spaces in each of those circles. If we find an empty space, we approach the owner.
Like they say in Japan, we are More steps, more sales (making sales with your legs), or working all day on your feet is one of our biggest hurdles.

Tanaka: I think the results we have had so far are fairly good. We were able to get significant results looking just at Sapporo and Fukuoka in terms of leads on potential clients, so I would consider this a big first step.

One of the big factors we were talking about with Ridge-i while working on this was whether we want to focus more on accuracy or coverage.
The more coverage we have, the less accurate it becomes, and vice-versa. We had to figure out which one we wanted to prioritize.
For sales personnel at akippa, the answer was definitely coverage.

We were able to get data that has almost 100% coverage of the areas we were looking at. I feel like there is a lot of potential in this program considering the numbers, 75% accuracy on top of 100% coverage which is more than enough for us to increase efficiency.

Another way of looking at it, this data allows us to take a landmark we are targeting, look at the parking lots around it, and figure out how many of them our sales personnel can approach. This is really important data to have when it comes to sales planning, so I feel pretty good about the results so far.

Yamazaki: Thank you very much. Next, let’s hear what Mr. Sugiyama has to say.

Sugiyama: We prepared a quality set of model data and used learning data to test our machine learning method.
I oversaw picking out the parking lots and marking them with color which we used to evaluating model leaning.

Sugiyama: We use this mean IoU as a score to show the average accuracy, which for this project was 75%, but considering the difficulty of the task and how quickly we did it, I think this is of pretty good standard.

Just expanding a bit on accuracy and coverage that Mr. Tanaka from akippa spoke about earlier, there are different sizes when it comes to parking lots.
A parking space for a house is going to be smaller, and many of the lots used for monthly parking space rentals are larger. These larger spots that can hold, say, 20 or more cars, make up about 80% of the total area used as parking space, and about 50% of parking spaces.
These larger parking areas account for around 80% to 90% of both the coverage and accuracy, which is something we are really happy with.

That being said, taking into consideration akippa’s business model, it is important that sales are able to cover both large parking lots and smaller, privately owned spaces.
Right now, one of the challenges we face is how we can improve our accuracy when it comes to finding smaller parking spaces.
The representative from Sharp spoke about this earlier, but we do have one way in mind to increase the quality of our data.

Another thing we need to consider before we can make this a service is how well our technology runs when looking at other parts of Japan that we haven’t tested yet. At the moment we have only proved our concept by analyzing areas in Fukuoka and Sapporo, but since akippa’s service covers the whole of Japan, this is something we are going to have to work out before we deploy our service.

Yamazaki: Thank you very much. So far, some of the key takeaways that have popped up are “increasing accuracy” and, as Mr. Sugiyama suggested, “versatility”.
Their business covers all of Japan, so the question he raises is whether their technology would work the way it did in Sapporo and Fukuoka in the rest of the country.

Another thing to consider when akippa’s sales personnel start using this technology is the “UI/UX interface” as I imagined they would use a smartphone or a tablet, to get the information they need.
Details like this will probably be worked out in future updates, but I do feel like the UI/UX interface will be a big part of sales personnel being able to effectively use the information they have access to.

On top of this, no matter how great a service is, if it is too expensive, I don’t think businesses will want to use it. Cost is another big hurdle when it comes to using satellite data.

ASNARO1 in particular is a satellite capable of producing some of the highest resolution data in Japan, but this comes with a steep price tag.
I believe through using Sharp’s super resolution approach, we will be able to use cheaper satellite data to create higher quality data, which will allow satellite data to be accessible to more businesses.
I would love to hear from Mr. Ito of Sharp about his thoughts on this.

(4) Applications for Sharp's Super Resolution Technology

Ito: I feel like what you said just now pretty much summarizes how our super resolution techniques play a role in all of this.

Ito: If you have the money for it, you can get some really high-quality satellite data.
For example, if you are trying to count all the cars on the road, you would need a resolution of about 1 m. But this resolution would be way too expensive for a business tool for daily use.

That said, low resolution satellite data allows us to have wider coverage at a lower cost. The issue is if you try to zoom in on an image with a lower resolution of about 10 m, it will get really blurry.

Within that spectrum, there is definitely a demand for zoomable satellite imagery, like in our car-counting example, which creates this conundrum in which we want high-quality satellite imagery but cheaply. I feel like super resolution imagery may be the answer to that conundrum.
In order to increase an image’s resolution, we need to use machine learning on satellite images with a high-resolution to teach them how to make images that the human eye can comprehend.
We actually did something last year when we used super resolution imagery, which you can see here.

Ito: These are images from a satellite called IKONOS. The one farthest to the left is what happens when you zoom in on the images.
As you can see it is really blurry and the pixels really stand out.

The one in the middle is what we get after performing super resolution with what would be considered a generic AI. I think you can easily tell how it is a lot less blurry compared to the image on the left.
If you look here though, in the upper right-hand corner there is a building with a parking lot beneath it. There should be cars parked there but you can’t really tell in this picture.
This is what happens when you use an algorithm specifically made for satellites.

Which brings us to the picture that we have been working on since last year, to the right.
Please take a look at the building with the white cars parked beneath it. You can see the slanted parts much more clearly using this technique.

Here is another example of super resolution.

Ito: The left shows an image that has been enlarged, and the right shows the same image only with our machine learning conducted on it.
You can see here as well how the cross walks and parked cars have come out a little clearer, without getting all messed up.

This shows you how you can take a set of satellite data, and make a pretty good looking picture out of it, though there are still a few hurdles we need to clear before we can make super resolution available on Tellus.

Satellite images vary greatly depending on which satellite took the images, so we need to figure out a way to reflect those differences in our machine learning before we can make practical use of our algorithms.
We also need to pair low- and high-resolution images in order to increase satellite data resolution, but we don’t necessarily have access to high-resolution satellite images in the first place.
Even more so, even if we have high-resolution images available, and are able to make smaller images from them, they tend to be different from how the images should actually look. This happens because when you change satellite, the results of the learning don’t always turn out so well, which is a problem we will continue to tackle for this fiscal year.

Ito: This diagram shows our plans for this fiscal year. First of all, we are presented with a problem as we don’t have images that handle low- and high-resolution, so we had to make images by shrinking the resolution from a high-resolution video to create a pair for learning.
What we ended up with was images that were quite different from the original image.
To combat this, we took an approach in which we created a pseudo-image of a low resolution that had all the properties of the original image for our learning.
This method seems to make super resolution imagery work even with different satellites, something that we haven’t been able to do up until now.

Ito: If you take a look at this diagram, the image we used this time is very dark, almost black, and the image in the middle is what we got when we used our old method.
I mean, it looks a little better than the original image, but take a look at this forest, for example. The road around it is still very blurry.
By using AI to learn from the picture in the middle, we currently are able to bring the resolution up to the level seen in the picture on the right.

Through using learning methods that match the characteristics for each satellite, we currently hope to be able to use super resolution imagery to process all the satellites covered by Tellus, to a certain extent.

Yamazaki: Thank you very much. This one right here is JAXA’s AVNIR-2, right?
This satellite has been in operation under JAXA for about 6 years, and its data is currently available on Tellus for free.
The raw data on the left looks like it is quite hazy, but it looks like the higher resolution data on the right will be available once we can use Sharp’s super resolution imagery to process it.

Ito: Yes, that’s right.

Yamazaki: Thank you very much.

(5) Free Discussion on Creating Better Services

What needs to be done to make this a viable service

Yamazaki: So how about we finish off with a free discussion about the entire project?
I’ll start off. I personally have been involved with space projects for a long time, and I personally want space data and knowledge of it to be used more throughout society.

A lot of the challenged companies face to create worthy businesses stand to gain a lot from the cutting-edge technology of Sharp and Ridge-i combined with Tellus data provided. I think that is why we are all involved in this project and I personally enjoy doing this work and think we are going to see great results.
I would love to hear what you guys think as well. How about we start with Mr. Sugiyama on behalf of Ridge-i?

Sugiyama: In terms of providing a service, we also feel that satellites have been receiving a lot more attention these past two years, which is especially evident in the pure increase of inquiries we receive, and how much more specific our clients’ demands have become.
I have to say that the more we talk about it, just like Mr. Yamazaki said just now, the more and more we are finding that if we have access to good images that we can properly analyze, we can produce results.
The only thing holding us back is the cost, which keeps satellite data from being profitable. This is a hurdle that we hope will be lowered with a combination of the Tellus platform and Sharp’s technology in the next 2 to 3 years. That would allow us to use this technology to create our own platforms as a service.

Another point I would like to make is that different companies will each have their own UI/UX interface requirements. The kind of product we provide our clients with will depend on what they need, perhaps a report instead of just raw data, and how often they will need this.
I think once we get a couple of services up and running, people will naturally start to realize that they can fully rely on satellite data, so I think the next few years are looking good for the satellite industry.

Yamazaki: Thank you very much. I think you hit the nail on the head just now. Everyone is watching data provided by the meteorological satellite, Himawari, and is doing so because they want to know the weather, not because they want to look at satellite data. I’m hoping we can make something similar to that.

What do you think, Mr. Tanaka, as someone who wants to use satellite data?

Tanaka: We are obviously amateurs when it comes to using AI to analyze images and knowledge about the space industry, so we are really grateful that we received help in making this project, which solves a major challenge our business was facing.

That being said, there are still a few things we would like to see improvement on.
To give some examples, now that we know where the parking spots are, the next step is figuring out whether those spaces are open, or if there is a driver using them?
For example, if we have data from 4 or 5 o’clock in the morning and could look at spots that don’t have cars parked in them around those times, it is safe to assume that they aren’t being used. I think adding a time aspect to the data would really increase its usability.

It would also be amazing from a sales standpoint if we could know who the owner of the parking space is? If this kind of data were to be paired with a solid UI/UX interface, I think a lot of businesses would want to use a sales tool like this.
This is something I hope we can keep working together on.

Yamazaki: Thank you very much. What you just pointed out is definitely important as far as creating a service goes. One of the good things about Tellus is that it is an open data platform.
For example, we have correspondence with a company involved in real estate. If they were to let us use their data, we may be able to add property ownership into our service as a way to further increase productivity.
Right now, we are only distributing satellite data, but we can definitely make it a better service by adding data made available by companies and the government. I would like to receive more advice like this on how we can make our service better.

Finally, let’s hear what Mr. Ito from Sharp has to say.

Ito: I speak with a lot of different clients, and a lot of them want to be able to see different things from the sky.
At one point we considered using drones before, and we never really thought we would be using satellite data like we have this time with Tellus. Even Google uses it. Satellite data didn’t seem like something the average company had access to for their own services.
The problem with drones is that there are areas where you can’t use them, and the value of any data depends on when, where, and what it is, so there are a lot of different scenarios where satellites are much more viable.

If I were to ask for something, besides what we mentioned before about resolution, how often the satellites collect data is another important aspect for our business. I think improving on this is key to making Tellus an even better service, and we want to continue working together to test different technologies to try and make things happen.

Yamazaki: Thank you very much. You mentioned that satellite data wasn’t easily accessible up until now. Do you think that is because most people don’t even know about satellite data in the first place?

Ito: I think that’s a fair assessment. I think most people probably don’t realize that satellite data is available for them to use for their own business.
We only really discovered how easy using satellite data is, and its potential, after working together with Tellus.

Yamazaki: The original concept behind Tellus was, we wanted to bring satellite data out into the open and lower the technical bar for companies and organizations that want to use it.
We also realize that the level of computing is necessary when analyzing satellite data, which is the reason we joined in on this project. It makes me happy to hear you say that.

One thing to keep in mind is Tellus is a project run by the Ministry of Economy, Trade and Industry (METI), so our real intention is we want to have Japanese industries around Japan to make good use of our data once it becomes an more obvious benefit.
This puts us in the position where we are asked what kind of services are actually using the assets available on Tellus. Therefore, I believe, it is part of our mission to help businesses out when they want to make create new services so that they can rely on Tellus.

I’m really glad we had the opportunity to work together with companies with that start-up spirit like akippa and Ridge-i, and also a world-renowned company such as Sharp to work on satellite data development projects.

Things We Struggled With During This Project

Yamazaki: So, Mr. Sugiyama, was there anything you felt challenging about the project?

Sugiyama: There were 2 things. The first is related to what Mr. Ito said about drones earlier. It was difficult for us to figure out how much we would be able to see using satellite data.
For this project, we were looking at something as specific as to whether or not a house had a free driveway, so in the future, it may be better for us to use images from space to create a big overall picture, and then fill in the blanks using drones or other aerial photography.

The second challenge was the open street map. It was difficult to mix our satellite data with maps that are already available to us.
We know where buildings and roads are with maps currently available to us, so we figured it would be a good idea to use that data in conjunction with satellite data. But in reality, they don’t match up that well.
When you try to use satellite data together with open street maps, there are parts of the maps that match up, and parts that don’t.
Satellite data creates an image of the earth by looking at it from space, and there is no standard map that matches up perfectly with this. How advanced the satellite is and how it takes photos also plays a role in how its map will look.
I didn’t think it would be that difficult to use two sets of data that are only slightly different, but this turned out to be one of the hardest parts.

Yamazaki: I see, that sounds pretty tough. When you use Google Maps, the larger image is taken by a satellite, but when you zoom in it becomes aerial photography, and then the street view is taken by a car with a camera mounted on it. It doesn’t really make a difference though as to how they take the pictures from the users’ perspective.
We were talking about small private homes with open driveways earlier. For small, private homes, we may not be able to release private information on the owners of individual homes, but this is definitely something that would make the service better if made available.

Sugiyama: Yeah, once our map is complete, it would be really useful if we could zoom in and get more details about each property, such as the owner’s phone number, or whether or not a salesperson has already approached them and when they did?

Yamazaki: That would really make it the perfect sales tool.
There are still hurdles that need clearing, like personal information, but I feel like if we can overcome them, this will become a great new service.

Final Thoughts

Yamazaki: Alright, that brings us to the end of the discussion. I’d like to hear each of your final thoughts for the day.
Let’s start with Mr. Tanaka from akippa.

Tanaka: We are still only about halfway through creating the service we are currently working on, and I’d like to do my best to make it even better before its release. Thanks everyone for your continued support.

Yamazaki: Next, a word from Mr. Sugiyama from Ridge-i.

Sugiyama: We plan to continue working on this project for this year too.
I think we all know what still needs improving, and I am hoping Ridge-i can play a big role in improving its accuracy and versatility.
I also think it is important to make this service easy to use, which includes a solid UI/UX interface. I have enjoyed working together with everyone in that regard, and so I’m really looking forward to continuing this.

Yamazaki: And finally, a word from Mr. Ito of Sharp.

Ito: We, here at Sharp, definitely intend on improving our super resolution imagery technology.
We can do this by making it work for more satellites, beyond the optical satellite images provided on Tellus. I’d like to try a lot of different kinds of satellites to see just how far we can take super resolution imagery.

Also, right now we are talking about making images easier to look at visually through just super resolution imagery, but there may be a way to combine other image processing techniques to see how it improves the process, so this is something I would like to continue working on together with everyone.

Yamazaki: Thank you very much. That is definitely something I would like to see you try as well.

I’ll give the closing remarks for the day. I think it is important to have a positive impact on the world, and though we are working with space, I would like to create a down-to-earth service that anyone can use. I hope we can work together to make this a great service.

Thank you very much.

(6) Summary

Just like the project we looked at today, there are still a lot of challenges faced by companies in Japan. Of the many challenges, there may always be a solution in satellite data.
Even if satellite data alone isn’t enough to answer a particular challenge, through combining it with other public data, or data owned by a company, you may be able to create a new solution.

There are hurdles when it comes to cost and technical skills, but these can be overcome by teaming up with a company that specializes in the skills you need as Sharp did with this project.

One of Tellus’ jobs as a platform may be to bring companies together. Data and technical capacity vary from company to company, but by breaking down barriers and working together, we may find new possibilities.

Find out for yourself by registering for Tellus and use it for free!

Want to try using satellite data? Try out Japan’s open and free to use data platform, “Tellus”!

You can register for Tellus right here

関連記事