宙畑 Sorabatake

How to use

[How to Use Tellus from Scratch] Get original data from the satellite-data-provider

In this article, we’ll tell you about original data (Standard Data Product, High-level Product, etc.) from the satellite-data-provider and how to get it.

In this article, we’ll tell you about original data (Standard Data Product, High-level Product, etc.) from the satellite-data-provider and how to get it.

What is original data from the satellite-data-provider?

As a satellite data platform, Tellus offers satellite data in two main formats.

The first format is map tile (PNG images on Mercator’s projection).
Map tiles are suitable for map handling on the web.

The other is unprocessed data supplied from satellite data providers.
When you handle data of a wide geographic area at one moment in time or you need precise locations, this format is recommended.

The following three types of data are available as of February 2020.

  • PALSAR-2 L1.1 (the CEOS format)
  • PALSAR-2 L2.1 (GeoTIFF)
  • AVNIR-2 Ortho Rectified Image Product (GeoTIFF)

The two types of data with a processing level called LX.X are both Standard Data Products created by JAXA.

The article below explains Standard Data Product and High-level Product in detail, including what L1.1 means.

We’ll continue to add scenes and data types.

In this article, we’ll explain PALSAR-2 L1.1, a special type of satellite data in the CEOS format.

What is the CEOS format?

The CEOS format is the data format defined by the Committee on Earth Observation Satellites, an organization set up to have coordination between space agencies from different countries as well as international institutions.

What binary data is put where depends on the satellite and sensor, so if you use observation data that’s not supported by existing software or libraries, you will need to read documents offered by the data provider.

PALSAR-2 Product Format Description (JAXA)

The CEOS format is supposed to mainly be used by scientists and professionals. It might be a bit hard to use for people who are not familiar with satellite data.

However, for those who are interested in what satellite-data-provider original data looks like, we’ll tell you in plain simple words how to get PALSAR-2 L2.1 (GeoTiFF) data on the development environment.

How to get data?

You can get the data file by following these three steps.

The API is only available on the development environment.

Do a file search
Do a file search to get metadata

Get the download URLs
Get the file download URLs from the ID of the needed file.

Download them
Download and save the files on the development environment.

Get PALSAR-2 L2.1 data

Now, let’s get data!

This time, we will get PALSAR-2 L2.1 data as an example.

See Tellusの開発環境でAPI引っ張ってみた (Use Tellus API from the development environment) for how to use the development environment (JupyterLab) on Tellus.

File search (PALSAR-2 L2.1)

https://file.tellusxdp.com/api/v1/origin/search/palsar2-l21

Arguments

All the arguments are optional. If you don’t specify arguments, you get unfiltered results.

Name Query Sample Search criteria
Search start time after ISO8601
2017-10-05T15:00:00.000000+00:00
Returns results with an observation_datetime value later than the specified time.
Search end time before ISO8601
2017-10-05T15:00:00.000000+00:00
Returns results with an observation_datetime value earlier than the specified time.
Polarization polarisations Possible values: HH, VH, VV, HV, HH+HV, VV+VH and HH+HV+VV+VH Returns datasets with the polarization combination specified with a string.
Observation mode code Possible values: SBS, UBS, UBD, HBS, HBD, HBQ, FBS, FBD, FBQ, WBS, WBD, WWS, WWD, VBS and VBD Exact match
Observation direction observation_direction Possible values: R and L Exact match
Orbit direction orbit_direction Possible values: A and D Exact match
Beam number beam_number Possible values: U2-6, etc. Exact match
Path number path_number int Exact match
Frame number frame_number int Exact match
Operation mode mode Possible values: SM-1, etc. Exact match
Search area left_bottom_lon Default: −180 Returns results if any part of their bounding box is within the specified range.
right_top_lon Default: 180
left_bottom_lat Default: −90
right_top_lat Default: 90
Number of results returned page_size Default: 100

Maximum: 1000

Cursor cursor For pagination

The API returns arrays of metadata.

Metadata in a return value

Name Property name Type and sample
Positions (the four corners) coordinates array of float

[[left_bottom_lon, left_bottom_lat],[right_bottom_lon, right_bottom_lat],[right_top_lon, right_top_lat],[left_top_lon, left_top_lat]]

Observation start time (UTC) begin_datetime ISO8601
Observation end time (UTC) end_datetime ISO8601
Scene center time (UTC) center_datetime ISO8601
Orbit direction orbit_direction string

D

Beam number beam_number string

U2-6

Off nadir angle off_nadir_angle float

29.5

Observation mode code string

UBD

Observation direction observation_direction string

R

Observation path number path_number int
Scene center frame number frame_number int

890

Operation mode mode string

SM1

Polarization polarisations array of string

[HH, HV]

Dataset name dataset_id Scene ID in the case of PALSAR-2
List of files in a dataset files array of string

[file name]

The observation data (GeoTIFF), thumbnail (JPG), summary.txt, etc. in the case of PALSAR-2 L1.1

Date of addition to Tellus date_added ISO8601
Representative time observation_datetime ISO8601
Area bbox [left_lon, bottom_lat, right_lon, top_lat]
Download link publish_url

 

Sample code

import requests
from pprint import pprint

TOKEN = 'ここには自分のアカウントのトークンを貼り付ける'

def search_file(params={}, next_url=''):
    if len(next_url) > 0:
        url = next_url
    else:
        url = 'https://file.tellusxdp.com/api/v1/origin/search/palsar2-l21'
        
    headers = {
        'Authorization': 'Bearer ' + TOKEN
    }

    r = requests.get(url, params=params, headers=headers)
    if not r.status_code == requests.codes.ok:
        r.raise_for_status()
    return r.json()

ret = search_file({'after': '2018-10-05T15:00:00'})

pprint(ret['count'])
pprint(ret['next'])
pprint(ret['items'])

Get the token from APIアクセス設定 (API access setting) in My Page (login required).

When you run the code, it will show you the output below.

 

We specified the argument to get data for October 15, 2018 or later, and got the corresponding data.

We couldn’t get all the corresponding results because there were more than 100 results (the default number of results returned is 100; you can change it to a maximum of 1000).

Use the URL returned in “next” for next_url to get the next results.

Get the download URLs (PALSAR-2 L2.1)

If you find the data you want, then get the download URLs for it.

Every time you send a request, it gives you random URLs. Be aware of the expiration time of those URLs.

If you find the data you want, then get the download URLs for it.

Every time you send a request, it gives you random URLs. Be aware of the expiration time of those URLs.

https://file.tellusxdp.com/api/v1/origin/publish/palsar2-l11/{dataset_id}

Assign the name of the needed dataset (the scene ID in the case of PALSAR-2) to dataset_id.

Return value

ISO8601

The download URL becomes invalid after this.

Name Property name Sample
Project name project ‘palsar2-l21’
Dataset name dataset_id Scene ID in the case of PALSAR-2
Download expiration time expires_at
File information files [{‘file_name’: file name, ‘file_size’: int, ‘url’: a random URL issued}]

 

Sample code

def publish_files(dataset_id):
    url = 'https://file.tellusxdp.com/api/v1/origin/publish/palsar2-l21/{}'.format(dataset_id)
    
    headers = {
        'Authorization': 'Bearer ' + TOKEN
    }

    r = requests.get(url, headers=headers)
    if not r.status_code == requests.codes.ok:
        r.raise_for_status()
    return r.json()

published = publish_files('ALOS2217642720-180604')

pprint(published)

We made a request using the value ‘ALOS2217642720-180604’ for dataset_id as an example and got an output as shown above.

We got a download URL for each file that belongs to the Scene of the ID ALOS2217642720-180604.

Download them

Finally, download the files using the URLs you received.

This time, we downloaded the IMG-HH-ALOS2217642720-180604-UBSR2.1GUD.tif (the second file), which belongs to the requested scene ALOS2217642720-180604.

def download_file(url):
    headers = {
        'Authorization': 'Bearer ' + TOKEN
    }
    r = requests.get(url, headers=headers)
    if not r.status_code == requests.codes.ok:
        r.raise_for_status()
    return r.content

def save_file(data, name):
    with open(name, "wb") as f:
        f.write(data)
        
file = download_file(published['files'][1]['url'])
save_file(file, published['files'][1]['file_name'])
print('done')

If you’ve successfully downloaded the file, it displays “done” (it takes a while due to its large size).

Because PALSAR-2 L2.1 data is GeoTIFF, you can easily read and display it using Python.

Sample code

from osgeo import gdal, gdalconst, gdal_array
from skimage import io
%matplotlib inline

tif = gdal.Open('IMG-HH-ALOS2217642720-180604-UBSR2.1GUD.tif', gdalconst.GA_ReadOnly)


# ファイル情報
print(tif.GetProjection())
print(tif.GetGeoTransform())
print(tif.RasterXSize, tif.RasterYSize)
print(tif.GetMetadata())
print('バンド数: {}'.format(tif.RasterCount))

# 画像表示
# 画像のサイズが27640 * 31340ピクセルと大きいため、すべてを一度に表示することができません。
# サンプルでは部分的に表示しています。
io.imshow(tif.GetRasterBand(1).ReadAsArray()[10000:11000, 10000:11000])
Credit : Original data provided by JAXA

In this article; how to get original data from the satellite-data-provider on the development environment of Tellus.

We plan to offer more original data from the satellite-data-provider in addition to PALSAR-2’s. We also plan to publish data analysis tools.

Check it out.

Reference

ALOS-2/PALSAR-2 Level 1.1/1.5/2.1/3.1 CEOS SAR Product Format Description