宙畑 Sorabatake

How to use

[How to Use Tellus from Scratch] Display AVNIR-2’s ortho rectified image product

The API for the original data from the AVNIR-2 sensor carried on JAXA's ALOS satellite has become available on Tellus! We'll tell you how to use it.

AVNIR-2’s original data (GeoTiff) provided by JAXA is now available on Tellus. You can deal with a wider area at a time by using the original data.

In this article, we’ll tell you how to get AVNIR-2’s original data (ortho rectified image product) on the development environment.

AVNIR-2’s data is satellite data provided by JAXA. You can use it on Tellus. When you create derivative works, you have to put “(c)JAXA” on them. See Tellus’ data catalog for more information.

Get AVNIR-2's original data (ortho rectified image product)

You can get the original data with the following three steps.

1. Do a file search
Perform a search for files that match your criteria using metadata.

2. Get the download URLs
Get the file download URLs from the files’ metadata.

3. Download the files
Download the files using the file download URLs and save them.

See Use Tellus API from the development environment for how to use the development environment (JupyterLab) on Tellus.

 

File search API

Use the API below for file search.

https://file.tellusxdp.com/api/v1/origin/search/avnir2-ori

Arguments

Name Query Type Search criteria
Search start time after str(ISO8601) Returns results with an observation_datetime value later than the specified time.
Search end time before str(ISO8601) Returns results with an observation_datetime value earlier than the specified time.
Cloud cover cloud_cover float Returns results with a cloud_cover value less than the specified value.
Doesn’t return the error value (99) if the argument is valid.
RSP path number rsp_path_number int Exact match
RSP frame number rsp_frame_number int Exact match
Minimum pointing angle min_pointing_angle float Returns results with a pointing_angle value greater than the specified value.
Maximum pointing angle max_pointing_angle float Returns results with a pointing_angle value less than the specified value.
Scene ID scene_id str Exact match
Search area left_bottom_lon
right_top_lon
left_bottom_lat
right_top_lat
Return results that have any part of their rectangular area outlined by min_lat, min_lon, max_lat and max_lon within the specified area.
https://tools.ietf.org/html/rfc7946#section-5
Number of results returned page_size int Specify the number of results returned in a request.
The default value is 100.
The maximum 1,000.

Return value

Data name Data name Type and sample
Positions
(the four corners)
coordinates array of float
[
[left_bottom_lon, left_bottom_lat],
[right_bottom_lon, right_bottom_lat],
[right_top_lon, right_top_lat],
[left_top_lon, left_top_lat]
]
Example)
[
[138.9756564, 36.3759473],
[139.8980412, 36.174298],
[140.1360529, 36.8693836],
[139.2058268, 37.0726605]
]
Scene center time (UTC) center_datetime str(ISO8601)

Example)2006-11-10T01:48:05.325279+00:00

Orbit direction orbit_direction str
D stands for descending orbit, A ascending orbit.
RSP path number rsp_path_number int
RSP frame number rsp_frame_number int
Total orbit number of scene center total_orbit_number int
Frame number of scene center frame_number int
Amount of scene shift scene_shift int
Cloud cover for reference cloud_cover int
Sun elevation angle sun_elevation_angle float
Sun elevation angle sun_azimuth_angle float
Pointing angle pointing_angle float
Gain of each band gain array of float

[band1, band2, band3, band4]

Example) [0.588, 0.573, 0.502, 0.557]

Offset of each band offset array of float

[band1, band2, band3, band4]

Example)[0.0, 0.0, 0.0, 0.0]

Scene ID scene_id str

Example) ALAV2A042302850

Dataset name dataset_id str

Example) ALAV2A042302850-OORIRFU-D075P0-20061110-001

List of files in a dataset files array of string
[
filename1, filename2, …
]Example)
[
‘IMG-02-ALAV2A042302850-OORIRFU-D075P0-20061110-001.tif’,
… ,
‘HDR-ALAV2A042302850-OORIRFU-D075P0-20061110-001.txt’
]
Date of addition to Tellus date_added str(ISO8601)
Representative time observation_datetime str(ISO8601)

Same as center_datetime.

Area bbox array of float

[left_lon, bottom_lat, right_lon, top_lat]

Example)

[138.9756564, 36.174298, 140.1360529, 37.0726605]

Download link publish_url Example)

https://file.tellusxdp.com/api/v1/origin/publish/avnir2-ori/ALAV2A042302850-OORIRFU-D075P0-20061110-001

1. Do a file search

First, write a function to call the file search API.

Get the token from APIアクセス設定 (API access setting) in My Page (login required).

import os, requests
import pprint

TOKEN = ‘ここには自分のアカウントのトークンを貼り付ける'
DOMAIN = 'tellusxdp'

# ファイル検索メソッド
def search_file(params={}, next_url=''):
    if len(next_url) > 0:
        url = next_url
    else:
        url = 'https://file.{}.com/api/v1/origin/search/avnir2-ori'.format(DOMAIN)
    
    headers = {
        'Authorization': 'Bearer ' + TOKEN
    }
    r = requests.get(url, params=params, headers=headers)
    if not r.status_code == requests.codes.ok:
        r.raise_for_status()
    return r.json()

This function receives a dictionary of parameters and then returns the content of header text of corresponding files in JSON format.

Let’s do a file search.
We use the following search criteria this time.
– Cloud cover of 10 percent or less
– Pointing angle of 20 degrees or less
– Three results in a request

# サンプル1
# 雲量10%以下 かつ ポインティング角が20deg以下、一度に取得する件数は3件
metas = search_file({'cloud_cover': 10, 'min_pointing_angle': 20, 'page_size': 3})
# 検索結果を表示
pprint.pprint(metas)

When you run the code, it will show you the output below.

{'count': 3,
 'items': [{'bbox': [138.9756564, 36.174298, 140.1360529, 37.0726605],
            'center_datetime': '2006-11-10T01:48:05.325279+00:00',
            'cloud_cover': 0,
            'coordinates': [[138.9756564, 36.3759473],
                            [139.8980412, 36.174298],
                            [140.1360529, 36.8693836],
                            [139.2058268, 37.0726605]],
            'dataset_id': 'ALAV2A042302850-OORIRFU-D075P0-20061110-001',
            'date_added': '2020-01-10T15:42:32.298772+00:00',
            'files': ['IMG-02-ALAV2A042302850-OORIRFU-D075P0-20061110-001.tif',
                      'IMG-04-ALAV2A042302850-OORIRFU-D075P0-20061110-001.tif',
                      'IMG-03-ALAV2A042302850-OORIRFU-D075P0-20061110-001.tif',
                      'HDR-ALAV2A042302850-OORIRFU-D075P0-20061110-001.txt',
                      'IMG-01-ALAV2A042302850-OORIRFU-D075P0-20061110-001.tif'],
            'frame_number': 2850,
            'gain': [0.588, 0.573, 0.502, 0.557],
            'observation_datetime': '2006-11-10T01:48:05.325279+00:00',
            'offset': [0.0, 0.0, 0.0, 0.0],
            'orbit_direction': 'D',
            'pointing_angle': 21.5,
            'publish_link': 'https://file.tellusxdp.com/api/v1/origin/publish/avnir2-ori/ALAV2A042302850-OORIRFU-D075P0-20061110-001',
            'rsp_frame_number': 2850,
            'rsp_path_number': 75,
            'scene_id': 'ALAV2A042302850',
            'scene_shift': 0,
            'sun_azimuth_angle': 168.9431399,
            'sun_elevation_angle': 35.5860298,
            'total_orbit_number': 4230},
           {'bbox': [138.3449177, 34.2143897, 139.4736153, 35.10965],
             <2件目。省略>
           },
           {'bbox': [138.1906085, 33.7231867, 139.3123251, 34.6170302],
             <3件目。省略>
           }],
 'next': 'https://file.tellusxdp.com/api/v1/origin/search/avnir2-ori?cursor=eyJjdXJzb3JfbGFzdF90aW1lIjogIjIwMDYtMTEtMTBUMDE6NDg6NDYuMzg2Mzg2IiwgImN1cnNvcl9iZWdpbl90aW1lIjogIjIwMDYtMTEtMTBUMDE6NDg6MDUuMzI1Mjc5IiwgInN0YXJ0X3RpbWUiOiAiMTk3MS0wMS0wMVQwMDowMDowMCIsICJlbmRfdGltZSI6ICIyMDIwLTAxLTE4VDEzOjQyOjIxLjQ1MTY5MSIsICJtaW5fcG9pbnRpbmdfYW5nbGUiOiAyMC4wLCAiY2xvdWRfY292ZXIiOiAxMC4wfQ=='}

When you run the code, it will show you the output below.

# サンプル2
# 同じ条件で続きを取得する、取得件数を1件にする
metas = search_file({'page_size': 1}, next_url = metas['next'])
pprint.pprint(metas)

Below is the output.

{'count': 1,
 'items': [{'bbox': [138.749406, 34.426666, 140.3650394, 35.4539698],
            'center_datetime': '2011-04-01T01:55:29.999478+00:00',
            'cloud_cover': 0,
            'coordinates': [[138.749406, 34.7618061],
                            [140.1085505, 34.426666],
                            [140.3650394, 35.1162994],
                            [138.9952686, 35.4539698]],
            'dataset_id': 'ALAV2A276132870-OORIRFU-D080P0-20110401-000',
            'date_added': '2020-01-10T15:42:32.298772+00:00',
            'files': ['IMG-02-ALAV2A276132870-OORIRFU-D080P0-20110401-000.tif',
                      'HDR-ALAV2A276132870-OORIRFU-D080P0-20110401-000.txt',
                      'IMG-01-ALAV2A276132870-OORIRFU-D080P0-20110401-000.tif',
                      'IMG-03-ALAV2A276132870-OORIRFU-D080P0-20110401-000.tif',
                      'IMG-04-ALAV2A276132870-OORIRFU-D080P0-20110401-000.tif'],
            'frame_number': 2870,
            'gain': [0.588, 0.573, 0.502, 0.557],
            'observation_datetime': '2011-04-01T01:55:29.999478+00:00',
            'offset': [0.0, 0.0, 0.0, 0.0],
            'orbit_direction': 'D',
            'pointing_angle': 38.0,
            'publish_link': 'https://file.tellusxdp.com/api/v1/origin/publish/avnir2-ori/ALAV2A276132870-OORIRFU-D080P0-20110401-000',
            'rsp_frame_number': 2870,
            'rsp_path_number': 80,
            'scene_id': 'ALAV2A276132870',
            'scene_shift': 0,
            'sun_azimuth_angle': 156.295176,
            'sun_elevation_angle': 57.2636783,
            'total_orbit_number': 27613}],
 'next': 'https://file.tellusxdp.com/api/v1/origin/search/avnir2-ori?cursor=eyJjdXJzb3JfbGFzdF90aW1lIjogIjIwMTEtMDQtMDFUMDE6NTU6MjkuOTk5NDc4IiwgImN1cnNvcl9iZWdpbl90aW1lIjogIjIwMTEtMDQtMDFUMDE6NTU6MjkuOTk5NDc4IiwgInN0YXJ0X3RpbWUiOiAiMTk3MS0wMS0wMVQwMDowMDowMCIsICJlbmRfdGltZSI6ICIyMDIwLTAxLTE2VDEwOjUyOjU4LjEwNzczOCIsICJtaW5fcG9pbnRpbmdfYW5nbGUiOiAyMC4wLCAiY2xvdWRfY292ZXIiOiAzMC4wfQ=='}

We got another result for the same condition. Yay!

Let’s try searching with other criteria.
This time, look for files created after the specified time.

# サンプル3
# 指定時刻以降
metas = search_file({'after': '2010-10-05T15:00:00.000000+00:00', 'page_size': 3})

pprint.pprint(metas)

Below is the output.

{'count': 3,
 'items': [<省略>],
 'next': 'https://file.tellusxdp.com/api/v1/origin/search/avnir2-ori?cursor=eyJjdXJzb3JfbGFzdF90aW1lIjogIjIwMTAtMTAtMDZUMDE6NDg6MTcuOTUzODEwIiwgImN1cnNvcl9iZWdpbl90aW1lIjogIjIwMTAtMTAtMDZUMDE6NDg6MDEuNTIwMjU5IiwgInN0YXJ0X3RpbWUiOiAiMjAxMC0xMC0wNVQxNTowMDowMCIsICJlbmRfdGltZSI6ICIyMDIwLTAxLTIxVDE1OjI2OjUyLjA5ODY4OSJ9'}

It gives three results with an observation time on or after October 5, 2010, according to the specified page_size value and time range.

2. Get the download URLs

You can download result files to see the actual data.
To do this, get the URLs for file download.

Make a function to get the file download URLs.

# ダウンロードURL発行メソッド
def publish_file(dataset_id='', searched_url=''):
    if len(searched_url) > 0:
        url = searched_url
    else:
        url = 'https://file.{}.com/api/v1/origin/publish/avnir2-ori/{}'.format(DOMAIN, dataset_id)
    
    headers = {
        'Authorization': 'Bearer ' + TOKEN
    }

    r = requests.get(url, headers=headers)
    if not r.status_code == requests.codes.ok:
        r.raise_for_status()
    return r.json()

Run the function to get the download URLs.
This function can be used in two ways.
In the first way, you use the file search function and then use the URL download function to get the download URLs for the search result files starting from the first one in the search results.

# サンプル4
# 取得用URL発行(検索結果から取得したい場合)
# ファイル検索
metas = search_file({'page_size': 10})
# ダウンロードURL発行
published = publish_file(searched_url=metas['items'][0]['publish_link'])
# 結果を表示する
pprint.pprint(published)

Below is the output.

{'dataset_id': 'ALAV2A013272710-OORIRFU-D066P3-20060425-000',
 'expires_at': '2020-01-18T20:29:06.307812+00:00',
 'files': [{'file_name': 'IMG-02-ALAV2A013272710-OORIRFU-D066P3-20060425-000.tif',
            'file_size': 58112471,
            'url': 'https://file.tellusxdp.com/api/v1/origin/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpYXQiOjE1NzkzMzYxNDYsImV4cCI6MTU3OTM3OTM0NiwiYXBpX3Rva2VuIjoiVEgwRS04UWRlTFZnIiwiZGF0YXNldCI6ImF2bmlyMi1vcmkiLCJrZXkiOiJBTEFWMkEwMTMyNzI3MTAtT09SSVJGVS1EMDY2UDMtMjAwNjA0MjUtMDAwIiwiZmlsZV9uYW1lIjoiSU1HLTAyLUFMQVYyQTAxMzI3MjcxMC1PT1JJUkZVLUQwNjZQMy0yMDA2MDQyNS0wMDAudGlmIn0.uI3qG28YKg6EbAnYVQiFVcMHQWeeWSTInIfKvXIsxbI'},
           <省略>
           ],
 'project': 'avnir2-ori'}

The “files” array has the name (file_name), size (file_size) and download URL (url) of each file.

“url” contains the URL to download the file from.

This URL expires after a certain time, which is assigned to expires_at. After this time, the URL becomes invalid and you will need to get another URL.

In the second way, you use dataset_id to get the download URLs.

dataset_id is included in the output of the file search API.

# サンプル5
# ダウンロードURL発行(dataset_idがわかっている場合)
published = publish_file('ALAV2A013272800-OORIRFU-D066P3-20060425-000')
pprint.pprint(published)

You can get the download URL from files.url.

{'dataset_id': 'ALAV2A013272800-OORIRFU-D066P3-20060425-000',
 'expires_at': '2020-01-18T20:29:17.943066+00:00',
 'files': [{'file_name': 'IMG-01-ALAV2A013272800-OORIRFU-D066P3-20060425-000.tif',
            'file_size': 58024471,
            'url': 'https://file.tellusxdp.com/api/v1/origin/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpYXQiOjE1NzkzMzYxNTcsImV4cCI6MTU3OTM3OTM1NywiYXBpX3Rva2VuIjoiVEgwRS04UWRlTFZnIiwiZGF0YXNldCI6ImF2bmlyMi1vcmkiLCJrZXkiOiJBTEFWMkEwMTMyNzI4MDAtT09SSVJGVS1EMDY2UDMtMjAwNjA0MjUtMDAwIiwiZmlsZV9uYW1lIjoiSU1HLTAxLUFMQVYyQTAxMzI3MjgwMC1PT1JJUkZVLUQwNjZQMy0yMDA2MDQyNS0wMDAudGlmIn0.BfqWgxI5Ns7JhveOCsK2wJdZ17RAjsmAibQHhIxI_XY'},
           <省略>
           ],
 'project': 'avnir2-ori'}

3. Download the files

Finally, download the files using the URLs you’ve got.

# サンプル6
# 画像データをダウンロードし保存
# ※サンプル5の実行後でないとエラーになります。
for file in published['files']:
    if re.match(r'.*tif$', file['file_name']):
        pprint.pprint(file['file_name'])
        download_file(file['url'], file['file_name'], published['dataset_id'])

It takes some time due to the big file size.

For other criteria for scene (data) search, download and check out the header text.

# サンプル7
# ヘッダーテキストをダウンロードし保存
# ※サンプル5の実行後でないとエラーになります。
for file in published['files']:
    if re.match(r'.*txt$', file['file_name']):
        pprint.pprint(file['file_name'])
        download_file(file['url'], file['file_name'], published['dataset_id'])

See プロダクトフォーマット説明書 (provisional English title: Product Format Details) for the description of the header text.

Now, let’s read and show one of the downloaded GeoTiff files.

# 読み込み
from skimage import io
from osgeo import gdal, gdalconst, gdal_array

# 青の波長
tif = gdal.Open('./ALAV2A013272800-OORIRFU-D066P3-20060425-000/IMG-01-ALAV2A013272800-OORIRFU-D066P3-20060425-000.tif', gdalconst.GA_ReadOnly)

# GeoTiffのタグコードを表示
pprint.pprint(tif.GetProjection())
pprint.pprint(tif.GetGeoTransform())
pprint.pprint((tif.RasterXSize, tif.RasterYSize)) 
pprint.pprint(tif.GetMetadata())
pprint.pprint(tif.RasterCount)

io.imshow(tif.GetRasterBand(1).ReadAsArray())

AVNIR-2’s image data consists of separate files for different wavelength bands.
In the case of AVNIR-2, blue, green, red and infrared are assigned to Band1, 2, 3 and 4, respectively.
(See https://www.tellusxdp.com/ja/dev/data/avnir-2 for details of AVNIR-2 data.)
The sample code above only deals with the blue wavelength file.

The image file looks like the picture below. It’s depicted in black and white because it is single-band imagery.

Credit : JAXA

Let’s combine the files for different wavelengths into a colored image. It should look more familiar to us.
Read the three files numbered 01, 02 and 03, which represents red, blue and green, respectively.

# 可視光
import numpy as np

tif_b = gdal.Open('./ALAV2A013272800-OORIRFU-D066P3-20060425-000/IMG-01-ALAV2A013272800-OORIRFU-D066P3-20060425-000.tif', gdalconst.GA_ReadOnly)
tif_g = gdal.Open('./ALAV2A013272800-OORIRFU-D066P3-20060425-000/IMG-02-ALAV2A013272800-OORIRFU-D066P3-20060425-000.tif', gdalconst.GA_ReadOnly)
tif_r = gdal.Open('./ALAV2A013272800-OORIRFU-D066P3-20060425-000/IMG-03-ALAV2A013272800-OORIRFU-D066P3-20060425-000.tif', gdalconst.GA_ReadOnly)

x_size = tif_b.RasterXSize
y_size = tif_b.RasterYSize
dtype = tif_b.GetRasterBand(1).ReadAsArray().dtype

rgb_array = np.zeros((y_size, x_size, 3), dtype=dtype)

rgb_array[:,:,0] = tif_r.GetRasterBand(1).ReadAsArray()
rgb_array[:,:,1] = tif_g.GetRasterBand(1).ReadAsArray()
rgb_array[:,:,2] = tif_b.GetRasterBand(1).ReadAsArray()
io.imshow(rgb_array)
Credit : JAXA

That is how to get AVNIR-2’s data (ortho rectified image product) supplied by its original provider on the development environment of Tellus.

We are planning to add original satellite data from other various providers on Tellus as well. Check it out.