[How to Use Tellus from Scratch] Display AVNIR-2’s ortho rectified image product
The API for the original data from the AVNIR-2 sensor carried on JAXA's ALOS satellite has become available on Tellus! We'll tell you how to use it.
AVNIR-2’s original data (GeoTiff) provided by JAXA is now available on Tellus. You can deal with a wider area at a time by using the original data.
In this article, we’ll tell you how to get AVNIR-2’s original data (ortho rectified image product) on the development environment.
AVNIR-2’s data is satellite data provided by JAXA. You can use it on Tellus. When you create derivative works, you have to put “(c)JAXA” on them. See Tellus’ data catalog for more information.
Get AVNIR-2's original data (ortho rectified image product)
You can get the original data with the following three steps.
1. Do a file search
Perform a search for files that match your criteria using metadata.
↓
2. Get the download URLs
Get the file download URLs from the files’ metadata.
↓
3. Download the files
Download the files using the file download URLs and save them.
See Use Tellus API from the development environment for how to use the development environment (JupyterLab) on Tellus.
File search API
Use the API below for file search.
https://file.tellusxdp.com/api/v1/origin/search/avnir2-oriArguments
| Name | Query | Type | Search criteria |
| Search start time | after | str(ISO8601) | Returns results with an observation_datetime value later than the specified time. |
| Search end time | before | str(ISO8601) | Returns results with an observation_datetime value earlier than the specified time. |
| Cloud cover | cloud_cover | float | Returns results with a cloud_cover value less than the specified value. Doesn’t return the error value (99) if the argument is valid. |
| RSP path number | rsp_path_number | int | Exact match |
| RSP frame number | rsp_frame_number | int | Exact match |
| Minimum pointing angle | min_pointing_angle | float | Returns results with a pointing_angle value greater than the specified value. |
| Maximum pointing angle | max_pointing_angle | float | Returns results with a pointing_angle value less than the specified value. |
| Scene ID | scene_id | str | Exact match |
| Search area | left_bottom_lon right_top_lon left_bottom_lat right_top_lat |
Return results that have any part of their rectangular area outlined by min_lat, min_lon, max_lat and max_lon within the specified area. https://tools.ietf.org/html/rfc7946#section-5 |
|
| Number of results returned | page_size | int | Specify the number of results returned in a request. The default value is 100. The maximum 1,000. |
Return value
| Data name | Data name | Type and sample |
| Positions (the four corners) |
coordinates | array of float [ [left_bottom_lon, left_bottom_lat], [right_bottom_lon, right_bottom_lat], [right_top_lon, right_top_lat], [left_top_lon, left_top_lat] ] Example) [ [138.9756564, 36.3759473], [139.8980412, 36.174298], [140.1360529, 36.8693836], [139.2058268, 37.0726605] ] |
| Scene center time (UTC) | center_datetime | str(ISO8601)
Example)2006-11-10T01:48:05.325279+00:00 |
| Orbit direction | orbit_direction | str D stands for descending orbit, A ascending orbit. |
| RSP path number | rsp_path_number | int |
| RSP frame number | rsp_frame_number | int |
| Total orbit number of scene center | total_orbit_number | int |
| Frame number of scene center | frame_number | int |
| Amount of scene shift | scene_shift | int |
| Cloud cover for reference | cloud_cover | int |
| Sun elevation angle | sun_elevation_angle | float |
| Sun elevation angle | sun_azimuth_angle | float |
| Pointing angle | pointing_angle | float |
| Gain of each band | gain | array of float
[band1, band2, band3, band4] Example) [0.588, 0.573, 0.502, 0.557] |
| Offset of each band | offset | array of float
[band1, band2, band3, band4] Example)[0.0, 0.0, 0.0, 0.0] |
| Scene ID | scene_id | str
Example) ALAV2A042302850 |
| Dataset name | dataset_id | str
Example) ALAV2A042302850-OORIRFU-D075P0-20061110-001 |
| List of files in a dataset | files | array of string [ filename1, filename2, … ]Example) [ ‘IMG-02-ALAV2A042302850-OORIRFU-D075P0-20061110-001.tif’, … , ‘HDR-ALAV2A042302850-OORIRFU-D075P0-20061110-001.txt’ ] |
| Date of addition to Tellus | date_added | str(ISO8601) |
| Representative time | observation_datetime | str(ISO8601)
Same as center_datetime. |
| Area | bbox | array of float
[left_lon, bottom_lat, right_lon, top_lat] Example) [138.9756564, 36.174298, 140.1360529, 37.0726605] |
| Download link | publish_url | Example)
https://file.tellusxdp.com/api/v1/origin/publish/avnir2-ori/ALAV2A042302850-OORIRFU-D075P0-20061110-001 |
1. Do a file search
First, write a function to call the file search API.
Get the token from APIアクセス設定 (API access setting) in My Page (login required).
import os, requests
import pprint
TOKEN = ‘ここには自分のアカウントのトークンを貼り付ける'
DOMAIN = 'tellusxdp'
# ファイル検索メソッド
def search_file(params={}, next_url=''):
if len(next_url) > 0:
url = next_url
else:
url = 'https://file.{}.com/api/v1/origin/search/avnir2-ori'.format(DOMAIN)
headers = {
'Authorization': 'Bearer ' + TOKEN
}
r = requests.get(url, params=params, headers=headers)
if not r.status_code == requests.codes.ok:
r.raise_for_status()
return r.json()
This function receives a dictionary of parameters and then returns the content of header text of corresponding files in JSON format.
Let’s do a file search.
We use the following search criteria this time.
– Cloud cover of 10 percent or less
– Pointing angle of 20 degrees or less
– Three results in a request
# サンプル1
# 雲量10%以下 かつ ポインティング角が20deg以下、一度に取得する件数は3件
metas = search_file({'cloud_cover': 10, 'min_pointing_angle': 20, 'page_size': 3})
# 検索結果を表示
pprint.pprint(metas)
When you run the code, it will show you the output below.
{'count': 3,
'items': [{'bbox': [138.9756564, 36.174298, 140.1360529, 37.0726605],
'center_datetime': '2006-11-10T01:48:05.325279+00:00',
'cloud_cover': 0,
'coordinates': [[138.9756564, 36.3759473],
[139.8980412, 36.174298],
[140.1360529, 36.8693836],
[139.2058268, 37.0726605]],
'dataset_id': 'ALAV2A042302850-OORIRFU-D075P0-20061110-001',
'date_added': '2020-01-10T15:42:32.298772+00:00',
'files': ['IMG-02-ALAV2A042302850-OORIRFU-D075P0-20061110-001.tif',
'IMG-04-ALAV2A042302850-OORIRFU-D075P0-20061110-001.tif',
'IMG-03-ALAV2A042302850-OORIRFU-D075P0-20061110-001.tif',
'HDR-ALAV2A042302850-OORIRFU-D075P0-20061110-001.txt',
'IMG-01-ALAV2A042302850-OORIRFU-D075P0-20061110-001.tif'],
'frame_number': 2850,
'gain': [0.588, 0.573, 0.502, 0.557],
'observation_datetime': '2006-11-10T01:48:05.325279+00:00',
'offset': [0.0, 0.0, 0.0, 0.0],
'orbit_direction': 'D',
'pointing_angle': 21.5,
'publish_link': 'https://file.tellusxdp.com/api/v1/origin/publish/avnir2-ori/ALAV2A042302850-OORIRFU-D075P0-20061110-001',
'rsp_frame_number': 2850,
'rsp_path_number': 75,
'scene_id': 'ALAV2A042302850',
'scene_shift': 0,
'sun_azimuth_angle': 168.9431399,
'sun_elevation_angle': 35.5860298,
'total_orbit_number': 4230},
{'bbox': [138.3449177, 34.2143897, 139.4736153, 35.10965],
},
{'bbox': [138.1906085, 33.7231867, 139.3123251, 34.6170302],
}],
'next': 'https://file.tellusxdp.com/api/v1/origin/search/avnir2-ori?cursor=eyJjdXJzb3JfbGFzdF90aW1lIjogIjIwMDYtMTEtMTBUMDE6NDg6NDYuMzg2Mzg2IiwgImN1cnNvcl9iZWdpbl90aW1lIjogIjIwMDYtMTEtMTBUMDE6NDg6MDUuMzI1Mjc5IiwgInN0YXJ0X3RpbWUiOiAiMTk3MS0wMS0wMVQwMDowMDowMCIsICJlbmRfdGltZSI6ICIyMDIwLTAxLTE4VDEzOjQyOjIxLjQ1MTY5MSIsICJtaW5fcG9pbnRpbmdfYW5nbGUiOiAyMC4wLCAiY2xvdWRfY292ZXIiOiAxMC4wfQ=='}
When you run the code, it will show you the output below.
# サンプル2
# 同じ条件で続きを取得する、取得件数を1件にする
metas = search_file({'page_size': 1}, next_url = metas['next'])
pprint.pprint(metas)
Below is the output.
{'count': 1,
'items': [{'bbox': [138.749406, 34.426666, 140.3650394, 35.4539698],
'center_datetime': '2011-04-01T01:55:29.999478+00:00',
'cloud_cover': 0,
'coordinates': [[138.749406, 34.7618061],
[140.1085505, 34.426666],
[140.3650394, 35.1162994],
[138.9952686, 35.4539698]],
'dataset_id': 'ALAV2A276132870-OORIRFU-D080P0-20110401-000',
'date_added': '2020-01-10T15:42:32.298772+00:00',
'files': ['IMG-02-ALAV2A276132870-OORIRFU-D080P0-20110401-000.tif',
'HDR-ALAV2A276132870-OORIRFU-D080P0-20110401-000.txt',
'IMG-01-ALAV2A276132870-OORIRFU-D080P0-20110401-000.tif',
'IMG-03-ALAV2A276132870-OORIRFU-D080P0-20110401-000.tif',
'IMG-04-ALAV2A276132870-OORIRFU-D080P0-20110401-000.tif'],
'frame_number': 2870,
'gain': [0.588, 0.573, 0.502, 0.557],
'observation_datetime': '2011-04-01T01:55:29.999478+00:00',
'offset': [0.0, 0.0, 0.0, 0.0],
'orbit_direction': 'D',
'pointing_angle': 38.0,
'publish_link': 'https://file.tellusxdp.com/api/v1/origin/publish/avnir2-ori/ALAV2A276132870-OORIRFU-D080P0-20110401-000',
'rsp_frame_number': 2870,
'rsp_path_number': 80,
'scene_id': 'ALAV2A276132870',
'scene_shift': 0,
'sun_azimuth_angle': 156.295176,
'sun_elevation_angle': 57.2636783,
'total_orbit_number': 27613}],
'next': 'https://file.tellusxdp.com/api/v1/origin/search/avnir2-ori?cursor=eyJjdXJzb3JfbGFzdF90aW1lIjogIjIwMTEtMDQtMDFUMDE6NTU6MjkuOTk5NDc4IiwgImN1cnNvcl9iZWdpbl90aW1lIjogIjIwMTEtMDQtMDFUMDE6NTU6MjkuOTk5NDc4IiwgInN0YXJ0X3RpbWUiOiAiMTk3MS0wMS0wMVQwMDowMDowMCIsICJlbmRfdGltZSI6ICIyMDIwLTAxLTE2VDEwOjUyOjU4LjEwNzczOCIsICJtaW5fcG9pbnRpbmdfYW5nbGUiOiAyMC4wLCAiY2xvdWRfY292ZXIiOiAzMC4wfQ=='}
We got another result for the same condition. Yay!
Let’s try searching with other criteria.
This time, look for files created after the specified time.
# サンプル3
# 指定時刻以降
metas = search_file({'after': '2010-10-05T15:00:00.000000+00:00', 'page_size': 3})
pprint.pprint(metas)
Below is the output.
{'count': 3,
'items': [],
'next': 'https://file.tellusxdp.com/api/v1/origin/search/avnir2-ori?cursor=eyJjdXJzb3JfbGFzdF90aW1lIjogIjIwMTAtMTAtMDZUMDE6NDg6MTcuOTUzODEwIiwgImN1cnNvcl9iZWdpbl90aW1lIjogIjIwMTAtMTAtMDZUMDE6NDg6MDEuNTIwMjU5IiwgInN0YXJ0X3RpbWUiOiAiMjAxMC0xMC0wNVQxNTowMDowMCIsICJlbmRfdGltZSI6ICIyMDIwLTAxLTIxVDE1OjI2OjUyLjA5ODY4OSJ9'}
It gives three results with an observation time on or after October 5, 2010, according to the specified page_size value and time range.
2. Get the download URLs
You can download result files to see the actual data.
To do this, get the URLs for file download.
Make a function to get the file download URLs.
# ダウンロードURL発行メソッド
def publish_file(dataset_id='', searched_url=''):
if len(searched_url) > 0:
url = searched_url
else:
url = 'https://file.{}.com/api/v1/origin/publish/avnir2-ori/{}'.format(DOMAIN, dataset_id)
headers = {
'Authorization': 'Bearer ' + TOKEN
}
r = requests.get(url, headers=headers)
if not r.status_code == requests.codes.ok:
r.raise_for_status()
return r.json()
Run the function to get the download URLs.
This function can be used in two ways.
In the first way, you use the file search function and then use the URL download function to get the download URLs for the search result files starting from the first one in the search results.
# サンプル4
# 取得用URL発行（検索結果から取得したい場合）
# ファイル検索
metas = search_file({'page_size': 10})
# ダウンロードURL発行
published = publish_file(searched_url=metas['items'][0]['publish_link'])
# 結果を表示する
pprint.pprint(published)
Below is the output.
{'dataset_id': 'ALAV2A013272710-OORIRFU-D066P3-20060425-000',
'expires_at': '2020-01-18T20:29:06.307812+00:00',
'files': [{'file_name': 'IMG-02-ALAV2A013272710-OORIRFU-D066P3-20060425-000.tif',
'file_size': 58112471,
'url': 'https://file.tellusxdp.com/api/v1/origin/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpYXQiOjE1NzkzMzYxNDYsImV4cCI6MTU3OTM3OTM0NiwiYXBpX3Rva2VuIjoiVEgwRS04UWRlTFZnIiwiZGF0YXNldCI6ImF2bmlyMi1vcmkiLCJrZXkiOiJBTEFWMkEwMTMyNzI3MTAtT09SSVJGVS1EMDY2UDMtMjAwNjA0MjUtMDAwIiwiZmlsZV9uYW1lIjoiSU1HLTAyLUFMQVYyQTAxMzI3MjcxMC1PT1JJUkZVLUQwNjZQMy0yMDA2MDQyNS0wMDAudGlmIn0.uI3qG28YKg6EbAnYVQiFVcMHQWeeWSTInIfKvXIsxbI'},
],
'project': 'avnir2-ori'}
The “files” array has the name (file_name), size (file_size) and download URL (url) of each file.
“url” contains the URL to download the file from.
This URL expires after a certain time, which is assigned to expires_at. After this time, the URL becomes invalid and you will need to get another URL.
In the second way, you use dataset_id to get the download URLs.
dataset_id is included in the output of the file search API.
# サンプル5
# ダウンロードURL発行（dataset_idがわかっている場合）
published = publish_file('ALAV2A013272800-OORIRFU-D066P3-20060425-000')
pprint.pprint(published)
You can get the download URL from files.url.
{'dataset_id': 'ALAV2A013272800-OORIRFU-D066P3-20060425-000',
'expires_at': '2020-01-18T20:29:17.943066+00:00',
'files': [{'file_name': 'IMG-01-ALAV2A013272800-OORIRFU-D066P3-20060425-000.tif',
'file_size': 58024471,
'url': 'https://file.tellusxdp.com/api/v1/origin/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpYXQiOjE1NzkzMzYxNTcsImV4cCI6MTU3OTM3OTM1NywiYXBpX3Rva2VuIjoiVEgwRS04UWRlTFZnIiwiZGF0YXNldCI6ImF2bmlyMi1vcmkiLCJrZXkiOiJBTEFWMkEwMTMyNzI4MDAtT09SSVJGVS1EMDY2UDMtMjAwNjA0MjUtMDAwIiwiZmlsZV9uYW1lIjoiSU1HLTAxLUFMQVYyQTAxMzI3MjgwMC1PT1JJUkZVLUQwNjZQMy0yMDA2MDQyNS0wMDAudGlmIn0.BfqWgxI5Ns7JhveOCsK2wJdZ17RAjsmAibQHhIxI_XY'},
],
'project': 'avnir2-ori'}
3. Download the files
Finally, download the files using the URLs you’ve got.
# サンプル6
# 画像データをダウンロードし保存
# ※サンプル5の実行後でないとエラーになります。
for file in published['files']:
if re.match(r'.*tif$', file['file_name']):
pprint.pprint(file['file_name'])
download_file(file['url'], file['file_name'], published['dataset_id'])
It takes some time due to the big file size.
For other criteria for scene (data) search, download and check out the header text.
# サンプル7
# ヘッダーテキストをダウンロードし保存
# ※サンプル5の実行後でないとエラーになります。
for file in published['files']:
if re.match(r'.*txt$', file['file_name']):
pprint.pprint(file['file_name'])
download_file(file['url'], file['file_name'], published['dataset_id'])
See プロダクトフォーマット説明書 (provisional English title: Product Format Details) for the description of the header text.
Now, let’s read and show one of the downloaded GeoTiff files.
# 読み込み
from skimage import io
from osgeo import gdal, gdalconst, gdal_array
# 青の波長
tif = gdal.Open('./ALAV2A013272800-OORIRFU-D066P3-20060425-000/IMG-01-ALAV2A013272800-OORIRFU-D066P3-20060425-000.tif', gdalconst.GA_ReadOnly)
# GeoTiffのタグコードを表示
pprint.pprint(tif.GetProjection())
pprint.pprint(tif.GetGeoTransform())
pprint.pprint((tif.RasterXSize, tif.RasterYSize))
pprint.pprint(tif.GetMetadata())
pprint.pprint(tif.RasterCount)
io.imshow(tif.GetRasterBand(1).ReadAsArray())
AVNIR-2’s image data consists of separate files for different wavelength bands.
In the case of AVNIR-2, blue, green, red and infrared are assigned to Band1, 2, 3 and 4, respectively.
(See https://www.tellusxdp.com/ja/dev/data/avnir-2 for details of AVNIR-2 data.)
The sample code above only deals with the blue wavelength file.
The image file looks like the picture below. It’s depicted in black and white because it is single-band imagery.
Let’s combine the files for different wavelengths into a colored image. It should look more familiar to us.
Read the three files numbered 01, 02 and 03, which represents red, blue and green, respectively.
# 可視光
import numpy as np
tif_b = gdal.Open('./ALAV2A013272800-OORIRFU-D066P3-20060425-000/IMG-01-ALAV2A013272800-OORIRFU-D066P3-20060425-000.tif', gdalconst.GA_ReadOnly)
tif_g = gdal.Open('./ALAV2A013272800-OORIRFU-D066P3-20060425-000/IMG-02-ALAV2A013272800-OORIRFU-D066P3-20060425-000.tif', gdalconst.GA_ReadOnly)
tif_r = gdal.Open('./ALAV2A013272800-OORIRFU-D066P3-20060425-000/IMG-03-ALAV2A013272800-OORIRFU-D066P3-20060425-000.tif', gdalconst.GA_ReadOnly)
x_size = tif_b.RasterXSize
y_size = tif_b.RasterYSize
dtype = tif_b.GetRasterBand(1).ReadAsArray().dtype
rgb_array = np.zeros((y_size, x_size, 3), dtype=dtype)
rgb_array[:,:,0] = tif_r.GetRasterBand(1).ReadAsArray()
rgb_array[:,:,1] = tif_g.GetRasterBand(1).ReadAsArray()
rgb_array[:,:,2] = tif_b.GetRasterBand(1).ReadAsArray()
io.imshow(rgb_array)
That is how to get AVNIR-2’s data (ortho rectified image product) supplied by its original provider on the development environment of Tellus.
We are planning to add original satellite data from other various providers on Tellus as well. Check it out.
