tckb / sentinel1-slc

Sentinel1-SLC IW imagery in AWS Cloud in an unzipped form

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Sentinel1- SLC Europe dataset in unzipped form

Table of Contents generated with DocToc

Sentinel1-SLC Product description

Synthetic Aperture Radar (SAR) is an active remote sensing technique, where electromagnetic pulses are emitted and received by an antenna. The Sentinel-1 satellite mission consists of two identical satellites, Sentinel-1A and Sentinel-1B, launched by the European Space Agency (ESA) in 03 April 2014 and 25 April 2016 respectively (ESA, 2021a). These satellites orbit the Earth while acquiring SAR images at a wave frequency of 5.405GHz (C-band). This mission exploits the usage of 2 satellites in order to have a fast revisit time (12 days and down to 6 days in some areas).The main applications of the Sentinel-1 SAR images are the monitoring of land use changes and surface deformation along with support for emergency management. Further applications include but are not limited to monitoring of sea ice, icebergs, land ice, inland waters, oil spills, ships, and others (ESA, 2021a).

ESA. (2021a). About copernicus sentinel-1

Dataset in S3 at glance

The Sentinel-1 SLC IW image collection provided by ESA comes in an archive format, which must be entirely downloaded and unzipped first, in order to be used. Often times users only need selective data inside these archived folders for their work.

  • The Earth On AWS dataset is stored in the unzipped form in the S3 bucket, offering users the option to selectively retrieve either the full imagery or only the parts of the data that is needed for a given study area.
  • Since the dataset resides on S3, depending upon the application, users can also directly read the object into memory and carry out their work without having to download, unzip and store them in on-premise or cloud storages.
  • The S3 bucket and objects in it are public. Anonymous access is also enabled. So users can access the data without aws account/credentials as well.

We have fully ingested Sentinel-1A/B Level-1 SLC over Germany, which is updated in the interval of 6 days, after they are made available by Alaska Satellite Facility (ASF).

Expanding data scope to EU region (work in progress)

Following up, as a next step, we aim to expand the service by increasing data coverage across the Europe region. The map below clearly describes the data coverage expansion plan.

The following figure provides an insight of the current percentage of EU dataset uploaded to the S3 bucket. As mentioned above, in addition to Germany, other regions are also being updated simultaneously, however, the process is still in progress with 34.31% of the total EU dataset already available in sentinel1-slc S3 bucket.

YearPercentage of EU dataset in S3
201494.55%
201566.48%
201624.08%
201728.33%
201829.53%
201968.47%
202019.80%
202117.30%
202243.21%
Total34.31%

AWS S3 bucket and object structure

The dataset in the S3 bucket is organized in a directory structure based on the start date of the acquisition for ease of retrieval. The table below provides information on product details, useful for querying the dataset.


s3://sentinel1-slc/YYYY/MM/DD/XXX_BB_SLC__1SPP_YYYYMMDDTHHMMSS_yyyymmddThhmmss_OOOOOO_DDDDDD_CCCC.SAFE


Example S3 URI of an imagery:

s3://sentinel1-slc/2022/01/01/S1A_IW_SLC__1SDV_20220101T053305_20220101T053332_041263_04E777_0EE8.SAFE/

Contents inside each imagery

Each Sentinel-1 SAR product folder (object in the bucket) includes:

  • a 'manifest.safe' file which holds the general product information in XML
  • a measurement folder with complex measurement data set in GeoTIFF format per sub-swath per polarisation
  • a preview folder containing 'quicklooks' in PNG format, Google Earth overlays in KML format and HTML preview files
  • an annotation folder containing the product metadata in XML as well as calibration data
  • a support folder containing the XML schemes describing the product XML.
Variable Description Details (code: code details)
XXX Denotes the satellite S1A: Sentinel-1A
S1B: Sentinel-1B
BB Acquisition Mode IW: Interferometric Wide-Swath
PP Polarisation SH:single HH polarisation
SV: single VV polarisation
DH: dual HH+HV polarisation
DV: dual VV+VH polarisation
HH: Partial Dual polarisation, HH only
HV: Partial Dual polarisation, HV only
VV: Partial Dual polarisation, VV only
VH: Partial Dual polarisation, VV only
YYYYMMDD Acquisition Start Date (UTC) Four-digit year, two-digit month, two-digit day
HHMMSS Acquisition Start Time (UTC) Two-digit hour, two-digit minutes, two-digit seconds
yyyymmdd Acquisition End Date (UTC) Four-digit year, two-digit month, two-digit day
hhmmss Acquisition End Time (UTC) Two-digit hour, two-digit minutes, two-digit seconds
OOOOOO Absolute orbit number at product start time In the range of 000001-999999
DDDDDD Mission data take ID In the range 000001-FFFFFF
CCCC Hexadecimal string generated from CRC-16 of the manifest file CRC-16 algorithm used to compute the unique identifier is CRC-CCITT (0xFFFF)

Querying for available imagery in the bucket

The data in the S3 bucket can be queried with AWS CLI or with boto3 library.

Some examples with AWS CLI.

For access without using the aws credentials, simply pass the --no-sign-request to the same command.

  • To retrieve list of all years for which the imagery are available:
    aws s3 ls sentinel1-slc
    aws s3 ls sentinel-slc --no-sign-request
  • To retrieve list of all imagery available for a given year, month and date:
    aws s3 ls sentinel1-slc/2022/01/01/
    aws s3 ls sentinel1-slc/2022/01/01/ --no-sign-request
  • To retrieve list of all files and folders inside a given imagery:
    aws s3 ls sentinel1-slc/2022/01/01/S1A_IW_SLC__1SDV_20220101T053305_20220101T053332_041263_04E777_0EE8.SAFE/
    aws s3 ls sentinel1-slc/2022/01/01/S1A_IW_SLC__1SDV_20220101T053305_20220101T053332_041263_04E777_0EE8.SAFE/ --no-sign-request
  • To retrieve list of all imagery with a given polarization type and mission type:
    aws s3 ls sentinel1-slc/2022/01/01/S1A_IW_SLC__1SDV_
    aws s3 ls sentinel1-slc/2022/01/01/S1A_IW_SLC__1SDV_ --no-sign-request

Examples with boto3 library

  • Anonymously read objects without downloading and without passing AWS credentials.

    import boto3
    
    from botocore import UNSIGNED
    from botocore.config import Config
    
    s3_client = boto3.client('s3', config=Config(signature_version=UNSIGNED))
    my_bucket = 'sentinel1-slc'
    file_to_read = '2022/01/01/S1A_IW_SLC__1SDV_20220101T053305_20220101T053332_041263_04E777_0EE8.SAFE/annotation/s1a-iw1-slc-vh-20220101t053305-20220101t053330-041263-04e777-001.xml'
    
    s3_response_object = s3_client.get_object(Bucket=my_bucket, Key=file_to_read)
    object_content = s3_response_object['Body'].read()
    print(object_content)
  • Retrieve list of all imagery for a given year and month with AWS credentials.

    import boto3
    
    client = boto3.client('s3', region_name='eu-west-1')
    my_bucket = 'sentinel1-slc'
    prefix_to_query = '2022/01/01/'
    results = client.list_objects(Bucket=my_bucket,
                                  Prefix=prefix_to_query,
                                  Delimiter='/'
                                  )
    for result in results.get('CommonPrefixes'):
        print(result.get('Prefix'))

Tools and ready to use scripts will be added to the repository in future updates.

Acknowledgements

We would like to thank Amazon Web Services for providing the storage resources for this program. We would also like to acknowledge the team at ASF for their collaboration.

About

Sentinel1-SLC IW imagery in AWS Cloud in an unzipped form

License:Apache License 2.0