You are here: Home User Guide Open Data Protocol (OData)

Open Data Protocol (OData)

The Open Data Protocol (OData) enables the creation of REST-based data services, which allow resources, identified using Uniform Resource Identifiers (URIs) and defined in a data model, to be published and consumed by Web clients using simple HTTP messages.

The OData protocol provides easy access to the Data Hub and can be used for building URI for performing search queries and product downloads offering to the users the capability to remotely run scripts in batch mode.

URI Components

A URI used by an OData service has up to three significant parts: the Service Root URI, the Resource Path and the Query Options.

  • the Service Root URI identifies the root of the OData service
  • the Resource Path identifies the resource to be interacted with. The resource path enables any aspect of the data model (Data Hub Products, Data Hub Collections, etc.) exposed by the OData service
  • the system Query Options part refines the results

 

Example of and OData URI exposed by the Data Hub Service broken down into its component parts:

https://data.sentinel.zamg.ac.at/odata/v1?$top=100

OData Service Root URI for the Data Hub:

  • https://data.sentinel.zamg.ac.at/odata/v1

Data Hub Resource Paths:

  • /Products
  • /Collections

Query Options admitted by the Data Hub service:

  • $format Specifies the HTTP response format of the record e.g. XML or JSON
  • $filter Specifies an expression or function that must evaluate to true for a record to be returned in the collection
  • $orderby Determines what values are used to order a collection of records
  • $select Specifies a subset of properties to return
  • $skip Sets the number of records to skip before it retrieves records in a collection
  • $top Determines the maximum number of records to return

Examples of OData URIs for the Scientific Data Hub:

  • https://data.sentinel.zamg.ac.at/odata/v1/Products?$orderby=IngestionDate desc&$top=100
    lists the records of the last 100 products published on the Data Hub
  • https://data.sentinel.zamg.ac.at/odata/v1/Products?$orderby=IngestionDate desc&$top=100&$skip=100
    skips the first 100 records of the products published on the Data Hub and then returns the next 100

Responses

When users query the Data Hub via the URI, the request is sent to the Data Hub OData service. This service provides the responses. The description of the responses is provided here below focusing on the response body and not covering the standard HTTP response envelope.

The default response format is Atom [RFC4287], an XML-based document format that describes Collections of related information known as “feeds”. The responses containing a single Product entity differ from those containing a collection of Product entities. Generally speaking, the single Product entity is returned as a bare Atom entry element, while for a collection the same Atom entries denoting the Product entities are wrapped into an Atom feed element.

The response format can however be controlled from the requests through the $format query option already introduced above. The formats currently implemented by Data Hub service are Atom (or XML) and JSON which is a non-standard output of OData but is an experimental behaviour useful for EO context.

The following sub-sections describe the response types for Atom/XML and JSON output formats. The very last sub-section deals with the specific responses returned in case of error.

XML Responses

The Atom/XML format is the default response format, though it can be forced by setting the $format query option to “atom” or “xml” indifferently as in the following examples:

  • https://data.sentinel.zamg.ac.at/odata/v1/Products?$format=xml

Service Metadata Document

In order to help clients discover the OData services, the Data Hub OData Service Metadata Document exposes the Entity Data Model of the service including among others, the Entities and Properties that can be queried. This document can be queried as with the following URL:

  • https://data.sentinel.zamg.ac.at/odata/v1/$metadata

The following example is extracted from the complete Service Metadata Document exposed by a Data Hub OData service. Some sections or XML Namespace declarations have been removed for brevity:

<edmx:Edmx Version="1.0">

<edmx:DataServices>

<Namespace="DHuS">

 

<EntityType Name="Node" m:HasStream="true">

<Key>...</Key>

<Property Name="Id" Type="Edm.String" Nullable="false"/>

<Property Name="Name" Type="Edm.String" m:FC_TargetPath="SyndicationTitle"/>

<Property Name="ContentType" Type="Edm.String"/>

<Property Name="ContentLength" Type="Edm.Int64"/>

<Property Name="ChildrenNumber" Type="Edm.Int64"/>

<Property Name="Value" Type="Edm.String"/>

...

</EntityType>

<EntityType Name="Product" m:HasStream="true">

<Key>

<PropertyRef Name="Id"/>

</Key>

<Property Name="Id" Type="Edm.String" Nullable="false"/>

<Property Name="Name" Type="Edm.String" m:FC_TargetPath="SyndicationTitle"/>

<Property Name="ContentType" Type="Edm.String"/>

<Property Name="ContentLength" Type="Edm.Int64"/>

<Property Name="ChildrenNumber" Type="Edm.Int64"/>

<Property Name="Value" Type="Edm.String"/>

...

</EntityType>

 

<EntityType Name="Collection">

<Key>

<PropertyRef Name="Name"/>

</Key>

Using OData to discover products in the Data Hub archive

Querying the content of the products archive: usage of the /Products resource path

The following OData URI addresses the resource "Products" exposed by the Data Hub Service which includes the records of all the products stored in the Data Hub archive. Each product record includes the Universally Unique Identifier (UUID), the product name, the link for download, the links for navigating through their nodes and attributes and some properties.

The syntax is:

<ServiceRootUri>/Products

Example:

  • https://data.sentinel.zamg.ac.at/odata/v1/Products

By default it provides a list of 50 records showing the properties of the latest 50 products ingested by the Data Hub.

Examples of properties are:

IngestionDate
date on which the Product was ingested by the contacted Data Hub instance. The date referential is UTC.
ContentDate (Start and End)
time period associated to the Product payload. This date regards the period of the subject of the content and not the content itself. For example, for an EO product, the ContentDate deals with the observation date and not to the downlink or processing dates.
ContentGeometry
footprint polygon coordinates

The universally unique identifier (UUID) is a standard identifier used in software construction to uniquely identify information without significant central coordination. A UUID is a 16-octet (128-bit) number.In its canonical form, a UUID is represented by 32 lowercase hexadecimal digits, displayed in five groups separated by hyphens, in the form 8-4-4-4-12 for a total of 36 characters (32 alphanumeric characters and four hyphens).

Querying a single Product in the archive

The responses containing a single Product entity differ from those containing a collection of Product entities. Generally speaking, the single Product entity is returned as a bare Atom entry element, while for a collection the same Atom entries denoting the Product entities are wrapped into an Atom feed element.

Syntax:

<ServiceRootUri> /Products('UUID')

The following example is an actual response of a Data Hub OData service that returned a single Product entity identified by the UUID: 18f7993d-eae1-4f7f-9d81-d7cf19c18378.

Again, this snippet has been reformatted for readability purpose but nothing has been removed from the actual response:

https://data.sentinel.zamg.ac.at/odata/v1/Products('18f7993d-eae1-4f7f-9d81-d7cf19c18378')

Response:

<id>https://data.sentinel.zamg.ac.at/odata/v1/Products('18f7993d-eae1-4f7f-9d81-d7cf19c1837 8')</id><title type="text">S1A_IW_SLC__1SDV_20141023T172123_20141023T172150_002960_0035D1_9743 </title> <updated>2014-12-07T17:06:00.324Z</updated> <category term="DHuS.Product" scheme="http://schemas.microsoft.com/ado/2007/08/dataser vices/scheme"/> <link href="Products('18f7993d-eae1-4f7f-9d81-d7cf19c18378')" rel="edit" title="Produc t"/> <link href="Products('18f7993d-eae1-4f7f-9d81-d7cf19c18378')/$value" rel="edit-media" type="application/octet-stream"/> <link href="Products('18f7993d-eae1-4f7f-9d81-d7cf19c18378')/Nodes" rel="http://schema s.microsoft.com/ado/2007/08/dataservices/related/Nodes" title="Nodes" type="applicatio n/atom+xml;type=feed"/> <link href="Products('18f7993d-eae1-4f7f-9d81-d7cf19c18378')/Attributes" rel="http://s chemas.microsoft.com/ado/2007/08/dataservices/related/Attributes" title="Attributes" t ype="application/atom+xml;type=feed"/> <link href="Products('18f7993d-eae1-4f7f-9d81-d7cf19c18378')/Products" rel="http://sch emas.microsoft.com/ado/2007/08/dataservices/related/Products" title="Products" type="a pplication/atom+xml;type=feed"/> <content type="application/octet-stream" src="Products('18f7993d-eae1-4f7f-9d81-d7cf19 c18378')/$value"/> <m:properties> <d:Id>18f7993d-eae1-4f7f-9d81-d7cf19c18378</d:Id> <d:Name>S1A_IW_SLC__1SDV_20141023T172123_20141023T172150_002960_0035D1_9743</d:Name > <d:ContentType>application/octet-stream</d:ContentType> <d:ContentLength>8544532822</d:ContentLength> <d:ChildrenNumber>2</d:ChildrenNumber> <d:Value m:null="true"/> <d:CreationDate>2014-12-07T17:06:00.324</d:CreationDate> <d:IngestionDate>2014-12-07T17:06:00.324</d:IngestionDate> <d:EvictionDate m:null="true"/> <d:ContentDate m:type="DHuS.TimeRange"> <d:Start>2014-10-23T17:21:23.23</d:Start> <d:End>2014-10-23T17:21:50.495</d:End> </d:ContentDate> <d:ContentGeometry> <gml:Polygon srsName="http://www.opengis.net/gml/srs/epsg.xml#4326" xmlns:gml="h ttp://www.opengis.net/gml"> <gml:outerBoundaryIs> <gml:LinearRing> <gml:coordinates>41.289764,6.891860 41.685265,9.900283 40.048470,10.244 14039.652199,7.311233 41.289764,6.891860</gml:coordinates> </gml:LinearRing> </gml:outerBoundaryIs> </gml:Polygon> </d:ContentGeometry> <d:Metalink> <?xml version="1.0" encoding="UTF-8" standalone="no"?> <metalink xmlns="urn:ietf:params:xml:ns:metalink"> <file name="S1A_IW_SLC__1SDV_20141023T172123_20141023T172150_002960_0035D1_97 43.zip"> <url>https://data.sentinel.zamg.ac.at/odata/v1/Products('18f7993d-eae1-4f7f-9d8 1-d7cf19c18378')/$value</url> </file> </metalink> </d:Metalink> <d:Checksum m:type="DHuS.TimeRange"> <d:Algorithm>MD5</d:Algorithm> <d:value>C4415763B3198B7A2874C2A60B2CDCDC</d:value> </d:Checksum> </m:properties>

Querying the products (paging)

The URI to be used for paging the list of products in the archive shall follow the syntax below:

<ServiceRootUri> /Products?$skip='N'&$top='M'

where ?$skip='N' is the number of records to skip before it retrieves records in a collection and $top is the maximum number of records to return.

Example:

  • https://data.sentinel.zamg.ac.at/odata/v1/Products?$skip=10&$top=50
    this OData URI allows to list 50 products skipping the first 10.

$top accepts as maximum value 100. For higher values the maximum number of records returned will still be 100.

Using OData to filter the products in the archive

It is possible to use the OData properties to perform search query to filter the products.

Filtering the Products on time-based criteria

For the metadata that are expressed in the UTC format such as

FilterDescription
CreationDate For future implementation
IngestionDate Time of the products archiving in the Data Hub
EvictionDate For future implementation
ContentDate/Start and ContentDate/End TBW

 

the filter can be performed on the day and/or month and/or year.

The symbols are not allowed in the OData protocol but they are substituted with the following syntax:

 

SymbolRegular ExpressionMeaning
< lt Lower than
le Lower or equal than
> gt Greater than
ge Greater or equal than
= eq Equal

 

Here below we provide some examples.

Filtering the products by Ingestion Date

To filter the products by Ingestion Date, the following syntax may apply:

<ServiceRootUri> /Products?$filter=year(IngestionDate) years[and month(IngestionDate) ] [and day(IngestionDate)

Examples:

  • https://data.sentinel.zamg.ac.at/odata/v1/Products?$filter=year(IngestionDate) eq 2014 and month(IngestionDate) eq 12
    This URI selects the products that have been published in the Data Hub on December 2014

 

  • https://data.sentinel.zamg.ac.at/odata/v1/Products?$filter=IngestionDate gt datetime'2014-12-06T00:00:00'
    This URI selects the products that have been published in the Data Hub after 06 Dec 2014

 

Filtering the Products using the file name

The products file name can be used for filtering the products. It shall be noticed that this query criteria is not based on the Medatata indexed from the products content but the criteria is search products matching a predefined string on the file name.

In this case, the Entity Data Model (EDM) string are used for metadata like Name which indicates the name of the product. Every metadata containing a string can be filtered using the substringof filter.

Examples:

  • https://data.sentinel.zamg.ac.at/odata/v1/Products?$select=Id&$filter=substringof('SLC',Name)
    This OData URI lists the products having SLC in the file name.

 

Filtering the Products by Sensing Time (file name)

To filter the products by the Sensing Time (start and stop) it can be used the S1 product filename, using the filter substringof. Next section will describe how to make query based on indexed metadata.

The syntax is:

<ServiceRootUri> /Products?$select=Id&filter=substringof('<string>', Name)

Example:

  • https://data.sentinel.zamg.ac.at/odata/v1/Products?$select=Id&$filter=substringof('_20141118T073253_',Name)
    This OData URI lists the products having in the file name the string 20141118T073253 . It means it returns the products with a predefined sensing time start (e.g. 2014-11-18, T073253).

 

Querying Product Attributes and Metadata Contents

OData can be used for querying the values of products metadata content.

Get product UUID from product name

To retrieve the UUID of a predefined product, the syntax is:

<ServiceRootUri> /Products?$filter=Name <Regular Expression> '<Filename>'

Example:

  • https://data.sentinel.zamg.ac.at/odata/v1/Products?$filter=Name eq 'S1A_IW_SLC__1SDV_20141101T165548_20141101T165616_003091_0038AA_558F'
    this URI retrieves the UUID of the product:
    S1A_IW_SLC__1SDV_20141101T165548_20141101T165616_003091_0038AA_558F

 

Download

OData URL can be used for download full products as well as partial products by using the combination of the Nodes and Attributes indexed for the products and adding $value at the end of the URI.

Download full product from its UUID

To download a full products the syntax is:

<ServiceRootUri> /Products('UUID')/$value

Example:

  • https://data.sentinel.zamg.ac.at/odata/v1/Products('18f7993d-eae1-4f7f-9d81-d7cf19c18378')/$value

 

Discover Product Nodes

Products from all processing levels (Level-0, Level-1 and Level-2) are disseminated in SENTINEL-SAFE format. The data delivered is packaged as a file structure containing a manifest file in XML format listing general product metadata and subfolders for measurement data, annotations, previews and support files. The SENTINEL-SAFE format wraps a folder containing image data in a binary data format and product metadata in XML. This flexibility allows the exploration of parts of the products without download the complete product.

In this section we will provide examples to how explore products nodes.

The following URI will return the XML file including the list of the first level nodes:

<ServiceRootUri> /Products('<UUID>')/Nodes

Generally the first node is the one whose name is [Product_Name.SAFE] and the URI to visualize the XML schema of this node is the following:

<ServiceRootUri> /Products('<UUID>')/Nodes('[PRODUCT_NAME.SAFE]')/Nodes

Example of the nodes of a level-1 Sentinel Product are:

  • Manifest.safe
  • Annotation
  • Measurement
  • Preview
  • Support

 

Download manifest file from the UUID

To download manifest file from the UUID the syntax is:

<ServiceRootUri> /Products('UUID')/Nodes('Filename')/Nodes('manifest.safe')/$value

Example:

  • https://data.sentinel.zamg.ac.at/odata/v1/Products('16902fd3-f323-4014-a950-853ac602e22f')/Nodes('S1A_IW_SLC__1SDV_20141101T165548_20141101T165616_003091_0038AA_558F.SAFE')/Nodes('manifest.safe')/$value

 

Download of Sentinel-1 quick-look file knowing the product UUID

To download the quick-look file from the UUID the syntax is:

<ServiceRootUri> /Products('UUID')/Nodes('Filename')/Nodes('preview')/Nodes('quick-look.png')/$value

Example:

  • https://data.sentinel.zamg.ac.at/odata/v1/Products('16902fd3-f323-4014-a950-853ac602e22f')/Nodes('S1A_IW_SLC__1SDV_20141101T165548_20141101T165616_003091_0038AA_558F.SAFE')/Nodes('preview')/Nodes('quick-look.png')/$value

 

Verifying Download integrity using the MD5 checksum

This can help to reveal if the download was incomplete.

Each product published on the Data Hub provides an MD5 checksum of the downloadable ZIP file. The Message Digit of the file can be discovered using the following OData query:

https://data.sentinel.zamg.ac.at/odata/v1/Products('UUID')/Checksum/Value/$value

where UUID is the Universally Unique Identifier of the product.

Example:

  • https://data.sentinel.zamg.ac.at/odata/v1/Products('a8dd0cfd-613e-45ce-868c-d79177b916ed')/Checksum/Value/$value

provides the following MD5:

  • 99F8BCE665CEF2587258F581873DFA53

The integrity of the downloaded product can be checked by comparing the MD5 value of the product at the source with the MD5 of the product located on the users’ s system. If they are identical the download has been succesfull.

Different Operating Systems require different methods for verifying the checksum of the downloaded products. Find below a list of the most common ones:

Linux

Open a terminal and type:

  • md5sum path/filename

OS X

Open a terminal and type:

  • md5 path/filename

Windows

The File Checksum Integrity Verifier (FCIV) utility does not come pre-installed on windows and needs to be downloaded.

Open a command prompt and type:

  • FCIV -md5 -sha1 path\filename

OpenSSL

This method can be done on any system with OpenSSL installed (i.e. Windows, Linux and MAC). Open a command prompt or terminal and type:

  • openssl md5 path/filename