1 INTRODUCTION¶
1.1 Purpose and Scope¶
The scope of this document is to be used as an annex for the CSC Common Entity Definition Document [RD-3] in order to describe the Product Entity Properties, the Product Attributes and the corresponding metadata elements that shall be catalogued and queryable for Sentinel products. This document provides the mapping and applicability for Sentinel-5P products.
1.2 Applicable Documents¶
| ID | Document Reference | Document Title |
|---|---|---|
| [AD-1] | ESA-EOPG-EOPGC-TN-58 | ESA EO Framework (EOF) – CSC – Data Flow Configuration |
Table 1-1 – Applicable Documents
1.3 Reference Documents¶
| ID | Document Reference | Document Title |
|---|---|---|
| [RD-1] | S5P-PDGS-DLR-ISP-3011 | Sentinel-5 Precursor PDGS L0 Product Format Specification |
| [RD-2] | S5P-KNMI-L01B-0014-SD | Metadata specification for the TROPOMI L1b products |
| [RD-3] | ESA-EOPG-EOPGC-IF-5 | CSC Common Entity Definition Document |
| [RD-4] | OGC 17-003r0 | OGC EO Dataset Metadata GeoJSON(-LD) Encoding Standard |
| [RD-5] | OGC 10-157r4 | OGC Earth Observation Metadata profile of Observations & Measurements |
| [RD-6] | ESA-EOPG-EOPGC-TN-13 | ESA EO Framework (EOF) – CSC – Glossary |
Table 1-2 – Reference Documents
1.4 Acronyms, Definitions and Abbreviations¶
| Acronym | Description |
|---|---|
| CSC | Copernicus Space Component |
| DA | Data Access |
| EOF | ESA Earth Observation Framework |
| ICD | Interface Control Document |
| JSON | JavaScript Object Notation |
| LTA | Long Term Archiving |
| NetCDF | Network Common Data Format |
| ODP | On-Demand Processing |
| PDGS | Payload Data Ground Segment |
| PR | Systematic Production |
Table 1-3 – Acronyms and Abbreviations
2 SENTINEL-5P PRODUCTS AND AUXILIARY DATA¶
2.1 Sentinel-5P Product List¶
The following table presents the list of Sentinel-5P products which are required to be circulated through the ESA EO Framework (EOF) Copernicus Space Component (CSC).
| Payload | Level | Product Type Code |
|---|---|---|
| TROPOMI | L0 | L0__ENG_A_ |
| L0__ODB_1_ | ||
| L0__ODB_2_ | ||
| L0__ODB_3_ | ||
| L0__ODB_4_ | ||
| L0__ODB_5_ | ||
| L0__ODB_6_ | ||
| L0__ODB_7_ | ||
| L0__ODB_8_ | ||
| L0__PDQ___ | ||
| L0__SAT_A_ | ||
| L1 | L1B_CA_SIR | |
| L1B_CA_UVN | ||
| L1B_ENG_DB | ||
| L1B_IR_SIR | ||
| L1B_IR_UVN | ||
| L1B_RA_BD1 | ||
| L1B_RA_BD2 | ||
| L1B_RA_BD3 | ||
| L1B_RA_BD4 | ||
| L1B_RA_BD5 | ||
| L1B_RA_BD6 | ||
| L1B_RA_BD7 | ||
| L1B_RA_BD8 | ||
| L2 | L2__AER_AI | |
| L2__AER_LH | ||
| L2__CH4___ | ||
| L2__CLOUD_ | ||
| L2__CO____ | ||
| L2__HCHO__ | ||
| L2__NO2___ | ||
| L2__NP_BD3 | ||
| L2__NP_BD6 | ||
| L2__NP_BD7 | ||
| L2__O3____ | ||
| L2__O3__PR | ||
| L2__O3_TCL | ||
| L2__SO2___ | ||
| Calibration | ICM_CA_SIR | |
| ICM_CA_UVN | ||
| ICM_CKDSIR | ||
| ICM_CKDUVN | ||
| VDAF | L2VOAER_AI | |
| L2VOAER_LH | ||
| L2VO_CH4___ | ||
| L2VO_CLOUD_ | ||
| L2VO_CO____ | ||
| L2VOHCHO__ | ||
| L2VONO2___ | ||
| L2VOO3_TCL | ||
| L2VOO3__PR | ||
| L2VOO3____ | ||
| L2VOSO2___ | ||
| PyCAMA | MPC_AER_AI | |
| MPC_AER_LH | ||
| MPC_CH4___ | ||
| MPC_CLOUD_ | ||
| MPC_CO____ | ||
| MPC_FRESCO | ||
| MPC_HCHO__ | ||
| MPC_NO2___ | ||
| MPC_O3_TCL | ||
| MPC_O3____ | ||
| MPC_O3__PR | ||
| MPC_SO2___ | ||
| MPC_NP_BD3 | ||
| MPC_NP_BD6 | ||
| MPC_NP_BD7 |
Table 2-1 – Sentinel-5P Products list
2.2 Sentinel-5p Products Packaging¶
All Sentinel-5P products follow the corresponding Product Format Specifications. All L1B products are distributed in netcdf compressed format.
2.3 Sentinel-5P Auxiliary Data List¶
The following table presents the list of Sentinel-5P Auxiliary (AUX) Data which are required to be circulated between the EOF CSC services.
| File Type | Data Provider | Description | Format |
|---|---|---|---|
| L2_FRESCO1 | PDGS | L2 TROPOMI | netCDF-4 |
| L2__AUXDEM1 | PDGS | L2 TROPOMI | netCDF-4 |
| AUX_BGHCHO | PDGS (UPAS-BC) | Background correction for HCHO product | netCDF-4 |
| AUX_BGSO2_ | PDGS (UPAS-BC) | Background correction for SO2 product | netCDF-4 |
| AUX_BGO3__ | PDGS (UPAS-BC) | Background correction for O3 | netCDF-4 |
| AUX_BGCLD_ | PDGS (UPAS-BC) | Background correction for CLOUD | netCDF-4 |
| AUX_CTMCH4 | KNMI | CTM model field of CH4 for use with CO and offline CH4 retrieval (CO for NRT, and CO and CH4 for OFFL) | Full tar archive, covering 6 – 9 months in daily files |
| AUX_CTM_CO | KNMI | CTM model field of CO for use with CO and offline CH4 retrieval (CO for NRT and CO and CH4 for OFFL) | Single netCDF4 file, containing 12 monthly fields |
| AUX_CTMFCT | KNMI | CTM model field of NO2, SO2 and HCHO for use with NO2 retrieval | Tar archive with 5 days (4-day redundancy). Five files in Net- CDF-4 |
| AUX_CTMANA | KNMI | CTM model field of NO2, SO2 and HCHO for use with NO2 retrieval | Single netCDF4 file, covering 1 day |
| AUX_MET_2D | ECMWF | Meteorological surface fields (wind, surface pressure, . . . ) | GRIB |
| AUX_MET_QP | ECMWF | Meteorological specific humidity profile | GRIB |
| AUX_MET_TP | ECMWF | Meteorological temperature profiles | GRIB |
| AUX_IERS_B | IERS | This difference between UT1 and UTC is published by IERS in Bulletin B | ASCII (plain text) |
| AUX_IERS_C | IERS | Leap second information | XML |
| AUX_ISRF | KNMI | Instrument spectral response function file. | netCDF-4 (*.nc) |
| AUX_L1_CKD | KNMI | The in-flight calibration key data product | HDF5 (*.h5) |
| AUX_O3_M | KNMI | O3 profile shape climatology | netCDF-4 (*.nc) |
| AUX_O3PPWL | KNMI | LUT for Time-dependent Soft-calibration in Ozone Profile Processor | netCDF-4 (*.nc) |
| CFG_L2_CMR | KNMI | CFI L2 semi-static configuration files | .tar |
| MPL NPIF_ | FOS | The NPIF is the output of the FOS Mission Planning System in response to the Nominal Baseline Schedule file (NPPF) provided by KNMI | TGZ contains two files, *.HDR and *.DBL |
| MPL SAF | FOS | The Station Availability file contains all PDGS scheduled Sentinel-5p ground station contacts | TGZ contains single *.EOF file |
| MPL SPF | FOS | The orbital events file is a XML file containing a list of S/C events derived from the S/C orbit histo-ry file | TGZ contains two files, *.HDR and *.DBL |
| MPL_ORBPRE | FOS | This file contains the Orbit State Vectors (OSV) predicted by the FOS | XML file, *.EOF |
| MPL_TLEPRE | FOS | This file contains the orbit predicted by the FOS, defined as a set of Two Line Elements | TGZ contains two files, *.HDR and *.DBL |
| VIIRS_CM | NOAA | VIIRS Cloud Mask EDR | .tar (containing .nc) |
| VIIRS_L1B_GEO | NOAA | PGE302a VIIRS L1 Moderate Resolution Geolocation | compressed HDF5 |
| VIIRS_L1B_RR | NOAA | VIIRS Moderate Resolution Radiance/Reflectance Data Band 7,9,11 | compressed HDF5 |
| VIIRS_CP | NOAA | VIIRS Cloud Phase EDR | .tar (containing .nc) |
| VIIRS_DCOMP | NOAA | VIIRS Daytime Cloud Optical and Microphysical Properties | .tar (containing .nc) |
| VIIRS_CTH | NOAA | VIIRS Cloud Top Height EDR | .tar (containing .nc) |
Table 2-2 – Sentinel-5P Auxiliary Data list
¹ L2_FRESCO and L2_AUXDEM are produced and distributed as Level-2 product by the Sentinel-5P PDGS. However, they are handled as Auxiliary data by the DA service, and therefore, they are traced in this list of S5P Auxiliary Data
3 SENTINEL-5P PRODUCT ENTITY PROPERTIES MAPPING¶
[RD-3] defines the Generic CSC Common Entity Properties. Table 3-1 below, provides the mapping between these Generic Product Entity Properties and the Sentinel-5P product metadata. Most of the properties defined in [RD-3] are generated by the CSC services. The following table describes the properties that map directly to the Sentinel-5P product metadata.
| OData Product Entity Properties | Type | Description | Cardinality | Corresponding Sentinel-5P Metadata | Example | Applicability | |||||
| L0 | L1 & L2 | AUX | PR | MPC | LTA | DA | |||||
| Name | String | Data file name (according to the Sentinel file naming conventions) plus the file extension | 1 | n/a | n/a | n/a | S5P_NRTI_L2__HCHO___20 190802T112419_20190802 T112919_09336_01_01010 7_20190802T120808.nc | X | X | X | X |
| ContentDate | TimeRange | The sensing range period. Compound property with start and end times in UTC in the format YYYY-MM- DDThh:mm:ss.sssZ | 1 | startTime stopTime | eop:EarthObservation/ om:phenomenonTime/ gml:TimePeriod/ gml:beginPosition eop:EarthObservation/ om:phenomenonTime/ gml:TimePeriod/ gml:endPosition | validity_start validity_stop | "ContentDate": { "Start":"2019-08- 02T11:24:19.022Z", "End":"2019-08- 02T11:29:19.872Z" } | X | X | X | X |
| Footprint | Geography | Footprint of the product | 0..1 | n/a | eop:EarthObservation/ om:featureOfInterest/ eop:Footprint/ eop:multiExtentOf/ gml:MultiSurface/ gml:surfaceMember/ gml:Polygon/ gml:exterior/ gml:LinearRing/ gml:posList | n/a | X | X | X | ||
| GeoFootprint | Geography | Footprint of the product following the GeoJSON format | 0..1 | n/a | eop:EarthObservation/ om:featureOfInterest/ eop:Footprint/ eop:multiExtentOf/ gml:MultiSurface/ gml:surfaceMember/ gml:Polygon/ gml:exterior/ gml:LinearRing/ gml:posList | n/a | X | X | X | X | |
Table 3-1 – Product Entity Properties mapping to Sentinel-5P metadata
4 SENTINEL-5P PRODUCT ATTRIBUTES MAPPING¶
All relevant metadata elements of the Sentinel-5P products shall be indexed in the Product Attributes, additional metadata elements may also be identified if appropriate. In order to provide a harmonised model across the Sentinel missions the JSON property naming from [RD-4] is preferred for the Attributes Ids. In case an attribute is not defined in [RD-4] the naming used in the mission specific metadata file is preferred. The Product Attributes mapping for all Sentinel-5P products is described in the Excel file “Sentinel-5P Product Attributes Mapping”, which is packaged with this document. A summary of the Sentinel-5P Product Attributes mapping across all products is presented in Table 4-1. It should be highlighted that, in order to harmonise the metadata model across the Sentinels, in some cases an “alias” for the value defined in the metadata is mapped to a standardised value to be used by the EOF CSC services, in other cases a standard value is defined where it is otherwise unavailable from the product.
The following table is filled with an x in case the attribute is used, additional symbols are used and the specific meaning is specified as:
“*” Attribute not available within the legacy PDGS
“#” Attribute is only available for auxiliary data generated by Processor CFI
| OData Attribute Id | Sentinel-5P Metadata Mapping | Applicability | ||||||||
| Level-0 | Level-1 | Level-2 | VDAF | PyCAMA | AUX | PR | MPC | LTA | DA | |
| platformShortName | x | x | x | x | x | x | x | x | x | x |
| instrumentShortName | x | x | x | x | x | x | x | |||
| processingLevel | x | x | x | x | x | x | x | |||
| beginningDateTime | x | x | x | x | x | x | x | x | x | x |
| endingDateTime | x | x | x | x | x | x | x | x | x | x |
| processingCenter | x | x | x | x | * | * | x | x | ||
| processorName | x | x | x | x | x | x | x | x | x | |
| processorVersion | x | x | x | x | x | # | x | x | x | x |
| processingDate | x | x | x | x | x | x | x | x | x | |
| processingLevel | x | x | x | x | x | x | x | |||
| processingMode | x | x | x | x | x | x | ||||
| orbitNumber | x | x | x | x | x | x | x | |||
| productClass | x | x | x | x | x | x | x | x | x | x |
| baselineCollection | x | x | x | x | x | x | x | x | ||
| productType | x | x | x | x | x | x | x | x | x | x |
| doi | x | * | * | x | x | |||||
| identifier | x | x | x | x | x | |||||
| parentIdentifier | x | x | x | x | x | x | ||||
| acquisitionType | x | x | x | x | x | |||||
| qualityStatus | x | * | * | x | x | |||||
Table 4-1 – Sentinel-5P Product Attributes mapping