1 INTRODUCTION¶
1.1 Objective of the Document¶
This document describes the organization of the Copernicus Sentinel data quality functions (calibration, validation, algorithms) funded by the EU Copernicus programme and under the responsibility of the Sensor Performance, Products and Algorithms (SPPA) section of the Earth Observation Mission Management and Product Quality Division (EOP-GM) in the EO Mission Management and Ground Segments Department (EOP-G).
The document is part of the Copernicus Technical Baseline package and complements the other documents providing the Copernicus operations baseline. It is applicable to all EO Copernicus Sentinel missions operated by ESA.
The document does not provide the specific mission data quality requirements, which can be found in other documents, in particular in the Sentinel product list document [RD-5].
The present document constitutes the 1st version of the Data Quality Functional Baseline document; it will be updated in 2021 in order to consolidate the various interfaces and to integrate it in a consolidated context. It is therefore expected than the 2nd version will be published by the mid-2021.
1.2 Scope of Document¶
The scope of this document is twofold:
It provides a high level description of the functions related to Sentinel data quality within the EU Copernicus Programme.
It gives an overview of their respective implementation within the EU Copernicus programme.
1.3 Documents¶
RD-1 A Quality Assurance Framework for Earth Observation: Operational Guidelines (http://qa4eo.org/docs/Guidelines_Framework_v3.0.pdf)
RD-2 Sensor Performance, Products and Algorithm (SPPA) Functional Baseline, PGSI-GSOP-EOPG-TN-05-0025, v2.1, 12/07/2013
RD-3 QMS Procedure for Sensor Performance Product and Algorithm, ESA-EOP-QMS-PR-3510, issue 2.4, Feb 2017
RD-4 CEOS Cal/Val Portal (http://calvalportal.ceos.org/)
RD-5 Sentinels User Level Data Portfolio, ESA-EOPG-EOPGC-TN-20, v1.0
RD-6 Copernicus Data Access Specifications of the space-based Earth Observation needs for the period 2014-2020, 5/3/2014, Issue 2, rev.0.
RD-7 Expression of User Needs for the Copernicus Programme, Brussels, 25.10.2019 SWD(2019) 394 final
RD-8 The next phase of Copernicus - Copernicus Space Component (CSC) Long Term Scenario" ESA/PBEO(2020)41, 09/09/2020
RD-9 CSC - ESA Copernicus Operations Framework: Specifications, ESA-EOPG-EOPGC-RS-1, issue 1.1
RD-10 CSC - ESA Copernicus Operations Framework: Architecture, ESA-EOPG-EOPGC-TN-7, issue 1.1
RD-11 CSC - ESA Ground Segment Operations Framework - Evolution Implementation Plan, ESA-EOPF-EOPGC-PL-1, issue 1.1
RD-12 Concept for the Cooperation of ESA and EUMETSAT regarding Copernicus Evolutions co-signed by J. Aschbacher (ESA) & A. Ratier (EUMETSAT), 2017
RD-13 Donlon, C. J, P. Minnett, N. Fox and W. Wimmer, (2015), Strategies for the Laboratory and Field Deployment of Ship-Borne Fiducial Reference Thermal Infrared Radiometers in Support of Satellite-Derived Sea Surface Temperature Climate Data Records, in Zibordi., G., C. Donlon and A. Parr (Eds.), (2015), Optical Radiometry for Oceans Climate Measurements, Vol. 47 Experimental Methods in Sciences, Elsivier, 697 pp., ISBN: 9780124170117
1.4 Acronyms¶
ARD Analysis Ready Data
ECE Expert Collaborative Environment
CCM Copernicus Contribution Mission
CSC Copernicus Space Component
FOS Flight Operating Segment
FMR Full Mission Reprocessing
FRM Fiducial Reference Measurement
GS Ground Segment
HR High Resolution
IPF Instrument Processing Facilities
LR Low Resolution
MFF Multi-Annual Financial Framework
MPC Mission Performance Cluster
MR Medium Resolution
PB Processing Baseline
RA Radar altimeter
SAR Synthetic Aperture Radar
SPPA Sensor Performance and Product Assessment
2 BACKGROUND¶
2.1 Data Quality Objective¶
A major asset of the Copernicus Space Component lies in the quality of the Sentinel data. The Copernicus programme must conduct appropriate activities to maintain its tradition of high quality data and services, partly enabled not only by the excellence of the Sentinel sensors but also by the Space Component on-ground efforts in terms of calibration, validation and algorithms performance. Without those activities, operational applications would hardly preserve their recognized accuracy, stability and therefore credibility (e.g.
climate change applications requiring a very high level of accuracy and traceability). Another benefit of those activities lies into the facilitated interoperability of Sentinel missions with other EO missions, to the benefit of the Copernicus programme. Therefore, data quality activities remain essential for the overall performance and credibility of the programme.
In terms of data quality, a key ESA objective is to meet and possibly exceed the initial mission requirements. In this framework, the SPPA section performs a wide spectrum of tasks with the ultimate objective of: Ensuring of the overall sensor dataset “best” performances. [RD-2]
2.2 Data Quality Functionalities¶
Data quality core functions ensured by the SPPA team includes:
Calibration and characterisation
Validation
Core product algorithm evolution
Processor and tools perfective maintenance
Expert product assessment and quality control
The sub-sequent sections are detailing these Copernicus Data Quality functionalities.
Secondary functionalities are also managed under SPPA team responsibility (e.g. communication related to data quality and calibration/validation through web pages or workshop communication, reporting…etc.). These secondary functions are not described hereafter.
2.3 Definition¶
These definitions are extracted from the Committee on Earth Observation Satellites (CEOS) Working Group on Calibration and Validation [RD-4] and ISO 9000 Definitions.
Calibration: The process of quantitatively defining the system responses to known, controlled signal inputs.
Validation: The process of assessing, by independent means, the quality of the data products derived from the system outputs.
Verification: The process leading to the confirmation that the specified requirements on a system have been satisfied.
Quality Control: The set of activities or techniques whose purpose is to ensure that all quality requirements are being met. In order to achieve this purpose, processes are monitored and performance problems are solved.
3 DATA QUALITY FUNCTIONAL BASELINE¶
3.1 Calibration and Characterisation¶
3.1.1 Function Scope¶
The goal of the calibration and characterisation function is to perform all necessary activities aiming at ensuring that the products resulting from the system composed by both the instrument and the processor can be traced to physical units, that is sensor dependent within the accuracy specified during the mission specification phase. One particular aspect of this function will be the characterisation of each step of the process, associating an uncertainty (error bar) that is combined (considering the internal correlation) in order to provide a final and global uncertainty associated to the measurement.
The CEOs agreed definition of Calibration is recalled: “Calibration is the process of quantitatively defining a system’s responses to known, controlled signal inputs.
“In the context of a constellation-based mission (e.g. Sentinel-A, -B, -C, -D), calibration is a fundamental task ensuring that different sensor units are comparable to each other.
Calibration and Characterisation function is an activity performed routinely during the entire mission operations lifetime through the implementation of the calibration plan defining the data need and the activities to be performed. The collection of calibration data over time allows identifying and characterising trends due to natural aging or anomalies related to hardware failure or calibration baseline update. The calibration and characterisation results are shared/discussed with other mission teams (e.g. Flight Operations Segment) during dedicated anomaly review board or satellite in-orbit progress meeting to identify specific activities or mitigation actions. Such information is also shared
with the Sentinels team project responsible for the development of the future units as valuable feedback for any possible improvement of the performance and reliability of the future Sentinel units.
This function fully relies on the availability of Fiducial Reference Measurements (FRMs) coming from ground instrumentation like buoys, transponders, tide-gauge, corner reflectors or spectrometers strictly mandatory to perform the calibration activities. FRMs are complemented by natural sites or cross-calibration of sensors belonging to the same family but from different missions (ex: Sentinel-2/Landsat-8, Sentinel-1/RCM, Sentinel-3 STM/Jason/CryoSat).
The calibration function might require defining and operating dedicated calibration processors that need a specific expertise (e.g. the Sentinel-1 PSCAL processor).
Accordingly, the sensor calibration and characterisation function covers the following main activities:
System calibration and characterisation
Generation of calibration auxiliary files or characterisation/calibration database
FRM analysis
This function is provided by a cluster of industrial/academic expertise, that has the necessary knowledge of the instrument characteristics and calibration model.
3.1.2 Sentinels Scope and Operations Highlights¶
The calibration/characterisation function relies intensively on the availability of an expert environment collocating data, computation power resource and a flexible data mining environment allowing to perform routine and/or ad-hoc time series analysis, enabling the necessary flexibility to cope with on-request analysis triggered by e.g. anomaly investigations. The implementation of such above working environment requirements is referred to as Expert Collaborative Environment (ECE).
The build-up of long-term instrument calibration records is performed in the Expert Collaborative Environment (ECE) where the necessary interfaces are created to feed the E2E
Operation monitoring function with the necessary information governed by [RD-9] and subsequent ICDs.
The calibration and characterisation results are available to the Copernicus Services and EU agencies (e.g. JRC) and at international level e.g. within CEOs or GSICS.
By its nature the calibration function is common to a family of missions. Significant optimization is achieved by grouping e.g. optical missions (e.g. S-2 and S-3). Horizontal scalability with future HPCMs is possible (e.g. S-1 & ROSE-L) ensuring the maximum reuse of operational procedures, experience and tools.
3.1.3 Interfaces and Context¶
The calibration and characterisation function interfaces in particular with:
The rest of the Ground Segment:
Data Access
The Reference System, using the environment made available for e.g. the reprocessed data acquired over sites with a specific interest to test the performance of a new calibration baseline
Auxiliary Data Gathering function
The Systematic Production services, providing revised calibration baseline
The E2E Operations Performance Monitoring element, providing performance information
Other Data Quality functions:
Validation
Core product algorithm evolution
Processor and tools perfective maintenance
Expert product assessment and quality control
The Quality Working Groups
The Sentinel Validation Teams
The Flight Operation Segment for the satellites orbital control
The Post Launch Support Office (PLSO) for the satellites maintenance
The Mission Planning for the definition of new observation requirements
The Mission Management
The Sentinel Development project (EOP-P)
The Copernicus services and the user communities
3.1.4 Function Performances¶
The performances of the calibration and characterisation function are directly related 1- to the fulfilment of the calibration requirements defined at mission specification level and updated as necessary during the mission lifetime, 2- to the timeliness for anomaly detection and the timely update of calibration baseline, 3- to the availability of dedicated FRM.
The performance is also related to the capacity to associate an uncertainty to the measurements; this uncertainty shall be traceable ideally to SI standard or community agreed best practices and fully documented.
In addition, specific meetings with Copernicus services are put in place to channel evolution of the calibration baseline and monitor user satisfaction.
3.2 Validation¶
3.2.1 Function Scope¶
The validation activities have two main goals: the first one is to provide products with documented and associated traceable error bars; the second goal is to gain more in-depth knowledge in the algorithm and sensor characteristics in order to improve their quality and reliability.
As for the calibration activities, the validation activities are a key component of a mission, as it is the foundation for user credibility into the mission data. Even if validation activities require a high level of scientific understanding of remote sensing techniques, validation activities are not performed for scientific purposes, but for operational purposes. Indeed, the level of acceptance of users, in particular operational users, will depend on the data accessibility (easy access, access price) and on the data quality (quality specifications, quality reliability). This is quite similar to any other goods or services.
It should therefore be clear that the validation activities (as the calibration activities) are not only performed during a commissioning phase, but continuously during the mission operations, and even after to ensure a correct bridging between similar missions. This is a long-term effort, precisely because uncertainties in the data quality will have a long-term impact for most EO applications and in particular for climate applications.
The validation activities have long been debated in the CEOS framework. Definitions are now well accepted:
Validation, on the other hand, is the process of assessing, by independent means, the quality of the data products derived from those system outputs.
A validation program is composed by a set of complementary activities bringing elements that need to be combined together in order to produce consolidated and confident validation results. In a generic manner the different components for a validation program are the following:
Validation against precise Fiducial Reference Measurements (FRM): few points but precise,
Validation against in-situ: more points but less precise,
Validation against others sources: inter-satellite comparison,
Validation against models: data assimilation rejection statistics, integrated model analyses
Validation using Level 3 data (i.e. merged data): statistical comparison between various Level 3 from various sensors constitutes an extremely useful tool (mean, median, standard deviation, bias, RMS… for selected zones, transects, latitudinal bands, seasonal trends…) for a cross-validation of the products,
Validation using monitoring tools: statistics, trend, systematic quality control, etc.
All the components are important and necessary; however, the FRM component is of particular importance because it gives a reference properly characterised and traceable to standard on which the validation results can be anchored.
This function relies on the availability of Fiducial Reference Measurements (FRMs) coming from ground instrumentation like buoys or radiometers, mandatory to perform the validation activities (see Section 4.2. for more details on FRMs purpose and on their provision).
The various validation approaches, methods follow best practises and reference methods that are defined in the CEOS WGCV framework.
They are reported in the CEOS Cal/Val portal [RD-4]
3.2.2 Sentinels Scope and Operations Highlight¶
The function related to Validation is specific for each mission. Actually, it is a function specific for each data product type or to variables within a data product. In this sense, synergy between missions could be found. In particular the approach for validating the optical sensors products will be similar. This is the case for Sentinel-2 and Sentinle-3 but also for the future potential LSTM (Land Surface Temperature Mission) and CHIME (hyperspectral) missions.
The validation function mainly covers the validation of the Level-2 data products but also the Level-1 data product. This is done usually by applying the vicarious calibration methods (for optical sensor) or by inter-comparison with well-established and characterised missions.
Validation activities have to be re-run following a Full Mission Reprocessing (FMR) campaign (, i.e. for any new product collection). It should be noted that validation is the most demanding function in terms of data reprocessing need, which depends on the mission/product type. In a generic manner we can classify the data reprocessing need for validation activities as follow:
HR data (SAR, OPT HR): they will need sample products (over specific area – Area Of Interest)
MR data (OPT): they will need sample products (over specific area – Area Of Interest AOI), but also large data set (global) that can be provided in the form of Level-3 (global average over time and space. The average period and size is to be defined).
LR: they need global data set (e.g. Altimetry)
As an example, one can mention Envisat MERIS (Sentinel-3 OLCI equivalent). The 3rd collection (3rd reprocessing) of MERIS was initially validated (processor acceptance) after having reprocessed globally 3 times a third of the mission lifetime.
Therefore, the validation function should have the capacity to pre/re-process the data according to the above specific needs. It can be data extraction (mini-file) over AOI, data averaging (L3) or others.
3.2.3 Interfaces and Context¶
Identical to section 3.1.3 for Calibration and Characterisation function.
3.2.4 Function Performance¶
As for the Calibration and Characterisation function, the performance is related to the capacity to associate an uncertainty to the measurements; this uncertainty shall be traceable ideally to SI standard or community agreed best practices and fully documented.
The validation activities shall be performed following the QA4EO principles [RD-1]. The comparison shall include the comparison with Fiducial Reference Measurements that provide the reference properly characterised and traceable to a standard on which the validation results can be anchored.
The validation results shall be presented and discussed in a validation forum involving science and expert users (e.g. Quality Working Group meetings and/or Validation Team Workshops).
In addition, specific meetings with Copernicus services and user community are put in place to channel evolution requirements on the data products accuracy baseline and monitor service satisfaction.
3.3 Core Product Algorithm Evolution¶
3.3.1 Function scope¶
Along the mission(s) lifetime, there is a need to maintain and evolve the processing algorithms in order:
To incorporate the results from the calibration and validation activities,
To follow the requirements evolution from the Copernicus Services and user community,
To stay on top of the scientific state of the art knowledge.
The Sentinel product evolution is managed through continuous dialogue with user communities in all phases of the evolution. The user communities encompass the existing and future public services, particularly the Copernicus services, but also national authorities, etc… Specific attention shall be devoted to the remote sensing scientists as a source of innovation, ideas and prototypes at the start of the product evolution loop.
The product evolution covers several phases including the definition, testing and development of products from experimental to operational qualification. For the purpose of simplification, ESA follows three phases in the product evolution, a prototyping phase, a pilot (or demonstration) phase and an operational phase:
In the prototyping phase, the prototype product simulates the final product or part of it (e.g. geographical or temporal subset). Prototypes are used to test the viability or usefulness of the product. In general, prototype activities are performed after a Proof of Concept was issued.
In the pilot phase (or demonstration phase), the product is generated with a preliminary version of the full processor and is tested against the final users (e.g. Copernicus services). The reason for doing a pilot is to get a better understanding on how the product will be used in order to refine the product and better qualify it.
In the operational phase, the core products, which are the official CSC operational products defined and approved by the European Commission, are available to all users according to the Copernicus data policy. Today all existing Sentinel core products are systematically generated, but in the future, the flexibility of the
production and data access services into cloud environment should allow having (new) core products available on-demand (“on-the-fly”).
Within the operational phase, evolution activities also take place for improving the current processing algorithm of an existing core product. Activities could also lead a fully new processing algorithm for an existing core product.
Finally, in the core product evolution chain, one shall also consider the potential synergy with non-Sentinel missions (i.e. Copernicus contributing missions) as demonstrated by the recent development of the concept of Analysis Ready Data (ARD) in which inter-calibration and sometimes inter-generation of products will play an increasing role.
3.3.2 Sentinels Scope and Operations Highlight¶
In the framework of Sentinels operation, this function is divided into two groups:
The existing core products
The new products
Regarding the existing Core Products, the function will encompass:
The evolution of algorithm for L1 and L2 core products providing an end-to-end view and ensuring consistency of the overall algorithm baseline through an up-to-date algorithm technical baseline,
Co-responsibility with the Ground Segment Team for the L1 product definition including metadata definition, format and performance specification,
Responsibility for the L2 product definition including metadata definition, format and performance specification,
Implementation of necessary prototype software capable of mirroring current operational product baselines,
Generation of ATBD and DPM in case of major evolution (changing the product collection), reference test data set and test results as a basis for implementation,
Definition of verification and acceptance criteria and test report,
Performing independent verification of the operation implementation.
Regarding the new products the function will encompass:
Tuning, testing of algorithm (baseline algorithm comes from ESA R&D programme),
Consolidating the ATBD, DPM and Product Specification
Developing the pilot processors (based on existing prototype) and fine-tuning the product definition,
Running a pilot demonstration,
Assessment of the quality (uncertainty) of the new products via a Cal/Val programme.
The new products development will follow the delineation process agreed with the European Commission.
As a general principle, the Algorithm Maintenance and Evolution should follow the principles of open science making openly available the data, methodology, results, algorithm and source code. The use of the Collaborative Environment (or the Product Algorithm Laboratory) shall facilitate this approach.
It shall also provide a high degree of scalability for new missions in order to procure algorithm updates related to changes implemented in recurrent units, to cope with new payloads if necessary ( e.g. AIS on-board S-1) and to the introduction of HPCM.
3.3.3 Interfaces and Context¶
Identical to Section 3.1.3
3.3.4 Function Performance¶
The performance is related to the capacity of performing an algorithm change responding to user need or mission evolution need in a timely manner.
The processing algorithm evolution results shall be presented, discussed in various forums involving science and expert users. The primary forum is the relevant Copernicus Sentinels Quality Working Group.
In addition, specific meetings with Copernicus services are put in place to channel evolution of the products accuracy baseline and monitor service satisfaction.
3.4 Tools and Processors Perfective Maintenance¶
3.4.1 Function Scope¶
The operational processing of the satellite data stream into Level-0, Level-1 and Level-2 products for the Sentinel missions is based on well-defined processing algorithms for which the operational implementation and maintenance depends strongly on the target operational environment and operational requirements (i.e. processing performance, scalability, optimization for a target computing environment, etc.).
The scope of this function is to maintain the processor implementation ensuring maximum suitability and operability with the processing environment and operational needs while guaranteeing the product quality standards.
Accordingly, the core product algorithm and operational processor maintenance encompasses the following key functions:
Perfective Maintenance of the Level-0, Level-1 and Level-2 instrument data processors, ideally as open source, software, including all necessary information to operate the processor (e.g. including the associated interfaces, the orchestration definition, the user manual, etc.),
Perfective Maintenance of the Cal/Val tools processors, ideally as open source, software, including all necessary information to operate the processors (e.g. including the associated interfaces, the user manual, etc.),
Assessment and acceptance of the product quality, including full verification of any new versions (i.e. Processing Baseline) of the data processors.
3.4.2 Sentinels Scope and Operations Highlight¶
It should be noted that in the Transformed Ground Segment Architecture, the Level-0 and Level-1 operational processors will be maintained by an operational service outside [RD-10] the SPPA team perimeter. SPPA team will have a role in validating the evolution and implementation of new Level-2 and Level-1 operational processor. Typically, this will be done by comparing the output products with reference products. The Level-2 operational
processor perfective maintenance and processor evolution will be a function of the Mission Performance Cluster Service.
As the Level-2 processor and processing depends on Level-1 format and contents, the interface between the Level-1 and Level-2 functions shall be clearly defined as any modification on Level-1 format and content has an impact on Level-2.
In addition, various key tools needed for Cal/Val shall also be developed and maintained on the long term. This is the case for example for vicarious calibration tools (DIMITRI) or Radiative Transfer Model (Eradiate). The list is not exhaustive; the tools will be given to the dedicated service (MPC service) as Custom Furnished Item.
3.4.3 Interfaces and Context¶
Being a specific Sentinel (or product) service, the associated operational interfaces and performance may be also specific, however a number of interfaces and performance measures will be common to all Mission Performance Cluster services.
The Core Processor perfective maintenance function interfaces in particular with:
The Reference System, using the environment made available for the e.g. reprocessed data acquired over sites with a specific interest to test the performance of a new calibration baseline
Other Data Quality functions
Auxiliary Data Gathering function
The Systematic Production services
The E2E Operations Performance Monitoring element, providing performance information
The interface between Level-1 and Level-2 shall be carefully and formally defined.
3.4.4 Function Performance¶
The function performance is measured by assessing the performance of the Operational processor versus the requirements. Particular attention is given to the processing time which is a key factor for the overall ground segment performance. Acceptance of processor will follow the standard acceptance procedures.
Performance is also assessed by measuring the time to perform a corrective or perfective maintenance and the quality of the delivery (typically, the number of delivery related to the correction/evolution of one processor).
In addition, specific meetings with Operation team are put in place to channel evolution of the processor baseline and monitor service satisfaction.
3.5 Expert Product Assessment and Quality Control¶
3.5.1 Function scope¶
Data product quality assessment can be performed through automatic routines and/or thanks to the support of specialised experts.
The automatic routines can be systematically applied on the data flow as it comes (Near Real Time or offline production). It is data driven. This Online Quality Control function will be performed by the Copernicus ‘Production Service’ as soon as the data is produced. The online automatic quality control functions are performed at the production facilities and shall be agreed during ground segment definition.
During nominal operations, these automatic routines implemented in the production line are performed systematically immediately after the products are generated. The checks can be limited to logs inspection but can also contain advanced data quality check of the data products generated with algorithms and tools installed in the processing chain (nb: in particular the development of automatic and systematic quality check based on Artificial Intelligence (AI) approach should have a great potential for performing this function).
Data Quality Assessment Reports are produced. In case of quality issues, non-conforming products are further investigated by the Mission Performance Cluster service dedicated to Product Assessment and Quality Control. These data products can be discarded from distribution or sent to users with a disclaimer; corrections and corrective actions are identified and applied as necessary.
As part of the Product Assessment and Quality Control function, quality checks can run offline the production chain for the systematic inspection of products. Data Quality Assessment Reports are produced. Further Expert analysis can be required for the inspection of these
reports. Any observation found during the report inspection shall be further investigated and reported to the ESA Data Quality Manager via analysis reports. If necessary one may request further processing using the reference facilities.
When an observation has been reported by an user through the Help Desk, the Mission Performance Cluster Service shall analyse the data products that have failed the quality control or have been defined as non-satisfactory by a user and define a plan of actions to provide a solution or an explanation as applicable. They may request the re-generation of data products to support these investigations. In this case, the adequate Facilities shall reprocess the products following specifications coming from the analysis.
As part of this function, a major component will be the systematic and long-term monitoring of key parameters (impacting the data quality). Time series, evolution and analysis of these parameters in the long term is mandatory to assess the overall quality of the mission and associated products.
3.5.2 Sentinels Scope and Operations Highlight¶
Data quality assessment is performed through automatic routines and/or thanks to the support of experts.
The automatic and systematic quality checks implemented in the production line are performed systematically on-line immediately after the products are generated. This function currently within the scope of the first generation of the Mission Performance Centre is moved out of the perimeter of the second generation of Mission Performance Cluster.
Instead, the Mission Performance Cluster Service remains responsible for the off-line analysis of the products. It includes:
Products anomaly investigation
Support to User Help Desk
Long Term Performance Monitoring
Systematic off-line product quality analysis
Reporting
3.5.3 Interfaces and Context¶
Similar to Section 3.1.3.
3.5.4 Function Performance¶
The performance associated to this function is measured by its capacity:
In detecting problems and anomalies;
In responding in timely manner to user requests and questions;
In performing product anomaly investigations;
In detecting suspicious trends associated to instrument anomalies, evolutions and product algorithms;
In proposing anomalies fixes as necessary;
In performing adequate and timely reporting.
In addition, specific meetings with Ground Segment Operations team are put in place to channel evolution of the processor baseline and monitor service satisfaction.
3.6 Function Inter-dependency¶
It should be noted that the above 5 data quality functions are interdependent. All together they constitute a virtuous circle allowing to provide the best quality data, with the associated knowledge and characterisation.
Figure 1: SPPA Functions inter-dependency
4 IMPLEMENTATION OF DATA QUALITY FUNCTIONS¶
The 5 data quality functions described above are implemented through various services or entities. As a general principle, these services are established through contracts; however, it should be noted in some cases, the provision of in-situ or FRM or model can be provided in-kind via agreement (bilateral with a Member States, international) or can be freely accessible.
The Data Quality functions are implemented through the following approach:
Mainly the Mission Performance Clusters (MPCs), through industrial contracts,
Provision of FRMs, trough industrial contracts or through agreements,
Specific Expert Activities, trough industrial contracts or through agreements,
Mission Quality Working Groups established by ESA,
Mission Validation Teams put in place following specific Announcements of Opportunity,
Relation with External Activities (CEOS, GSICS, …),
MPC Integrator Service, through an industrial contract.
4.1 The Mission Performance Clusters¶
The second generation of Sentinel MPCs (i.e. from 2022 onward) are structured in four separate service contracts distributed in four families of sensor types, as illustrated in Figure 1. They are selected by open competition (ITTs) and should be in place during 2021 to be fully operational from January 2022.
The sensor families are in line with the CSC Long Term Scenario [RD-02]:
Figure 2: Overview of the transition from first to second MPC generations.
The distribution in sensor families allows to reinforce the domain expertise and to better exploit synergies across similar missions/sensors, therefore decreasing the MPC service
costs on the long-term. This applies to the current fleet of Sentinel missions with a perspective towards future potential Sentinel missions (HPCM).
Figure 3: Mission Performance Cluster Organisation and Potential Evolution
The Sentinel radar imaging (SAR) MPC initially includes Sentinel-1 C-SAR.
The Sentinel optical imaging (OPT) MPC initially includes Sentinel-2 MSI (Multispectral Instrument), Sentinel-3 OLCI (Ocean and Land Colour Instrument) and SLSTR (Sea and Land Surface Temperature Radiometer).
The Sentinel altimetry measurements (ALT) MPC initially includes Sentinel-3 SRAL (Synthetic Aperture Radar Altimeter) and MWR (Microwave Radiometer).
The Sentinel atmospheric sensors (ATM) MPC includes Sentinel-5P TROPOMI (Tropospheric Monitoring Instrument).
In addition an evolution including support to Copernicus Contributing Missions data quality aspects is envisaged on the mid-term (at the end of the current PRISM service).
The second generation of Sentinel MPCs addresses the 5 functions described in Section 3 of this document, i.e.:
Calibration and characterisation
Validation
Core product algorithm evolution
Processors and tools perfective maintenance
Expert product assessment and quality control
While the MPC services shall as far as possible encompass all operational activities supporting the data quality of Sentinel products, the following four elements stay outside the Sentinel MPC services perimeter:
For a better efficiency, the operational implementation and maintenance (both corrective and perfective) of the Level-0 and Level-1 processors, is performed by the “Instrument data processing algorithm and Operational Processor maintenance” function within the Ground Segment, and the online quality control runs on the “Systematic production and routine quality control” function (‘New Production Services’) within the Ground Segment (while the offline quality control remains within MPC service perimeter).
For guarantying the independence of measurements, the operational generation of Fiducial Reference Measurements (FRM) is performed outside the MPCs by other entities. The FRM data are then analysed by MPCs as part of the validation activities. Exceptions to this approach are the radar transponder operations, which are included in the SAR and ALT MPC.
For benefiting from national investment and expertise related to Sentinel-5P, the processing algorithm knowledge and implementation is provided by external public entities (KNMI, BIRA, DLR).
For exploiting unique facilities, e.g. expertise and Cal/Val historical databases, some operational support is provided by CNES for Sentinel-2 and Sentinel-3 (see chapter 4.3).
The Sentinel MPCs are the essential services for ensuring the best possible algorithm expertise on the Sentinel core products (distributed by ESA). With this respect, the Sentinel MPCs have a fundamental role within the process of Sentinel products evolution, both for existing core products (improvement/update) and for new core products. For new products, MPCs are not involved in the first step of the process i.e. prototype products, as this step is generally funded by R&D programmes (ESA FutureEO or ESA CSC-4, EU Horizon Europe).
Instead MPC services are involved in the final operational step (core products), respecting the product delineation rules of the Copernicus programme, but also in the pre-operational step where is verified the maturity of the product algorithm, tuning it to be fully adapted to the needs of the users (in particular the Copernicus services) and preparing the start of its operations.
The Sentinel MPCs shall offer a degree of flexibility to do the qualification and support of new core products, to support unforeseen/contingency activities or for expanding validation activities (e.g. using new FRM data) whenever necessary.
For sake of cost saving and efficiency, the second generation of Sentinel MPCs shall exploit as much as possible cloud-based solutions. This includes e.g. the usage of cloud-based virtual expert collaborative environments for activities such as Cal/Val and algorithms/processors qualification/testing. The MPCs follow the principle expressed in the Long Term Scenario [RD-02] to “Streamline interfaces and minimise inter-services dependencies”, and interfaces with other Services shall be clearly defined (e.g. with the Reference System that provides a replica of the operational environment for testing new processor versions).
The MPC is therefore a cluster of expertise, encompassing expertise in calibration, characterisation, validation algorithm and engineering. The experts rely on tools developed in house or provided as CFI. They keep a continuous dialog with the user community - CEOS Working Group Cal/Val, GSICS, Validation Team, Users Fora – allowing to stay on top of the state of the art in the remote sensing from satellites.
The main interfaces are shown below:
Figure 4: Main MPC external interface
4.2 Fiducial Reference Measurement Provision¶
The concept for the provision of Fiducial Reference Measurements (FRMs) was elaborated jointly by ESA and EUMETSAT, then shared and approved by the European Commission. The joint concept was prepared taking into account feedback from EEA, currently in charge of in-situ measurements within the Copernicus programme, as well as through interaction with the other Copernicus Entrusted Entities and the ESA and Eumetsat Member States.
The calibration and validation (Cal/Val) activities are a key component of an EO mission, as it is the foundation for credibility for the mission data. Cal/Val activities require continuous efforts during and after the mission life time. Calibration and Validation functions are composed of different complementary activities that need to be combined together to produce fully documented and consolidated performances (see chapter XXX). One of these activities consists in: Inter-comparison against tailored and precise Fiducial Reference Measurements (FRM). All the Cal/Val components are important and necessary; the first component (FRM) is of particular importance because it gives a reference properly characterised and traceable to standards and/or community best practises on which the Cal/Val results can be anchored.
The FRM can be defined as [RD-13]:
“The suite of independent tailored and fully characterised measurements that provide the maximum Return On Investment (ROI) for a satellite mission by delivering, to users, the required confidence in data products, in the form of independent validation results and satellite measurement uncertainty estimation, over the entire end-to-end duration of a satellite mission.”
The defining mandatory characteristics for FRM are:
FRM measurements should ideally have documented SI traceability (e.g. via round-robin characterisation and regular pre- and post-deployment calibration of instruments) using metrology standards and/or community recognised best practices;
FRM measurements are independent from the satellite geophysical retrieval process;
An uncertainty budget for all FRM instruments, and derived measurements, is available and maintained;
FRM measurement protocols, procedures and community-wide management practices (measurement, processing, archive, documents, etc.) are defined, published and adhered to by FRM instrument deployments;
FRM are accessible to other researchers allowing independent verification of processing systems;
FRM are required to determine the on-orbit uncertainty characteristics of satellite geophysical measurements via independent validation activities.
In summary, Fiducial Reference Measurements (FRM) are tailored and fully characterized measurements in support for satellite Cal/Val and may be considered a sub-set of ‘in-situ’ measurements. They are a prioritized (cost-benefit ROI) suite of measurements required to demonstrate that mission products meet the mission’s requirements.
The FRM are essential for the validation of satellite measured parameters, in particular for missions aiming to respond to long-term public needs (e.g. Sentinel missions, meteorological missions). The FRM shall meet specific mission requirements in terms of accuracy and traceability (i.e. fully characterized FRMs, operating throughout the lifetime of
the mission under different weather conditions and varying geophysical areas, directly mimicking the satellite sensor measurements when possible).
The FRM also provide a means facilitate the inter-operability between sensors, to help bridge potential data-gaps, and anchor Fundamental Data Records (FDRs).
Like any element of a space programme (satellite, ground segment, data products), FRMs follow a process ranging from concept and research, to development, operations, maintenance and evolution.
Figure 5: FRM development to operation processes
The SPPA team ensures the access to a network of FRM infrastructures to ensure that reference measurements are available to the MPC services and therefore for the long-term monitoring of the Sentinel mission performances. To do so, the ESA SPPA team:
identify and maintain a list of FRM infrastructures and networks that can be made available for Copernicus programme, either existing or to be developed outside the programme envelope, to be identified from the beginning of the satellite mission definition in the mission Cal/Val plan [see current list below, RD xx];
establish the appropriate partnerships to be in position to access existing FRM infrastructures (mainly thanks to European and international partnerships) and operate and maintain these FRM infrastructures for the purpose of the Copernicus Sentinel Mission performance assessment.
shall take all measures to access additional in-situ data at large for the purpose of calibration and validation of the missions. Access to in-situ data for Cal/Val purposes shall not be charged to the programme and be available full, free and open from contributing Member States and international partners of opportunity.
Within the Copernicus programme, the funding of the operations of FRM infrastructures shall follow the approach below:
The research, development and deployment of new FRM infrastructures shall not be part of the Copernicus programme and shall be funded through appropriate R&D funding mechanisms (e.g. ESA, EUMETSAT R&D budgets, Horizon Europe…);
In case some existing in-situ infrastructures are not enough characterized to be considered as FRM, the activities and related costs to upgrade to become FRM characterized infrastructures, shall remain also outside of the Copernicus funding;
In case the above would be reached but the network of FRM infrastructures is not completely operational (e.g. missing central historical database, issue of timeliness, insufficient quality procedure…), upgrades to make it operationally managed should remain also outside of the Copernicus funding;
For FRM networks owned by an EU Member State and compliant with the above constraints, ready for operational use, the SPPA team ensures the tasks (with related contracting) to make the European FRM data ready to be delivered by the European Member State to the Copernicus Cal/Val activities, with the appropriate operational requirements (e.g. format, timeliness, delivery mechanism) without including any tasks or costs that would be required by the network related to its regular monitoring (other than for Sentinel Cal/Val purpose), research purposes or for the duties of internal network operators;
Access to international FRM networks should be at no costs and the ESA SPPA team shall seek to establish the appropriate partnerships (including through exchanges of data between European capacities and international ones) to access these international networks including by benefiting from Contribution Agreements placed by the Copernicus programme with third-party countries.
A list – non exhaustive – of FRMs for Sentinel missions is given below:
SAR:
Active Transponder
Passive Corner Reflector
Optical:
Permanent Instrumentations for Vegetation – FRM4VEG
Permanent Instrumentations for Hyperspectral measurements – HYPERNET
Permanent Instrumentations (radiometer) for Land Surface Temperature
Altimetry:
Active Transponder
Instrumentation for altimetry in inland, land and sea ice – S3tART
Atmospheric sensor:
Pandora instrument
4.3 Specific Expert Activities¶
Specific expert activities might be needed in some specific cases. While it is quite difficult to anticipate, once can mention the following reasons why specific support might be needed:
when the expertise is not represented in the MPCs on specific issues,
for benefiting from national investment or historical implementation,
for exploiting unique facilities,
where for scientific, technical or economic reasons contracts for additional for supplementary supplies or services cannot be separated from a previous contract,
if the supplies or services required are the subject of intellectual property rights and can, as a consequence, only be procured from one particular source.
The following examples can be anticipated:
Due to worldwide recognised expertise and Cal/Val historical databases, some operational support are provided by CNES for Sentinel-2 and Sentinel-3 optical and altimetry missions. Such activity is funded by the Copernicus programme.
For benefiting from national investment and expertise related to Sentinel-5P, the processing algorithm knowledge and implementation are provided by external public
entities (KNMI, BIRA, DLR). This activity is provided in-kind by these entities at no cost for the Copernicus programme.
4.4 External Activities¶
4.4.1 International Groups¶
Constant dialog between International User Community and SPPA team and representatives is necessary. Indeed, in order to stay on top of the state of the art and to ensure coordination between various Agencies around the world, exchange of data, methods, best practise, knowledge and expertise is mandatory. This continuous dialog also allows to go toward harmonisation between various data set and cal/val approaches; it will facilitate interoperability, which remain a goal expressed in GEOSS.
Three major types of group are instrumental in this process:
CEOs
GSICS
International Conference/Workshop
Generic descriptions of these groups is given in the following chapters.
4.4.1.1 CEOs¶
The mission of the CEOs Working Group on Calibration & Validation (WGCV) (http://ceos.org/ourwork/workinggroups/wgcv/) is to ensure long-term confidence in the accuracy and quality of Earth Observation data and products and to provide a forum for the exchange of information about calibration and validation, including the coordination of cooperative activities.
CEOS Working Group Cal/Val addresses the need for standardizing the ways that different data sources are combined to ensure interoperability among existing and future Earth Observing systems. The CEOS WGCV promotes the international exchange of technical information and documentation, joint experiments, and the sharing of facilities, expertise, and resources. This makes the WGCV the focal point for all things related to calibration and validation in the international user community. The WGCV Subgroups also conduct many activities around specific topics and sensor types.
4.4.1.2 GSICS¶
Global Space-based Inter-Calibration System (GSICS) (https://gsics.wmo.int/en/welcome) is an international collaborative effort initiated in 2005 by the World Meteorological Organization (WMO) and the Coordination Group for Meteorological Satellites (CGMS) to monitor, improve and harmonize the quality of observations from operational weather and environmental satellites of the Global Observing System (GOS).
GSICS aims at ensuring consistent accuracy among space-based observations worldwide for climate monitoring, weather forecasting, and environmental applications.
This is achieved through a comprehensive calibration strategy which involves:
monitoring instrument performances,
operational inter-calibration of satellite instruments,
tying the measurements to absolute references and standards, and
recalibration of archived data.
GSICS delivers calibration corrections needed for accurately integrating data from multiple observing systems into products, applications and services. GSICS contributes to the integration of satellite data within the WMO Integrated Global Observing Systems (WIGOS) and within the Global Earth Observation System of Systems (GEOSS) of the Group on Earth Observations (GEO).
4.4.1.3 International Conferences and Workshops¶
International conferences, workshop and/or symposium are natural places to pursue dialog with user community; Calibration, Validation and Algorithm are topics that are consistently discussed and presented in major international forum. Constant feedback from users are necessary to improve the overall process.
Among others, one should mention the following International conferences and workshops. The list is given as illustration, it does not pretend to be exhaustive.
Living Planet Symposium: Organised by the European Space Agency, Living Planet Symposium is held every three years usually with the support of the local space agency. This symposium focuses on how Earth Observation contributes to science and society, and how disruptive technologies and actors are changing the traditional Earth
Observation landscape, which is also creating new opportunities for public and private sector interactions. This event is one of the most popular event related to remote sensing and earth observation.
The European Geosciences Union (EGU): EGU is the leading organisation for Earth, planetary and space science research in Europe. EGU foster fundamental geoscience research, alongside applied research that addresses key societal and environmental challenges. It is a non-profit international union of scientists with about 20,000 members from all over the world. Membership is open to individuals who are professionally engaged in or associated with geosciences and planetary and space sciences and related studies, including students and retired seniors. The EGU publishes a number of diverse scientific journals, which use an innovative open access format, and organises topical meetings, and education and outreach activities. The annual EGU General Assembly is the largest and most prominent European geosciences event, attracting over 16,000 scientists from all over the world in the year 2019. The meeting’s sessions cover a wide range of topics, including volcanology, planetary exploration, the Earth’s internal structure and atmosphere, climate, as well as energy and resources.
IGARSS: The Geoscience and Remote Sensing Society (GRSS) is a technical society of the Institute of Electrical and Electronics Engineers (IEEE). GRSS fosters engagement of its members for the benefit of society through science, engineering, applications, and education as related to the development of the field of geoscience and remote sensing. During IGARSS, GRSS members and non-members share their latest results and novel developments in the area of geoscience and remote sensing. IGARSS is a big conference, and all the technical communities that form the GRSS community are gathering in different sessions, meetings and technical activities. Diversity is an advantage, and cross-fertilization of different ideas and points of view has always brought to new ideas and new research projects.
One can also mention dedicated or thematic groups, conferences, symposium and workshops: IOCCG, GHRSST, OSTST, FRINGE, POLINSAR, LAND PRODUCT VALIDATION EVOLUTION, ATMOSPHERIC COMPOSITION VALIDATION EVOLUTION…
4.4.2 In-situ and Network of In-situ data¶
The Calibration and Validation activities, managed by the Mission Performance Clusters, also exploit Cal/Val in-situ data other than FRMs, freely available from various sources.
This is the case for example of AERONET data, which are very popular for validating Aerosol products but also surface reflectance products. AERONET data are freely and easily accessible and thanks to its geographical distribution and long-time availability, they provide valuable source of validation; they have been used in almost all validation programmes (NASA, NOAA, ESA, EUMETSAT…etc…).
One can also mention the availability of GRUAN (GCOS Reference Upper Air Network) radiosondes and Network for Detection of Atmospheric Composition Change (NDACC).
A number of in-situ network that can be used for validation are given under the CEOS WGCV Cal/val portal : http://calvalportal.ceos.org/calvalsites.
Mission Cal/Val plans shall list the sites/network that will be used for Cal/Val and if needed, specific interface or agreement will be defined.
4.5 Quality Working Groups and Validation Teams¶
4.5.1 Quality Working Groups¶
Quality Working Groups (QWG) are groups of experts who collectively provide independent advice and recommendations on quality aspects of the Sentinel data products, their algorithms and validation and on the instrument calibration and validation. The aim of a QWG is to assure a sustained high quality for the data products for a specific mission throughout the mission lifetime, meeting the mission requirements.
The Sentinel Quality Working Groups are composed of :
Senior scientists who have - as a collective - an end-to-end view of the measurements made and have the scientific knowledge to ensure a high end-to-end data quality also seen from a multi-program (Virtual Constellation) context;
Key users experts/scientists who can represent the stakeholder’s actual product use and advise accordingly. In particular, representation of the Copernicus Services in the QWGs is required;
Experts representatives of the MPC;
Experts representatives of the Validation Team;
Additional experts from the R&D community.
The QWGs often incorporate previous members of the Mission Advisory Groups (MAGS) as the MAGs are generally closed at end of mission development.
The Sentinel Quality Working Groups (QWGs) are chaired by ESA (co-chaired by ESA and EUMETSAT in case of Sentinel-3). The QWGs provide recommendations to the Mission Managers, on the below topics :
Support a high Sentinel product quality throughout the mission lifetime and beyond through providing advice on the data product quality, their retrieval algorithms and validation,
Provide recommendations for improvements to the baseline of the Sentinel core products, their algorithms and validation and if necessary suggest specific studies to support these,
Assess the instruments’ performance and advise ESA (ESA and EUMETSAT in case of Sentinel-3) on improvements to the instrument calibration and validation;
Assess the data product quality in the context of the mission requirements from other multi-mission, multi-program and (virtual) constellations.
Each QWG meets approximately 2 times per year. The discussions are primarily based on feedback from members of the QWG, expert (services and scientific) users, from the Sentinel MPCs and the Sentinel Validation Teams.
4.5.2 Validation Teams¶
Through the establishment of the Sentinel Validation Teams, ESA seeks the involvement of the international community with experience in conducting scientific verification and
validation of Sentinel type data, field experiments and campaigns. The aim of the Validation Team is:
“To engage world-class expertise and activities, through mutual benefit collaboration, that support the implementation of the Sentinel validation activities and ensure the best possible outcomes for the Sentinel Mission”
The validation teams are open to relevant and interested groups and individuals worldwide; group responses are particularly welcome. Each Sentinel Validation Team (SVT), established through an open call, provides structured coordination of international activities that contribute to Sentinel validation, in particular during the Commissioning Phase (Phase E1), but also beyond during the operations Phase (Phase E2). The SVT bring together world-leaders in relevant mission validation activities to provide independent validation evidence, experimental data and recommendations from such work that will be reported formally to ESA (and EUMETSAT in case of Sentinel-3) to characterise the quality and performance of the mission.
The results and output from the validation team is analysed, synthesized within the Mission Performance Clusters. Results are reported in the Quality Working Group meetings.
In the current implementation, Validation Teams were created for:
Sentinel-2 → S2VT (~50 projects)
Sentinel-3 → S3VT (~90 projects covering all validation expertise for Land Surface Temperature, Sea Surface Temperature, aerosol, fire radiative power, ocean colour, altimetry)
Sentinel-5P → S5PVT (~40 projects)
The validation teams meet typically once per year. Regular ad-hoc, or subgroup meeting/telecom, are organized to monitor the progress and exchange information between the various actors.
4.6 MPC Integrator Service¶
Under the direct ESA SPPA leadership, the MPC Integrator service coordinates the various MPC service activities, but also the preparation of the quality working groups and validation teams meetings as well as the various inputs relevant for the operations related to Cal/Val
and algorithm maintenance and evolution. A huge flow of information is generated from these activities which need to be processed, analysed and synthesized.
The Integrator Service contractors also contribute to the overall Cal/Val strategy evolution and maintain independence from the existing MPC and FRM contracts as they consist of impartial analysis, review and synthesis of outputs from the abovementioned contracts.
The MPC Integrator service is an industrial contract selected in open competition and set up as a service contract in line with operational and legal constraints prevailing today. It requires a very large domain of expertise ranging from SAR, Optical High Resolution, Optical Medium Resolution, Altimetry, Atmospheric domain.
5 OVERVIEW OF THE RELATIONS AND INTERFACES BETWEEN FUNCTIONS¶
The interfaces between various actors are summarised below. It is important to recall that the process is a dynamic process with a constant dialog between the various actors. The MPC plays a central role, as it is backbone of the SPPA functional baseline. It is fed by the various actors under ESA contracts or not.
Figure 6: Overview SPPA Functional Baseline Implementation and Interfaces