1. Introduction

1.1 Purpose

This System Design Document (SDD) provides a comprehensive view on the design and overall architecture of the DestinE Core Service Platform and in particular on the Platform Management Services (section 4).

1.2 Reference Documents

1.3 Acronyms and Abbreviations

1.4 Terms and definitions

Ecosystem:

Ecosystem defines a governed perimeter leveraging data and services plus a set of members (participants) that are interlinked for reciprocal interactions and for data and services exploitation.

Data Space:

Open ecosystem of distributed and federated actors sharing data, applications, services, and infrastructure.

In our framework, the Ecosystem concept is applied to DestinE Ecosystem. Please refer to Section 2.

System/Engineered System:

A system is an arrangement of parts or elements that together exhibit behaviour or meaning that the individual constituents do not. An engineered system is a system designed or adapted to interact with an operational environment to achieve one or more intended purposes while complying with applicable constraints.

The system’s properties (as a whole) result, or emerge from:

•the parts or elements and their individual properties; AND

•the relationships and interactions between and among the parts, the system and its environment.

In our framework, the engineered system is the DestinE Platform.

System Element:

A system element is a member of a set of elements that constitutes a system. It can be any or all of people, products, services, information, processes, and natural elements which constitute it (the term subsystem is sometimes used instead).

In this document, system elements are all the Services and Infrastructure elements constituting DestinE Platform Management Services. The architecture of the elements composing the Data Management Services is not discussed in this document.

System Component:

System component forms a system element. They can be software package, a web service, a web resource, etc. Each component is an essential part of the system element and is necessary for it to work properly.

Workflow:

A Workflow is any user coding architecture, including the data management with inputs, retrieval, transformation and writing outputs.

2. DestinE Platform design overview

DestinE Core Service Platform (DestinE Platform) is a user-friendly platform, funded by the European Commission in the frame of the DestinE initiative, for the delivery of DestinE services to users.

DestinE Platform is attracting user communities for data exploitation on local platform or on remote environment. It is based on an open, flexible, federated, scalable and evolvable secure cloud-based architecture. In phase 2 of DestinE, the platform connects to existing and future HPC resources as well as public cloud computing infrastructures.

2.1 DestinE Platform Design Drivers

The main drivers, i.e., the guidelines used to define the architecture are derived from the following documentation:

•[DP-SOW], which provides the full set of contractual requirements, and [AD-DSP-TSR] providing the technical requirements.

•Services Portfolio, described in “[DSP-USR-SDP] DestinE platform Services Portfolio” which document all services provided as part of the contract.

•Service Level Agreement which defines the expected levels for service delivery.

•the Architectural Principles of the European Initiative SIMPL.

Analysis of these set of documents and information is conducted to derive design drivers and constraints for subsequent development of the system.

Main drivers related to system design activities are detailed in Table 1:

Table 1: DestinE Platform Design Drivers

2.2 DestinE Platform context

In Figure 1, the System context diagram of the DestinE Platform is depicted.

Looking at the system as a black box (depicted in dark grey), its external interfaces are depicted as black dashed arrows, listed in Table 2 (Section 2.3). The light-grey background draws the boundaries of the DestinE Ecosystem. External elements to the platform are represented in orange. Relevant external interfaces are depicted as black dashed arrows.

DestinE Platform is the entry point to the DestinE ecosystem, i.e., it is the DestinE entry point for users.

Figure 1: DestinE Platform context diagram.

2.3 External elements

DestinE Platform external elements are defined as systems/services deployed outside the DestinE Platform cloud infrastructure and run by administrators according to an agreed service level.

Table 2: DestinE Platform External elements

3. DestinE Platform system design

Design is performed via the following steps:

ogeneration of System Functional model (Section 3.1) which defines how the System satisfy requirements and user scenarios;

oidentification of System Elements able to implement the identified functions.

oassociation of System Functions to Elements.

3.1 Functional model

The features of the DestinE Platform are realized through various functions, which are enumerated in the subsequent paragraphs and distinguished by a selected identifier.

Their implementation through the System Elements chosen to compose the DestinE Platform Management Services is fully described in Section 4, which reports the functional decomposition models.

The chosen approach is to define high-level functional areas, which group main functions.

3.1.1 F1 Identity and access management

3.1.1.1 F1.1 User Registration

The registration function covers the registration of any kind of user into DestinE Platform, thanks to self-registration via Information Dissemination & Onboarding Support Service - which redirects to the IAM Service dedicated panel.

The registration function also includes the possibility to find on the Web Portal the DestinE Platform “Code of Conduct”, “Terms and Conditions”, “Privacy Policy”, “Cookie Policy” and “Legal Notice”, read and access them.

3.1.1.2 F1.2 Authentication & authorization

This function implements authentication (it confirms the validity of user identities) and authorization (it gives users permission to access DestinE Platform functions or part of them upon approval), ensuring that no unauthorized access is allowed.

The access to services is managed through the definition of user groups and service groups which are link by an authorization schema.

Upon registration, users are authorized by default to access a specific group of DestinE Platform services. Additionally, users can request authorization to access other groups of services, such as those providing access to DestinE data and derived information. The authorization process is asynchronous and involves the EC as the approving entity.

3.1.1.3 F1.3 Login

The login function allows users to access their accounts on the DestinE Platform by providing their credentials.

Once a user is logged-in, he/she may access:

•User Profile area storing personal account information

•All services which are not providing users the possibility to access DestinE data

•Onboarding request form

•Upgraded access request form

•“Submit an idea” form, to suggest new features

Moreover, depending on the level of authorization, users can access:

•Services allowing access to the original DestinE data and

•Service allowing access to information/data derived from DestinE data.

3.1.1.4 F1.4 User Profile management

This function implements the user accounts management by the DestinE Platform Registered users and Administrators.

In particular:

1)Users will be able to manage their user attributes.

2)Users will be able to change their password and email.

3)Users will be able to manage/monitor Tenancy feature.

4)Administrator will have the grants to edit user information including its token and roles.

5)EC will have the authority to grant DestinE Platform users access to services that are not available by default (i.e., services immediately accessible after registration).

3.1.1.5 F1.5 Federation

This function allows to federate the access to Third party services. This translates into the possibility for Service Providers to allow the publication of their services to the DestinE Platform user community i.e. third-party services could be accessed by DestinE Platform users using their DestinE Platform accounts.

3.1.1.6 F1.6 Management Web Interface

This function allows DestinE Platform Administrators to:

•Register services (i.e. client, applications) that can be used for authentication by users

•Register service resources

•Manage the user authorization over its resources bases on policies over custom attributes and permissions schemes

3.1.1.7 F1.7 Tenancy

This functionality enables group of Users to access dedicated Resources provided by DestinE Platform Services.

It relies on:

-Resource Access Control Layer provided by the IAM to ensure dedicated and secure access

-Clearing House and Accounting component to track DestinE Platform Resource Consumptions

-Tenancy Management component provided by the IAM to support Tenancy Feature scenarios:

oTenancy Members and Project Participants management

oTenancy Projects Management

3.1.2 F2 Accounting/Clearing House

Figure 2 Accounting and Clearing House diagram

3.1.2.1 F2.6 Transaction and Consumption Logging

From Accounting/Clearing House perspective, a Service is composed by one or more Transactions which are the description of every Service functionality that may trigger User Consumptions. An Accounting Transaction describes the characteristics of a Service functionality wrapped as a Tenancy Resource and enables the Service App to log related Consumptions.

3.1.2.2 F2.7 Query User Consumptions

This function allows Users to check their Resource Consumptions through all the Resources provided by every Service of the DestinE Platform.

3.1.2.3 F2.8 Query Service Consumptions

This function allows Service Providers to check the Resource Consumptions happened through the provided Resources.

3.1.2.4 F2.9 Query Tenancy Consumptions

This function allows Tenancy/Project Admins to check the Resource Consumptions happened through the managed Projects for the related Allocated Resources.

3.1.3 F3 Service Registration and Discovery

Figure 3: Service registration and discovery Functional Area

3.1.3.1 F3.2 Service Registration

This function covers the registration of a new service into the Service Registry – after its successful revision and approval by the Integration Manager. A registered service can also be unregistered.

3.1.3.2 F3.3 Services repository management

This function implements the storing and maintaining of the list of the available DestinE Platform registered Services list and their relevant metadata

3.1.3.3 F3.4 Service Discovery

This function - allowed to both unregistered and registered users - covers the search and identification of available registered services to be consumed.

3.1.3.4 F3.6 Service Feedback (Phase 2)

This function covers the process of leaving feedback for a service. Only DestinE Platform users (registered or federated) and Service Providers can leave feedback for a service.

3.1.4 F4 User Community Management

Figure 4: User Community Management Functional Area

3.1.4.1 F4.1 Access to Web Portal

This function covers the access to the Information Dissemination and Onboarding Support Service (Web Portal), without registration and authentication.

3.1.4.2 F4.2 Website content search

This function allows users to search for any content published in the DestinE Platform Information Dissemination and Onboarding Support Service (Web Portal), thanks to dedicated filters and search engines.

Included in the documentation published and available via the Web Portal, registered users can find the DestinE Platform Information Model and its relevant Ontology.

3.1.4.3 F4.3 Website content management

This function comprehends the creation and management of the DestinE Platform Information Dissemination and Onboarding Support Service (Web Portal) digital content, covering:

•publication of information concerning DestinE Platform and its Services, including relevant news.

•users access to the content, including “DestinE platform Code of Conduct”, “DestinE platform Terms and Conditions” and “DestinE platform Privacy Policy”.

•knowledge base management enabling users to easily search information (e.g. User Guides).

3.1.4.4 F4.4 Community feedback management

This function covers the feedback and suggestions features managements on the platform (community ideas) andservice ratings, ensuring the collections of valuable data to improve services and support. It fosters users’ interaction and is supported by sharing of information via online messages (Forum – Phase 2).

3.1.4.5 F4.5 Issue tracking and management

This function covers:

•handling incoming requests. These requests may include technical support issues, questions about products or services, or requests for information or assistance.

•tracking and resolving issues, to track and manage incoming requests, assigning them to the appropriate staff member, and monitoring their progress until resolution.

•Resolution confirmation to the ticket originator automatically sent with the option to provide feedback.

3.1.4.6 F4.6 User feedback analysis and reporting

This function covers the collection, analysis and reporting of user’s feedback.

In their reply to a ticket resolution confirmation, users will have the choice to express their level of satisfaction of the service, with the choice of varying degree of satisfaction levels. Additionally, a dedicated feedback field will allow users to provide their level of satisfaction on a scale of 1 to 5 and leave suggestions for improvement or other comments relating to each of the services.

Collected feedback will be analysed and will feed the envisaged reports (quarterly/annual). Moreover, such data will be published on the Executive Dashboard in a page accessible to ESA.

3.1.5 F5 Monitoring and Reporting

Figure 5: Monitoring and reporting Functional Area

3.1.5.1 F5.1 Monitoring data gathering

This function implements the data gathering, performed by agents, of raw input data from Platform Management Services and Data Management Services for monitoring purposes.

3.1.5.2 F5.2 Monitoring data brokering and manipulation

This function covers the data brokering and collection of gathered information as a queue of message, as well as data filtering and pre-processing before storing them for reporting purposes in the Real Time Monitoring (Service Operational Monitoring Dashboard) and in the Long-Term Monitoring (Executive Dashboard).

3.1.5.3 F5.3 Monitoring data storage

The purpose of this function is to store the information in suitable datastores structures.

Data storages can be used for different purposes, such as:

•provide a persistent and virtually infinite store of records.

•provide access with minimal delay to the parameters of interest.

3.1.5.4 F5.4 Data Mining

This function implements the retrieval of data archived in the datastores and their aggregation and population into a dedicated DWH to reduce the execution time of a predefined list of data mining queries acting at producing a report. This ensures the provisioning of statistical data for reporting purposes.

3.1.5.5 F5.5 Reporting

This function implements the handling of statistical data, provided by the mining, to create automatic reporting and service reports.

3.1.5.6 5.6 Alerting

This function aims at feedback authorized users about significant events related to monitored systems.

3.1.5.7 5.7 Real-time monitoring

This function implements publication and visualization of metrics allowing up-to-date real-time monitoring of the status and performance for all Services, via the Service Operation Monitoring Dashboard Service. It allows to create, explore, share dashboards via GUI or API and export the published information to a format suited to the type of exported information (e.g., csv, excel, pdf) interactively.

3.1.5.8 F5.8 Statistics visualization

This function ensures unregistered and registered users to visualize statistics exposed by by the DestinE Platform public Dashboard

The Executive Dashboard which is accessible to ESA supports bar-charts, curves, histograms, pie-charts, Gantt Chart tables, textual tables, etc. It will be configurable by means of filtering the visualized information per mission/unit/service etc. It will allow to select any time range for the retrieved information.

3.1.5.9 F5.9 Statistics export

This function ensures DestinE Platform registered users to export the Executive Dashboard statistics, in different formats, for ad-hoc further analysis.

3.1.6 F10 Infrastructure Management

Figure 6: Infrastructure Management Functional Area

3.1.6.1 F10.1 Infrastructure provisioning

This function covers the infrastructure provisioning of:

•the Runtime Platform – integrated with the OVH Cloud Infrastructure layer.

•Cloud Services provided by the OVH Cloud Infrastructure layer.

3.1.6.2 F10.2 Container orchestration

This function allows to deploy, manage, and scale containers across a cluster of machines. It automates container lifecycle management, including deployment, scaling, and recovery.

3.1.6.3 F10.3 Storage Orchestration

This function provides mechanisms to manage storage resources and their allocation to containers.

3.1.6.4 F10.4 Deployment management

This function supports smooth updates and rollbacks of containerized applications without impacting the overall system.

3.1.6.5 F10.5 Load Balancing

This function handles load balancing capabilities to distribute traffic across the containers providing a particular service.

3.1.6.6 F10.6 Scaling and Auto-scaling

This function enables horizontal scaling by allowing adding or removing instances of containers based on workload demands. It can automatically scale the number of replicas based on metrics such as CPU utilization or custom-defined metrics.

3.1.6.7 F10.7 Fault Tolerance and Self-healing

This function enables the monitoring of the health of containers and nodes within the Runtime platform. It ensures the availability and resilience of applications by automatically restarting or rescheduling containers in case of failures.

3.1.6.8 F10.8 Resource Allocation and Management

This function allows specifying resource requirements and limits for containers, ensuring optimal allocation of resources within the cluster. It also provides monitoring and metrics for resource utilization.

3.1.6.9 F10.9 Security and Access Control

This function offers security features and network policies. It helps enforce security best practices and provides a secure environment for containerized applications.

3.1.7 F11 Service Handling

3.1.7.1 F11.1 Calculation and estimation process

This function oversees the estimation of the pricing needed to host services on DestinE Platform. Reserved to Service Providers, it allows them to select OVH Cloud or Runtime Platform resources they intend to use, choose hourly or monthly type, and compute an estimation of the price in a specific time range.

3.1.7.2 F11.2 Carbon Calculator Estimation

This function offers registered users the capability to calculate carbon footprint estimation based on user service configuration.

3.1.7.3 F11.3 Service boosting

This function provides to eligible DestinE Platform Services access to additional resources allowing Service Providers to boost their Service and enhance user experience.

4. Overall architecture

The final architecture we are going to show in Table 3 reflects the mapping among:

oFunctions (Section 3.1)

oElements involved in the function.

The functions described in Section 3.1 are implemented by DestinE Platform external and internal elements (this latter, the Core Services).

Table 3: Functions, Services & Elements mapping matrix

In accordance with Table 3, a general block view of the DestinE Platform architecture is shown in Figure 4: functions and involved external / internal elements are depicted together within functions they belong to.

The colour of the arrows identifies the functions belonging to one of the Functional areas (see Section 3.1). The arrow direction represents the direction of the action from the subject to the object. When the same software component is deployed in close interaction with another component, it is represented with a small shape. Badges represent in an intuitive way functions involving more services – while badges in parentheses are facultative.

Figure 7: DestinE Platform Overall Architecture.

DestinE Platform supports an open ecosystem of services for DestinE data exploitation and information sharing, including the provision of Core Services, referred to as Platform and Data Management services, for the benefit of the DestinE users and Third-Party entities. Services shall be available for all potential users as well as for any potential Third-Party applications or services.

•Platform Management Services: trusted entities within the ecosystem, competing in creating a business model around services, establishing trust among participants. These services are the pillars of the platform set up and are unique.

• Data Management Services: all services allowing consumers to access data and applications. The access to these services is protected.

The Platform management services, on which this document focuses on, are intended to be used by the other services to integrate inside the platform. The Data Management Services, on the other hand, are services highly dependent on the user demand. They will strongly leverage the scalability and elasticity of the platform.

In following Section 4.1, a high-level description of DestinE Platform system elements is provided.

4.1 DestinE Platform Elements

4.1.1 Platform Management Services

The DestinE Platform elements implementing the functions described in Section 3.1 are the Platform Management Services.

These, described in the following paragraphs, are:

• Identity and Access Management (IAM) Service

•Accounting Service/Clearing House

•Service Registry

•Information Dissemination & Onboarding Support Service

•Service Desk

•Dashboard Services

4.1.1.1 IAM Service

The Identity and Access Management (IAM) Service is the unified DestinE Platform user management service, granting access to all authorised registered Services based on the same digital identity.

It provides the means of authenticating end-users and authorizing their access to resources depending on the specific resource and access privileges.

The IAM Service provides also means for other Entities and Ecosystems to federate with DestinE Platform, leveraging standard protocols like SAML and OpenID Connect.

Tenancy Management Component

Tenancy Management Component is a section of the User Profile management web application provided by the IAM Service.

This component provides graphical user interface to support Tenancy scenarios which include:

-Tenancy Management

-Project Management

-Consumptions Tracking

4.1.1.2 Clearing House (Accounting)

The Clearing House collects and stores all the transactions executed by DestinE Platform Services. This will track information about the services consumption, by platform active users.

4.1.1.3 Service Registry

The Service Registry catalogues all services, integrated in DestinE Platform, including the Core Services delivered as part of this contract, onboarded services, DEDL Services operated by EUMETSAT and DT Services operated by ECMWF.

It maintains a searchable database of all the available services and data portfolio, as per the DestinE Platform Service Catalogue.

As part of the Service Onboarding, information and metadata of new Services are registered in the Service Registry and made available to DestinE users.

4.1.1.4 Information dissemination & onboarding service

The Information dissemination & onboarding support service take cares of:

•Web information Presence layer (website, knowledge base, video tutorials, technical documentation, …) of the DestinE Platform

•user experience and community engagement activities Onboarding support to users (aiming to provide information, suggestions or help to register services on the platform).

4.1.1.5 Service Desk

The Service Desk is available for users to interact with DestinE Platform and Community by submitting issues, support and/or information request, and has the responsibility to manage the relevant ticketing system. Moreover, it oversees user feedback analysis and reporting.

The Service Desk will also take care of the interaction with the DestinE elements and/or third-party application services as part of the anomaly and maintenance management processes. The User Support Team will raise internal tickets for specific issues, e.g., when they discover anomalies or faults in the system or want to share suggestions on improvements for workflows or processes.

4.1.1.6 Dashboard Services

DestinE Platform Dashboard

The DestinE Platform Dashboard provides live information about the DestinE Platform usage in terms of total registered users and trend by user profile, registered users by country, Monthly active users per service, Number of tasks of the public roadmap for each phase, number of services. This page is public and accessible to anyone visiting the platform.

Executive Dashboard Service

The Executive Dashboard Service provides live information about the DestinE Platform usage, performance and status as well as relevant statistics coming from the platform registered Services.

Pages of the executive dashboard present a set of dedicated statistic panels designed to match reporting needs and will be accessible to the authorized users (ESA, EC).

Service Operations Monitoring Dashboard Service

The Service Operations Monitoring Dashboard Service allows real-time monitoring of DestinE Platform operations and performance, with restricted access targeted to Serco as Service Administrator and ESA Technical Officer (including individuals authorised by the ESA Technical Officer).

All the Events are delivered by parsing relevant logs and system metrics useful for operations.

4.1.2 Infrastructure elements

Figure 8: Deployment model of DestinE Platform system elements

4.1.2.1 Infrastructure as a Service (IaaS)

The DestinE Platform is hosted on the infrastructure provided by OVH Cloud. This infrastructure is composed by several Kubernetes clusters managed by TAS and hosted on VMs in OVH Public Cloud (OpenStack-based [RD-1]).  

4.1.2.2 Runtime platform (PaaS and Caas)

The DestinE Platform Runtime Platform, built by Thales Alenia Space, is a common layer for the deployment and operation of all the DestinE Platform services.

The Runtime Platform integrates the following components:

•The cloud infrastructure layer provided by OVH;

•The Virtual Machines built on top of the cloud infrastructure using TAS provided secured operating System;

•The Orchestration platform (Kubernetes-based [RD-2] CaaS) provided by TAS. Containers are the infrastructure core building block on which the exposed services physically reside, in which one or more applications may be run;

•The Orchestration support services provided by TAS.

On the Runtime Platform, the Platform Management Services will provide the needed functions to enable Data Management Services to perform their ones.

The Runtime Platform will ensure the automation relevant to the configuration, coordination, and management of computing resources and applications. The Orchestrator distributes the Containers above the Compute, Storage and Network layers, providing each Container with the necessary Compute, Storage, Network.

The Orchestration Platform may also leverage also other infrastructure providers, being these DEDL computing services, HPC providers or other cloud providers that aim to federate with the DestinE Platform.

4.1.3 New registered Services

Within the DestinE Platform, a Service is a specific function operated by a Service Provider based on a service-level agreement.

A Destine Platform Registered Service is listed in the Service Registry, and available with free access to authorized registered users. Each registered Service providing data access, information, or software shall state the applicable access conditions and licenses and required registered Users acceptance.

External service providers, data providers and infrastructure providers can grow and expand the DestinE Platform offering by registering their own data and services on the platform or by offering their infrastructural resources to host the Services.

From a design perspective, a new registered service should adhere to the following principles:

•NRSD#1: It should follow the FAIR (Findability, Accessibility, Interoperability, Reuse) principles for scientific data management.

•NRSD#2: It should foster interaction and collaboration amongst the user community by providing dedicated collaborative interfaces and frameworks.

•NRSD#3: It should enable seamless access to a complete suite of services, which shall serve expert development activities while allowing discovery of the Earth system and interaction with its representation by the general public.

•NRSD#4: It could enable smooth integration of resources not funded by DestinE but that provide operational continuity and create a seamless scalability for registered user operations.

As a minimum, a new registered service:

•Shall be registered on the Service Registry.

•Shall be integrated with the IAM Service.

5. Interface Control Document

This Section provides the Master ICD of DestinE Platform, listing the internal and external interfaces

Each Interface is labelled with a unique code as follows:

<component ID>-<EXT=external | INT=internal>-IF-<P/R>-<progressive number>

Where:

1)<component ID> is the letters acronym of DestinE Platform element. The identified acronyms, listed in alphabetic order, are:

•ACC = Clearing House/Accounting Service

•CACHE = Data Cache Management Services

•DARS = Data Access & Retrieval Services

•DCDS = Data Catalogue & Discovery Services

•DCMS = Data Cache Management Service

•DSK = Service Desk

•DSP = entire DestinE Platform system (as a black box)

•EXED = Executive Dashboard

•IAM = Identity and Access Management Service

•IDOS = Information dissemination & Onboarding Service

•OVH = Cloud Infrastructure layer

•RUN = Runtime Platform

•SOMD = Service Operations Monitoring Dashboard Service

•SR = Service Registry

•TRAC = Traceability Services

•TRFO = Data Transformation Services

•UWS = User Workflow Services

•VIS = Visualization Services

2)<EXT=external | INT=internal> defined the interface as EXT if external or INT if internal.

3)<P/R> indicated the interface type, P if provided or R if required.

4)<progressive number> is a zero-padded four digits number (e.g., 0001, 0010, 0020, etc.).

Table 4 provides the list of the interfaces with their short description.

For each identified interface, the following properties are reported:

•Interface unique identifier

•Description

•Source (i.e. provided service, and owner of the interface details description in the relevant applicable service documentation)

•Destination

Table 4: List of the External and Internal interfaces of DestinE Platform

‎

End of the document