View release on GitLab →

Evolutions of 2.6.4

• Sentinel 1 and Sentinel 2 L0 mappings have been updated/created
• PDF generation of documentation
• Zarr store now handles relative paths
• Triggering payload parser was refactored
• EOContainers can be walked/browsed
• EOProcessing unit validation refactoring

Product related:

• multiple minor corrections for the majority of products
• S01:
• add support for old Level-0 products
• Level-1 product corrections
• S02:
• add support for old Level-0 products
• L1C product corrections
• L2A product corrections
• S03:
• fix error on EOSafeStore
• fix incorrect fill_value attribute
• fix Altimetry mappings anomalies
• fix slant_range_time coordinates
• update of SLSTR mappings

View release on GitLab →

Patch for 2.6.2 for dask gateway issue in dask_context

View release on GitLab →

Evolutions of 2.6.2

• upgrade of Xarray to 2025.9.0
• upgrade to EOPF-asgard 1.3.0
• CLI QualityControl handles EOContainers
• fix missing "short_name" attribute when converting Xarray.DataTree to EOProduct
• update of product-type pydantic models

Product related:

• set meteo variables optional
• S01:
• correct zero doppler time decoding for S01 L2 OCN products
• add support for old Level-1 and Level-2 products
• S03:
• new SRAL L1 and L2 products: S03ALTL1A, S03ALTL1R, S03ALTL2H, S03ALTL2L, S03ALTL2S;
• new MWR L1 products: S03MWRL1\_;
• STAC related:
• fix STAC instruments attribute
• fix STAC processing:software
• replace of STAC processing:expression with processing:lineage

Dask iprovements:

• fix dask_gateway dependecy

View release on GitLab →

View release on GitLab →

Evolutions of 2.6.0

• mapping updates for S01 L1 (GRD/SLC) and L2 products (OCN)
• mapping updates for S02 L1C and L2A products
• mapping updates for S03 products with the exception of Altimetry L1 and L2
• real delayed computation in triggering: write a product while continuing the workflow
• detect dask computation error and exit cleanly
• add ErrorPolicy management: FAIL_FAST, FAIL_ON_CRITICAL and BEST_EFFORT
• add exit code on exception to be able to set the CLI exit code in the triggering : calibration mode, errors
• add dask_config settings in the triggering to set dask parameters and clean them at the end of execution
• add STAC validation on EOProduct
• add STAC validation in EOQCValidCheck
• intermediate product cache
• EOTemporaryDirectory shared with workers
• DaskClusterMonitor
• EOProduct validation ( STRUCTURE / STAC / MODEL )
• Regex/Folder product search -> List of products in ProcessingUnit
• possible external module load
• convert is now available also as a CLI command
• merging tool for Sentinel-2 L1C/L2A tiles (with no meteo data)
• same name for coordinates and dimensions is now possible
• improvements of nominal product type mapping models

API Breaking changes:

• Dask context gateway "workers" params become "n_workers" to be in accordance with the localCluster n_worker param
• ProductStore get a cancel parameter on the close call to cancel all ongoing operations (mostly dask stuff but can be any async)
• Not a real API change but the exit code on the triggering can vary depending on the exception raised. Still 0 in case of no error
• Processors have to provide their input and output validation schemas. It can be empty json if no rule
• Processors now have an optional mode parameter on the run method
• triggering always uses run_validating but the validation mode is turned to STRUCTURE if triggering__validate is false
• EOProduct.shortnames returns a Mapping, i.e the user can't change the content directly; it has to use the setters of shortnames of EOProduct
• EOQCProcessor takes a list of config folders. If it is none it will uses the qualitycontrol__folder only
• EOQCProcessor no longer allows specifying a checklist, always uses the type search mechanism, except it default config builder is replaced
• default_filename_convention 3 digits CRC is now computed on the attrs dict hash, thus it gives the same value at each consecutive calls with the same product

Deprecated:

• remove EOProduct and EOGroup add_group methods, they are deprecated
• remove EOSafeStore, EOZarrStore, EONetCDFStore, EOCogStore load_container methods, they are deprecated
• remove EOPRoduct set_type, deprecated
• remove enter and exit on EOProduct as it is not a context manager
• remove update_attrs and replace_attrsfrom eo_abstract
• ProcessingStep :CPM no longer enforce a specific API for function calls for the algorithm, as it doesn't add anything but constraints. Free choice on xarray/numpy function signature
• Conveniences.get_product_type and set_product_type set as deprecated; one should directly use the EOProduct/EOcontainer methods to avoid circular dependencies
• parameter dask__compute__step is no longer needed as the dask computation flow has been corrected

Please find the technical changes since the previous version in the file.

View release on GitLab →

Minor patch on numpy version to unblock Asgard install

View release on GitLab →

This release introduces mappings for S02 L1C/L2A as well as S03 SLS RBT/LST/FRP. However, this should be treated as a beta release. Further corrections to the mappings are underway and will be released in subsequent versions, 2.5.9 and 2.6.0.

View release on GitLab →

View release on GitLab →

• update numpy to version 2, and related dependencies upgrades: see MR 825
• mask and scale updates to match Xarray: see issue 695
• Triggering: add as_datatree switch also for output writing; see issue 703
• Triggering: setup a temporary working folder; see issue 659
• Triggering: input folder with a regex selector; see issue 656
• write some specifications on intermediate product naming convention in PDFS; see issue 613

View release on GitLab →

• update uvicorn: see issue #648
• move Ipython depency to notebooks tag: see issue #630
• handle specific SCALE_FACTORS: see issue #547
• handle masked arrays as Xarray: see issue #534
• EOObject tree method was refactored: see MR !811
• temporary fix for Sentinel 1 OCN product: see MR !809
• allow additional attributes under Xarray attributes: see MR !808

View release on GitLab →

• add additional QC checks: see MR 779
• fix coordinates association with variables, see issue 653
• fix dtype conversion with mapping: see issue 652
• fix support old Sentinel products: see issue 651
• fix update PyPi packaging infos: see issue 649
• fix Zarr writing for datetime data: see issue 637
• update docs on EOProduct Update/Append mode: see issue 647
• update docs on triggering parameters: see issue 640

View release on GitLab →

View release on GitLab →

• refactoring of quality control ; see MR !692
• refactoring of EOZarrStore.load() to address container loading problems; see issues #604, #612 and #623
• stac_discovery corrections; see issue #624
• conversions within pytest are now working; see issue #621
• added missing dependencies, i.e. bs4 and IPython; see issue #618
• added EOPF-CPM version in the pyproject.toml; see issue #616
• add a temporary fix to support DataTree.to_zarr(), when the DataTree is obtained from an EOProduct; see issue #608
• add support for automatic decoding of datetime EOVariables for the EOZarrStore; see issue #611
• upgrade dependencies to avoid security issues, i.e. python-multipart and aiohttp; see MR !769

View release on GitLab →

Features:

• Added support for S02 L1A and L1B conversions to zarr #598
• Mappings developed till version 2.4.5 were added to eopf/store/legacy_mapping. One can switch between mappings by changing the eopf.toml under eopf/config/default !749
• EOObject can be directly comparend as __eq__ and __ne__ have been implemented for EOContainer, EOProduct, EOGroup and EOVariable !754

Fixes:

• add overrides package as a dependency #596
• spelling corrections !757
• updates of libraries due to security issues !756
• removed several eopf: metadata, no longer part of the eopf stac extension !752
• corrected the EOVariable assignment of dimensions !750
• fixed altering of EOVariable by some fill_value !745

View release on GitLab →

Major updates to converted products structure for Sentinel 1 L1 and L2, Sentinel 2 L1C and L2A, Sentinel 3 L1 and L2 (except SRAL)

Features:

• Sentinel 2 L1C and L2A products now follow the product definition envisioned by Best Practices
• Setntine 3 L1 and L2 products now follow the product definition envisioned by Best Practices; except S03AHRL1B and S03AHRL2B which are not currently supported

Fixes:

• Sentinel 1 L1 and L2 products have been updated, conversion will output EOContainers as top level, and corrected.

Regression:

• Access to EOVariables via short names is currently not working; to be corrected with future releases.

View release on GitLab →

View release on GitLab →

Fixes

• S3 SLC LST corrections
• Stac discovery corrections for all mappings
• Removed Scipy dependency, no longer used

View release on GitLab →

Features:

• Sentinel L1 and L2 mapping updates

View release on GitLab →

Fixes:

• Mask and Scale correction on the EOZarrSTore
• S2 L1C support for PSD 15

Feature:

• Processing Units with Self-Descriptive Static Methods for Enhanced Orchestration

View release on GitLab →

Fixes:

• EOPF-CPM formatter raises exception when data value is NaN #577
• Sentinel L1C and L2A Zarr naming incorrect #572
• Downgrade eccodes to avoid test issues

Features:

• S2 L1C support for PSD 15 #541
• POC : Use function to initialize EOProduct in SafeStore

Docs:

• Update of Mapping documentation

View release on GitLab →

2.4.0

# Features:

• Introduced product versioning in the mappings, i.e. same product with different baselines. Mapping are now uniquely identified by << type >> and << processing_version >>. Some changes were necessary in the EOSafeStore, EOPFMappingManager, EOPFMappingFactory
• Introduced an alternative method to recognise SAFE products via custom code. Hence, new concepts were introduced: EOPFProductRecognitionFactory and EOProductRecognition. Hence, if the regex recognition pattern is not capable of recognizing a particular SAFE product type and processing_version, one can develop a custom EOProductRecognition
• EOZarrStore supports updating an existing product
• EOProduct short_names can be set via a custom dictionary or by providing the product type and processing_version
• EOSafeStore
• << load >> method supports only retrieving product metadata via <<metadata_only>> parameter
• can << load >> EOContainers
• Add new formatters for product metadata: auto, to_number

# Fixes:

• AnyPath
• << basename >> method corrected for Windows
• << glob >> method correctly identifies zip protocol
• Git repository broken on Windows corrected
• S3 LST & RBT are not missing product attributes
• S2 L2A
• support old reprocessed products
• products respecting PSD14 and PSD15 are now supported
• b03_10m was re-added to the product
• EOContainer, EOProduct, EOGroup, EOVariable string representation improved
• upgrade of some internal dependencies to avoid security issues
• DataTree returned by Processing Units is now supported

# API changes

• With the new EOPFMappingFactory it is easier to register custom mappings or mapping folder; one can directly pass the file or directory path at initialization without needing to modify the EOConfiguration file.
• With the introduction of productversioning mechanism, all functionalities using mappings need also to pass the << processing_version >>
• As EOSafeStore supports loading containers, all mappings should contain the <<eopf_category>> attribute in the << other_metadata >> section for each EOProduct and EOContainer described

# Deprecation warnings:

• << load_container >> method from EOSafeStore and EOZarrStore are now deprecated; their functionality is incapsulated in the << load >> method.

View release on GitLab →

View release on GitLab →

View release on GitLab →

View release on GitLab →

View release on GitLab →

• Upgrade of xarray and dask dependencies
• Improvement of some warning messages
• Possibility to copy EOvariable, EOGroup and EOProduct
• Conversion of all common ADF into xarray datasets
• Creation of the CADU store
• Creation of map_overlap function for xarray dataset
• Compatibility with xarray datatree

View release on GitLab →

View release on GitLab →

• Compatibility with windows
• Dask graph function restored
• Upgrade eopf[docs] dependencies
• Deprecated set_type
• A product is no longer needed to get shortnames from mappings, use parse_shortnames on EOPFMappingManager
• Enable copy() and deep_copy on EOProduct/EOGroup/EOVariables

Internal changes:

• ProductType is no longer stored in the EOProduct but in the Attributes of EOProduct in eopf:type ( this was implicitly the case as we automatically set type=eopf:type)
• Removed add_variable/add_group internal calls
• Homogeneize chunks settings in accessors

View release on GitLab →

• Documentation updates
• EOVariable.data now points to Xarray
• Resolve regression in breakpoints and triggering
• Update Sentinel 1 Level 1 GRDH maps
• Corrected issues on impeding the load of Sentinel 1 Level 0 products
• Correction on displaying EOProduct.tree in Jupyter notebooks
• Resolve EOZarrStore has asymmetric open API OPEN/CREATE

View release on GitLab →

Corrections on version 2.0.0, mainly related to Sentinel 2 conversions

EOVariable Xarray DataArray can now be accessed as EOVariable.data (previously it was ._data)

EOVariable Dask Array can be accessed via EOVariable.data.data

View release on GitLab →

Synopsis:

• Coordinates of EOVariables are no longer stored separately under the coordinates EOGroup, they are now moved within the EOVariables.

• Support Xarray and Zarr libraries reading of EOVariables written on disk as zarr.

Major changes:

• EOSafeStore and EOZarrStore were rewritten to support coordinate changes & simplify functionality.

• Introduced dynamic mappings handled by EOPFMappingManager. That is, mappings can expand eo_obj_descriptions and values of eo_obj_descriptions based on the legacy products provided.

• All mappings were updated to support the move of coordinates and dynamic mappings.

• Stores and accessors now use the common file utilities, i.e. AnyPath

• Delegated EOVariable sub-setting/selection/filtering to Xarray functionalities

• add_variable and add_group are deprecated, replaced by standard [] set_item function

Corrections:

• Mask & Scale corrections for EOVariables

• S03OLCEFR and S02MSIL2A conversion corrections

• S02MSIL0_ html tree representation generation corrected

• Documentation updates and corrections

• S3 product retrieval corrections

• Metadata harmonization

• CI/CD corrections

Removal:

• Numerical module was removed

• SafeHierarchy was removed

View release on GitLab →

• Mappings are renamed to correspond with EOProduct types
• STAC metadata are clearly separated in "stac_discovery" and "other_metadata"
• Triggering configuration is improved
• Multiple outputs are now possible in the triggering configuration

View release on GitLab →

Fixes & Updates

• update and document DaskContext
• mapping updates:
• update S1 L0 mappings with new mechanism for finding files based on patterns with regex
• update S1 L1 mappings by allowing RFI variables to be optional
• update S3 L1 and L2 mappings based on ACRI mappings
• add S3 L2 SLSTR FRP mapping
• add S3 L2 OLCI ERR mapping
• correct Zarr writing with chunks under Dask concatenate operations
• add a new common mechanism to treat file/folder locations
• EOVariable now uses coordinates for the following functions:
• loc()
• sel()
• plot()
• add detailed representation of EOProduct.tree() under CLI

API changes:

• No longer possible to call "coordinates" on EOVariable to retrieve the list of coordinates groups associated to this variable, instead call "coordinates_dict" to retrieve the dictionary of associated coordinate group

View release on GitLab →

Features:

• upgrade to Python 3.11, i.e. Python 3.11.7
• upgrade of dependecies to recent versions
• masking and scaling mechanism has been re-written, now under EOVariable
• documentation corections/updates
• added warning for defining multiple mappings for the same product type
• S2 MSIL1C, S2 MSIL2A, S3 L0, S3 OL 1, S3 SL 1, S3 SL 2, S3 OL 2 and S3 SY 2 mappings were updated:
• corrected long_name
• Fill_Value corrections
• add dtype attributes
• add short_name attributes
• add dimensions attributes
• target_path corrections
• reduced complexity for L0 writers
• refactoring of EOConfiguration

Known issues:

• Conversions to EOCog format via EOCogStore need corrections (issue 341)
• S3 L2 OL conversion to zarr takes a long time ~2400s (issue 343)
• S3 L1 SL conversion to zarr takes a long time ~2500s (issue 343)

View release on GitLab →

API impact:

• EOProduct no longer accepts a store in storage but a ProductStore class to handle internaly the lifetime of the store : storage_driver and url to be used instead of storage

non-API impact:

• Store must no be manually instantiated but the storeFactory should be used
• Store have an Open static function to create a product initialized with the correct Store class
• Imports in module are reworked

Product-specific changes:

• S1 Reduce high memory consumption
• S1 EW/IW/SM/WV minor mapping corrections
• S1 L1 GRD/SLC:
• major mapping corrections
• conversion from zarr to safe added
• S1 L2 OCN IW/SM: major mapping corrections and sliceProductFlag corrected
• S3 SL: mapping updates to avoid naming EOGroups as python reserved words, e.g., in -> inadir
• S3: corrected scale_factor to be as per legacy products

General changes:

• Replace deprecated pkg_resources
• Fix EOProduct tree function for command line interfaces
• Refactoring of accessor recovering EOP attributes from .dat files
• Refactoring accessors to provide dask array with chunks
• Refactoring of unit testing
• Add more explicit errors for mapping and zarr path
• Test for filenotfound before opening a file

Docs updates:

• Add a table on supported legacy products
• Add coding rule on documentation dev guide

View release on GitLab →

• Security updates
• Documentation updates
• Improve RAM usage on conversions
• Improve time on Sentinel-1 Level-1 products
• Add design justification mechanism
• Add release note mechanism
• Fixes on EOZarrStore and EOCogStore
• Refactoring:
• configuration module
• logging module
• accessors
• ProcessingStep
• Major map updates
• Sentinel-1 Level-0 maps
• Sentinel-1 Level-1 maps
• Sentinel-1 Level-2 maps
• Minor Map fixes:
• Sentinel 3 Level-1 SLSTR & Level-2 SLSTR maps

View release on GitLab →

• Full Python 3.10 support

View release on GitLab →