Three years ago, we started the phase-0 study of a mission named Sentinel-HR : S-HR would be a satellite mission to complement Sentinel-2 new generation (S2-NG) with

  • better resolution images (2 m),
  • a lower revisit (around 20 days),
  • 4 spectral bands (Blue, green, red and near-infrared),
  • and the capacity to make stereoscopic observations

In that study, one of our assumptions was that a lower revisit frequency than that of Sentinel-2 was sufficient. This seems possible because, in most cases, the high frequency features of the images (the details) do not change as quickly as the low frequency features (the colors).

During that study, Julien Michel spent quite some time in verifying this assumption. To do so, he studied if it is possible to fuse the time series from S2-NG and from S-HR in order to get hybrid time series with the high resolution of S-HR (2 m), and the high revisit of S2 (5 days) or S2-NG (3 days). These results have already been shown in our report one year ago, but we are not fully sure everyone reads reports with 150 pages.

In this first article, we show the methodology, validation data set and first results, and in a second post, we study the influence of the revisit frequency. A second post studies the influence of the revisit.

Data sets

Different techniques allow to merge time series of S2NG and VHR data, to get at the same time the frequent revisit of S2NG, and the very high resolution of the S-HR, with a good accuracy. Our study compared different methods, using Sentinel-2 and VENµS images to simulate S-HR and S2NG data. As VENµS images are acquired every second day, for each time series, one image every 20 days was used for the data fusion, and the other ones to validate the fused data.

The following series gives an example of the data set we used (which has been published [4]). We have such data on 110 sites, observed by VENµS for two years.

Series of images obtained from Sentinel-2 (top row) and VENµS (bottom row). For each column, S2 and VENµS data were acquired one day apart or even on the same day. The first S2 image is shown in black as it was fully cloudy. As VENµS resolution is 5 m, the S2 data have been downgraded to 12.5 m, in order to keep the 2.5 resolution factor we will have between S2 NG and S-HR. In this data set, the first and last S-HR images are separated by 20 days. The real Sentinel-HR data would not provide the data in between. The S-HR images acquired in between are therefore used to validate the data fusion.

Data fusion methods

For sake of conciseness we do not detail the methods here, as they are fully described in the report. We studied five methods :

Two trivial methods for reference:

  • GF : Gap filling the VHR data, without using any information from the S2 or S2NG sensors. It is just a temporal interpolation, performed pixel wise
  • Zoom : Zooming the S2NG or S2 using resampling techniques. In that case S-HR data are not used.

Three more methods where tested:

  • STARFM [3] or *FM is a classical multi-linear approach to data fusion, that uses information from both missions
  • CARN [2] is an efficient single image super-resolution method, that only uses the S-HR data as reference data for the machine learning, while the inference of super-resolution images is obtained with the S2 (NG) data.
  • DDI is a guided interpolation approach. Neural networks are trained to learn how to interpolate the S2(NG) data at S2(NG) resolution, and then, the same neural network is used to interpolate the VHR data at full resolution. The DDI method was tested with two different normalizations. DDI, developed at CESBIO ash not been published yet, but should be published soon.


The results displayed in next figure show that differences lower than 0.01 with the reference data set have been reached in the red, green and blue bands with the DDI methods. It has been tested for uniform pixels, where temporal variations are preponderant, and with the 25% of pixels that include the highest gradients, for which the spatial features are more important. The results are not as accurate for the near infra-red bands because of larger differences in the spectral response functions between VENµS and Sentinel-2. Standard deviations of 0.01 are close to the performances of atmospheric correction on Sentinel-2 images. It may therefore be considered as a really good result.

Comparison of data fusion methods for 4 bands, obtained on a data set of Sentinel-2 images degraded to 25 m GSD, and VENµS images at 5m, representing the VHR images with a low revisit of 20 days. The factor between resolution is the same as in Sentinel2 vs VHR mission.   GF is gap filling, or temporal interpolation, using only VENµS images, ZOOM is a spatial interpolation using only S2 images, STARFM is a classical but old multilinear data fusion method, CARN is a single image super resolution method based on deep learning, using S2 and VENµS for training but only S2 for inference, and DDI is a modern data fusion method. The results show that an accuracy on reflectance of 0.01 can be reached even for the pixels with sharp variations. Gap Filling provides good results for the rmse, but terrible results for the temporal criteria.


The next figure shows, for just one case, the differences observed in the fused data depending on the methods. The final report includes much more results.


The first row shows the predicted HR images, with first, the reference imaged by VENµS, and then the prediction obtained from the S2 image (bottom image in the first column). In that case, we simulated the data fusion between S2 and S-HR, with a factor 5 in resolution. The gray images show the differences between the predicted image and the reference, for 4 methods, gap filling, CARN, STARFM and DDI.

It is interesting to note that the gap filling methods predicts the contours very well, but the linear interpolation in time is not able to capture the complex evolution of landscape colors, while the other methods manage to provide the low frequency evolution, but do not obtain the contours as well as the gap filling. However, the DDI methods does quite a good job for both criteria, as shown by the statistics above. These results show that our assumption of the possibility of a lower revisit for the Sentinel-HR is correct. Of course, in terms of cost, a constellation with a revisit of 20 days is probably 4 times cheaper than the same constellation with 5 days revisit. In a second post we studied if we could further reduce the revisit.



[1] Julien Michel, Olivier Hagolle, Jean-Marc Delvit, Martin Thierry, Lebegue Laurent, et al.. Sentinel-HR Phase 0 Report. CNES – Centre national d’études spatiales; CESBIO. 2022.

[2] Li, Yawei, Agustrsson, Eirikur, Gu, Shuhang, et al. Carn: Convolutional anchored regression network for fast and accurate single image super-resolution. In : Proceedings of the European Conference on Computer Vision (ECCV) Workshops. 2018. p. 0-0.

[3]Hilker, T., Wulder, M. A., Coops, N. C., Seitz, N., White, J. C., Gao, F., … & Stenhouse, G. (2009). Generation of dense time series synthetic Landsat data through data blending with MODIS using a spatial and temporal adaptive reflectance fusion model. Remote Sensing of Environment113(9), 1988-1999.

[4] Michel, J., Vinasco-Salinas, J., Inglada, J., & Hagolle, O. (2022). SEN2VENµS, a dataset for the training of Sentinel-2 super-resolution algorithms. Data, 7(7), 96.


Laisser un commentaire

Votre adresse e-mail ne sera pas publiée. Les champs obligatoires sont indiqués avec *

Ce site utilise Akismet pour réduire les indésirables. En savoir plus sur comment les données de vos commentaires sont utilisées.