Theia's Sentinel-2 production zones

Sentinel-2 L2A products (with a good cloud mask and atmospheric correction) are produced by THEIA using the MAJA software on approximately 1000 different tiles. But this is only a small part of our big world, and we do not want to let the users alone to deal with Sen2cor poor cloud masks, outside Theia production zones.

Since October 2018, it has been possible to run MAJA on PEPS, the French Copernicus collaborative ground segment, to produce L2A products, on demand. But until now, the way MAJA had been implemented within PEPS was rather inefficient, as the implementation only produced one single L2A at a time with MAJA, even if 8 products were generated to obtain that product. Processing at least 8 products is necessary because MAJA, which is a multi-temporal processor, requires to process time series to improve the quality of the output L2A products. But in its usual implementation, the 8 or more products are provided as output, while in PEPS, only the last one was provided. So if you wanted to obtain a whole time series this way, it turned out each product had to be produced 8 times. It had been implemented this way because of constraints of the PEPS on line processing interface.

Thankfully, my colleague from CNES, Christophe Taillan, who works on PEPS, recently improved the implementation of MAJA with PEPS.

Time series of L2A images generated with MAJA on PEPS. Clouds are circled in green, cloud shadows are outlined in yellow. and snow in pink. MAJA implements a cirrus cloud correction, which explains why you might not see any clouds in regions outlined in green.

With the new FULL-MAJA implementation, we have abandoned the clickable interface, and only offer a python interface, but the processing is much more efficient. With only one command line, you can start the processing of a time series of MAJA images. You can get the python interface program from my github repository. Here is how we obtained the nice images presented to the right of this text, that correspond to Sebastopol in Crimea (I picked a random tile)

You can get the python interface program from my github repository. You will also need to create an account on PEPS. Here is an example of how it works, for more information, please go the the github page. The examples below show how I obtained the nice images presented above, that correspond to Sebastopol region (I picked a random tile).

1. Request processing

To start processing a 6 month time series starting in July 2017, on Sebastopol tile (31TCJ), using only orbit 51, just use the following command line :

python ./full_maja_process.py -a peps.txt -t 36TWQ -o 64 -d 2019-03-01 -f 2019-09-01 -g 36TWQ_2019-03-01.log

In this command line, -a, gives the password file, and -g is a log file, that will be used subsequently to download the data. Do not use the same log file for two different processings, as you will not be able to download the data from the first. The time period to process must be longer or equal to two months, and shorter that 12 months.

2. Check processing status, and if finished, download data

To check if the processing is finished, and retrieve the data, just use the following command line, and the generated products will be stored in the directory provided in with -w option.

python full_maja_download.py -a peps.txt -g 36TWQ_2019-03-01.log -w /path/to/36TWQ

As MAJA only provides data for dates with less than 90% of cloud cover, some of the dates are not downloaded. With Sentinel-2, in some cases, products are split in two, with eigher the upper or the lower part with no_data values. The implementation of MAJA within PEPS only processes one of those two, so far. This corresponds to products which are stated at missing in the log-file

Warning: 4 products have not been processed For more information, 
please check Full_MAJA_Sebastopol_2019.json

How long does it take ?

If PEPS is not too busy, it may take up to 24 hours to process a one year time series. But, as the computing resource is limited, the number of simultaneous processes with MAJA on PEPS is limited to 10. Above that number, processes will be queued. As a result, your request could be "pending" for a while. It will mean our on-demand processing facility is a success 😉 To see the progress, use the full_maja_download_tool.py tool, or have a look at the processing page (the gear icon) on https://peps.cnes.fr.

 

A big thank you to the PEPS team, and especially to Christophe Taillan !

 

3 thoughts on “It has never been so easy to obtain Sentinel2 L2A data processed with MAJA.

  1. Hello,
    first of all thank you all for this. I am trying to process some scenes with MAJA on PEPS and so far no success. My first and second processing requests probably failed, I am not sure why with the first one (but there were some FTP errors in the logs), but during the second the PEPS web site became unavailable, so perhaps the job stopped because of server maintenance or overload. I started the job again, but I still see in My Jobs on the website the two older jobs, one of them already two days old - I just suppose these two failed, because there is no indication. The status shows "Process started" and there is percent indicator (not increasing for the two older jobs it seems). I also do not see any way to get more info, kill a job or continue processing in case the older ones just stopped, not failed. Maybe instead of running the job again, it would be possible to revive the second one, saving resources... Also, in the log of the first job with the errors there was notice to "contact the support", but no indication where to contact what support and I do not see a support link in the web site of PEPS. So basically, where to ask questions or read some more detailed documentation, and this in English, since i unfortunately do not understand French?

    1. I forgot to say, I have read the instructions in this article and on the GitHub repository. But there is nothing about job control..
      Thanks again
      Tomas B.

Laisser un commentaire

Votre adresse de messagerie ne sera pas publiée. Les champs obligatoires sont indiqués avec *