EuroCropsML Dataset¶

Project Background¶

The EuroCropsML dataset and associated eurocropsml code repository are provided and developed as part of the joint PretrainAppEO research project by the chair of Remote Sensing Technology at Technical University Munich and dida.

The goal of the project is to investigate methods that rely on the approach of pre-training and fine-tuning machine learning models in order to improve generalizability for various standard applications in Earth observation and remote sensing.

The ready-to-use EuroCopsML dataset is developed for the purpose of improving and benchmarking few-shot crop type classification methods.

EuroCropsML is based on Version 9 of EuroCrops, an open-source collection of remote sensing reference data for agriculture from countries of the European Union.

Data Acquisition¶

In order to obtain the observation data for a given country and year, the following steps were caried out:

  1. Collection of relevant Sentinel-2 tiles: Alignment of \(\texttt{EuroCrops}\) version 9 reference data with Sentinel-2 raster data files.

    1.1 .SAFE files collection: Request the necessary .SAFE files for the given year via an API request to the EO-lab Data Explorer. All tiles that overlap the land surface of the country for a given year are collected.

    1.2 .SAFE files collection: Mapping collected .SAFE files to each parcel individually by their geometries.

    1.3 Band image path collection: Move all necessary .SAFE files to a local directory (if specified) to fasten up polygon clipping. Collect the individual band image paths of each .SAFE file.

  2. Clipping of satellite data and calculation of median pixel values.

    Polygon clipping: Clip parcels from the .SAFE files to obtain time series of corresponding reflectance data. As the dataset is intended to be used for crop type classification, we aggregated the collected pixel values. For every parcel and each available time step observation, we calculated the median pixel value for each of the 13 spectral bands of the Sentinel-2 raster tiles, as also done in the tiny EuroCrops dataset.

  3. Regional mapping: To enhance the precision of geographical data and facilitate the effective partitioning of the dataset, we utilized the Eurostat GISCO database to link the \(\texttt{EuroCrops}\) parcels with their corresponding NUTS region.

    NUTS regions: Add NUTS1-NUTS3 regions. The shapefiles for the NUTS-regions have been obtained from Eurostat.

Note

During step one, some parcels that lie at the intersection of two or multiple raster tiles, are first assigned to all of them. In this case, only the raster tile with the lowest cloud coverage is kept for the following steps. Thus, during clipping, only parts of the polygon are clipped and included in the calculation of the median pixel value. However, since we are only relying on the median pixel value and not on individual pixel values, clipping only a part of the polygon in rare edge cases is sufficient. Furthermore, the \(\texttt{EuroCrops}\) sometimes contains duplicate parcel geometries. If this is the case, only one entry is kept.

Data Acquisition Pipeline.

We provide all scripts that are necessary to perform the above steps.

Note

The scripts could be adapted accordingly in order to get similar data for other countries or years present in \(\texttt{EuroCrops}\), either through EOLab or any other source that has Sentinel-2 .SAFE files available. as long as you have access to the necessary .SAFE files. Please create a separate configuration file in the directory eurocropsml/configs/acquisition/cfg for this. The config module already contains the necessary information for the other available \(\texttt{EuroCrops}\) countries. Please refer to the official EuroCrops reference dataset for more reference data. Examples on how to customize the pipeline for collection additional data can be found in Examples.

To run the data collection, you can use the provided command-line interface (CLI).

The following commands provide further assistance:

$ eurocropsml-cli --help

$ eurocropsml-cli acquisition eurocrops --help

The default configuration eurocropsml/configs/acquisition/config.yaml collects data for Portugal. For the \(\texttt{EuroCropsML}\) ROI countries Estonia, Latvia, and Portugal, there are predefined configuration files available in the eurocropsml/configs/acquisition/cfg directory that can be selected. For example, to collect data for Estonia:

$ eurocropsml-cli acquisition eurocrops get-data cfg=estonia

Note

If you want to get the data for another country, please first create a new acquisition configuration in the directory eurocropsml/configs/acquisition/cfg. You can then simply replace the default configuration with the one you created (\({\textit{cf.}\,}\) Examples).

Data Preprocessing¶

The collected data needs further preprocessing in order to be used with most machine learning models.

To run the data preprocessing, you can use the provided command-line interface (CLI).

The following command provides further assistance:

$ eurocropsml-cli datasets eurocrops --help

During preprocessing, each data point is saved separately as a \(\texttt{NumPy}\) .npz file along with metadata such as the spatial coordinates of the centroid of the parcel and the timestamp of each observation. The .npz files use the naming convention <NUTS3-region>_<parcelID>_<EC_hcat_c>.npz, where EC_hcat_c is the \(\texttt{EuroCrops}\) HCAT crop class code.

Cloud Removal (for Sentinel-2)¶

Additionally, we perform a cloud removal step for Sentinel-2 data following the scene classification approach of the Level-2A Algorithm. To detect clouds, we rely on the brightness thresholds of the red band (B4). If the median reflectance of the band is lower than the threshold \(t_1=0.07\), we consider it as cloud-free and assign a cloud probability of 0%. If it is higher than the threshold \(t_2=0.25\), it is considered cloudy and is assigned a cloud probability of 100%. Similarly, we linearly interpolate values between the aforementioned thresholds and assign probabilities between 0% and 100%. Consequently, all observations with a cloud probability greater than 50% are removed. The removal of the cloudy observations as well as the individual thresholds can be adjusted in the preprocess config (\({\textit{cf.}\,}\) Examples).

Further Notes¶

Please note that when creating training/validation splits for machine learning algorithms, there is an option for downsampling the class \({\texttt{pasture_meadow_grassland_grass}}\) during the pre-training phase of a transfer-learning scenario to the median frequency of all other classes. The downsampling can be turned off by removing the meadow_class parameter from the respective split configurations in the directory eurocropsml/configs/dataset/split.