Dataset structure

A dataset is a directory with the following structure:

data.yaml
data/
    data_dir_1 ...
index/
conf/
    areas.yaml
    map_styles.yaml
    params.yaml
    param_styles.yaml

data.yaml

This is the main data configuration file describing all the dataset components (aka experiments).

mars_params: &mars_params
    "93.128": tdyn
    "95.128": trad
    "98.128": tvdiff
    "105.128": tcon
    "109.128": tcl
    "94.128": qdyn
    "99.128": qvdiff
    "106.128": qcon
    "110.128": qcl
# use __ROOTDIR__ as a placeholder for the actual root data path
experiments:
    - an:
        label: "an"
        dir: __ROOTDIR__/an
        fname : re"an_[A-Za-z0-9]+_[A-z]+\.grib"
    - oper:
        label: "oper"
        dir: __ROOTDIR__/oper
        fname : re"oper_[A-Za-z0-9]+_[A-z]+_\d+_\d+\.grib"
    - control:
        label: "control"
        dir: __ROOTDIR__/control_hhe4
        fname : re"hhe4_[A-Za-z0-9]+_[A-z]+_\d+_\d+\.grib"
        mars_params: *mars_params

It contains the paths to the data files. The __ROOTDIR__ placeholder is resolved to the full path of the data directory.

data

This is the directory containing the data files. There is ideally one sub-directory in it for each experiment.

index

This is the directory containing the data indexing information. It contains one directory for each experiment. The contents is typically pre-generated and if it is missing or incomplete automatically re-generated.

conf

This is the directory containing the map and style configurations. It can be empty, however if we place files in it they will extend the built-in default configurations.

conf/areas.yaml

This file contains the definition of the map areas.

- base:
    map_area_definition: "corners"
    area: [15,-70,80,40]
- atl:
    map_area_definition: "corners"
    area: [30,-60,70,0]

Areas defined here can be used like this:

view = mv.make_geoview(area="atl")
mv.plot_maps(v, view=view)

conf/map_styles.yaml

This file contains the definition of the map plotting styles (mcoast()). E.g.:

grey_light_base:
        map_coastline_resolution: "low"
        map_coastline_land_shade: "on"
        map_coastline_land_shade_colour: "grey"
        map_coastline_sea_shade: "on"
        map_coastline_sea_shade_colour: "RGB(0.86,0.94,1)"
        map_boundaries: "on"
        map_boundaries_colour: "RGB(0.21,0.21,0.21)"
        map_disputed_boundaries: "off"
        map_administrative_boundaries: "off"
        map_grid_latitude_increment: 10
        map_grid_longitude_increment: 10
        map_grid_colour: "RGB(0.294,0.294,0.2941)"
        map_label_colour: "RGB(0.294,0.294,0.2941)"

Styles defined here can be used like this:

view = mv.make_geoview(area="atl", style="grey_light_base")
mv.plot_maps(v, view=view)

conf/params.yaml

This file defines the data parameters and assigns the plotting styles to them.

conf/param_styles.yaml

This file defines the plotting styles for data.